Module parser :: Class Lexer
[show private | hide private]
[frames | no frames]

Class Lexer


Class for lexical analyser to use with the parser
Method Summary
  __init__(self, rules_list)
By now lexer is kept as simple as possible, so order is really essential: i.e.
  readscan(self)
Scans a string read from stdin
  rebuild(self, st)
Re-assembles the structure resulting from scanning as a list.
  scan(self, string)
Performs the lexical analysis on string
  scanOneRule(self, rule, st)
Scans space st according only one rule
  scanUnknown(self, st)
Scans the resulting structure making Unknown strings

Instance Variable Summary
dictionary operators: precedence and associativity for operators
  rules: lexical rules

Method Details

__init__(self, rules_list)
(Constructor)

By now lexer is kept as simple as possible, so order is really essential: i.e. if a keyword is substring of another its rule must appear after the larger keyword for the obvious reasons...
Parameters:
rules_list -

contains pairs (re,funct,op?) where:

re: is an uncompiled python regular expression

funct: the name of a funcion that returns the pair (TOKEN, SPECIAL_VALUE), where TOKEN is the token to be used by the parser and SPECIAL_VALUE an eventual associated value. The argument is the matched string. If funct equals "" the token is ignored. This can be used for delimiters.

op: if present, is a tuple with operador information: (TOKEN,PRECEDENCE,ASSOC) where PRECEDENCE is an integer and ASSOC the string 'left' or 'right'.

readscan(self)

Scans a string read from stdin

rebuild(self, st)

Re-assembles the structure resulting from scanning as a list.
Returns:
a list of tokens (pairs of token-value).

scan(self, string)

Performs the lexical analysis on string
Returns:
a list of tokens (pairs (TOKEN , SPEcial_VALUE )), for recognized elements and ("@UNK", string ) for the others

scanOneRule(self, rule, st)

Scans space st according only one rule
Parameters:
rule - one rule (re,fun,op)
st - can be a string or a more complex structure product of previous scans steps ( an tuple (match, left, right))

scanUnknown(self, st)

Scans the resulting structure making Unknown strings

Unknown parts will be of the form ("@UNK", string )

Instance Variable Details

operators

precedence and associativity for operators
Type:
dictionary

rules

lexical rules

Generated by Epydoc 2.1 on Sun Feb 26 02:29:20 2006 http://epydoc.sf.net