Main Page | Namespace List | Class Hierarchy | Alphabetical List | Class List | File List | Namespace Members | Class Members | File Members | Related Pages

_InternalFlexScannerBase Class Reference

Internally used common parts of scanner base classes. More...

#include <scannerbase.h>

Collaboration diagram for _InternalFlexScannerBase:

Collaboration graph
[legend]
List of all members.

Public Member Functions

virtual ~_InternalFlexScannerBase ()
int SetInput (const TLString &path, SError &error, bool included, int first_token=-1, XYPos alt_pos=XYPos(1, 0), SourcePositionArchive *alt_pos_arch=NULL)
 Set input from where to read data to be lexically analyzed.

TokenEntry * LexNextToken ()
 Primary lexer function; get next token.

SCLocation CurrLoc () const
 Get current lexer location.


Protected Member Functions

void __TidyUpZombieList (bool force)
 Internally used to tidy up zombie list; use _TidyUpZombieList().

void _TidyUpZombieList (bool force=0)
 Tidy up zombie list.

void _pi_tok (size_t l)
 To be called after having read a token which does not contain a newline or other spaceing (such as tabs).

void _pi_tab ()
 To be called when "eating" a tab character.

void _pi_line ()
 To be called after having read a newline.

void _pi_char (char c)
 To be called after having read the passed character.

void _pi_str (const char *str, size_t len)
 To be called after having read the passed string which possibly contains special chars (newline, tab, return).

void _pi_start ()
 Make current pos the beginning of a token. Call after whitespace and comments.

void _pi_eof ()
 Called when EOF (on current input) is "read".

TokenEntry * _tok_ptr ()
 Get pointer where to store token info (string/integer/float val...).

SCLocation _MakeCurrLoc () const
 Get current (start) location as SCLocation.

SCLocation _MakeCurrEndLoc () const
 Get current start location as SCLocation.

SCLocationRange _MakeCurrLocRange () const
 Get current location range as SCLocationRange.

virtual TokenEntry * AllocTokenEntry ()
 Allocate a new TokenEntry.

virtual void reset ()
 Reset scanner.

void _RawYYLex (TokenEntry *dest)
 Raw lexing routine as provided by flex(1).

void _reset (bool no_virtuals=0)
 Reset scanner.

virtual int _lex (void *scanner)
 Actual lex function; needs to be overridden by PREFIX_ class.

virtual int _lex_init (void **scanner_p)
 Initialize scanner; needs to be overridden by PREFIX_ class.

virtual void _lex_destroy (void *scanner)
 Cleanup scanner; needs to be overridden by PREFIX_ class.

virtual int _wrap ()
 This implements the _wrap() function; is done in PREFIX_ class.

virtual int _SetInputBH (LexerInput *inp, bool included, XYPos alt_pos, SourcePositionArchive *alt_pos_arch)
 Implement "bottom half" of SetInput(); is done in PREFIX_ class.

 _InternalFlexScannerBase ()
 Never directly construct object of this type.


Protected Attributes

uint32 magic
 For simple detection of memory corruption or evil pointer mess.

void * scanner
 The scanner used for scanning; only one; NULL if inactive.

TokenEntry * lex_token
LexerInput * top_inp
 Which input the scanner is currently reading from:.

LexerInput * read_inp
int file_depth
 Current include hierarchy depth:.

int n_errors
 Accumulate number of errors.

LinkedList< TokenEntry > zombie_list
 Zombie token list.

int zombie_list_nents
 Number of entries in the zombie_list.

int last_zombie_list_nents
XYPos p0
 Current token: start position; while parsing: previous position.

XYPos p1
 Current token: end position; while parsing: current position.

SourcePositionArchive * pos_arch
 Used to generate source position objects; see there.

bool pos_arch_allocated
Config cfg
 Basic scanner configuration.


Static Protected Attributes

const uint32 MyMagic = 0xdeadbeefU
 What the magic value should be:.

const int zombie_list_threash = 256
 Zombie list tidy up threshold. Don't use too small values.


Private Member Functions

 _InternalFlexScannerBase (const _InternalFlexScannerBase &)
 Do not use.

void operator= (const _InternalFlexScannerBase &)
 Do not use.

Detailed Description

Internally used common parts of scanner base classes.

For internal use only.

Author:
Wolfgang Wieser ] wwieser (a) gmx <*> de [
This class is for internal use, only. Do not use explicitly.

This class is not "C++-safe".

Definition at line 103 of file scannerbase.h.

Constructor & Destructor Documentation

_InternalFlexScannerBase::_InternalFlexScannerBase (  const _InternalFlexScannerBase &   )  [private]
 

Do not use.

_InternalFlexScannerBase::_InternalFlexScannerBase (   )  [protected]
 

Never directly construct object of this type.

Definition at line 541 of file iscannerbase.cc.

References last_zombie_list_nents, lex_token, magic, MyMagic, pos_arch, pos_arch_allocated, read_inp, scanner, top_inp, and zombie_list_nents.

_InternalFlexScannerBase::~_InternalFlexScannerBase (   )  [virtual]
 

Definition at line 565 of file iscannerbase.cc.

References Assert, magic, MyMagic, read_inp, scanner, and top_inp.

Member Function Documentation

void _InternalFlexScannerBase::__TidyUpZombieList (  bool  force  )  [protected]
 

Internally used to tidy up zombie list; use _TidyUpZombieList().

Definition at line 437 of file iscannerbase.cc.

References Assert, _InternalFlexScannerBase::TokenEntry::clear(), LinkedList< TokenEntry >::count(), LinkedList< TokenEntry >::dequeue(), LinkedList< TokenEntry >::last(), last_zombie_list_nents, _InternalFlexScannerBase::TokenEntry::MayBeCleared(), LinkedListBase< TokenEntry >::prev, Warning(), zombie_list, zombie_list_nents, and zombie_list_threash.

Referenced by _TidyUpZombieList().

int _InternalFlexScannerBase::_lex (  void *  scanner  )  [protected, virtual]
 

Actual lex function; needs to be overridden by PREFIX_ class.

Definition at line 582 of file iscannerbase.cc.

References CritAssert.

Referenced by _RawYYLex().

void _InternalFlexScannerBase::_lex_destroy (  void *  scanner  )  [protected, virtual]
 

Cleanup scanner; needs to be overridden by PREFIX_ class.

Definition at line 594 of file iscannerbase.cc.

References CritAssert.

Referenced by _reset().

int _InternalFlexScannerBase::_lex_init (  void **  scanner_p  )  [protected, virtual]
 

Initialize scanner; needs to be overridden by PREFIX_ class.

Definition at line 588 of file iscannerbase.cc.

References CritAssert.

Referenced by SetInput().

SCLocation _InternalFlexScannerBase::_MakeCurrEndLoc (   )  const [protected]
 

Get current start location as SCLocation.

Definition at line 176 of file iscannerbase.cc.

References cfg, SourcePositionArchive::GetPos(), _InternalFlexScannerBase::XYPos::line, _InternalFlexScannerBase::Config::loc_use_lpos, _InternalFlexScannerBase::XYPos::lpos, p1, and pos_arch.

Referenced by _MakeCurrLocRange().

SCLocation _InternalFlexScannerBase::_MakeCurrLoc (   )  const [protected]
 

Get current (start) location as SCLocation.

Definition at line 170 of file iscannerbase.cc.

References cfg, SourcePositionArchive::GetPos(), _InternalFlexScannerBase::XYPos::line, _InternalFlexScannerBase::Config::loc_use_lpos, _InternalFlexScannerBase::XYPos::lpos, p0, and pos_arch.

Referenced by _MakeCurrLocRange(), and CurrLoc().

SCLocationRange _InternalFlexScannerBase::_MakeCurrLocRange (   )  const [protected]
 

Get current location range as SCLocationRange.

Definition at line 183 of file iscannerbase.cc.

References _MakeCurrEndLoc(), and _MakeCurrLoc().

Referenced by _RawYYLex().

void _InternalFlexScannerBase::_pi_char (  char  c  )  [protected]
 

To be called after having read the passed character.

Definition at line 190 of file iscannerbase.cc.

References _pi_line(), _pi_tab(), _InternalFlexScannerBase::XYPos::lpos, and p1.

Referenced by _pi_str().

void _InternalFlexScannerBase::_pi_eof (   )  [protected]
 

Called when EOF (on current input) is "read".

Definition at line 207 of file iscannerbase.cc.

References Assert, _InternalFlexScannerBase::LexerInput::read_eof, and top_inp.

void _InternalFlexScannerBase::_pi_line (   )  [inline, protected]
 

To be called after having read a newline.

Definition at line 322 of file scannerbase.h.

References _InternalFlexScannerBase::XYPos::line, _InternalFlexScannerBase::XYPos::lpos, and p1.

Referenced by _pi_char().

void _InternalFlexScannerBase::_pi_start (   )  [inline, protected]
 

Make current pos the beginning of a token. Call after whitespace and comments.

Definition at line 331 of file scannerbase.h.

References p0, and p1.

void _InternalFlexScannerBase::_pi_str (  const char *  str,
size_t  len
)  [protected]
 

To be called after having read the passed string which possibly contains special chars (newline, tab, return).

Definition at line 201 of file iscannerbase.cc.

References _pi_char().

void _InternalFlexScannerBase::_pi_tab (   )  [inline, protected]
 

To be called when "eating" a tab character.

Definition at line 319 of file scannerbase.h.

References cfg, _InternalFlexScannerBase::XYPos::lpos, p1, and _InternalFlexScannerBase::Config::tab_width.

Referenced by _pi_char().

void _InternalFlexScannerBase::_pi_tok (  size_t  l  )  [inline, protected]
 

To be called after having read a token which does not contain a newline or other spaceing (such as tabs).

Definition at line 316 of file scannerbase.h.

References _InternalFlexScannerBase::XYPos::lpos, and p1.

void _InternalFlexScannerBase::_RawYYLex (  TokenEntry *  dest  )  [protected]
 

Raw lexing routine as provided by flex(1).

Only some additional code is added for position handling.

Token is stored in passed token entry struct.

Definition at line 401 of file iscannerbase.cc.

References _lex(), _MakeCurrLocRange(), Assert, lex_token, _InternalFlexScannerBase::TokenEntry::lloc, magic, MyMagic, n_errors, p0, p1, _InternalFlexScannerBase::LexerInput::read_eof, scanner, _InternalFlexScannerBase::LexerInput::special_next_tok, _InternalFlexScannerBase::TokenEntry::token, and top_inp.

Referenced by LexNextToken().

void _InternalFlexScannerBase::_reset (  bool  no_virtuals = 0  )  [protected]
 

Reset scanner.

State afterwards like after construction but not changing config/tuning.

Will call reset() to reset the derived scanner unless no_virtuals is set. Virtual functions in PREFIX_FlexScannerBase will always be called as the design is in a way that _reset() gets called from within the PREFIX_FlexScannerBase con/destructurs.

Definition at line 487 of file iscannerbase.cc.

References _lex_destroy(), _TidyUpZombieList(), _wrap(), Assert, file_depth, lex_token, _InternalFlexScannerBase::XYPos::line, _InternalFlexScannerBase::XYPos::lpos, magic, MyMagic, n_errors, p0, p1, pos_arch, pos_arch_allocated, read_inp, reset(), scanner, and top_inp.

int _InternalFlexScannerBase::_SetInputBH (  LexerInput *  inp,
bool  included,
XYPos  alt_pos,
SourcePositionArchive *  alt_pos_arch
)  [protected, virtual]
 

Implement "bottom half" of SetInput(); is done in PREFIX_ class.

Definition at line 607 of file iscannerbase.cc.

References CritAssert.

Referenced by SetInput().

void _InternalFlexScannerBase::_TidyUpZombieList (  bool  force = 0  )  [inline, protected]
 

Tidy up zombie list.

Definition at line 290 of file scannerbase.h.

References __TidyUpZombieList(), last_zombie_list_nents, zombie_list_nents, and zombie_list_threash.

Referenced by _reset(), and LexNextToken().

TokenEntry* _InternalFlexScannerBase::_tok_ptr (   )  [inline, protected]
 

Get pointer where to store token info (string/integer/float val...).

Definition at line 336 of file scannerbase.h.

References lex_token.

int _InternalFlexScannerBase::_wrap (   )  [protected, virtual]
 

This implements the _wrap() function; is done in PREFIX_ class.

Definition at line 600 of file iscannerbase.cc.

References CritAssert.

Referenced by _reset(), and LexNextToken().

_InternalFlexScannerBase::TokenEntry * _InternalFlexScannerBase::AllocTokenEntry (   )  [protected, virtual]
 

Allocate a new TokenEntry.

This must be overridden if you use your own derived version.

Definition at line 67 of file iscannerbase.cc.

Referenced by LexNextToken().

SCLocation _InternalFlexScannerBase::CurrLoc (   )  const [inline]
 

Get current lexer location.

Normally not needed at the current location is passed with the TokenEntry returned by LexNextToken().

Definition at line 521 of file scannerbase.h.

References _MakeCurrLoc().

_InternalFlexScannerBase::TokenEntry * _InternalFlexScannerBase::LexNextToken (   ) 
 

Primary lexer function; get next token.

This doest the lexical analysis. It may read some more input, and will finally return the next token. The returned data is allocated by this class (using the virtual AllocTokenEntry()) and may not be freed by outside functions.

Returned tokens may be used by a parser and must return 1 in the MayBeCleared() functions once they were used and may now be cleared and re-used.

For example, a TokenEntry may contain a "char *string_val;" which gets allocated using a strdup()-like manner in the lexer. (We cannot use a string class such as TLString because bison wants to put the tokens into a union{}.) Then, the token can be queued in the bison stack for some time and finally get reduced. Upon reduction, a rule is exectued which will, e.g. make an identifier tree node out of the token. This should then set some flag in the token that it was used and can be re-used, e.g. by setting *string_val='\0'. (Note that setting string_val=NULL or something like that will NOT function since the token in the bison stack is a memcpy()-like copy of the original.)

Note that there is no problem for simple types as they are copied into the bison stack using memcpy() or the like; the problem lies in the pointer value.

All allocated tokens are also in a list created by the lexer and will be asked if they can be cleared once. If the token answers "no" (0) (e.g. token with string_val which is not yet NULL, i.e. used), the token is put into the zombie_list. It stays there until end of parsing.

This is needed especially for the case that there is a parse error and bison pops off tokens from the stack (reduction) in "error" rules. These tokens will never get used and if the lexer would not take care of them, this would yield to a memory leak in string_val (in the example above).

Hence, at end of parsing, the TokenEntry::clear() function will be called with argument force=1 meaning that the arg has to be cleared even if MayBeCleared() still indicates no. In this case, the token got lost somewhere (probably in bison error recovery).

NOTE!! Never pass the TokenEntry directly as token type to bison. TokenEntry (or any derived) class is a C++ type and cannot be stuffed into an union; furthermore it is not "C++-safe" (due to linked list header). If you need an YYSTYPE, simply put the bison-provided one _inside_ the TokenEntry like that: 

   struct MyTokenEntry : TokenEntry
   {
       YYSTYPE lval;  // <-- contains string_val. 
   };

Return value is NULL on EOF.

Definition at line 253 of file iscannerbase.cc.

References _RawYYLex(), _TidyUpZombieList(), _wrap(), _InternalFlexScannerBase::Config::ahead_toks_wanted, AllocTokenEntry(), LinkedList< TokenEntry >::append(), Assert, cfg, _InternalFlexScannerBase::TokenEntry::clear(), LinkedList< TokenEntry >::count(), LinkedList< TokenEntry >::IsEmpty(), magic, _InternalFlexScannerBase::TokenEntry::MayBeCleared(), MyMagic, _InternalFlexScannerBase::LexerInput::next_toks, _InternalFlexScannerBase::Config::old_toks_wanted, LinkedList< TokenEntry >::PopFirst(), _InternalFlexScannerBase::LexerInput::prev_toks, _InternalFlexScannerBase::LexerInput::read_eof, read_inp, scanner, _InternalFlexScannerBase::TokenEntry::token, top_inp, Warning(), zombie_list, and zombie_list_nents.

void _InternalFlexScannerBase::operator= (  const _InternalFlexScannerBase &   )  [private]
 

Do not use.

void _InternalFlexScannerBase::reset (   )  [protected, virtual]
 

Reset scanner.

Scanner state afterwards should be like after construction but not changing config/tuning.

Need not be overridden if not used; do NOT call _reset() from this function.

Called by _reset().

Definition at line 534 of file iscannerbase.cc.

Referenced by _reset().

int _InternalFlexScannerBase::SetInput (  const TLString &  path,
SError &  error,
bool  included,
int  first_token = -1,
XYPos  alt_pos = XYPos(1, 0),
SourcePositionArchive *  alt_pos_arch = NULL
) 
 

Set input from where to read data to be lexically analyzed.

Parameters:
path: The file to be read
error: Used to return error details
included: If set, treat passed input as "included", i.e. stop reading the current input and parse this input before continuing with the previous one.
first_token: If >=0, this is the first token to be returned by LexNextToken() before any tokens from the data to be lexically analyzed. Useful to bring the parser into the correct initial state.
alt_pos: You can pass the start position in the last 2 arguments if they are non-standard.
alt_pos_arch: In case you specify an alternate position, you may also pass a SourcePositionArchive to use. YOU THEN HAVE TO make sure that the current path is equal to the passed path.
Returns:
  • 0 -> OK
  • 1 -> must reset; (another input is active and "included" was not set).
  • -2 -> failed to open file (see error)
  • -3 -> may not use alt_pos_arch if included=1.
  • -4 -> includes nested too deeply

Definition at line 218 of file iscannerbase.cc.

References _lex_init(), _SetInputBH(), Assert, cfg, DELETE(), _InternalFlexScannerBase::LexerInput::down, file_depth, magic, _InternalFlexScannerBase::Config::max_file_depth, MyMagic, _InternalFlexScannerBase::LexerInput::OpenFile(), scanner, _InternalFlexScannerBase::LexerInput::special_next_tok, and top_inp.

Member Data Documentation

Config _InternalFlexScannerBase::cfg [protected]
 

Basic scanner configuration.

Definition at line 310 of file scannerbase.h.

Referenced by _MakeCurrEndLoc(), _MakeCurrLoc(), _pi_tab(), LexNextToken(), and SetInput().

int _InternalFlexScannerBase::file_depth [protected]
 

Current include hierarchy depth:.

Definition at line 269 of file scannerbase.h.

Referenced by _reset(), and SetInput().

int _InternalFlexScannerBase::last_zombie_list_nents [protected]
 

Number of entries after last tidy up.

Definition at line 288 of file scannerbase.h.

Referenced by __TidyUpZombieList(), _InternalFlexScannerBase(), and _TidyUpZombieList().

TokenEntry* _InternalFlexScannerBase::lex_token [protected]
 

Exclusively used to pass current TokenEntry() to the flex-based lex() routine (yylex); may be accessed to get access to token information (e.g. put parsed integer value of integer token there).

Definition at line 260 of file scannerbase.h.

Referenced by _InternalFlexScannerBase(), _RawYYLex(), _reset(), and _tok_ptr().

uint32 _InternalFlexScannerBase::magic [protected]
 

For simple detection of memory corruption or evil pointer mess.

Definition at line 248 of file scannerbase.h.

Referenced by _InternalFlexScannerBase(), _RawYYLex(), _reset(), LexNextToken(), SetInput(), and ~_InternalFlexScannerBase().

const uint32 _InternalFlexScannerBase::MyMagic = 0xdeadbeefU [static, protected]
 

What the magic value should be:.

Definition at line 250 of file scannerbase.h.

Referenced by _InternalFlexScannerBase(), _RawYYLex(), _reset(), LexNextToken(), SetInput(), and ~_InternalFlexScannerBase().

int _InternalFlexScannerBase::n_errors [protected]
 

Accumulate number of errors.

Definition at line 272 of file scannerbase.h.

Referenced by _RawYYLex(), and _reset().

XYPos _InternalFlexScannerBase::p0 [protected]
 

Current token: start position; while parsing: previous position.

Definition at line 298 of file scannerbase.h.

Referenced by _MakeCurrLoc(), _pi_start(), _RawYYLex(), and _reset().

XYPos _InternalFlexScannerBase::p1 [protected]
 

Current token: end position; while parsing: current position.

Definition at line 300 of file scannerbase.h.

Referenced by _MakeCurrEndLoc(), _pi_char(), _pi_line(), _pi_start(), _pi_tab(), _pi_tok(), _RawYYLex(), and _reset().

SourcePositionArchive* _InternalFlexScannerBase::pos_arch [protected]
 

Used to generate source position objects; see there.

Definition at line 303 of file scannerbase.h.

Referenced by _InternalFlexScannerBase(), _MakeCurrEndLoc(), _MakeCurrLoc(), and _reset().

bool _InternalFlexScannerBase::pos_arch_allocated [protected]
 

If _we_ allocated the pos archive or if it was passed to us via some input setting function.

Definition at line 307 of file scannerbase.h.

Referenced by _InternalFlexScannerBase(), and _reset().

LexerInput* _InternalFlexScannerBase::read_inp [protected]
 

Which input has the token buffer we're currently processing. Is different from top_inp around an #include.

Definition at line 266 of file scannerbase.h.

Referenced by _InternalFlexScannerBase(), _reset(), LexNextToken(), and ~_InternalFlexScannerBase().

void* _InternalFlexScannerBase::scanner [protected]
 

The scanner used for scanning; only one; NULL if inactive.

Definition at line 255 of file scannerbase.h.

Referenced by _InternalFlexScannerBase(), _RawYYLex(), _reset(), LexNextToken(), SetInput(), and ~_InternalFlexScannerBase().

LexerInput* _InternalFlexScannerBase::top_inp [protected]
 

Which input the scanner is currently reading from:.

Definition at line 263 of file scannerbase.h.

Referenced by _InternalFlexScannerBase(), _pi_eof(), _RawYYLex(), _reset(), LexNextToken(), SetInput(), and ~_InternalFlexScannerBase().

LinkedList<TokenEntry> _InternalFlexScannerBase::zombie_list [protected]
 

Zombie token list.

In short, TokenEntries of that type get queued here until they are used. For more information, see the docu of LexNextToken().

Definition at line 280 of file scannerbase.h.

Referenced by __TidyUpZombieList(), and LexNextToken().

int _InternalFlexScannerBase::zombie_list_nents [protected]
 

Number of entries in the zombie_list.

Definition at line 282 of file scannerbase.h.

Referenced by __TidyUpZombieList(), _InternalFlexScannerBase(), _TidyUpZombieList(), and LexNextToken().

const int _InternalFlexScannerBase::zombie_list_threash = 256 [static, protected]
 

Zombie list tidy up threshold. Don't use too small values.

Definition at line 284 of file scannerbase.h.

Referenced by __TidyUpZombieList(), and _TidyUpZombieList().

The documentation for this class was generated from the following files:
Generated on Sat Feb 19 22:35:16 2005 for Ray by doxygen 1.3.5