Provides an incremental tokenizer

     TOKENNEXT(<idTokenInitVar>) --> cToken


     <idTokenInitVar>  Designates the name of the character string
     previously initialized using TOKENINIT().  This is the only way to
     access this string once it has been stored by the Virtual Memory Manager


     TOKENNEXT() returns the next token from the string variable designated
     by TOKENINIT().  If there are no more tokens, a null string is returned.


     In conjunction with TOKENINIT(), this function provides a speed
     optimized variation of the normal TOKEN() function.  The increase in
     speed is achieved in two ways.

     TOKENINIT() exchanges all delimiting characters for the first delimiter
     in the list.  This means the entire delimiter list does not have to be
     searched every time.  The second advantage is that TOKENNEXT() does not
     always begin its search for the token that is extracted at the beginning
     of the string.

     The TOKENAT() function allows you to determine the position of

     Although the address of the string that is processed and the internal
     counter have already been determined or initialized by TOKENINIT(),
     Clipper Tools still needs the variable name.  The function does work
     without this parameter, but only as long as the initialized character
     sequence has not been stored to VMM.  You should enter the variable name
     in each case because there is no explicit control over this!  If there
     is no access to the variable, a runtime error occurs.


     .  When you use TOKENINIT() or TOKENNEXT(), you cannot use the
        TOKENSEP() function.  The required information can be determined
        using TOKENAT() in conjunction with the original string (status
        before TOKENINIT()).

     .  To determine the delimiter position before the last token, set
        TOKENAT() to -1.  To determine the delimiter position after the last
        token, set TOKENAT to .T..


     Break down a string:

     cDelim   :=  "!?.,-"
     cString   :=  "A.B-C,D!E??"

     TOKENINIT(@cString, cDelim)      // "A!B!C!D!E!!"

        cWord  :=  TOKENNEXT(cString)
        ? cWord


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.