question about tokens?

Parsing is a very general process - with regard to compiling it''s breaking down the textual code into lexical tokens (program language things, like a for loop, or a multiplication, assignment, etc...)

A token is the vague concept of a single item of something that represents something else. A Chucky-Cheese token represents 1/4 the amount of money you need to play a game. A compiler token represents one thing of language (again, things like for loops, assigments, an addition, etc...)

Magmai Kai Holmlor
- Not For Rent

Continuing Magmai''s explanation, an English language token is a word.

You''re probably interested in string tokenization. There''s a Standard C Library function called strtok() especially for this purpose, while the Standard C++ Library requires you to use find_..._of() family of functions.

// C code
// tokenize takes the string, the array in which to store the
// tokens and the string of separators, and returns the number
// of tokens found
int tokenize(char *string, char **tokens, char *seps)
{
  int nTokens = 0;
  char *tok = strtok(string, seps);
  while(tok != NULL)
  {
    // note that I''m simply assigning pointers here. You might
    // want to memcpy() the string instead
    tokens[nTokens] = tok;
    ++nTokens;
  }
  return nTokens;
}
.
// C++ code
// tokenize takes a string object, a vector of strings in which
// to store the tokens and the string of separators, and returns
// the number of tokens found
int tokenize(string str, vector<string> &tokens, string seps)
{
  int nTokens = 0;
  int pos0 = 0, pos1 = 0;
  while(true)
  {
    pos1 = str.find_first_of(seps.c_str(), pos0);
    if(pos1 == string::npos)
      break;
    tokens[nTokens] = str.substr(pos0, (pos1 - pos0));
    // get ready for next extraction
    pos0 = str.find_first_of(seps.c_str(), ++pos1);
    ++nTokens;
  }
  return nTokens;
}
 

If any of the above code doesn''t perform as expected, let me know so I can send you some code somewhere on my D drive that correctly tokenizes STL strings.

---

Start Here!

now in usage of scripting, how would i compare those tokens to something?

meaning if the word "resize" was tokenized, how would i have my program know to resize a matrix? will i hundreds of if/case statements to do all possible commands?

One way would be to use some form of table to map keywords to functions/objects. Something like this, f.ex:

class ICommand { public: virtual void execute( const Arguments& args ) = 0; }; //... std::map< std::string, ICommand* > commandMap; //... std::string command = getNextCommand(); Arguments args = getArgs( ); ( commandMap[ command ] )->execute( args );

Psalm 137:9: "Happy shall he be, that taketh and dasheth thy little ones against the stones."