🔒 Locked

Script Lexer

Started by BJRowlett Jul 20, 2007 at 9:07 AM 2 replies 1.1k views

Original Post

Jul 20, 2007 at 9:07 AM

Hey, Im new to the fourms, Ive been programming for a couple of years. (i havent taken any classes because my high school never offers the programming classes cause not enough students sign up for them...) Most of my programming has been in autoit, ive made a few programs for myself just to make things easier, and a couple for my dad whos the vice president of a credit union. Ive also made a few bots in C++ for Diablo II, nothing major like d2jsp or anything, they just use SendMessage to control the game by mouse clicks and sending keys, and very simple memory reads to detect health/mana and things. But lately, ive been interested in a much larger project. Something i understand will be difficult, and time consuming. Its a language lexer, written in C/C++ that will parse a scripting language similar to &#106avascript. Before you flame! i know it already exists, its called SpiderMonkey, but i think it could be a good, challenging project. Ive found a tutorial, that ive read about half of so far, (http://www.flipcode.com/articles/scripting_issue01.shtml), this tutorial uses FLex to generate the code, but i was actually planning on writing my own, to get the full experience. And i was wondering if anyone had anymore tutorials, or any information that would help me. Thanks alot everyone, Brian

ToohrVyk

238

ToohrVyk

Jul 20, 2007 at 9:26 AM

Writing a naive lexer is very simple. You write, for each token type, a function which takes an iterator as argument by reference and attempts to read the token starting from that iterator, changing it so that it is placed at the end of the recognized token (or doesn't move, if the token isn't recognized) and returns the token.

For instance:

namespace Tokenizer{  typedef std::string Identifier;  typedef std::string Operator;  typedef int Literal;  struct EOF {};  struct ERROR {};    typedef boost::variant<ERROR,Identifier,Operator,Literal,EOF> Token;  template<typename It>  Token lexIdentifier(It *& input, It end)  {    It begin = input;    while (      (input != end) && (       (*input > 'a' && *input < 'z') ||        (*input > 'A' && *input < 'Z')     ))    {      ++input;    }    return std::string(begin,input);      }  template<typename It>  Token lexOperator(It *& input, It end)  { /* implement operator lexing */ }  template<typename It>  Token lexLiteral(It *& input, It end)  { /* implement literal lexing */ }  template<typename It>  Token tokenize(It *& begin, It end)   {    if (begin == end) return EOF();       It cursor = begin;    Token t;    t = lexIdentifier(begin,end);    if (begin != cursor) return t;    t = lexOperator(begin,end);    if (begin != cursor) return t;    t = lexLiteral(begin,end);    if (begni != cursor) return t;    return ERROR();  }}

This might be slow, but it's the fastest you'll get with a tutorial. If you want to have access to serious techniques for lexer optimization, you'll need to buy and read a book on compiler design.

BJRowlett

Author

BJRowlett

Jul 21, 2007 at 7:54 AM

Okay, i havent had alot of time to work on it, because of work. But i got up early today to get some time in before work, and as of now, it reads a file and can distinguish between uppercase letters, lowercase letters, numbers and operators. I havent written the functions to return the tokens for each token type, as of now all it does is know what type of data its dealing with.

Hopefully sometime soon ill get some free time.

Thanks for your help,
Brian

umbrae

Jul 28, 2007 at 5:16 AM

As an idea, perhaps you could think of ways to improve and or simplify &#106avascript (ECMA). I don't think it's the cleanest and nicest of languages and it would be a shame for you to implement something that has already been done.<br/><br/>I'm working on a language at the moment but work constraints have restricted any recent advancements. I think it's great when anyone challenges the satus-quo - think of features and things that you would like in a language, discuss it here or on other dedicated forums and implement them.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.