Skip to main content
GameDev.net gamedev.net
🔒 Locked

Lex & Yacc

Started by PumpkinPieman Dec 8, 2005 at 1:45 AM 13 replies 2k views
Original Post
PumpkinPieman
PumpkinPieman
I'm starting to think about adding scripting to my small engine. Actually I have been thinking of it for a while, and it was only until a week ago that I started revisiting the topic. Taking a quick glance at the articles and topics I notice that Lua seems to be a popular scripting language amongst most people. Now, what about Lex & Yacc? (I'll hold the phrase scripting language very loosely because I'm still not entirely sure what I can do with it) A while ago I was talking about game scripting with a friend of mine, and he recommended me a book called "Lex & Yacc - O'Reilly by John Levine, Tony Mason, & Doug Brown". I started reading through it and saw plenty of material that I could apply, but is this a wise choice when it comes to scripting? It's been a while since I've gone through the book, but from what I can recall it takes a script file and generates a C file out of it. I was just wondering if I could use it as well to run methods at run time from a script file, as part of some on the spot debugging. Also, on a side note... Right now I have my engine laid out where you have to initialize the graphics class before any other graphics linked classes can be used (texture, etc). The problem I have with it is that I have it so the graphics class is passed to each created object that requires the use of the device. Would it be in my best interest to have a global pointer to the graphics class seeing as there will only be one initialized? (May be easier for a scripting engine to handle a global pointer.) Thanks
haegarr
haegarr
lex & yacc (or flex & bison if going to go with C++) are a lexical analyser and a parser _generator_. Input to these is a language grammer (_not_ a script file), and output is the lexer and parser in C/C++ source code.

The actual scripting is something different. E.g. the resulting lexer/parser are able to work on a input file written in a scripting language (from which the aforementioned grammer is) and to make C routine invocations of them.

So using lex & yacc is only a kind of "preprocessing", yielding in the feasibility to work on a script file. So lex & yacc would be fine in a first order if you want to realize your own scripting language syntax.

EDIT: However, be aware that you still have to write glue code. Using a complete scripting language w/ a suitable binding to the C (C++, or whatever you are using) would be definitely less work.

[Edited by - haegarr on December 8, 2005 2:03:41 AM]
haegarr
haegarr
Quote:
Original post by PumpkinPieman
What I want to do is having a scripting language for debugging purposes, but it will eventually be compiled.

Compiled into what kind of thing? Do you mean it should be compiled into the machines native CPU code? Or do you mean it should not be interpreted from text but run in a kind of byte code similar to Java?

IMHO the ad-hoc investigation of the state of the app (by using an interpreter) driven by scripting language is definitely fine, but the need to compile debugging stuff into native CPU code from a scripting language isn't clear to me. (But I never have thought about that before, so maybe I simply missing something.)

It is possible to do so using lex & yacc, right you are. There're also some more or less complete C and C++ grammar files available on the internet suitable for lex & yacc (or flex & bison? don't remember), if you don't want to define your own language syntax (others may be available, too).
evolutional
evolutional
Download GameMonkey Script and take a look at the lexx / yacc files it uses. GMScript is a Lua-like language with a C-style syntax, the flex/bison files are pretty simple to understand and you can study how they hooked it up to the bytecode generator.

Hope this helps - good luck with your language!

(PS: You might also wish to investigate the GOLD parser too)
PumpkinPieman
PumpkinPieman
evolutional: Thanks, I'll take a look.

Quote:
Original post by haegarr
Quote:
Original post by PumpkinPieman
What I want to do is having a scripting language for debugging purposes, but it will eventually be compiled.

Compiled into what kind of thing? Do you mean it should be compiled into the machines native CPU code? Or do you mean it should not be interpreted from text but run in a kind of byte code similar to Java?

IMHO the ad-hoc investigation of the state of the app (by using an interpreter) driven by scripting language is definitely fine, but the need to compile debugging stuff into native CPU code from a scripting language isn't clear to me. (But I never have thought about that before, so maybe I simply missing something.)

It is possible to do so using lex & yacc, right you are. There're also some more or less complete C and C++ grammar files available on the internet suitable for lex & yacc (or flex & bison? don't remember), if you don't want to define your own language syntax (others may be available, too).
I want to avoid byte-code for final releases and have it compiled into native CPU code, but for debugging purposes byte code would do fine. Something along those lines.

Extrarius
Extrarius
PumpkinPieman: Very few scripting langauges are compiled to native code. Unless your game targets legacy or otherwise limited hardware, byte code should be fast enough if you work a little on optimizing the VM. Most (probably all) commercial games that use scripting languages use bytecode or worse (things like direct interpretation aren't entirely unusual). Realize that to compile to native code, you essentially have to write an assembler. Writing the basics isn't too difficult, but making a to-assembly compiler and the assembler work together to generate good native code is very difficult.
If you don't mind releasing your game under the GPL, you could incorporate code from projects like nasm or gcc, but then you open your game to cheating and make any possible exploits easier to find (much more likely to exist if your program generates actual executable code to run based on scripts), which are a big deal if your game is multiplayer.
"Walk not the trodden path, for it has borne it's burden." -John, Flying Monk
PumpkinPieman
PumpkinPieman
To be honest I really don't like the idea of running bytecode with a VM, but what are actually the benifits of using bytecode \ scripts besides that you can avoid compiling your application? and what exactly would the scripts control (are they more of a layer on top of a game, or a layer on top of an engine?
evolutional
evolutional
Quote:
Original post by PumpkinPieman
To be honest I really don't like the idea of running bytecode with a VM, but what are actually the benifits of using bytecode \ scripts besides that you can avoid compiling your application? and what exactly would the scripts control (are they more of a layer on top of a game, or a layer on top of an engine?


You can sandbox the VM bytecode, catching any errors in the actual code and stop them from crashing the application, exploiting the application or even worse... If you allow compilation to native code, there's a good chance that someone will figure out how to execute code you didn't intend them to unless you can guarantee it's sandboxed. Errors in native code can crash the application, whereas errors in scripted bytecode will just cause the current script to be terminated.

As for how it's organised, a scripting VM will be made aware of your engine functions, allowing your scripts to call them. In my recent work I've built a basic 2d engine API that can be used with C/C++ but on top of that there's a whole scripting layer. The idea is that you can do anything the C API can do without having to 'compile' your code; it's very safe, mod-friendly and quick for prototyping things :)
Extrarius
Extrarius
Quote:
Original post by PumpkinPieman
To be honest I really don't like the idea of running bytecode with a VM, but what are actually the benifits of using bytecode \ scripts besides that you can avoid compiling your application? and what exactly would the scripts control (are they more of a layer on top of a game, or a layer on top of an engine?
The main benefit of running bytecode (besides those mentioned by evolutional) is that you don't have to go to the trouble of making an assembler or the last few steps in a compiler, because you can suit the bytecode to the language and that can save a LOT of work. Otherwise, SwiftShader wouldn't be such a big deal - all it does is translate from graphics shader languages to native code like you want to do, but because fast native code generation is not easy it is quite an acomplishment. Note that simply generating native code doesn't magically increase speed - it takes a lot of work to make it significantly faster.

As far as what script normally does - it ususally uses high level engine functions to make the logic for a game. It doesn't handle every minute detail, but instead delegates responsibility to the engine and just tells it when and what to do. For example, you wouldn't have a function in script that updates the physics of an object each frame. Instead, you would set the object's physics type (ie does gravity affect it, etc) then set the initial parameters (such as velocity and acceleration) and then only occasionally adjust the parameters while the engine automatically applies the parameters each time it updates physics.

A good way to see how scripting is used would be to look for mods for any Unreal game, because unreal script isn't too difficult to understand. You can find a lot of information about unrealscript at the Unreal Wiki
"Walk not the trodden path, for it has borne it's burden." -John, Flying Monk
PumpkinPieman
PumpkinPieman
evolutional:
I'm not completely sure what you mean by 'sandboxing' but I do have an idea what you're trying to get at. My question is (since I don't know much about hacking or topics there of) couldn't someone potentially modify the byte code (given they understand it) and access methods that way?

Is GameMonkey Script the closest type of small script language to C? To be honest I'd rather script in something with C type syntax, or something familiar.

Extrarius:
My problem will be how I'm going to achieve custom sprites for a 2D engine. Right now my focus is on small sprite based games, but as you might know there are quite a few variables required to create various still\animated sprites.

Sorry for being dense, but on the topic of high level engine functions and the example you give. Since I haven't done a whole lot of physics in my application I can't really picture how it would work. Would that mean that the methods that are called work independant of the engine itself. Say for example I add the methods addTexture, and removeTexture would those methods use a resource manager independant of other created resource managers in the engine? (maybe I'm just confusing myself) Okay, back to physics. I don't have any physics resources in place right now, so if I were to create objects of a class that physical properties could be applied to. Methods like addForce() would use a resource manager that would be used only by the scripting engines methods.

flangazor
flangazor
Sandboxing just means having a lot of security and validity checks to make sure code being run isn't malicious. Having a well defined interface into your game engine means you can catch the places where malicious code might be run. Also, a watchdog timer to make sure the script isn't running too long (infinite loop, etc).

By malicious, I mean intentional and unintentionally unsafe code. And by unsafe, I mean code that might crash the main process or make it do things you don't want it to (e.g. through buffer overflows causing specific code to be run which will expose your computer to security risks).

GameMonkey does have the superficial resemblence to C that will make you comfortable. Please check the link evolutional gave you.

From what you describe as your needs, you sound like you only want some data files for packaging sprite sets. That shouldn't require a scripting language.

[Edited by - flangazor on December 13, 2005 7:39:23 AM]
PumpkinPieman
PumpkinPieman
Well, not totally. I have other things I could implement for my engine. In terms of sprites I already have a format and an editor, I'm just not sure if the way my sprite code works is flexable and good for a scripting langauge.
evolutional
evolutional
I mention GameMonkey Script a lot because that's my main scripting language and is like Lua in many ways, except IMHO it feels more comfortable to work with from both a language point of view and as an API. AngelScript (www.angelcode.com) also has a C-style syntax and should be fairly easy to expose some of your functions to its environment (I believe it's also strongly typed, like C). There's also "Small", which again is supposed to be a lot like C - there's several C-like languages around as you can see :)
PumpkinPieman
PumpkinPieman
I've been looking over LUA\GameMonkey\AngelCode and I think AngelCode is the best route for me, it also looks very straight foward and flexable.

Thanks for your help guys.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.

Sign in to reply to this topic.