Skip to main content
GameDev.net gamedev.net
🔒 Locked

LUA serialization and debugging questions

Started by bubububu Oct 18, 2008 at 9:36 PM 10 replies 5.5k views
Original Post
bubububu
bubububu
Hiya! I'm new to the forums! I want to use LUA for an upcoming 3d engine but there are two questions that preocupy me: 1. How do you deal with LUA serialization(persistence)? I see there are no official built-in libraries in LUA. All I found is this small entry in the LUA docs: http://www.lua.org/pil/12.1.html but I need custom serialization(to define what I want and what I don't want to save/load).... and I'm not completely sure that mechanism could store really complex tables with a lot of circular and "pointer" references. I've seen 3rd-party serialization libraries like Pluto but I am not completely convinced about them. I really cannot understand why LUA does not have a built-in serialization mechanism like python or .NET. I'm curious about how Crytek and Blizzard solved the LUA serialization problem. 2. A scripting system without debugging support won't be very useful. I need embedding debugging support too(which adds an extra difficulty). I don't know any good LUA debuggers for this.... but if the game level designers can't debug the scene entities(Actors in UE3) I don't think that script language could be good and productive... [Edited by - bubububu on October 18, 2008 10:10:48 PM]
ddn3
ddn3
I've also looked at Pluto, from looks of it, its pretty complete. I'm using LuaJit and I've heard there are issues. Once I do some test with it I'll post my findings.

As for a good debugger, I use Decoda for step through debugging, it has saved me alot of time. I highly recommend it. If you have any issues getting it working, just post it here.

Good Luck!

-ddn
bubububu
bubububu
Quote:

Is there a version of Decoda for Linux or Macintosh?
At the moment Decoda only supports Windows



Hehe, I forgot to mention we work in linux and macosX too :p byt it seems very complete for the Windows platform.
Sneftel
Sneftel
First, a disclaimer: I'm the author of Pluto.

Quote:
but I need custom serialization(to define what I want and what I don't want to save/load).... and I'm not completely sure that mechanism could store really complex tables with a lot of circular and "pointer" references.
Other than coroutines and functions, it is possible to serialize arbitrarily complex data using a pure-Lua approach. Check out Lua Pickle for an example.

Quote:
I've seen 3rd-party serialization libraries like Pluto but I am not completely convinced about them.
What would you like to know?

Quote:
I really cannot understand why LUA does not have a built-in serialization mechanism like python or .NET.
Boy howdy. Pluto has to do a fair amount of ugly behind-the-scenes stuff in order to serialize all types. It would make much more sense, IMHO, for such a thing to be at least partially integrated into the core... but the authors of Lua have decided not to go that direction.

Quote:
I've also looked at Pluto, from looks of it, its pretty complete. I'm using LuaJit and I've heard there are issues. Once I do some test with it I'll post my findings.
You won't be able to serialize coroutines with Pluto under LuaJIT. Other than that, I'm not aware of any problems (though I personally haven't done it, I've gotten reports from a few who have).

bubububu
bubububu
Quote:
Original post by Sneftel
First, a disclaimer: I'm the author of Pluto.

hello! :D

Quote:
Other than coroutines and functions, it is possible to serialize arbitrarily complex data using a pure-Lua approach. Check out Lua Pickle for an example.

Oh good! I didn't realize that was there! But... can it serialize complex circular references and "pointers"? And.. that seems not to allow custom serialization ( python's pickle uses getstate/setstate )... and neither to pickle simple values or strings.

Quote:
What would you like to know?(about Pluto)

Can I use it in a commercial engine? Can it manage circular-references?
Is it resistant to versioning? Can it use custom serialization(define what is serialized and what not)? Does it allow non-intrusive serialization like boost?

Just need the serialization for this: I want to create LUA "actors"(scene entities) for my game engine. Then, save the complete scene actors data(attributes, elements, circular references... in a "custom" way) to a file so it can be restored completely when the game level is loaded. Ideally, I would like a method like the python's pickle(getstate/setstate) or the .NET GetObjectData() custom keyed serialization mechanism. The serialization format is irrelevant for me(can be text, XML or binary)... but need to be resistant to versioning and I would prefer keyed serialization vs version sequential storage.

thx

[Edited by - bubububu on October 19, 2008 4:56:12 PM]
Sneftel
Sneftel
Quote:
Original post by bubububu
Can I use it in a commercial engine?
I assume so. Non-permissive-licensed Lua modules are rare.
Quote:
LuaPickle: Can it manage circular-references? Is it resistant to versioning? Can it use custom serialization(define what is serialized and what not)? Does it allow non-intrusive serialization like boost?
For the first one: Yes. For the rest: no idea. Anything it doesn't have, though, shouldn't be too difficult to add.

Quote:
Pluto: Can I use it in a commercial engine? Can it manage circular-references? Is it resistant to versioning? Can it use custom serialization(define what is serialized and what not)? Does it allow non-intrusive serialization like boost?

Yes, yes, kind of, yes, kind of. Pluto is versioning-agnostic... if an object has a particular field when it is saved, that field will be there when the object is reloaded. Handling unexpectedly missing fields due to version differences is left to client code. As for non-intrusive serialization, custom serialization routines for objects are through the metatable instead of the object itself. So you don't need to change the object's interface or its behavior, but you do need to change its metatable. (There's a way around this, but it's probably not worthwhile.)

For more info, see the README. Tell me if there's anything unclear in it.
bubububu
bubububu
First, thanks for all the help, Sneftel.

Quote:

For more info, see the README. Tell me if there's anything unclear in it.


Well, I don't know much about LUA metatables... could you indicate me how this simple python object could be serialized/deserialized using Pluto, please?

import cPickle #this is the python serializing mechanism#this is the basic scene actor/entity/node/object, call it like you wishclass Actor:    def __init__(self):        #set object's hit points to 1000 by default        self.HP = 1000     def __getstate__(self):         #this is the serialize method        #returns a dictionary with key:HP,value:hit points        return { 'HP':self.HP }        def __setstate__(self,state):        #this is the deserialize method        if 'HP' not in state:            #versioning resistance... if this attribute is not found, set a default value            self.HP = 1000        else:            self.HP = state['HP']#This is an "actor" implementation for a player. The player inherits from Actorclass Player(Actor):     def __init__(self): #this is something like a C++ constructor        #call base class "constructor"        Actor.__init__(self)                #Set properties's default values        self.IsDead = False        self.MshModel = 0        self.DieStr = "I'm alive"    def __getstate__(self):        #deserialize base class        basePickeData = Actor.__getstate__(self)        #return a dictionary with the serialization data        return {'base':basePickeData, 'IsDead':self.IsDead, 'MshModel':self.MshModel, 'DieStr':self.DieStr}    def __setstate__(self,state):        #call base class's serialization        if 'base' not in state:            Actor.__init__(self)        else:            Actor.__setstate__(self,state['base'])        if 'IsDead' not in state:            self.IsDead = False        else:            self.IsDead = state['IsDead']			        if 'MshModel' not in state:            self.MshModel = 0        else:            self.MshModel = state['MshModel']			        if 'DieStr' not in state:            self.DieStr = "I'm alive"        else:            self.DieStr = state['DieStr']    def OnDamage(self,dmg):        self.HP -= dmg;        if ( self.HP<0 ):            self.DieStr = "I'm dead! You killed kenny!";    def Tick(self):        self.OnDamage(10)a = Actor()a.Tick()serializedData = cPickle.dumps(a)unserializedData = cPickle.loads(serializedData)


thx, a big and important decision depends on this :D
Sneftel
Sneftel
What that code looks like in Lua depends on how you handle things like inheritance. As for serializing and deserializing things with Pluto, there's no need for the getstate/setstate stuff. Just call pluto.persist(a).

BTW, if you're not familiar with metatables under Lua, I'd suggest that you defer this particular big important decision until you understand the language more fully.
bubububu
bubububu
Quote:

As for serializing and deserializing things with Pluto, there's no need for the getstate/setstate stuff. Just call pluto.persist(a).

Nope, because I need to control what to serialize and what not. For example, imagine I store in memory a password unencrypted... I don't want that data serialized.... or a pointer to a GL display list... I don't want that serialized neither.... Ofc, I could just serialize that and ignore the data I don't need... but I don't wanna waste space(and loading time) and neither to write some potentially unsecure data. That's why I was asking for "custom serialization" getstate/setstate or equivalent.... so I kill two birds in a shot: don't write/read things I don't need and to deal with the versioning problem.

Quote:

BTW, if you're not familiar with metatables under Lua, I'd suggest that you defer this particular big important decision until you understand the language more fully.

Yep, the problem is that I have no idea about python and I got that example more or less working in seconds... while I'm still trying to unsertand LUA... that's exactly the problem :D
Sneftel
Sneftel
I don't mean to be unkind, but answering these sorts of questions for you is going to be a poor use of time if you don't understand the language I'm using to answer them. People I've discussed Pluto with have, without exception, found it to be trivial to learn and internalize, which is unsurprising, because it is an extremely simple library (at least on the outside). All of these people, however, already knew how to program in Lua. Seeing examples of the use of Pluto is not going to teach you how to program in Lua. If you're looking for code to learn from example, a serialization library is not going to provide much of use.
bubububu
bubububu
No matter, thanks for the help.
We're gonna use use python because I think it fits much better to our needs and it has a much better learning curve for the level designers.
Sneftel
Sneftel
If you've decided to take your own first impressions of a language as gospel rather than bother to understand it, then I don't think I've successfully provided you with any actual help. But thanks for the thanks, I suppose.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.

Sign in to reply to this topic.