Linux x86-64 not loading or saving bytecode correctly.

Version 2.22.1 did not yet have a platform independent bytecode, this was only implemented in version 2.23.0.

In version 2.22.1 Linux 64bit should be able to load bytecode compiled on Windows 64bit and vice versa, as should Linux 32bit be able to load bytecode from Win 32bit and vice versa. Linux 64bit won't be able to load Linux 32bit bytecode, and vice versa.

The bytecode does not depend on the registered functions' calling convention, as long as they are registered with the same declaration, the bytecode loader should work correctly.

I'm not aware of any bugs in version 2.22.1 that causes bytecode not to load properly on Linux 64bit. The bug fixes that have been done in later releases are all related to the changes done in version 2.23.0 to make the code platform independent.

Would it be possible to narrow down the script that fails to load properly, so that I can try to determine the cause of the failure? Unfortunately I do not have access to a 64bit Linux environment to test the library on anymore, since Jeremy's not responding anymore, but depending on the situation I may be able to figure out the cause anyway.

May I ask what game it is?

I tried bumping to 2.23.1 or 2.24a and now I get new issues when saving.. I'm getting an offset != currOffset in asCWrite::AdjustGetOffset in 64bit!

Though running 32bit saves and loads the bytecode perfectly fine.. and the 64bit loads that bytecode fine..

Odd.

This is using the 2.24.0 tag from SVN.

Wait.. scratch that.. it's acting REAL funcy now in 64bit .. causing odd errors in game.. Ugh!!!!

it's not easy to see which exact source file is causing issues as the whole thing is compiled into one file via the main.cpp.

And I can send you a PM w/ the game name

Andreas, if its of any help debugging this issue, test_feature doesn't fail but reports the following on x86_64 OS X:

The saved byte code is not of the expected size. It is 1888 bytes
The saved byte code contains a different amount of zeroes than the expected. Counted 620
The saved byte code has different checksum than the expected. Got 0xCC85F81D
-- TestSaveLoad passed[/quote]

Hi quarnster, this tells me that there is indeed something wrong in the 64bit save/load feature. I'll try to see if I can reproduce this on my Win64 machine. Hopefully it is the same problem that urkle is facing.

This is using the 2.24.0 tag from SVN.

If you are going to use 2.24.0 then you should get release 2.240a. There was a rather serious bug in the 2.24.0 release causing memory invasions.

I managed to get access to a 64bit Linux environment to try this on. Unfortunately I was not able to reproduce the problem, neither with version 2.22.1 nor with the latest revision from the SVN.

There are a few minor issues with the latest revision from the SVN that I'll check in tonight, but they are unrelated to bytecode saving/loading.



I will need more information to be able to fix this problem.

First, will you be working with the latest release or stick with version 2.22.1? For me the latest would be best, but I understand if you'd prefer to stick with the version already used by the game on other platforms.

Second, I really need for you to narrow down the script that is failing to load, so I can debug it in my own environment in order to figure out the problem. Try commenting out part of the script, then test the save/load part. If it still fails, comment out more of the script, it it doesn't fail, uncomment the script and comment the other part instead. With a few iterations like this you should be able to get a reasonably small script.

I understand the problem shows immediately as the script is loaded, i.e. the LoadByteCode() method returns an error, and not just when running the game, right? This ought to make it easier to pinpoint the problem.

The minor issues that I found in the tests along with the patches provided by quarnster in the other thread have been checked in to revision 1370.

I backported the tests that I have for the latest WIP to version 2.22.1 and some of the problems that have been fixed in the latest releases were already pre-existent in version 2.22.1 even before the major changes in 2.23.0.

The situations that I found out that fails on 2.22.1 are the following:

- Arrays initialized with function pointers (fails to properly store the function pointers that the array is initialized with)
- Non-shared classes that inherit from shared classes (fails when loading the second module with the same shared class)

However, both of these problems also occur on 32bit so I do not believe they are related to the problem you're facing with Linux 64bit. I won't work on fixes for these problems in version 2.22.1, unless it is determined that they really are what is causing your problem.

Hey Andreas,

I've managed to isolate the problem!
Happens on both Windows and Linux 64 bit with 2.24.0a
Looks like it's caused by an enum followed by a non-primitive type in a script function...:

enum TestEnum
{
TestEnum_A
}
class NonPrimitive
{
}
// Problem function
void Foo( TestEnum e, NonPrimitive o )
{
}
void main()
{
NonPrimitive o;
Foo( TestEnum_A, o ); // Crashes saving bytecode for where it's called
}

"NonPrimitive" can be value or reference, script or application class and it will crash, but primitive types work fine.

Looks like it breaks in as_restore.cpp around line 3369 when it gets to the enum parameter.
Maybe some kind of discrepancy in how enum parameters are treated?
for( asUINT p = 0; p < calledFunc->parameterTypes.GetLength(); p++ )
{
if( offset <= currOffset ) break;
if( calledFunc->parameterTypes

.GetObjectType() ||
calledFunc->parameterTypes

.IsReference() )
{
numPtrs++;
currOffset += AS_PTR_SIZE; // offset is 1 on 32 bit and 64 bit, but this causes currOffset to be 2 on 64 bit
}
else
{
asASSERT( calledFunc->parameterTypes

.IsPrimitive() );
currOffset += calledFunc->parameterTypes

.GetSizeOnStackDWords();
}
}

Great!

I'll give this a try, and provide a fix as soon as possible.

I've reproduced the problem on the latest WIP and will be working on the fix. However, the same scenario is working OK in version 2.22.1, so it doesn't seem to be the cause of the problem that urkle reported.

The fix in 2.24 is easy.


for( asUINT p = 0; p < calledFunc->parameterTypes.GetLength(); p++ )
{
if( offset <= currOffset ) break;
// if( calledFunc->parameterTypes

.GetObjectType() || <-- exchange this line for the following
if( !calledFunc->parameterTypes

.IsPrimitive() ||
calledFunc->parameterTypes

.IsReference() )
{
numPtrs++;
currOffset += AS_PTR_SIZE;
}
else
{
asASSERT( calledFunc->parameterTypes

.IsPrimitive() );
currOffset += calledFunc->parameterTypes

.GetSizeOnStackDWords();
}
}

The function AdjustGetOffset() doesn't exist in version 2.22.1. It was introduced in version 2.23.0 so the problem in 2.22.1 is definitely a different one.

Regards,
Andreas

I've checked in this fix in revision 1372.

Thanks for the prompt fix Andreas!

The bytecode saves out ok from 64 bit now.
Unfortunately I'm still having issues with loading bytecode on 64 bit.

The bytecode seems to load ok and it runs, but when it executes a function with an enum parameter, the other parameters get mangled.
For example, I have a script function:
void change_state(state_types new_state, int new_face)
When the scripts are compiled on x64 it works fine, but when the bytecode is loaded again, the second parameter becomes the same value as the enum.
Replacing state_types with int then casting it back to an enum to assign it fixes the problem.
There was another case with a few enums and a string that caused a crash in the string assignment operator when the function was called. This was also fixed by replacing the enums with ints.
The function I was originally having problems with also causes problems, so I'm assuming this is the same problem but manifesting itself in a different way.

I'll see if I can isolate the problem again, I suspect it will just be a matter of reloading the bytecode in the test I posted earlier.
Edit: Yep, using the same script as before but saving then loading the bytecode before executing it causes a crash, but it works if the enum is swapped for an int.

Thanks for your help!

The same fix needed to be applied in the asCReader::AdjustGetOffset() method too. I checked in that fix in revision 1373 now.

Regards,
Andreas

Thanks Andreas,
I was having the same issue with revision 1373, but I made the same changes in asCWriter::CalculateAdjustmentByPos() and asCReader::CalculateAdjustmentByPos()
And everything seems to be working now!
Does that sound right?

Thanks!

Yes, it sounds correct. Hopefully that was the last of them.

I'll add some further test cases to try to catch these problems and avoid that they re-occur with future changes.

I checked in these last changes in revision 1374. Let me know if you find any further problems with the latest release.

Andreas you are AWESOME! Thanks so much for turning around this bug fix (and every other bug fix you've ever turned around for me in the past ! )