Skip to main content
gamedev.net
🔒 Locked

Debug Code Faster than Optimized?!?!

Started by scottdewald 6 replies 1.4k views
Original Post
scottdewald
scottdewald
I am helping someone with their code, and I am experiencing strange results. The debug version of the code takes much longer to load (I understand this), but it is running 2-3x faster than the optimized version (I don't understand this). We are using IBM's Visual Age 6 compiler (don't ask why). The compiler flag that is causing the great speed improvement is -qheapdebug. I read on IBM's site, and it says this flag replaces all the malloc(), realloc(), and calloc() calls with ones that initialize the memory to known values. This program has no GUI, it strictly does calculations. It uses over 22MB of memory. One thought I had is that these debug calls allocate the chunks of memory closer together so there are less page swaps. Another thought is that when they initialize the memory, more/all of it is preloaded in the cache. Any other ideas? Has anyone seen this kind of behavior? Thanks, Scott
outRider
outRider
Some more info would be helpful. What OS, processor, language? Since it's VA and based on the bit of info you gave I'd guess AIX, PPC, C, but who knows. In general terms how does this program behave, how does it allocate memory? There could be many reasons for this.

If you want to quickly test your theory replace all calls to malloc with calloc, which should touch the memory after allocating it, or use a loop to init the memory to some value.
iMalc
iMalc
I think it is far more likely that these 'known values' are values that happen to prevent this persons program from having to to do extra work.
For example code that sorted an array initialised to all the same value would normally take much less time than sorting an array that contains uninitialised memory.
Once you step through what is happening in each case, I'm positive you will find that it is a difference of what the code is doing, not how optimised the assembly code is. I have been there before.
LorenzoGatti
LorenzoGatti
I would compare the debug and release programs to find what they do differently. In order of increasing complication:
* Which executables and static or dinamic libraries are different.
* Linking information from dumping tools (NOT from compiler options etc.) to find different libraries: the release version might somehow get slower ones.
* Which individual object files of the application are different.
* A sample of different assembler code for such compilation units; judging optimization quality or lack thereof is very difficult, but localized or systematic extra work is easy to find.
Omae Wa Mou Shindeiru
scottdewald
scottdewald
AIX, PPC, c
Quote:
Original post by iMalc
I think it is far more likely that these 'known values' are values that happen to prevent this persons program from having to to do extra work.

Yeah, this is probably correct because I just wrote a test app that runs much faster in optimized mode compared to debug mode. Even when I replaced the calls to malloc with calloc so the data would be initialized, the optimized code still ran much faster.

However, let me know if anyone has any other ideas.
outRider
outRider
Why would he be doing work on uninitialized memory in the first place? If, for example, he's sorting a freshly allocated array without initializing it, then that's an error in his methodology. Maybe you should look over his code to see what he's doing. Try replacing malloc with calloc in his optimized build to see if you get the same behaviour, not in your test program.

Also, how are you timing your code? You can use the time command, which has verbose output that gives you # of page faults, context switches, etc. AIX also has a profiler called tprof that might tell you roughly where the extra time is spent.

I still have access to VA 6 so if you can give me some idea of what he's doing I might be able to replicate it.
scottdewald
scottdewald
Quote:
Original post by outRider
Why would he be doing work on uninitialized memory in the first place?
Definitely not on purpose.

Quote:
Original post by outRider
Maybe you should look over his code to see what he's doing.
I'm in the process of doing that, buts its about 90k lines of code.

Quote:
Original post by outRider
Also, how are you timing your code?
The process takes about 15 mins to run. The ps command can give the time a process has been on the CPU. While it runs, every 10 seconds we run the ps command and diff the current CPU time with the previous CPU time to determine the amount of time the process has been on the CPU over the past 10 seconds.
Figgles
Figgles
I had this problem once and it was related to floating point exceptions.

Basically, the debug version gave variables sane values such as 0.0, while the release (optimized) version didn't. When I performed any floating point operation, I got an FPU exception, but that was handled transparently. Still, triggering the exception meant user -> kernel -> user context switching, which really was pretty bad for performance. If you are doing floating point math, check to make sure that there aren't uninitialized variables used in calculations. Compilers are good at detecting it most of the times, but some subtle things they cannot catch:

float add(float* a, float* b, int count)
{
int i;
for(i=0;i<count; i++)
a += b;
}

int main()
{
float a[16], b[16];

add(a, b, 16); //Unlikely to get warning, esp. if add() in diff source file.

}

I don't know if you are using FP-math, but just something to check out.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.

Sign in to reply to this topic.