Terrain Loading

Quote:

Original post by Geometrian Hi, I've been working on a game which requires terrain. I have made several versions of something to load it, but they are all too slow, and some do not work. I'm confident I can code it, so I think theory is most relevant here. The code must load the terrain data into an OpenGL mesh as fast as possible and it must have a lot more than 256 possible heights. What is the best way to do this? Thanks, Geometrian

That's odd.. Usually loading terrain data from a height-map isn't hugely slow. What sort of additional processing are you doing on the terrain data ? If you are not happy with speed then there is always the option of re-saving the terrain data in the exact format you need it in for run-time and then just whisking it off the disk when required.

If you want more than 256 height values you can simply make use of all the color components in the height-map image (RGB) rather than just the red component for example.

There are many different ways of doing terrain but here's one scheme I adopted for an XNA racing game that I worked on:

1 - The terrain itself uses 4 diffuse textures and 2 bump map textures, which are blended together at each terrain vertex. An additional texture unit is also used for a lightmap, and another unit for a detail texture.
2 - The terrain heights are loaded from an ordinary heightmap texture. Only the red component is used. I also have four 'blend maps' which are used to determine the weight of each texture (and the two bump maps) at each vertex.
3 - The data itself is stored in a vertex buffer and drawn using indexed triangles.

Hope this helps,
Darragh

Well, I tried using the data in ways other than simply using one channel. I tried, for instance:

...converting the heights to floats, on a scale of [0.0 to 1.0], then encoding the floats to red, green, and blue values. A height 0.629541 would be the color (62,95,41).

...Using a heightmap. The color (41,41,41) = a height of 41. This doesn't work because it is slow and because it only allows 256 heights.

...and a whole bunch of other solutions. Most of which worked, but didn't go quickly at all. I'm loading a mesh of size 1024x1024 and my computer is 1.7GHz duo and 2000MB RAM. I'm thinking, then, that the code should run faster--it takes around a minute or 30 seconds to load it.

Quote:

Original post by Geometrian I'm loading a mesh of size 1024x1024 and my computer is 1.7GHz duo and 2000MB RAM. I'm thinking, then, that the code should run faster--it takes around a minute or 30 seconds to load it.

Hard-drive speed is of more concern for loading times than either processor or RAM, particularly if you have a laptop (which have slower drives on average). However, that seems a little lengthy, so I wonder if you are storing too much data. The two ways to get a height field greater than 256 steps is to either logarithmically scale your byte height values (WoW uses this I believe), or to store it on disk as an array of floats.

Quote:

Original post by swiftcoder Hard-drive speed is of more concern for loading times than either processor or RAM, particularly if you have a laptop (which have slower drives on average). However, that seems a little lengthy, so I wonder if you are storing too much data.

That is true, but it still seems like an inordinately long time for a height-map to take loading- even if too much data is being stored. My personal hunch would be that some expensive processing is being done with the data once it's been loaded. That is of course speculation since I don't know any details about the implementation, but it would make sense.

Have you tried profiling the loading code to see where the bulk of the time is being spent ?

Here's the code (it's in Python):Surface is a pygame.surface which is the heightmap. The heightmap contains heights encoded by putting the height on a range: [0.0 to 1.0) and then encoding the first two digits of the height as red, the next two as green, and the next two as blue. Thus, the heights are encoded to 6 decimal places. It took 31 seconds to load.

5 seconds to get the colors from the surface

Quote:

Original post by Geometrian Here's the code (it's in Python)

Therein is your problem! You should have mentioned that earlier. [smile]

Python is an interpreted language and as such is considerably slower than compiled languages like C/C++. I don't have an exact figure for how much slower but somewhere in the region of 20-50 times would be reasonable. It's good for web and scripting technologies but not suitable for heavy duty data processing like what you are doing with your terrain.

Secondly, I would be a bit suspicious of that array / list you are storing your data in. Forgive my ignorance of the Python language, but I am unsure whether it is a variable sized array (like a STL vector in C++) or a linked list. Either way putting data into it would be much slower than using a fixed size array.

I'm not sure what options you have here other than switch to a compiled language or even an interpreted language with a JIT compiler like Java. Maybe it would be possible to code this module in C or something similar and integrate it with the rest of your python code.. Again I can't say for sure since I am not that well familiar with the language

How about this?
http://psyco.sourceforge.net/
would that help?
I suspect it would, but I don't know how I should use it.

Also, would this problem go away if I compiled it with py2exe?

Quote:

Original post by Geometrian How about this? http://psyco.sourceforge.net/ would that help? I suspect it would, but I don't know how I should use it. Also, would this problem go away if I compiled it with py2exe?

That psyco looks good; yeah, you should definitely give that a try before considering migrating your code base to another language- which would be a pain to say the least!

I don't think Py2Exe will help though unfortunately. I've used a similar program for java which just basically dumps all the java byte code into an executable that can be clicked on to run; it doesn't actually do any compiling or change the underlying code, just provides a nicer way to launch the program other than a .bat file. I suspect this program would be something similar.

That's annoying.
How do I use psyco, then?

Quote:

Original post by Geometrian That's annoying. How do I use psyco, then?

I think you just install the thing and it will run on top of your Python interpreter. There seems to be libraries associated with it also which will allow you to control the JIT compiling and be more selective about how it works.

I think at minimum you have to stick the following two statements in your main python source file for it to work:

import psyco
psyco.full()

That doesn't work.
I mean the program runs, just not any faster.

Well, you may try using vertex arrays (glVertexPointer,glTexCoordPointer..), but really, processing million polygons with python won't get much faster. It may be possible to precompute data for the vertex arrays and store them directly instead of using heightmap, but i'm not sure if pyopengl can then pass it to OpenGL directly, without at least some reprocessing.

You should try using python's low-level arrays (module array) as they should be considerably more efficient than generic lists.