🔒 Locked

Terrain streaming

Started by Mephs Jun 6, 2004 at 10:28 AM 7 replies 1.9k views

Original Post

Jun 6, 2004 at 10:28 AM

Hi, I''m having some problems getting my terrain streaming to work correctly, every method I''ve tried seems to have sme big negative points, so I''m wondering if anyone can suggest a way to get it working as smoothly as possible? Basically, I have a buffer of landscape chunks. The buffer is composed of 8x8 chunks. Each chunk is composed of 33x33 vertices (as I''m using splatting following Charles Blooms method). Now, a chunk also contains a 128x128 base texture, a 64x64 alpha map for blending between textures (both of which are 32bit colour). To stream in my data, I''ve created a worker thread, which I intend to give low priority so it doesn''t disrupt my frames per second too much. Every frame, my render loop checks to see if the player has moved past a streaming boundary. If the player has moved past that point (currently the distance of a single chunk) then it starts to update the chunk buffer. 8 chunks are loaded from disk dependant upon which direction the player has crossed the boundary. The other chunks are simply copied from the existing chunks in memory (no point reloading chunks if they already exist). Now I''m having problems in 2 areas. Firstly, to avoid the player seeing the whole update take place, I would have to perform the update on a temporary set of chunks, and update the pointer to the chunk buffer, to point to the temporary set of chunks once it is finished. This way, the update is almost instantaneous. My first problem though is that this almost doubles a pretty hefty memory consumption, having to effectively store 2 of every chunk, and I''m really fighting to get this all to run without errors due to being out of memory with only 256 megs RAM. My second problem is to do with the whole chunk copying process. I''m wondering what is the best method to copy across the chunks? I''ve found that changing the chunk pointers to point to the chunk that will be replacing it, sometimes ends up creating duplicate chunks, and also leaves me with an incorrect set of second texture co-ordinates, as I need these to remain the same so I can map one large texture over the entire terrain. An alternative method, rather thn changing pointers, would be to simply copy across all data stored in the chunk. I''ve managed to do this, but I''m pretty sure that it is going to be far too slow to do. To reliably copy across all data, I''d have to lock 64 index buffers, 64 vertex buffers, 64 alpha textures, 64 base textures... etc etc. But is there another way of copying the information in the chunks, while ensuring that I don''t accidentally end up with duplicate chunks or other data? Has anyone else faced these problems who might be able to point me in the right direction? Cheers, Steve

Etnu

Jun 6, 2004 at 11:05 AM

Why do you have to duplicate the ones that are already in memory? That just seems counter intuitive. You could simply replace the chunks that are no longer necessary with the new ones once they were loaded.

Mephs

Jun 6, 2004 at 11:16 AM

Okay,

Say I''m moving upwards or North on the map. The chunks that would be replaced would be the 8 chunks on the top row (array elements 0-7). The chunks no longer necessary would be array elements 56-63. If I replaced elements 0-7 with the newly loaded chunks, I would lose elements that I still need. If I replace elements 56-63 with the new elements, they would be in the incorrect location in my chunk buffer, as chunks are placed in the chunk buffer in an element corresponding to its actual position, though I guess I could work to remove the dependancy of the program on having to have the chunks render position correspond to it''s actual position, so I guess that could work.

I suppose because I''ve had that dependancy in the back of my mind, I''d not considered doing that... but I''ll give it a shot, so thanks!! That could actually solve both problems in one, as I''d only have to create 8 temp chunks to load the generated chunks into, saving me some memory.

Still though, if anyone has any further thoughts, I''d appreciate that too.

Thanks,

Steve

Instruo

Jun 6, 2004 at 11:23 AM

That makes sense to me. Instead of having the location depend on where they are in the array, perhaps just keep a small index representing the chunk''s position in grid.

Etnu

Jun 6, 2004 at 11:40 AM

Instruo is on the right path. Maybe have each chunk store a vector representing the grid''s origin in world space, and sort them based on that (I''m assuming you had the dependency to facilitate occlusion?)

Trienco

Jun 6, 2004 at 1:01 PM

squeeze in another layer thats constant. and 8x8 array containing either pointers or indices to the actual chunk. that way you just have to update 16 pointers/indices. pretty much what i did except it was only 3x3.

hiding the updates.. fog? seeing the update happen or just having it pop up both dont sound too great.

samgzman

Jun 6, 2004 at 2:00 PM

I'm working on something pretty much identical. What i did is have a sytem working like this:

My chunk cache/buffer system used pointers instead. Don copy the chunks... thats a serious waste of memory and CPU power. A good solution is to have a function in your paging manager that will efficient handle pointers for each range of direction moving. (N, NE, E, SE, S, etc....) You can do diagonal moves more effieciently by not first moving everything up/down and then left/right (Think about that one for a few mins).

This buffer management system works awsome by using a a triple pointer system:



// Triple indirection

CTerrainChunk***   m_ChunkBuffer;
m_ChunkBuffer*** = new CTerrainChunk[x_size];
for(int i=0; i < x_size; x++)
     m_ChunkBuffer** = new CTerrainChunk[y_size];


//now just acces your pointersas: 

m_ChunkBuffer[x][y]* = bla;



// deallocate properly:

for(int i=0; i < x_size; i++)
{
    for(int j=0; i < y_size; j++)
          if (m_ChunkBuffer[i][j])
               delete m_ChunkBuffer[i][j];
    delete[] delete m_ChunkBuffer[i];
}
delete[] m_ChunkBuffer;
/*
u can optimize the moving around of the pointers by messing around with pointers.  For example, to move up a whole horizontal line of pointers just do:
*/

m_ChunkBuffer[0] = m_ChunkBuffer[1];
// Nice...

i think thats about right....

**NOTE** something I found to reduce memory:
u dont need to load all the terrain squares... At least not the ones in the corners. if u can only see a certain distance, then u dont need to. Example:

   
X X X X X X X X                       X X  
X X X X X X X X                     X X X X 
X X X X X X X X                   X X X X X X 
X X X X X X X X   Can become    X X X X X X X X
X X X X X X X X                 X X X X X X X X
X X X X X X X X                   X X X X X X 
X X X X X X X X                     X X X X  
X X X X X X X X                       X X

just keep the pointers in the corners empty at all times.

While the code maybe become a little more involved to manage the buffer such as shiftint pointers, you can reduce your memory consumption a good bit

Especially if each square contained multiple alpha maps for your texture blending , a base texture, and other meta-data descriping the layers. In the example a above, i removed 24 Cells! for large terrains, the benefit is even more drastic.

By paging cells this way, you provide a nice access method to all cells at any given moment. This can allow u to remove all cracks easily, even for what could become an infinitely sized terrain! Clever.

On top of that, i also built in a nice quadtree system, for effiecnt culling.

I hope that leaves you with some good ideas. In my implmementation, i still have issues with the format of the data on disk. I want to allow for a truely unlimited terrain size (with all terrain being modeled... not randomly generated) In order to do this, i would need some sort of database implementation. So i said to myself: "If im gonnna design some kind of DB to store terrain, then why not just store all my game content this way?" So now, i have consumed all my spare time w/ database research. I want to develop a Object Oriented DB to store serialized versions of my objects, so i can later retrive them again and expand them back into the original objects. Tricky stuff. On top of the terrain, I struggle with dealing with spacial relationships. I want to use some sort of a scene graph for all objects, but i would need to build it off of the quad tree so it all worked fluidly and seemlessly.

Anyhooo.. If u find that last part especially interesting I would love a partner!! My AIM s/n is also samgzman If, not i would still love to know what you think about this clever idea. Thanks.

[edited by - samgzman on June 6, 2004 3:01:02 PM]

[edited by - samgzman on June 6, 2004 3:04:50 PM]

[edited by - samgzman on June 6, 2004 3:08:26 PM]

Mephs

Jun 7, 2004 at 7:43 PM

Excellent. It's all working fine now. I've simply removed the position->elementposition relationship, though I may use the indices system for allowing things in future to be somewhat more simple (much less confusion with indices than pointers for virtually no cost). I'll look into the whole reducing the number of chunks thing. I can't really g to a 3x3 system though, as each chunk is 33x33 vertices and 3x3 chunks means my terrain would be awful blocky and small!!

My way of hiding chunk loading is to use fog, and also set the view distance such that you can never possibly see the chunks being loaded, though for slower computers that can't keep pace with the streaming I'll implement a loading progress indicator alternative.

I've also experimented with octrees in the past. Now, correct me if I'm being clueless here (probably!!), but isn't an octree only going to be of use if you have a massive terrain in memory already? ie When streaming, your buffer is only ever going to be a certain size, using an octree to cull polygons seems a bit pointless when I can simply frustum cull my 64 chunks and so far as I'm aware, that isn't too costly to do, as I'm never having to check a chunk unless it's loaded. An octree may be useful if I had, say, 640 chunks in my buffer (ie loads and loads of chunks to check against the frustum), but due to the streaming process I never have that many... or am I benig thick and missing the point?

Anyhoo, I'd be glad to work with a partner, though I dont use AIM, but you can contact me via (mephs at hotmail . com) or my personal website forum (being built, but 1 functional forum at the moment!) on Liquidigital Online

Cheers,

Steve

[edited by - mephs on June 7, 2004 8:43:59 PM]

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.