Original Post
🔒 Locked
Reading binary from memory?
Advertisement
Quote:
Original post by lephyrius
Yes I know, but lets say that
I have:
unsigned long x;
float y;
double z;
How do I read the the variables x,y and z from the data array as fast as possible?
unsigned char *data = new unsigned char[filesize];...struct Foo { unsigned long x; float y; double z;};Foo * foo = (Foo *)data;The only slightly slower approach involves proper serialization, which has been discussed around here often enough, so search would help.
Quote:
Original post by lephyrius
Yes I know, but lets say that
I have:
unsigned long x;
float y;
double z;
How do I read the the variables x,y and z from the data array as fast as possible?
You can't necessarily just point at values in the array like Antheus shows because of alignment issues (some platforms, for example, will only let you treat 4 bytes as an int if they "line up" so that the address of the "first" byte is divisible by 4). Typically, we want a copy of the data anyway (so that it isn't all stuck together). What we do is copy the data a byte at a time from its location in the data array into the location where our copy will be. For example:
// determine the file sizestd::vector<char> data(filesize); // std::vector will handle the memory management for us.// Fill the vector with data from the file (you know how to do this part)// ...// Say that the first long-sized chunk of the file should be treated as a long:long x;// We get the address of the destination 'x' as a char*, and use iterators in// the vector to make the source range. We are copying from the beginning of// the vector, and we will copy as many bytes as the size of the variable.std::copy(data.begin(), data.begin() + sizeof(long), reinterpret_cast<char*>(&x));Of course, keeping track of where each variable is located can get tricky, but we know they're sequential, so we can just keep an iterator around that represents our "read position" (in the old days with raw allocated memory, we would use a raw pointer for this), and make a helper function to take the iterator by reference and advance it:
template <typename InputIterator, typename T>T read_and_advance(InputIterator& position) { T result; InputIterator original = position; std::advance(position, sizeof(T)); std::copy(original, position, reinterpret_cast<char*>(&result)); return result;}// ...std::vector<char>::const_iterator read_it = data.begin();unsigned long x = read_and_advance<unsigned long>(read_it);float y = read_and_advance<float>(read_it);double z = read_and_advance<double>(read_it);But this only works for very basic types, and you need to be aware of endianness still. For the general case, see serialization.
And by the way, reading the whole file into a chunk of memory first like that might not actually be faster. It depends what you're planning to do with it. The standard library iostream classes already do some buffering.
Quote:
Original post by Zahlman
You can't necessarily just point at values in the array like Antheus shows because of alignment issues (some platforms, for example, will only let you treat 4 bytes as an int if they "line up" so that the address of the "first" byte is divisible by 4).
Antheus already pointed out that you must becareful of alignment issues, structure padding and endianess. If you take that into consideration by setting the correct compiler flags/pragma packs, allocate on aligned memory, and the output file is written in the correct endian ordering then there is no reason you should have to iterate over each element and serialize it in.
Attaching directly to memory via a cast is then perfectly fine and more importantly its much faster then serializing each data member at a time. I couldn't imagine having a streaming world that serialized in each 4 bytes but just because I haven't seen it done doesn't mean some studio somewhere hasn't done it. Though, for the code I have seen the data was output in the proper endianess, the structures were all probably packed and the memory the file was read into was allocated on aligned memory.
Though you are correct about reading in the whole file not being the fastest solution. Typically if the chunks are big enough then you'll probably want to compress the data, so you have
read chunk => uncompress chunk -> put in finished buffer
Then you can attach your data to the memory.
The only reason you should serialize each member is when you need to transform each member. Though, if there are no transformations I can't imagine why you would want to do that.
-= Dave
Quote:
Original post by David Neubelt Quote:
Original post by Zahlman
You can't necessarily just point at values in the array like Antheus shows because of alignment issues (some platforms, for example, will only let you treat 4 bytes as an int if they "line up" so that the address of the "first" byte is divisible by 4).
Antheus already pointed out that you must becareful of alignment issues, structure padding and endianess. If you take that into consideration by setting the correct compiler flags/pragma packs, allocate on aligned memory, and the output file is written in the correct endian ordering then there is no reason you should have to iterate over each element and serialize it in.
If we also say that the data was written out the same way (i.e. includes the same kind of padding), then sure. :) Otherwise the word in the file might not be aligned in memory even though the overall buffer is.
Quote:
The only reason you should serialize each member is when you need to transform each member.
Or when you don't have POD data. Which is a lot of the time in C++. Even strings pose a problem for serialization if you're not willing to (a) set an upper limit on string length and (b) waste all the rest of that space when the strings are short. To say nothing of object graphs.
There wouldn't be libraries for these things if it were rare to need the functionality. :)
Quote:
Original post by Zahlman
If we also say that the data was written out the same way (i.e. includes the same kind of padding), then sure. :) Otherwise the word in the file might not be aligned in memory even though the overall buffer is.
Yes that is what we're saying - which isn't a hard assumption to make. Though it might not be the case if you aren't dealing with the outputter for the file and if thats true then you may need to do serialization per member.
Quote:
The only reason you should serialize each member is when you need to transform each member.
Or when you don't have POD data. Which is a lot of the time in C++. Even strings pose a problem for serialization if you're not willing to (a) set an upper limit on string length and (b) waste all the rest of that space when the strings are short. To say nothing of object graphs.
There wouldn't be libraries for these things if it were rare to need the functionality. :)
The string example is an easy custom container, e.g. you have an array of chars and an index map. The index map points to the first character in your string. The interface could look like this
unint Grow(const CString& string);
unint Grow(cpointer string, unint len = kNeverIndex);
cpointer StringPtr(unint index) const;
CString StringCopy(unint index) const;
Object graphs can be flattened. Anything with a pointer can be changed to an offset from a base location.
There may be a library for it but most games just point a structure or array of structures/classes to a memory chunk and call it a day. Let the tools/build pipeline worry about putting the data structures into the correct format.
Also, we used boost library for intermediate serialization of binary data to and from during our build pipeline. We switched it out for our own binary serializer and a typical build time for a level went from 1 hour to 4 mins. It was one of our greatest optimizations :P
-= Dave
Topic Locked
This topic has been locked by a moderator. New replies are not allowed.
Sign in to reply to this topic.
Advertisement