Temporary string literals

Of course the string has to be stored somewhere in your executable file, likely in a readonly data section, as you conjectured. The heap is a purely runtime thing and will only be populated during execution of your program.

That being said, depending on what you're doing with the string and your optimization options, it's possible that the string doesn't actually end up on the heap (e.g. all functions you call on the string are getting inlined and you only do read-only accesses).

2 hours ago, rnlf_in_space said:

Of course the string has to be stored somewhere in your executable file, likely in a readonly data section, as you conjectured. The heap is a purely runtime thing and will only be populated during execution of your program.

Ok, so if I make something like this:

void function()
{
  std::string test1("Some very long test string 1 ...");
  std::string test2("Some very long test string 2 ...");
  std::string test3("Some very long test string 3 ...");
  // ... more string
  std::string test10000("Some very long test string 10000 ...");
}

Then I will end up with an exe file "bloated" up with 10000 long strings that are not used anywhere besides in one function. It looks to me like an overkill, it must be something that takes care of this?

If the strings get used (and not optimized out), of course they need to be stored somewhere. Where else would they be stored if not in the executable? You could store them in a data file, but then you have to load them at runtime.

I don't know how to clarify that any better... if the strings are used OF COURSE they need to be stored somewhere, how else could you use them? Maybe an analogy: If you move to a different apartment and you want to take your books with you, you have to store them in a box. Are the boxes "bloated" just because you want to take your 10000 books with you? Your new apartment is the running process, the boxes are the executable file, the books are the strings.

If you don't want the bloat, find a way to code your program without using the strings.

1 hour ago, rnlf_in_space said:

If the strings get used (and not optimized out), of course they need to be stored somewhere. Where else would they be stored if not in the executable? You could store them in a data file, but then you have to load them at runtime.

It makes sense. One other thing that comes to my mind:

void function()
{
  char text1[] = "Some very long string 1 ...";
  char text2[] = "Some very long string 2 ...";
  // ... more char arrays
  char text10000[] = "Some very long string 10000 ...";
  
  std::string test1(text1);
  std::string test2(text2);
  // ... more string
  std::string test10000(text10000);
}

This way we would not increase the size of .rodata but instead we would store the chars directly on a stack and then on heap when we assign them to std::string.
Although this solution would increase .text segment (code segment) as this is where the chars would be stored now. So I guess in the end we can't skip increasing the exe size.

No, it's the same, really. Going back to the moving apartments analogy, if you want to sort the books into your shelves (the stack in your code) in the new apartment, you first have to get them there somehow.

.text and .rodata both end up in the exe (and again, this would put the strings into the read only data section, not the code section).

The only way you get something onto the stack or heap is to put it there using code (that the compiler generates for you, you don't see it directly in C++). And that generated code needs to copy it from somewhere. This "somewhere" is you exe file. I cannot ask you to hand me a certain book from your bookshelves if you didn't bring that book from your old apartment. And that means you need to have had that book in one of the boxes while moving.

The exe file must contain all the data required to execute your program (unless you load the data from a different file, but then again you need to ship that file and it doesn't make a big difference in the overall size of things). If your program contains code that relies on a certain string being present, then that string must somehow get to the place where you execute the program.

1 hour ago, rnlf_in_space said:

.text and .rodata both end up in the exe (and again, this would put the strings into the read only data section, not the code section).

I would have to disagree on this one. If you take a look at this link from beginning of the topic, it says this:

Quote

If we do the same for char[]:
char s[] = "abc";
we obtain:
17: c7 45 f0 61 62 63 00 movl $0x636261,-0x10(%rbp)
so it gets stored in the stack (relative to %rbp), and we can of course modify it.

So the chars "abc" end up directly in .text segment as I guess the above "char text1[] = "Some very...";" and other arrays.

This is because $0x636261 is not an address, it's directly a map of chars that get stored with other code in .text segment.

So it's still stored inside the exe, just in this case as part of a movl instruction. That works for short strings (8 characters incl. terminator at most). This is the piece of code that copies the string onto the stack!

Yes, 0x00636261 is a 32 bit value that is equal to the string "ABC\0". So yes, for short strings it's part of the code itself. But that doesn't work with longer strings. When you have a longer string, it will copy it from where it is stored (.rodata or .text doesn't really matter) onto the stack first. See it here: https://gcc.godbolt.org/z/KBHEBm

The .LCn: are the labels where the compiler puts the string data, all those movdqa xmm0 ... movaps [rsp] are the unrolled memcpy, where the compiler added the code to move the string from the (in this case) .text section onto the stack. The first line, sub rsp, 104 is where the compiler makes space for the string on the stack.

The stack is completely empty when the program starts running. The only way to get something onto the stack is during runtime using code! The same goes for the heap. You cannot store something on the stack or heap directly so that it's just there when your program starts running.

@ryt I think you're hunting a non-existing objective. There is no universal "string behavior for x86 systems". It depends on the compiler (and compiler version) and library (library version) at least, and likely on other things as well.

If you want to enforce some behavior, build your own string class. Otherwise, just assume the string designers were sane people and made wise choices, and spend your time on more useful things than text storage inside string objects.

@Alberth, I think what they're really trying to do is to have constant string data in a program without having it in any file they're shipping. I'm a bit lost as to how to explain that that's not possible.

If you XOR the string with itself byte--by-byte, it will consist of only 0 bytes. Since it's all a known length of zeroes, you just need to store the length. Then, at runtime, you just need to allocate the length amount of space on the stack and XOR it once again with the original string, and voila, no need to store the string in the DATA segment, it can go in the BSS segment instead.

It's called the "turtles all the way down" method.

Please don't!

If your strings are so similar, why can't you generate them at run-time?

::sprintf_s( szBuffer, MAXLEN, "Some very long  string %u  . ..", ui32Index );

Or similar. This sounds as though this problem is meant to be tackled via out-of-the-box thinking rather than by working around computer-science issues.

For what it's worth, using constant strings as just pointers into a data section is reliable enough that it is used in ASKA Engine (tri-Ace) to implement "print-once" error messages. To avoid spamming the console, the string pointer is kept, not an actual copy of the string. This is important for run-time as we do need running builds of the games that also print (or silence) debug messages quickly. So you will use a macro such as "ASSERT_ONCE( someCondition, "The condition failed. Checkmate, atheists." )", and as long as "pool strings" is set then this string pointer will have 1 pointer no matter where it is used, so redundancy checks on the string can be done just by checking the pointer, not the string.

This has been very tested over many years and is reliable on all consoles and relevant platforms (and most irrelevant platforms), however do keep in mind that it is only used for debug purposes, as it is not guaranteed by the standard. Code that relies on this should not be put into the wild.


L. Spiro

10 hours ago, Bregma said:

If you XOR the string with itself byte--by-byte, it will consist of only 0 bytes. Since it's all a known length of zeroes, you just need to store the length. Then, at runtime, you just need to allocate the length amount of space on the stack and XOR it once again with the original string, and voila, no need to store the string in the DATA segment, it can go in the BSS segment instead.

It's called the "turtles all the way down" method.

? Thanks

I agree with @rnlf_in_space that I should not but I never done something like this and I just need to try it.
I can't say that I'm much of that expert, wouldn't some code immediately when it encounters " " store the string in read-only segment if it's long enough or store it in an instruction if it's short enough as commented in previous posts?
Could you please show me how would you do it?

3 hours ago, L. Spiro said:

If your strings are so similar, why can't you generate them at run-time?

::sprintf_s( szBuffer, MAXLEN, "Some very long  string %u  . ..", ui32Index );

I didn't know I can do this. I red a bit about ::sprintf_s() in the documentation. I see it stores some string into some buffer, that is szBuffer. But what I don't understand is if I write it like you did, wont "Some very long string %u . .." be immediately stored in read-only segment, before I even call this function, at least the part "Some very long string" without %u?

You need to be familiar with printf, sprintf, and related functions if you are going to be a programmer. All the information on what they do and how they behave is very heavily documented.
In my example, szBuffer is a local heap buffer (or you could allocate with new [] and free later with delete []) and MAXLEN is the length of that buffer. It should be long enough to handle the longest possible string you could put into it. If it goes over 1024 bytes you should strongly consider allocating szBuffer rather than putting it on the stack.

The source string ("Some very long string %u . .." in this case) will exist once in your executable and then format-copied into szBuffer with the changes you make during the format.

The rest you have to learn on your own via the documentation.


L. Spiro

Oh sorry, I forgot that you were talking about similar strings. Somehow I thought that this way it wont go into exe file.

Just out of curiosity, if you don't want your strings stored in the exe, and you don't want to read them from a file, where exactly do you plan to get them from?

You're going away on an expedition to the south pole. Besides some stuff to protect yourself from the cold, you think you may get bored one evening, and you will need something to read duriing the evening/night.

Question: Can you do that without bringing a number of books with you?

My guess is, you cannot. The only things you have when you're at the south pole, is whatever you bring with you, and lots and lots of snow and ice.

A computer works is not the south pole, but it works a lot like it. If you give a program to someone to use, you must give everything that is required for functioning of the program. If you leave out a part, the computer cannot re-invent it, or wizard it out of thin air, just as much as you cannot read a book at the south pole that you left at the dinner table of your home, but forgot to pack.

@ryt, just to be clear, what @Bregma suggested is a joke. You see how they want you to "XOR with the original string"? That means you need to store that in your file.

2 hours ago, rnlf_in_space said:

@ryt, just to be clear, what @Bregma suggested is a joke. You see how they want you to "XOR with the original string"? That means you need to store that in your file.

Ok, thanks for clarifying that ?, it was strange to me though I was not sure it was possible.

Just to be clear I never intended to skip storing strings from an exe file. I started the topic based just of curiosity. When I red the link from the beginning and some other sources it was still not clear to me completely how strings were stored so I asked it here.
For e.g. the link mentioned that the strings could be stored in read-only segment or in some other place like in an instruction for small strings. So this was pretty undefined for me so I thought that there are also some other possibilities to store hard-coded temporary string literals.
I also thought that if we use a local temporary string that gets stored in an array that they wont be at all in read-only section but coded in an instruction or somehow else. Now I know that only smaller strings get stored in an instruction and that other get stored in read-only segment whatever we use, a pointer to it or an array.