[.net] StackOverflow: .Net compiler error?

It's more common that these situations that appear completely nonsensical are caused by something you're not noticing (corrupt debugging information, for example, can cause similar weirdness). What's the code look like? What's the generated IL look like (use Reflector)? Et cetera.

This is occurring during a very deep but finite recursion. The foreach probably would just require a few call stack frames too many to fit on the 1 MB .net stack. I also get a stack overflow on large data sets when the recursion on that method passes 300 or so levels deep.

Pretty hard to diagnose without seeing the actual code, so I'll throw in some general info.

In my experience, the single most common reason for stack overflow in C# is that someone made a property that returns itself in the getter method due to a typo, because the actual member has the same name, but doesn't start with a capital.

That's true, but in this case the recursion is not infinite. The overflow occurs specifically on recursion level 304 of my tree-traversing method collectRelevantNodes(). However, when I increase the stack size to 10 MB using Thread.Thread(ParameterizedThreadStart start, int maxStackSize) the stack overflow does not occur. I suppose I could attempt to lessen the amount of space taken up per call by decreasing the number of stack allocated primitives, or just leave the stack size at 10 MB (which is less than ideal since that constructor argument is ignored on windows versions predating XP). I still think it's odd though that the overflow could be invoked when the stack is almost at its max by a call that's never actually made.

Quote:

Original post by DanX2002 That's true, but in this case the recursion is not infinite. The overflow occurs specifically on recursion level 304 of my tree-traversing method collectRelevantNodes(). However, when I increase the stack size to 10 MB using Thread.Thread(ParameterizedThreadStart start, int maxStackSize) the stack overflow does not occur. I suppose I could attempt to lessen the amount of space taken up per call by decreasing the number of stack allocated primitives, or just leave the stack size at 10 MB (which is less than ideal since that constructor argument is ignored on windows versions predating XP). I still think it's odd though that the overflow could be invoked when the stack is almost at its max by a call that's never actually made.

Ok, apparently I didn't read your post about to finiteness of the recursion, sorry...

If you can fill up the 1 MB stack with a ~300 level deep recursion than every level would have to consume ~3 KB, which seems a lot to me.

Nevertheless, there's nothing wrong with increasing the stack size if you run out. But from what you write it doesn't seem to be that straightforward in C# (in C/C++ it's a linker setting for the main thread and the stack for other threads is allocated by the application), so you could try to move some data to the heap. Probably at the expense of performance. Don't know if that would be an issue for you.

As for the exception itself:
Of course it's thrown _before_ the allocation is made that would overflow the stack, otherwise it would potentially leave the program in an unrecoverable state (terminated by the OS or worse).

Well, first off: this is why you don't use recursion but prefer iterative methods.

Secondly: just because you dont expect a stack allocation there doesn't mean one doesnt happen. The JIT is free to allocate stack space when it needs to, and many operations that are NOT related to calling a function, may require space to be allocated. (A prime example is floating point mathematics).

Quote:

Original post by Rattenhirn Of course it's thrown _before_ the allocation is made that would overflow the stack, otherwise it would potentially leave the program in an unrecoverable state (terminated by the OS or worse).

Right but in my original post I was wondering about it because the foreach that invoked it was never actually hit, nor could it have been hit, because the condition of the if block it was in was always false. That's why I was initially surprised that commenting out this unreachable foreach fixed the error for small data sets.

Quote:

Original post by Rattenhirn If you can fill up the 1 MB stack with a ~300 level deep recursion than every level would have to consume ~3 KB, which seems a lot to me.

Quote:

Original post by Washu Well, first off: this is why you don't use recursion but prefer iterative methods.

It does seem like alot. I don't know how so much could be taken up per call. Part of it is because my method collectRelevantNodes() doesn't call itself, it calls relevantPaths(), which calls collectRelevantNodes(). The whole algorithm is somewhat complicated, which is why I haven't been using loops instead of recursion: they would be at least hundreds of lines long and require huge amounts of data to be stored in a makeshift stack between iterations, which would circumvent stack overflows only at the cost of higher code complexity, which I don't really want.

As an update, I'm now developing a bottom-up algorithm as an alternative to this one, to circumvent deeply nested definitional methods

Quote:

Original post by Washu Well, first off: this is why you don't use recursion but prefer iterative methods.

Or use tail recursion, which is sadly not very well supported under .Net.

The fact that the foreach statement is never executed, does not mean that it won't consume stack space. Note that C# yield, foreach, anonymous delegates and lambdas involve an amount of compiler magic that might have unexpected side-effects in some occasions.

Without actual code it's difficult to say what's going on here.

While it's not impossible to uncover a compiler bug (I have encountered/reported a number of bugs on both Microsoft's and Novell's compilers), it's a rather unlikely occurrence. I'd suggest cross-checking with Mono to see if the behavior is the same (download the windows version and use MonoDevelop to build your project). If it's the same, then most likely this isn't a bug. Otherwise, it might be a good idea to isolate the issue in a small test case and report it to Microsoft Connect.