Confusion re. handedness and LookAt() implementations

Stay as far as possible away from left-handed coordinate systems. You will find no greater source of headache in game development.

If you must use it (please don't), the key thing to realize is that only the projection matrix is left-handed (to make depth positive). The camera viewing matrix is right handed, but must be constructed so that Z+ points in the desired direction. Everything else (object transformations) should be right-handed. Left-handedness implies a reflection across an axis, which would break lots of things (e.g. physics) if applied to all objects. The choice of left vs. right handedness is independent of the choice of “front” direction (e.g. Z+ in your case). You can make Z+ front in a right-handed coordinate system by making X+ right and Y+ down.

In a left-handed game engine, the projection matrix causes a mirroring of the scene. The dirty secret of left-handed engines (Unity, Unreal, etc.) is that they must apply the reverse mirroring to all geometry when it is imported so that it looks correct when viewed through a left-handed projection matrix. If you don't believe me, try importing a 3D model into Unity, then use script functions to access the imported mesh and write it out as an OBJ mesh. You will find that the resulting OBJ has been mirrored along the X axis relative to the original mesh (though it will look OK in Unity).

Left-handedness is truly an insane choice that only persists for historical reasons. The side effects are numerous. It's like viewing the scene through a mirror. On the other hand, with a fully right-handed system, everything “just works”.

Thanks for the reply!

The camera viewing matrix is right handed, but must be constructed so that Z+ points in the desired direction.

Ah okay, I think I (somewhat) understand now. So basically, if I have a “non-standard” view direction (Z+ forwards in my case), I have to construct the view matrix with that in mind, rather than just using a standard right-handed LookAt function? I had thought that the LookAt function was just “generic” and could be used for /all/ cases, but I forgot to account for my change of view direction.

I have to leave for work in less than a minute so I'll have to give this a further look into once I get back, but thanks for the pointer!

I always use left-handed, because X goes right and Y goes up, and Z goes forward intuitively for me. That's how I like it. I think you should never mix left-handed and right-handed if possible. Pick one that is easy for you to reason about, and stick with it. Same with 2D coordinate systems and stuff like vertically flipped BMP files and all that. Just pick the one you like and stick with it.

The only real reason you could come up with is as Aressera said: make sure to be aware in which coordinate system your assets are stored. But even then, you should make your own asset format that you use at runtime, and write loaders that convert all the output files from your tooling into your own format. For example if you use .obj files, which AFAIK are text-based, just parse them once and convert them into a raw file that has the exact same memory layout as the thing you will be putting into GPU buffers, so that you can just do a file read operation right into a mapped buffer. And the same goes for image files or audio files. Decide on an internal data format to use, and convert whatever files you created into that internal format.

So the matter of whether your model editor's coordinate system is left-handed or right-handed is a total non-issue because you should export your model files into a custom binary format anyway, and the exporter would take care of the handedness.

I never had any issues with mirrored scenes or any of that, so I have no idea what Aressera is talking about. I never used an engine, though, and always rolled my own tech.

RmbRT said:
X goes right and Y goes up, and Z goes forward intuitively for me

That is completely wrong from a math point of view. Z always points out of the page if X = right and Y = up, and cross( X, Y ) == Z. Does your cross product function produce the opposite result too? It must to be consistent. All of math and physics uses the right-hand rule for good reason. Computer graphics only sometimes uses left handed coordinates for historical reasons (someone long ago decided to make depth positive and it stuck). That was a mistake.

Typical issues with LH matrices are that their determinant is negative, or that conversation to quaternions does not work.
All this can be adapted to the convention ofc., but will take more time and is more error prone.

Ok, I've dug a bit deeper… and I think the problem may be deeper in my renderer pipeline than I thought (I'm using a software rasterizer that I've written myself, as the target platform is /very/ constrained and has no 3D hardware whatsoever). I'd actually been flipping the y axis during the final conversion from NDC to screen coordinates (I need y=0 at the top of the screen), but I'd forgotten that this has the side effect of /inverting the winding order/. Attempting to display what I had before /without/ this flip displays nothing, as now all the polygons are backfacing… so now I have to figure out what exactly I've broken haha. :/

JoeJ said:
Typical issues with LH matrices are that their determinant is negative, or that conversation to quaternions does not work. All this can be adapted to the convention ofc., but will take more time and is more error prone.

I always write my own maths libraries anyway, so it makes no difference to me. So there is no such thing as things just working out of the box for me anyway. I don't trust quaternions because I never bothered to look at the maths behind them, but if I did want to use them, I would also roll my own maths for it from scratch anyway (for example with a focus on being SIMD-isable / bulk-processing, etc.), which I guess some random maths library out there would not have.

Aressera said:
That is completely wrong from a math point of view. Z always points out of the page if X = right and Y = up, and cross( X, Y ) == Z. Does your cross product function produce the opposite result too? It must to be consistent. All of math and physics uses the right-hand rule for good reason. Computer graphics only sometimes uses left handed coordinates for historical reasons (someone long ago decided to make depth positive and it stuck). That was a mistake.

Math is based on first principles thinking, on counting, measuring, and similar things. Coordinate systems are a high level convention, not a fundamental first principle of reality / metaphysics. And to begin with, the letters X, Y, and Z are completely arbitrary, too, and the associated meaning with those letters. All you do is use 3 cardinal directions to span a 3D space, and it doesn't matter which way those directions face. And even the cross product is just a formula, not a fixed fundamental principle. You could just as well come up with a mirrored cross product formula and claim that that is the “real” or “official” cross product in some alternate reality where history took a different path. The problem only starts when you want to use stuff out of a textbook without adjustment, in a system that simply has other basic conventions. Just like calling conventions in languages. They are not inherently right or wrong or perfect. You just gotta stick with one on both the caller and callee side.

RmbRT said:
I don't trust quaternions because I never bothered to look at the maths behind them, but if I did want to use them, I would also roll my own maths for it from scratch anyway

I've spent quite some time on understanding quaternions. I could understand how rotating a vector works, but i could not fully understand quaternion multiplication. And i don't know anybody who does. All the attempts of explaining quaternions usually don't go that far.

So i'm sure you will do what we all did: Just copying code with a working implementation. I doubt such code for LH matrices exists, since it does not make much sense. Ofc. you can just negate one vector of your LH matrices, but doing so for a general math library would feel just wrong, so likely you have to add LH and RH versions of related functions, which is confusing and thus still bad practice nodody wants to bother with.

This is just the best example i personally know.
Usually, in cases of optional conventions (e.g. column major vs. row major), you can indeed pick what you like. There are pros and cons about performance maybe, but it goes both ways.
But regarding LH vs. RH it's not like that. A choice should not even exist, because math guys could agree on a general convention early. Same for the cross product. We all use the convention they have decided long before computers existed, and we never run into related problems. While column major vs. row major still causes us problems again and again.
You have been warned… ; )

Funny thing: if you want to support GL ES 2.0 (WebGL 1), you cannot transpose shader uniform matrices. So that forces you into using a matrix structure that has the four axis vectors subsequently in memory. Which is what I like more anyway, since I want to see matrices as a collection of 3 axis vectors, or 3 axis vectors and an origin vector.

But you already know my takes on math conventions, so I'll spare you the repetition.

RmbRT said:
So that forces you into using a matrix structure that has the four axis vectors subsequently in memory. Which is what I like more anyway, since I want to see matrices as a collection of 3 axis vectors

Yeah i like it this way too.
The downside is, with the other convention the last vector is just (0,0,0,1) all the time, so we can spare it easily.
Though, because matrices have so many numbers even that is still too much, so i prefer quaternion + position vector where possible.

Looking up shader coding tutorials you really get the impression you could use matrices without any worries. But in my experience it's really a performance killer and should be last resort (at least for shaders that do more than just a simple vertex transformation).

But you already know my takes on math conventions, so I'll spare you the repetition.

I won't insist either. But i was thinking: Probably he does not use LH matrices at all, but just hacks projection matrix to get the final expected output, which would be usual practice.

To clarify i have a question: How does the diagonal of your identity matrix look like? Is it all ones, or is there a single minus one?
If it's the former, you do not use LH matrices, i guess.
If it's the latter, doesn't this exceptional negative number feel somewhat wrong to you?

The Grassman product is defined quite well.

https://www.theworld.com/~sweetser/quaternions/intro/tools/tools.html

This may help?

https://github.com/sjhalayka/blind_poker/blob/master/blind%20poker/bp/matrix_utils.mm

I realize this thread got a little derailed, but I did manage to find my problem in the end, so I thought I'd write this up in case someone else runs into a similarly annoying problem.

The first (and most annoying to discover) issue was that I had my winding order the wrong way around. The second is that I was blindly assuming that I could still use my Z+ forward world coordinates without modifying anything (I suppose I could multiply the z axis by -1 before the LookAt, but that feels far more janky than just negating my z axis altogether in all my calculations).

JoeJ said:
To clarify i have a question: How does the diagonal of your identity matrix look like? Is it all ones, or is there a single minus one?

If that was a question directed at me: I use a normal identity matrix with all 1s. But the only real usecase I have for those matrices is to supply the shader with a vertex transformation matrix and a camera/projection matrix. But because I don't do anything complicated, and especially since I don't operate through a theoretical lens, but rather simply view the mathematical operations as a toolbox, I don't have any necessity to make any established mainstream axioms hold true. For my use cases, I don't need to preserve neat properties or relations between concepts. All I want is to transform geometry into view space. For FPS camera movement, I use the forward+up vector combination. I never liked how complex a matrix multiplication is, since usually you got lots of zeroes in there, which is wasteful on a CPU, if you can help it in any way.

RmbRT said:
If that was a question directed at me: I use a normal identity matrix with all 1s. But the only real usecase I have for those matrices is to supply the shader with a vertex transformation matrix and a camera/projection matrix. But because I don't do anything complicated, and especially since I don't operate through a theoretical lens, but rather simply view the mathematical operations as a toolbox, I don't have any necessity to make any established mainstream axioms hold true.

Well, extracting the relevant information between the self defense, it shows you use a mix of LH and RH yourself, but the things you've done with it so far were not complex enough to reveal any issues from doing so. And thus the realization of this being bad practice is still in front of you.

You could get there sooner if you would just believe in the advise from obviously competent and experienced people (meaning Aressera), instead sabotaging their attempt to help. We don't have so many of such people left on the forum.
And i would not need to derail threads and behaving like an ass. Don't hate me for that.

Aikku said:
I realize this thread got a little derailed, but I did manage to find my problem in the end, so I thought I'd write this up in case someone else runs into a similarly annoying problem.

Yeah, sorry for the derailing. But likely there is not much we could do to help from a distance, with so many details being potentially affected.

I'm still affected from accidental LH matrices pretty often. Often we can construct two axis geometrically, and the 3rd axis is the the cross product of the two we already know. Look at matrix is just one example.
Personally i fail to memorize the order the final cross product should have to give me a RH matrix.
So i place an assert ensuring the determinant of the matrix is positive. If it triggers i reverse the order.

Besides, iirc you mentioned to use NDC space. But that's not really needed for a software rasterizer. A projection matrix also isn't needed. Transforming the vertices to camera space is enough, and you can set focal point and perspective divide directly from there.
Maybe that's some potential optimization you could still do.

Yeah, sorry for the derailing

That's alright, I was curious to see the reasoning behind both sides anyway, since I had been on the fence about LH vs RH before deciding that RH is so ubiquitous that if I wanted to do something really niche at some point, I would find it easier to find examples if everything was already set up for RH processing.

Besides, iirc you mentioned to use NDC space. But that's not really needed for a software rasterizer.

Well, my processing pipeline is designed to be a bit more flexible than other cases. The biggest difference to more “hardwired” rasterizers is that I go from camera space to clip space because my clipper code is self-modifying to clip against the relevant axes (I'm on a slight memory constraint to hold the code, and this was the best solution I could come up with). Going from clip space necessitates going to NDC and then to screen space coordinates, because even if you fold as many of the common factors together, you still have to multiply everything by 1/w; you can't fold the multiply by Width/2 or Height/2 into the 1/w beforehand, so I'm just stuck with doing that (but honestly, that is nowhere near the top of my performance concerns; a much bigger concern is my backface culling logic that triggers far too late into the polygon transformation pipeline).

Aikku said:
code is self-modifying

Hehe, last time i've heard this word was on C64. : )

Aikku said:
a much bigger concern is my backface culling logic that triggers far too late into the polygon transformation pipeline

If many of your polygons share the same plane, you could cull them all with a single test.
A modern approach of this idea is 'cone culling', which works on triangle clusters of similar normals / positions.

Not sure if you can have a level of detail where those things become practical, though.

JoeJ said:

Not sure if you can have a level of detail where those things become practical, though.

Haha yeah nah, I have very, very few polygons being rendered.

For one of the use cases, I'm averaging anywhere between 50-100 polygons (mostly quads, with some triangles mixed in), and the full geometry is no more than about 300 or so polygons. I could definitely optimize some of the culling logic for this (eg. subdivide into a 8x8 grid on the xz planes), but at least for now it's not too big of an issue. Backface culling performance is still a concern, though, since this use case is a “room” geometry, with walls, floors, etc., and I do need backface culling here.

The other use case for the renderer is the one I was trying to fix here; this one is generating a terrain from a heightmap and a tilemap for the graphics (so backface culling is very rare, and limited to the back faces of mountains, etc). The full map is large (144x112 grid), so I can't just blindly send all 16,000+ polygons the renderer; my solution is to figure out the edges of the camera projecting on the floor (including the volume between the floor and the peak of the heightmap displacement), and sending /only/ the cells that the camera will see; since I gave my renderer a hard limit of 200 polygons, the render distance doesn't go very far (and increasing the cap and rendering more geometry has a massive performance drop, since you're rendering tiny polygons as you approach the horizon but still have to fully process them), but it's enough to get a decent enough rendering.

Aikku said:
Backface culling performance is still a concern, though, since this use case is a “room” geometry, with walls, floors, etc., and I do need backface culling here.

When i was working on a 3D engine for early Symbian smart phones (300 MHz ARM cpu), i ended up using 2D BSP similar to Doom instead triangles. It comes with 2.5d limitations (can't look up or down), but it can push much more detail than polygons (which i have used only for characters).

Drawing walls with vertical scanlines and floors with horizontal scanlines, z remains constant so texture mapping is much faster.
There was no need for backface culling, and i also had zero overdraw for the level geometry using a portal spans data structure.
It could not render slopes, though. Just walls or floors.

I also avoided triangles for the terrain:

Instead i was drawing colored edge lines of the height map in front to back order. When drawing the red lines after the blue ones, i was simply drawing vertical spans to connect them, interpolating the colors. This also avoided any cracks when terrain lod has changed across one line to the next. No overdraw either, and it was very fast. Terrain also worked without 2.5 limitations.

Well, then the iPhone came out and shocked me with… what? fucking GPUs on mobiles???
I didn't see that coming, but it meant the end of my project. <:/