I'm coding an Orientation class for my objects and I just decided I wanted to improve the one I had.

I was using the GLM library and I was doing this for each object:

mat4 modelCenter = glm::mat4(1, 0, 0, 0
                             0, 1, 0, 0
                             0, 0, 1, 0
                             obj.cx, obj.cy, obj.cz, 1);
//obj.rotX and rotY are floats that I use cos and sin to calculate
quat modelRotationQ = quat(vec3(0, obj.rotY, obj.rotX));
mat4 modelRotation = mat4_cast(modelRotationQ);
mat4 modelTranslation = glm::mat4(1, 0, 0, 0
                                  0, 1, 0, 0
                                  0, 0, 1, 0
                                  obj.x, obj.y, obj.z, 1);
mat4 modelTransformation = modelTranslation * modelRotation * modelCenter;
 
mat4 modelViewProjection = mat4Projection * mat4View * modelTransformation;     

My camera also was using a sin/cos for the rotations and I used the glm::lookAt function to get it's matrix.

It works, but I don't think it's good and I'd like to try and improve it.

I'm trying to ditch GLM and get my own classes so I can understand exactly what's going on with my objects.

class Vector3
{
    public:
        Vector3()
        {
            x = y = z = 0;
        }
        float x, y, z;
};
class Matrix4x4
{
    public:
        float matrix[4][4];
};
class Orientation
{
    public:
    private:
        Matrix4x4 _transformation;
        Vector3 _position;
        Vector3 _scale;
};

So each object will have an Orientation, and I'll only calculate the matrix of transformation if it's required (the object has been updated) so I avoid doing all those calculations every frame, and I'll just work with the values in the vectors since it's easy to handle and I don't need to generate matrices to update an object's transformation.

Object* newObject = new Object();
newObject->_rotation.x += 90.0f; //adds 90 degrees in X
newObject->_position.x += 10.0f; //translate
 
//During render
 
//Get the vectors values and set up a matrix4x4 once
Matrix4x4 objectTransformation = newObject->GetTransformation();

The problem is, I've read a bunch of tutorials about matrices, vectors and quaternions, but I only managed to implement what was given to me. I think I don't really "understand" matrices transformations, so I'm looking for some help running me through this or any source/tutorial that does it in a very newbie level.

The easy way to think of a matrix (3x3 anyway, just the upper 3x3 part of a 4x4 matrix) is a collection of 3 axes (one in each of the 3 rows), with length.

So the identity matrix which does nothing is just 3 vectors along the x, y and z axes with length 1. Now think of those as 3 of the corners of the unit cube. (The other points in the cube are the origin and the result of adding combinations of the different rows together).

Think of the transform matrix as what it does to that cube. The 3 rows are what happens to the rows in the identity matrix when the matrix is applied. So rotating around the z-axis by 45 degrees (anticlockwise, which is usual for maths - because of the way the trigonometric functions are defined) you will end up with the first row as what happens to the x axis (it goes to [1/sqrt(2), 1/sqrt(2), 0]), the 2nd row is what happens to the y axis (it goes to [-1/sqrt(2), 1/sqrt(2), 0]), the 3rd row is what happens to the z axis (it stays the same, [0, 0, 1]).

You can combine simpler transformations by multiplying the matrices together (the order is important though since matrices are not commutative).

For more understanding check out books or websites on linear algebra, they should start out with the definition of a vector space.

EDIT: Something else easy to understand about matrices, the determinant of a matrix is the volume of the unit cube after the transformation has been applied, although transformations which flip the cube faces inside out negate the determinant.

Something also that might help is actually multiplying some points by hand to see what the transformation is doing. If you have a column vector P = (x,y,z,w) and you multiply it by a 4x4 matrix T, then you'll get another point P' = (x',y',z',w'), which is the transformed point. If you take some simple object and sample a few points, you'll get a general sense of how the transformation works. I would start with simple rotations and translations, then move on to scaling, reflections, etc. If you're not comfortable doing this in 3D, do it first in 2D with P = (x,y,w) and the 3x3 matrix transformation.