D3DXVec3TransformCoord

Quote:

I need to know how it works, and how it transforms a point by a matrix.

The function simply matrix multiplies the given matrix with the given vector and returns the result.
If you do not know how matrix multiplication works, you can read about it in the above link. The algorithm is pretty simple, for element i,j in the resulting matrix you take the inner product of row i of the matrix to the left with column j of the matrix to the right. For example
If A =

 a<sub>11</sub>, a<sub>12</sub>, a<sub>13</sub>a<sub>21</sub>, a<sub>22</sub>, a<sub>23</sub>a<sub>31</sub>, a<sub>32</sub>, a<sub>33</sub>

And you have a vector v

v<sub>11</sub> = v<sub>x</sub>v<sub>21</sub> = v<sub>y</sub>v<sub>31</sub> = v<sub>z</sub>

Then the multiplication will result in a new vector u given by

u<sub>11</sub> = u<sub>x</sub> = <a<sub>1</sub>, v> = a<sub>11</sub> * v<sub>11</sub> + a<sub>12</sub> * v<sub>21</sub> + a<sub>13</sub> * v<sub>31</sub>u<sub>21</sub> = u<sub>y</sub> = <a<sub>2</sub>, v> = a<sub>21</sub> * v<sub>11</sub> + a<sub>22</sub> * v<sub>21</sub> + a<sub>23</sub> * v<sub>31</sub>u<sub>31</sub> = u<sub>z</sub> = <a<sub>3</sub>, v> = a<sub>31</sub> * v<sub>11</sub> + a<sub>32</sub> * v<sub>21</sub> + a<sub>33</sub> * v<sub>31</sub>

That's how it works, then you need to define the matrix in such a way so that when applied to a bunch of vertices these will be transformed in a way which would apply a certain transformation to the object which they represent.

I'm starting to loose track of how many times I've repeated this on this forum ^^

Quote:

But the thing I want to find out is why my D3DXVec3TransformCoord call spits out a point (1,1,1) no matter what I feed into it.

Could we see your code for calling the function please?

I would guess that your matrix that you are transforming by is invalid. I use this call all the time, works for me...