Where can I start (3D math)

Here are a couple of things I use all the time and find very useful.

Dot Product
given two vectors a and b the dot product is defined as follows

length(a) * length(b) * cos(angleBetweenAandB) = a[sub]x[/sub] * b[sub]x[/sub] + a[sub]y[/sub] * b[sub]y[/sub] + a[sub]z[/sub] * b[sub]z[/sub] = a dot b
[sub]
[/sub]
The dot product is very useful. You can use it to find the angle between two vectors
angle = acos(a dot b) / (length(a) * length(b))

You can also use this to find what is called vector projection.


In the diagram above. A is projected onto B resulting in the part labeled |A|cos [font="sans-serif"]?[/font] . To find the projection vector, I will call it c, we do the following
c = b * (a dot b) / (b dot b)
This has many uses. You can use this to find where a point is closest to a line. It is used frequently in overlap detection algorithms like you want to do between triangles and quads.


Cross Product
given two vectors a and b. The cross of them is the vector perpendicular to both a and b and is defined as
c[sub]x[/sub] = a[sub]y[/sub] * b[sub]z[/sub] - a[sub]z[/sub] * b[sub]y[/sub]
c[sub]y[/sub] = a[sub]z[/sub] * b[sub]x[/sub] - a[sub]x[/sub] * b[sub]z[/sub]
c[sub]z [/sub]= a[sub]x[/sub] * b[sub]y[/sub] - a[sub]y[/sub] * b[sub]x[/sub]
[sub]
[/sub]
[sub]
[/sub]
Another very important thing you need to understand are transformations. Most commonly in the form of matrix transformations.
The basic idea is this. Say you want to transform a set of points meaning you want to rotate, move, or expand or contract them(scaling). To transform them you pass it through a transformation.

Trivial example.

Vector3 movePoint(Vector3 point, Vector3 offset)
{
return Vector3(point.x + offset.x, point.y + offset.y, point.z + offset.z)
}



Obviously the above example is a very simple transformation. Most transformations do more than just move objects around. You also want to be able to rotate and scale them. You also want to be able to append multiple transformations meaning you start with point a, pass it through one transform called X resulting in point b then passing that trough another transformation, Y, resulting in point c. It would be much better to join transformations X and Y to only have to transform the point once going from point a directly to point c in one transform. Matrices can do that.

Here are three simple (but not all) transformations that can be done with matrices.
Translation (Move by x, y, z amount)
Rotation
Scaling Makes this bigger or smaller. Can flip objects if you specify a -1 for one of the axis.

To join any two transformations simply multiply them.

C = A * B

Any number of matrices can be multiplied together taking a large number of translations, rotations, and scales and turning them into a single matrix.

M = A * B * C * D

Then to transform a point simply multiply the point by the matrix treating the point as a 1x4 matrix with the last element in the 1x4 matrix being a 1

b = M * (a.x, a.y, a.z, 1)

Multiplying the above matrix M by a is the same as multiplying a by D then by C then by B then by A. As you can see, combining matrices is much more efficient. Remember, the right most transformations will be done first meaning with M, D is first, A is last. DirectX matrices are opposite of that, but I will go into more detail why in a second.

Using matrices as transformations is used at the heart of 3D graphics. 3D graphics data is made up of many 3D points connected in triangles. Both DirectX and OpenGL require a matrix to control where an object is drawn in the 3D scene.

DirectX, as I briefly mentioned, has a minor difference in matrix transformations. The matrices are all transposed, meaning they are flipped on the diagonal going from top left to bottom right. Because they are transposed the point needs to be on the left of the matrix when multiplying.

b = (a.x, a.y, a.z, 1) * M

And the leftmost matrix is the first transformation to be done when stacking multiple matrices.

M = D * C * B * A

I just threw a whole lot at you. Don't be frustrated if you don't grasp it all right away it is a lot to take in. All of this stuff is covered in a college level class on the subject of Linear Algebra. So I suggest you study Linear Algebra whether in school or on your own. If you want me to show how to solve any of the problems you listed above or need any further clarifcation let me know.