Coordinate system conversion

No one?

I would like to answer your question :) But, I'm not sure of the answer myself. It sounds from your original post like whatever you're doing is working, but you want to make sure it's correct in theory. Is that right?

So you have Euler angle triples. What do you mean when you say you're multiplying these by your conversion matrix? Do you mean you're converting the Euler angles to a rotation matrix, and then multiplying this matrix by the conversion matrix?

Let's say the conversion matrix is M, the rotation matrix (constructed from the original Euler angles) is R, and you have a point p in the original space. The image of p under M, i.e. p after conversion, is:

p' = Mp

Let q be the image of p under R, that is, p rotated by R in the original space:

q = Rp

We want a rotation matrix R' that rotates p' to q' the same way R rotates p to q:

q' = R'p'
Mq = R'Mp
MRp = R'Mp
MR = R'M
MRM^-1 = R'MM^-1
R' = MRM^-1

Since M is orthogonal in your case:

R' = MRM^T

Or in row-vector form:

R' = M^TRM

This is a similarity transform, and has the effect of rotating a point q from transformed space back into the original space, performing the original rotation, and then rotating back into the 'new' space. This makes sense to me intuitively, but I don't know for sure that the above analysis or the solution it leads to is correct. Also, I gather this isn't what you're doing currently, and yet you say it seems to be working ok. So I may be missing something.