Skip to main content
GameDev.net gamedev.net
🔒 Locked

visual c++ SIMD problems

Started by cignox1 Jul 29, 2007 at 11:27 AM 15 replies 3.4k views
Original Post
cignox1
cignox1
Hi, recently I noticed that my program actually slows down when compiled with SIMD optimization enabled on VC++ 2005. I bagan to test my matrix/vector libray and with SIMD enabled performance fall down up to 30% for matrix-matrix multiplication. AFAIK matrix mult should be considerably faster when calculations are done in parallel, but it is instead way slower. Do someone know if this is a known bug in VC++, if there are code setups to be avoided in order to gat advantage of SIMD optimization or if there is something wrong in my code? This is my matrix class (and the multiplication method):

//In the form of ROWxCOL
class matrix
{
    public:
        real mat[4][4];
        
    public:
        matrix() { Identity(); }
        matrix(const matrix &src) 
                { memcpy(this, &src, 16 * sizeof(real)); }       
.
.
.
}

const matrix matrix::operator * (const matrix &m) const
{
    matrix t;
    register int i;
    for(i = 0; i < 4; ++i)
    {
        t.mat[0] =  mat[0]*m.mat[0][0] + mat[1]*m.mat[1][0] + 
                mat[2]*m.mat[2][0] + mat[3]*m.mat[3][0]; 
                
        t.mat[1] =  mat[0]*m.mat[0][1] + mat[1]*m.mat[1][1] + 
                mat[2]*m.mat[2][1] + mat[3]*m.mat[3][1];
                
        t.mat[2] =  mat[0]*m.mat[0][2] + mat[1]*m.mat[1][2] + 
                mat[2]*m.mat[2][2] + mat[3]*m.mat[3][2];
                
        t.mat[3] =  mat[0]*m.mat[0][3] + mat[1]*m.mat[1][3] + 
                mat[2]*m.mat[2][3] + mat[3]*m.mat[3][3];
    }    
    return t;
}


Spoonbender
Spoonbender
Well, it doesn't seem like you actually have any SIMD code.
The compiler isn't clever enough to generate that by itself.

You'll have to use the compilers SSE intrinsics (or write the asm yourself) to take advantage of SIMD.

That said, I'm not sure why your code becomes so much slower when you enable SIMD How exactly do you do that? Is it the Enable Enhanced Instruction Sets option?
That only tells the compiler to use the SSE unit instead of the x87 one. But it'll still generate scalar floating point operations only.

However, it still shouldn't be slower.
I guess you might have to look at the assembly generated by the compiler, and see if you can spot any big differences.
Antheus
Antheus
Quote:
Original post by Spoonbender

However, it still shouldn't be slower.
I guess you might have to look at the assembly generated by the compiler, and see if you can spot any big differences.


It happens in some cases. I believe it's loading of floating point data that causes that, whereas in trivial cases simpler, non SSE code is faster.

Quote:
AFAIK matrix mult should be considerably faster when calculations are done in parallel


Only if you write your own intristics to perform it in parallel. Generic compiler will be unable to perform algorithmic optimization. Even concurrency-enabled compilers require hints as to what can and what can't be parallel.
cignox1
cignox1
Well, I hoped the compiler to be able to recognize a suitable SIMD code (IIRC there is a compiler, Vector C, that does it, but I'm not sure..) but when I wrote the code I didn'even thought of a possible SIMD optimization :-)
That said, I have also noticed that if I set real to double instead than float and set the SSE2, results are better. Tomorrow I will test this claim (I don't have a SSE2 processor at the moment. I don't need all that optimization either since I'm working on a off-line ray tracer, not a linear algebra library, but I simply will use these classes to better understand optimization for future needs :-)
Thank you!

As about looking the ASM code: I could, but I never studied ASM seriusly and I don't use it since years. I will give it a try, though.
cignox1
cignox1
I tried the code on a SSE2 enabled cpu. The fastest code is with float and no optimization, then double with SSE2 optimization and then float with SSE optimization...
Julian90
Julian90
Quote:
I tried the code on a SSE2 enabled cpu. The fastest code is with float and no optimization, then double with SSE2 optimization and then float with SSE optimization...


Did you try it with /fp:fast? When using /fp:precise with SSE under VS the compiler will convert all floats values to doubles every time it loads them to register and converts them back when it saves them so that may be partly responsible for the slow down with SSE.
cignox1
cignox1
Quote:
Original post by Julian90
Quote:
I tried the code on a SSE2 enabled cpu. The fastest code is with float and no optimization, then double with SSE2 optimization and then float with SSE optimization...


Did you try it with /fp:fast? When using /fp:precise with SSE under VS the compiler will convert all floats values to doubles every time it loads them to register and converts them back when it saves them so that may be partly responsible for the slow down with SSE.


You win! :-) This should explain (partially) why doubles were faster. Now I have to decide if the precision loss will hurt too much. First, I will add asm sse/sse2 code to see how it perform with precise (or strict) modes, then I will decide. Thank you very much!
cignox1
cignox1
I did some more tests and the results are quite strange to me...
Here they are:

-float w/o SSE option enabled with ASM SSE code (see below): 454ms
-float with SSE enabled with ASM code: 422ms
-float with SSE enabled w/o ASM code: 329ms
-float w/o SSE enabled w/o ASM code: 376ms

-double w/o SSE2 enabled: 328ms
-double with SSE2 enabled: 172ms

These tests were done on 10000000 matrix*vector multiplications, 1 thread and "fast" for floating point option. Where no ASM code was used, I used the plain algorithm in c++. I didn't make averages with more tests, so those numbers are inherently imprecise. The ASM code, that I took from a web tutorial, is the following (I modified it to work with my specific vector and matrix classes):
const vector3 matrix::operator * (const vector3 &v) const{    vector3 t;#ifdef USE_DOUBLE	Apply(t);#else	//Fast SSE code when float specified	float* const row0 = (float*)mat;	float* xin = (float*)&v.x	float* xout = (float*)&t.x	__asm 	{		mov         esi, xin		mov         edi, xout		// load columns of matrix into xmm4-7		mov         edx, row0		movups   xmm4, [edx]		movups   xmm5, [edx+0x10]		movups   xmm6, [edx+0x20]		//   movups   xmm7, [edx+0x30]		// load v into xmm0.		movups   xmm0, [esi]		// we'll store the final result in xmm2; initialize it		// to zero		xorps      xmm2, xmm2		// broadcast x into xmm1, multiply it by the first		// column of the matrix (xmm4), and add it to the total		movups   xmm1, xmm0		shufps   xmm1, xmm1, 0x00		mulps      xmm1, xmm4		addps      xmm2, xmm1		// repeat the process for y, z and w		movups   xmm1, xmm0		shufps   xmm1, xmm1, 0x55		mulps      xmm1, xmm5		addps      xmm2, xmm1		movups   xmm1, xmm0		shufps   xmm1, xmm1, 0xAA		mulps      xmm1, xmm6		addps      xmm2, xmm1		/*     movups   xmm1, xmm0		shufps   xmm1, xmm1, 0xFF		mulps      xmm1, xmm7		addps      xmm2, xmm1*/		// write the results		movups   [edi], xmm2	}#endif    return t;}  


Looking at the huge difference between SSE2 disabled and SSE2 enabled for double type make me think that the VC++ is indeed able to optimize the code to make use of SIMD parallelization with doubles...
I can understand that ASM code, but I'm not sure if it uses SIMD the way it should be used. There is anyway only a very small timing difference between it and the c++ code I wrote.
Note howewer that there is a difference between the ASM version and the c++ one: the c++ on directly works on the result variable (passed as argument), thus avoiding the creation and the copy of a temporary vector3 (perhaps 2?) object. Could this be so slow to completely hide ASM optimizations?
Could someone please help me understanding these results?

Thank you!

mattnewport
mattnewport
Using inline assembly for optimization isn't generally a good idea these days. Microsoft's recommendation is to use intrinsics if you want to write code that explicitly uses SSE instructions. The reason inline assembly is not recommended is that it generally stops the compiler from performing any optimizations on the inline assembly and limits the optimizations it can perform on the surrounding code. That may partially explain the results you're seeing.

Another issue to be aware of is that with modern optimizing compilers you have to be very careful about how you benchmark code. Simply doing an operation many times in a loop and timing it is generally not going to give you very useful timing results. This is particularly true with Visual C++ if you have global optimizations enabled. If you're not careful the optimizer may be throwing out work that it recognises as redundant even though you actually want the code to be run so you can time it. To ensure somewhat meaningful and accurate timing results you need to write your code in a way that prevents the compiler optimizing out what you're trying to test and you really need to look at the generated assembly to verify that you're testing what you think you're testing and to understand unexpected timing results.
Spoonbender
Spoonbender
Quote:
Original post by cignox1
I did some more tests and the results are quite strange to me...
Here they are:

-float w/o SSE option enabled with ASM SSE code (see below): 454ms
-float with SSE enabled with ASM code: 422ms
-float with SSE enabled w/o ASM code: 329ms
-float w/o SSE enabled w/o ASM code: 376ms

-double w/o SSE2 enabled: 328ms
-double with SSE2 enabled: 172ms

These tests were done on 10000000 matrix*vector multiplications, 1 thread and "fast" for floating point option. Where no ASM code was used, I used the plain algorithm in c++. I didn't make averages with more tests, so those numbers are inherently imprecise. The ASM code, that I took from a web tutorial, is the following (I modified it to work with my specific vector and matrix classes):
*** Source Snippet Removed ***

Looking at the huge difference between SSE2 disabled and SSE2 enabled for double type make me think that the VC++ is indeed able to optimize the code to make use of SIMD parallelization with doubles...

Should be easy to check if you make the compiler output the asm it generates. It would surprise me though.
However, one reason why your asm versions don't perform terribly well is that they use unaligned loads. (movups). If you can arrange your data so it is aligned on 128-bit boundaries, you can use movps, which is a *lot* faster.

Personally, I prefer using compiler intrinsics instead of asm to write SIMD code, but I doubt that makes a big difference.
But when using SIMD instructions, you have to be really careful to get good performance. Unaligned loads can really wreck your performance (so much that non-SIMD code might end up faster)
I can understand that ASM code, but I'm not sure if it uses SIMD the way it should be used. There is anyway only a very small timing difference between it and the c++ code I wrote.
ldeej
ldeej
SSE optimizations on the VC compiler are fairly conservative, other compilers do better jobs at taking advantage of these type of optimizations (i.e. the Intel compiler)

Using the restrict keyword on the arguments might help reduce the copies of the arguments passed in, but make sure you understand what you are getting into if you use restrict.

Note also that using intrinsics reduces the ability of the compiler to optimize your code (i.e. the compiler has less freedom to reorder or replace your instructions), so if you use instrinsics make sure that your code is as optimal as possible.

If you are going the inline assembly route you might be able to use SSE3 which has some instructions that make dot product type of computations easier (HADDP), as somebody suggested earlier be really careful with your test case since the VC compiler is very aggressive on removing redundant and dead code.
cignox1
cignox1
Thank you all very much for the help!

Quote:

Using inline assembly for optimization isn't generally a good idea these days. Microsoft's recommendation is to use intrinsics if you want to write code that explicitly uses SSE instructions.

I should really give them a look (can I use intrinsics with gcc as well? Lets't see...)

Quote:

If you're not careful the optimizer may be throwing out work that it recognises as redundant even though you actually want the code to be run so you can time it. To ensure somewhat meaningful and accurate timing results you need to write your code in a way that prevents the compiler optimizing out what you're trying to test and you really need to look at the generated assembly to verify that you're testing what you think you're testing and to understand unexpected timing results.

I was aware about this possibility, but never investigate how to avoid that. The only thing I did was being sure that calulation results are used to modify the value of a variable (I multiply then time the matrix and the vector, so in oder to calculate the final vector value I have to compute all the 9 previous multiplications...) but I don't know if this is enaught...

Quote:

If you can arrange your data so it is aligned on 128-bit boundaries, you can use movps, which is a *lot* faster.

Except than forcing the compiler to align data, what should I do? Since my vector is a 4 float object (well, after I modified it, with only x,y,z the ASM code crashed) it should be aligneable... am I right?

As about SSE3: well my pc supports only SSE (at work SSE2) si it is for my next machine :-)
Spoonbender
Spoonbender
Quote:
Original post by cignox1
I should really give them a look (can I use intrinsics with gcc as well? Lets't see...)

Yeah. I can't guarantee that they use exactly the same syntax, but they exist, and they look very similar, at least.
mattnewport
mattnewport
Quote:
Original post by cignox1
Quote:

If you're not careful the optimizer may be throwing out work that it recognises as redundant even though you actually want the code to be run so you can time it. To ensure somewhat meaningful and accurate timing results you need to write your code in a way that prevents the compiler optimizing out what you're trying to test and you really need to look at the generated assembly to verify that you're testing what you think you're testing and to understand unexpected timing results.

I was aware about this possibility, but never investigate how to avoid that. The only thing I did was being sure that calulation results are used to modify the value of a variable (I multiply then time the matrix and the vector, so in oder to calculate the final vector value I have to compute all the 9 previous multiplications...) but I don't know if this is enaught...

That's a necessary but not always sufficient step to ensuring the compiler isn't avoiding redundant work. You usually have to at the very least output the value that you calculate, not just assign it, so that it's not thrown away. You sometimes also have to read things like loop iterations and source data in from a file or from stdin in order to avoid the compiler being too clever. Ultimately checking the assembly is the only way to be sure that you're testing what you think you're testing.
Quote:

Quote:

If you can arrange your data so it is aligned on 128-bit boundaries, you can use movps, which is a *lot* faster.

Except than forcing the compiler to align data, what should I do? Since my vector is a 4 float object (well, after I modified it, with only x,y,z the ASM code crashed) it should be aligneable... am I right?

The easiest way to get correct alignment is to declare the data member of your vector as an __m128 member which the compiler knows to place in SSE registers and to enforce 16 byte alignment. You still have to be careful about alignment for dynamically allocated memory though - the compiler only guarantees 16 byte alignment for stack allocated memory, not for dynamically allocated memory. If you want to manually specify alignment you can use __declspec(align(16)) on VC.
cignox1
cignox1
Ok, here a first (stupid) test with intrinsics.
vector3 ScaleOld(){	float x = 1.1, y = 1.1, z = 1.1, w = 1.1;	float val = 1.00001;	for(int i = 0; i < M_COUNT; ++i)	{		for(int j = 0; j < 100; ++j)		{			x *= val;			y *= val;			z *= val;			w *= val;			x -= val;			y -= val;			z -= val;			w -= val;						x /= val;			y /= val;			z /= val;			w /= val;			x += val;			y += val;			z += val;			w += val;		}	}	return vector3(x, y, z + w);}vector3 ScaleNew(){	__m128 xyzw = _mm_set_ps1(1.1);	__m128 val = _mm_set_ps1(1.00001);	for(int i = 0; i < M_COUNT; ++i)	{		for(int j = 0; j < 100; ++j)		{			xyzw = _mm_mul_ps(xyzw, val);			xyzw = _mm_sub_ps(xyzw, val);			xyzw = _mm_div_ps(xyzw, val);			xyzw = _mm_add_ps(xyzw, val);		}	}	return vector3(xyzw.m128_f32[0], xyzw.m128_f32[1], xyzw.m128_f32[2] + xyzw.m128_f32[3]);}


Well, this algorithm could be easily thrown away by the compiler, but apparently it does not (the timings increase as I increase the loop end condition...) and it shouldn´t, since I print the resulting vector on screen...
They always perform quite the same: with SSE compiler option enabled the time required is half the time required with the option off. This lead me to think that indeed Vc++ recognizes some places where the computation can be helped by SIMD parallelism... should that option be on also with intrinsics? Because apparently it must be...
It could also me using intrinsics in the wrong way (I began to look into them a copule of hours ago :-(

I will also give a look at the ASM code generated by the compiler, though I don´t really know ASM very well (more like to "not at all", actually).

Sorry for being all that annoying, I´m moving my first steps and I wasn´t able to find a good tutorial on this topic.. (some good link, reference documents aside?).

Thank you again
Skizz
Skizz
Quote:
Original post by cignox1
*** Source Snippet Removed ***
Thank you!


Your asm is not particularly optimised - you have a lot of sequential dependancies, that is, using the result of one instruction in the following instruction. I reordered some instructions and fiddled around a bit and came up with this:
__declspec (align (16)) class Vector4{public:  Vector4 ()  {    memset (vec, 0, sizeof vec);  }  Vector4 (float x, float y, float z, float w)  {    vec [0] = x;    vec [1] = y;    vec [2] = z;    vec [3] = w;  }  float vec [4];};__declspec (align (16)) class Matrix{public:  Matrix (float *src)  {    memcpy (mat, src, sizeof mat);  }  float mat [16];  void multiply (const Vector4 &src, Vector4 &dest)  {    __asm    {      mov esi,src      mov edi,dest      lea edx,[ecx].mat      movaps xmm7,[esi]      movaps xmm0,[edx]      pshufd xmm4,xmm7,0      movaps xmm1,[edx+0x10]      pshufd xmm5,xmm7,0x55      movaps xmm2,[edx+0x20]      pshufd xmm6,xmm7,0xaa      movaps xmm3,[edx+0x30]      pshufd xmm7,xmm7,0xff      mulps xmm0,xmm4      mulps xmm1,xmm5      mulps xmm2,xmm6      mulps xmm3,xmm7      addps xmm0,xmm1      addps xmm2,xmm3      addps xmm0,xmm2      movaps [edi],xmm0    }  }    // this is the original code  const Vector4 operator * (const Vector4 &v) const  {    Vector4 t;    float* const row0 = (float*)mat;    float* xin = (float*)&v.vec[0];    float* xout = (float*)&t.vec[0];    __asm     {      mov esi, xin      mov edi, xout      // load columns of matrix into xmm4-7      mov edx, row0      movups xmm4, [edx]      movups xmm5, [edx+0x10]      movups xmm6, [edx+0x20]      // load v into xmm0.      movups xmm0, [esi]      // we'll store the final result in xmm2; initialize it to zero      xorps xmm2, xmm2      // broadcast x into xmm1, multiply it by the first      // column of the matrix (xmm4), and add it to the total      movups xmm1, xmm0      shufps xmm1, xmm1, 0x00      mulps xmm1, xmm4      addps xmm2, xmm1      // repeat the process for y, z and w      movups xmm1, xmm0      shufps xmm1, xmm1, 0x55      mulps xmm1, xmm5      addps xmm2, xmm1      movups xmm1, xmm0      shufps xmm1, xmm1, 0xAA      mulps xmm1, xmm6      addps xmm2, xmm1      // write the results      movups [edi], xmm2    }    return t;  }  };void main (){  float ms [16] =  {    1, 0, 0, 0,    0, 3, 0, 0,     0, 0, 2, 0,    0, 0, 0, 1  };  Matrix m (ms);  Vector4 s (1, 2, 3, 4),d,e;  int t1 = timeGetTime();  for (int i = 0 ; i < 1000000 ; ++i)  {    m.multiply (s, d);  }  int t2 = timeGetTime();  for (int i = 0 ; i < 1000000 ; ++i)  {    e = m * s;  }  int t3 = timeGetTime();  int s1 = t2-t1;  int s2 = t3-t2;  return 0;}

On my PC, in a debug build (so no optimisations), s2 is about 250 and s1 about 65. Using movaps instead of movups makes no difference (the memory's aligned). Note that the faster version is doing a full 4x1 vector by 4x4 matrix calculation.
If I change your code to the following, so the test is just comparing the asm (all other code being the same):
  void multiply2 (const Vector4 &v, Vector4 &t) const  {    float* const row0 = (float*)mat;    float* xin = (float*)&v.vec[0];    float* xout = (float*)&t.vec[0];    __asm     {      mov esi, xin      mov edi, xout      // load columns of matrix into xmm4-7      mov edx, row0      movups xmm4, [edx]      movups xmm5, [edx+0x10]      movups xmm6, [edx+0x20]      // load v into xmm0.      movups xmm0, [esi]      // we'll store the final result in xmm2; initialize it to zero      xorps xmm2, xmm2      // broadcast x into xmm1, multiply it by the first      // column of the matrix (xmm4), and add it to the total      movups xmm1, xmm0      shufps xmm1, xmm1, 0x00      mulps xmm1, xmm4      addps xmm2, xmm1      // repeat the process for y, z and w      movups xmm1, xmm0      shufps xmm1, xmm1, 0x55      mulps xmm1, xmm5      addps xmm2, xmm1      movups xmm1, xmm0      shufps xmm1, xmm1, 0xAA      mulps xmm1, xmm6      addps xmm2, xmm1      // write the results      movups [edi], xmm2    }  }  

the timing drops to 85 which is still 20 short of my code (30% slower), which shows the overhead of returning the value1. Don't forget that my code does an extra multiply as well.

There's more to SIMD (and asm in general) than just creating a sequence of instructions - you need to understand how the CPU works internally to get the best out of it. Unless you get the absolute best out of the CPU with hand written assembler, it really isn't worth doing since the compiler can usually do a better job.

Skizz

Note 1: Return value optimisation is done by the compiler which effectively converts the operator* function into the multiply function, i.e. pass in a pointer to the result rather than create and copy an instance.
cignox1
cignox1
Quote:
Original post by Skizz
Quote:
Original post by cignox1
*** Source Snippet Removed ***
Thank you!


Your asm is not particularly optimised - you have a lot of sequential dependancies, that is, using the result of one instruction in the following instruction. I reordered some instructions and fiddled around a bit and came up with this:
*** Source Snippet Removed ***



Thank you very much, I will look at it, and compare it with the code I used...

Quote:

There's more to SIMD (and asm in general) than just creating a sequence of instructions - you need to understand how the CPU works internally to get the best out of it. Unless you get the absolute best out of the CPU with hand written assembler, it really isn't worth doing since the compiler can usually do a better job.


Don't worry, I'm perfectly aware of that :-) Actually, I'm not trying really to outperform compiler code, but only to understand a bit better what lower level optimization techniques are: For my raytracer I don't need at all a faster matrix/vector library, but I think that some of these techniques may become usefull later, then it is better to try them with a easier and working code like matrix and vector classes than experimenting with kd-tree or shaders...
Furthermore, it's an interesting thing to experiment with :-)

Quote:

Note 1: Return value optimisation is done by the compiler which effectively converts the operator* function into the multiply function, i.e. pass in a pointer to the result rather than create and copy an instance.

[/quote]
That's a good new :-)

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.

Sign in to reply to this topic.