Cutting polygons

Depends on what you want to do:

Changing geometry permanently -> subdivision
Just insert new points, delete the old polygon an create the cut polygons from the old and new points. For example you want to devide the triangle ABC. You insert a point D at AC the delete the triangle ABC and create 2 new triangles ABD and DBC - et voilá - you've devided the polygon.

Only a "visual" cut
You want to have a polygon cut visually by an arbitrary plane but want to keep the original geometry. Then you use clip planes which do the "visual" cut when rendering. Try a google on OpenGL clip planes.

I'll change the geometry permanently. But I'm working with complex models(3DS models). Subdivision, I think, works for simple models(square, triangles) and the processing will be very slow if I use subdivision.
Is there atr other way

You could pre-cut the model in a 3d program and then swap it out when you want it to be cut. But this, of course, would only work for specific fractures.

Check out Sutherland Hodgman clipping. One good presentation at:
http://www.cs.princeton.edu/courses/archive/fall00/cs426/lectures/clip/sld035.htm

In order to map the concept to 3D, just use a plane/multiple planes as the "boundary" and use ray-plane intersection to find the "clip-points". May be faster ways to do this, but it's a simple concept. I'm using it to cut up large TIN-terrains in several patches...

Edit:
Also, in case of a very large mesh you can speed up things by culling away polygons that are clearly outside the clip-area and won't be clipped anyway.

/Mike