Fabric simulator

Hello,
If you need more information please ask, I really need to solve this problem, thank you for your help.

It's a very hard topic since it involves non-convex geometry. Solutions I have seen is to use positional coherence between local geometry to form groups of convex geometry and use a regular bounding box, plane or or whatever on those. You then collide triangles once a hit is determined to get the accuracy. There has been a number of topics on this, both on flipcode and gamedev, try searching.

To detect the collisions just do tri-tri collision against the fabrics own mesh tris. This won't be cheap however for large fabrics and you may want to use some sort of spacial partioning technique to speed things up. On the other hand, if your mesh is not able to fold back on itself in a distance comparable or smaller than the mesh tris (ie: your fabric has a cetain rigidity to it) you should be able to get away with doing just halfspace testing between the meshes verts and tris. The difficult part is how you handle self collision and here are a couple of things I can think of off the top of my head.

1: Use penalty based collision. What this means is that when a vert is penetrating the fabric create an extra spring between the vert and the fabric at the point it penetrated (you can just use a rough ray intersection to determine this). These extra springs will pull the cloth back apart and once seperation has been acheived the spring is destroyed.
Advantages) This method is simple to implement and fast computationaly, and will fit in well with the current fabric solver.
Disadvantages) A certain amount of interpenetration is permitted (depending on the spring strength), the system will have difficulties settling to a steady state (which may or may not be visually acceptable with damping applied), and the extra spring forces could alter the fabrics movements in unrealistic ways (eg: by preventing sliding off itself or even causing the fabric to slide the wrong way in extreme cases).

2: Apply constraints. When a vert penetrates the fabric either rewind the simulation to the point the vert first touches the fabric surface (expensive but exact) or simply move the vert alone back until surface contact is acheived. Then as long as the vert and suface contact point are trying to move towards each other apply a rigid constaint which removes all relative degrees of freedom between these points (or even better a constraint which removes all degrees which are non-tangential to the surface so that sliding is still allowed). I've never used Havok but I imagine it will come with some pretty good constaint solvers.
Advantages) No interpenetration, will settle to a stable steady state, more realistic.
Disadvantages) Harder to implement, more computationaly expensive, requires using a method removed from the current fabric solver.

Hope this helps

Two options, one easy and one state-of-the-art:

1) make sure that no two particles (vertices) are too close to each other. that is, effectively insert spheres around each cloth particle and do somewhat normal sphere-sphere collision handling
2) there's a paper called "robust treatment of collision, contact and friction for cloth animation". that method is used in a couple of hollywood movies. it's very accurate, very slow, and very hard to code

Thanks for all of you.
I can imagine what you told me, but I can not imagin it properly inside my head. I probably need to seat down and think about it.
Do you have any virtual image showing these techniques?
uutee, I was thinking about your first point myself and I will test it to see how it goes. Speed is really important for this program, a minimum spec computer should be able to run it smoothly, the one I have already done is pretty cool and fast and I want to keep it like that. any more ideas?

thanks