Pathfinding animation question

A nice unified way to do swarming is with interagent potentials. I.e. have them exert repulsive/attractive forces on one another, with minima at the interagent distances you want to maintain.

You can also incorporate, in such a framework, global potentials in the environment -- and these are what will prevent the agents from leaving the navmesh. In other words, the boundaries of your environment would radiate a repulsive force.

Some papers include Rimon & Koditschek (for the single-agent case) and Tanner & Jadbabaie Part I and Part II for the multiagent case, though, honestly, I don't really recommend building navigation functions the way Rimon & Koditschek do; instead, there are some natural ways to build potentials that are defined piecewise over your triangles. I don't have a source off the top of my head for this, but it's pretty straightforward; it's the kind of thing you'd just make up yourself if you paused to think about it for a little bit. Maybe I can track down some sources later though.

Thanks, Will take a look at these