How do professional game developers deal with enemies entering doors?

Audio_Bellum said:
but, I don't know how to do that.

That's the hard part. That's also why most games have really bad AI. Just like almost every racing game under the sun has extreme rubberbanding and mechanics-cheating AI for enemies.

I have no advice to give, especially no general advice. It comes down to thinking deeply about the specific problem and finding a way to somehow translate your intuition into code.

First off, remember that the goal of game AI's is not to create a perfect response. If that were the case, player characters would die from multiple headshots any time they left cover.

Designers tend to build up logic around them. Doors are a great choke point and they can allow for a fun player experience. Doors can allow for inexperienced / bad / lazy players to get an easier time, and for many games designers choose to let them be exploitable by players.

There are generally two ways to think of doors, one is as a portal system with a zone around the entry, the other as a wide navigable area. I've found the portal system is more common. With a portal system and steering system when an NPC steers close to the door there is often a range-based trigger; if the NPC intends to go through the door the steering system is briefly replaced with a different control “walk cleanly through the doorway”, then restored too regular steering; if the NPC didn't intend to go through the door the steering system directs the NPC away from the doorway trigger area.

I'll even show some of the the code for it on the enemy side:

…

That code is far too hard-coded for what is typically done. It also involves more combined pieces all at once.

Usually in character design there are a set of patterns that get assigned to different AI characters. Designers look at what fits the game and what they think is fun. They're put together into modular ways, and the “hunt down the player” logic is distinct from the “give up hunting” logic. Hunting down the player may involve specific logic for doorways, overhangs, cover, explosive/flamable objects, and more, depending on details of the game.

Some characters are designed to charge forward blindly. Often this will be built into a “charge in a straight line and continue even if the player ducks out of the way”, a predictable pattern that allows players to create a strategy. It works no different in a doorway, although the logic of the portal through the doorway varies. There is no universal system, but often as an NPC approaches a doorway they reserve the portal as a slot, in a wide doorway that may have 2 or 3 slots they reserve any of the positions as they charge. They'll route once to the doorway portal, run through it, and continue running on the other side, releasing the portal as they exit.

Some characters are more strategic. Some look for cover, they will slide up adjacent to the doorway and peek over the edge. In a wide doorway they'll reserve the edge slot. In a narrow one-slot doorway portal they may be next to the door, reserve the slot, peek around, possibly take a shot, then move back and release the portal. I've been involved in a couple where designers wanted that to stack, the characters would seek cover on one or both sides, and then they'd take cover behind each other slightly back, much like a flock of geese in a “V” formation behind the cover. This type of character also tends to seek cover around corners and other blind spots on maps as part of their design. These characters as part of their character design often have communication with other NPCs they share line-of-sight with. These offer a different gameplay challenge to players, so designers can work with it.

Another type of characters looks for ways to ambush. Doorways are built with these in mind, level designers have two or three ways around so the player goes through the doorway and the NPC is triggered by something shown in the game, perhaps a security camera or motion detection lights or audio triggers, but they become something of a stealth mission. Players can use stealth to avoid triggering the NPC, but once triggered the NPC identifies the route the player is taking, then routes the opposite direction around the obstacle. Level designers can build this with doorways, mazes, columns / pillars, and more. These either communicate with other NPCs through line-of-sight directly or through the sensing mechanisms, like if they see the player in a security camera they'll choose to take the alternate routes that others also watching the security camera didn't take, as though the NPC's had a quick a “you go left and I'll go right” discussion. If the door is a a gated portal system, they'll reserve the slot when their steering behavior routes them near the door, reserve it as they pass through, and release it when

This logic is generally distinct from the logic about when to give up. They may be based on distance, time since last activated / sensed, or only as long as the player is doing whatever provoked the NPC, like a guard that only gets mad when you're inside their guarded area, and loses all interest the moment you step outside the area.

Audio_Bellum said:
A more proper solution would involve having the enemies take turns to enter the door or have them go through the door in a line but, I don't know how to do that.

I would divide this into two options.

A ‘quantized’ approach, where things move in a strict order in discrete steps, often within a simplified domain e.g. a grid of tile cells.
Examples would be the classic 8-bit game Boulder Dash, where the player can dig away tiles of soils, then rocks above the soils eventually fall down. The rocks follow simple logic. If there is a stack of two rocks on top of a cell of soil which was just digged away, only the bottom rock can move down to the empty cell. After it moved, the top rock detects it's space below is now empty and it can move down too. Due to simple logic, we don't need to calculate any forces or pressure, and we don't need real numbers but integers are enough.
Another example of such system is the simulation of sand and fluid in the recent game Noita, which does the same thing just at higher resolution.
And similar methods are still used in modern games to model behavior of npcs or enemies.
I would also put path finding into this category, since it's graph data structure is always a quantized representation of a world and paths describe movement in discrete large steps ignoring time.

The other option is the ‘smooth' approach, attempting to avoid quantization and trying to simulate things more in a way they happen in the real world. Example would be rigid body simulation solving for resting contact forces in a stack of rocks, or fluid simulation to model sand or water particles, or flock simulation to model a swarm of birds avoiding collisions. It's way more advanced math of calculus and linear algebra with real numbers, and it requires more processing power to do such simulations.
But - since that's the option you want - i would not say it's much harder to do, assuming you're familiar with math basics such as vectors and integrating time. There was a similar question years a go and i made a simple code example. The question was how could we get out many agents from a crowded room if they all have to go through a narrow door.
My code example does this with a really basic physics simulation where agents collide with walls and each other, so they simply get pushed out until each agent has enough space and there are no more intersections. And i would do the same thing to model npc behavior for problems like the one you ask for.
Some images showing how the code example evolves over time:

Initially the randomly green agents intersect each other and the white walls.

After one iteration of resolving collisions, wall penetrations are gone but they still intersect each other. (some debug vis in other colors still shows their initial / previous states)

After 30 iterations agents on the outside no longer intersect, but in the crowded room there are still too much of them.
We need more iterations to push them all out…

After 700 iterations the result is good enough.

In a game i would need only one iteration per frame to minimize the illegal penetrations over time.
And each agent could still have it's own velocity to move towards a target e.g. the next door along a path.
I could have a larger radius of soft collisions to avoid colliding with nearby obstacles.
And so forth. With such methods i can model natural behavior without any logic, decision making or intelligence.
For an FPS game it becomes harder. My agents would stick on the polygons of a nav mesh, requiring more advanced tests to detect collisions against the boundary edges. And i might need a function to trace a ray sticking on the polygons while crossing internal edges connecting them. I might represent doors with a line projected down to the nav mesh polygons, becoming another dynamic obstacle. And at this point npcs also need some logic to respond on closed vs. open doors, and they need ability to change this state. etc.
Also, this system would run beside the engines default physics simulation handling collisions with walls too. My system is only meant to model behavior, so i can keep it much simpler than a rigid body simulator for example.

Personally i would NOT attempt to model behavior with path finding. Path finding only tells me through which doors i need to go in which order, but it does not help me with realistic and smooth behavior.
But different people come up with different solutions for different games, ideally.

Here the code for the example:

		static bool testParticleAgents = 0; ImGui::Checkbox("testParticleAgents", &testParticleAgents);
		if (testParticleAgents)
		{
			struct Agent
			{
				vec2 pos;
				float mass;
				float radius;

				void Vis (float r, float g, float b)
				{
					RenderCircle (radius, pos, vec(0,0,1), r,g,b);
				}
			};

			struct Obstacle
			{
				vec2 minCorner;
				vec2 maxCorner;

				void Vis (float r, float g, float b)
				{
					//assert(minCorner[0] < maxCorner[0] && minCorner[1] < maxCorner[1]);
					RenderAABBox(minCorner, maxCorner, r,g,b);
				}
			};

			struct Collision // making this struct only to have a local function (back then i was no yet familiar with C++11 lambdas)
			{
				static void ResolvePairOfAgents (Agent &a0, Agent &a1, float colorize)
				{
					float minDist = a0.radius + a1.radius;
					vec2 diff = a1.pos - a0.pos;
					float dist = diff.Length();
					if (dist < minDist) // colliding
					{
						float massSum = a0.mass + a1.mass;
						vec2 sep = diff / dist * (minDist - dist); // vector to resolve the intersection
						RenderVector(a0.pos, -sep * (a1.mass / massSum), 1-colorize,0,colorize); // visualize trajectory
						RenderVector(a1.pos,  sep * (a0.mass / massSum), 1-colorize,0,colorize);
						a0.pos -= sep * (a1.mass / massSum); // we distribute the vector to both agents, weighted by mass
						a1.pos += sep * (a0.mass / massSum);
					}
				}
			};

			auto ResolveAgentObstaclePair = [](Obstacle &o, Agent &a) // lambas also give us local functions, but it's more convenient
			{
				if (a.pos[0] < o.minCorner[0]-a.radius ||
					a.pos[1] < o.minCorner[1]-a.radius ||
					a.pos[0] > o.maxCorner[0]+a.radius ||
					a.pos[1] > o.maxCorner[1]+a.radius)
						return; // skip if no collision
					
				vec2 center = (o.minCorner + o.maxCorner) * .5f;
				vec2 half = o.maxCorner - center;
				
				// calculate closest point on obstacle boundary
				vec2 closest;
				bool inside;
				{
					vec2 diff = a.pos - center;
					vec2 clampedDiff (	min(max(diff[0], -half[0]), half[0]),
										min(max(diff[1], -half[1]), half[1]) );
					inside = (clampedDiff == diff);

					if (inside) // circle center is inside rectangle
					{
						closest = a.pos;
						int d = (half[0] - fabs(clampedDiff[0]) > half[1] - fabs(clampedDiff[1]));
						//RenderLabel (a.pos, 0,0.5f,1, "%i", d);
						closest[d] = center[d] + half[d] * (diff[d]<0.f ? -1.f : 1.f);
					}
					else // outside, in the voronoi regions of either edges or vertices
					{
						closest = center + clampedDiff;						
					}
				}

				// resolve collision
				{
					vec2 diff = a.pos - closest;
					float dist = diff.Length(); // todo: if the agent is axactly at the wall this becomes zero and won't work. fix: introduce a contact normal
					if ((inside || dist < a.radius) && dist > FP_EPSILON)
					{
						float mag = (!inside ? a.radius - dist : -dist - a.radius);
						a.pos += diff / dist * mag;
					}
				}
			};

			// create some obstacels

			csl::vector<Obstacle> obstacles(4);
			int i=0;
			obstacles[i].minCorner = vec2(-0.5f, -0.5f); obstacles[i].maxCorner = vec2(-0.3f, 0.5f); i++;
			obstacles[i].minCorner = vec2(0.5f, -0.5f); obstacles[i].maxCorner = vec2(0.7f, 0.1f); i++;
			obstacles[i].minCorner = vec2(-0.75f, -0.55f); obstacles[i].maxCorner = vec2(0.55f, -0.35f); i++;
			obstacles[i].minCorner = vec2(-0.75f, 0.35f); obstacles[i].maxCorner = vec2(0.55f, 0.55f); i++;		
			for (auto &r : obstacles) r.Vis(1,1,1);

			// create random agents
			static vec2 dbgOffset (0.f); ImGui::SliderFloat2("dbgOffset", (float*)&dbgOffset, -0.5f, 0.5f);
			//static float dbg (0.f); ImGui::DragFloat("dbg", &dbg, -0.01f);
			static int agentCount = 200; ImGui::DragInt("agentCount", &agentCount);
			std::srand(0);
			csl::vector<Agent> agents;
			for (int i=0; i<agentCount; i++)
			{
				Agent a;
				vec2 pos(0.f);
				for (int d=0; d<2; d++)
				{
					float c = float(std::rand()) / float(RAND_MAX);
					c = (c-0.5f) * 2.f; 
					pos[d] = c;
				}
				a.pos = pos + dbgOffset;
				a.radius = 0.05f + 0.05f * float(std::rand()) / float(RAND_MAX);
				a.mass = a.radius * a.radius;
				agents.push_back(a);
			}
			for (auto &a : agents) a.Vis(0.5f,0.5f,0.5f);



			static int iterationCount = 500; ImGui::DragInt("iterationCount", &iterationCount);
			for (int n=0; n<iterationCount; n++)
			{
				float colorize = pow(float(n+1) / float(iterationCount), 0.25f);

				// collide each agent against each other
				for (int i=0; i<agentCount-1; i++)
				for (int j=i+1; j<agentCount; j++)
				{
					Collision::ResolvePairOfAgents(agents[i], agents[j], colorize);
				}

				// collide each agent against each obstacle 
				for (int j=0; j<obstacles.size(); j++)
				for (int i=0; i<agentCount; i++)
				{
					ResolveAgentObstaclePair (obstacles[j], agents[i]);
				}
			}
			for (auto &a : agents) a.Vis(0,1,0);
		}

@frob

frob said:
if the NPC intends to go through the door the steering system is briefly replaced with a different control “walk cleanly through the doorway”, then restored too regular steering; if the NPC didn't intend to go through the door the steering system directs the NPC away from the doorway trigger area.

I like what's being said here. The system I have partially does that. Each of the doors in my games have two Marker3D nodes that represent the back of the door and the front of the door(the mesh I placed in them is only visible while developing in the engine).

If the enemy is in the chase state and their “DoorCast” Raycast hits out of these doors close to the enemy; then the door will run a open_door_for_enemy() function. Here's the enemies open_opener() function:

func door_opener(door_range = 2):
	if parent.door_cast.is_colliding() and parent.door_cast.get_collider().owner is Door:
		var d = parent.door_cast.get_collider().owner
		var dist_2_door = parent.global_transform.origin.distance_to(d.global_transform.origin)
		if dist_2_door <= door_range:
			d.open_door_for_enemy(parent)

and here's the door' open_door_for_enemy() function:

func open_door_for_enemy(target):
	enemy_opening_door = true
	var enemy_position = facing_(target)
	update_enemy(
		target,
		front.global_transform.origin if enemy_position else back.global_transform.origin,
		back.global_transform.origin if enemy_position else front.global_transform.origin
	)
	if anim_player.is_playing():
		await anim_player.animation_finished
	if not door_open:
		door_animation(enemy_position)
	enemy_opening_door = false

The open_door_for_enemy() function checks if the enemy is opening the door from the back or front, then places the appropriate door opening animation for the enemy; it also gives the enemy the locations of one of the front/back Marker3D then let the enemies move towards them to get through the door where the Marker3D would be.

There's also more to it. Like if the enemy's “DoorCast” touches the door when it's open, the enemy would make it's good to move to the Marker that points the enemy in front of the door way and then move to the other Marker that makes in goes through the door. With all of this, my enemy AI still manage to get stuck behind doors.

That said, did what I try to do approximate what you were talking about?

frob said:
That code is far too hard-coded for what is typically done. It also involves more combined pieces all at once.

That seems true. I definitely think it's true. I'm still a beginner, so I don't know where to start when it comes to a solution to that.

I enjoyed reading your response. I'm just wondering, are their any books that address this problem in-depth for non-professionals? Is their any GDC video/article that address the issues of enemies going through doors?

@JoeJ

The quantized solution seems more preferable, as the game I'm working on is supposed to be able to run on low end pcs while also looking good. Still, the problem I have with too many enemies getting such in doorways currently involve only 3 or 4 enemies.

Audio_Bellum said:
The quantized solution seems more preferable, as the game I'm working on is supposed to be able to run on low end pcs while also looking good. Still, the problem I have with too many enemies getting such in doorways currently involve only 3 or 4 enemies.

Performance should not be an argument at all. Low end PCs can do entire ragdoll simulations for multiple characters, so a much simpler system for avoiding obstacles, representing characters as capsules or even just points, is no performance problem. (Assuming efficient spatial queries to get all characters / walls within a given bounding box.)

In FPS games movement is not quantized in general. Characters can move freely, e.g. doing some strafing to dodge player shots. They are not stuck at fixed paths or grid cells. Thus you always need smooth movement.
The motivation for my circles example is to have a more general obstacle avoidance system to make doors less of a difficult special case.

But there can be other reasons forcing you to make doors a special case. E.g. if you have animations requiring to move through doors in some predefined path, my proposal would not help much. Imo, animation is the primary reason why such things can end up more difficult than they should be.

Audio_Bellum said:
Multiple enemies try to enter a single doorway at once and they get stuck behind each other

When enemies decide to cross a door (or other choke point) they should be able to compute a precedence ordering among themselves and the other enemies that want to go through the same door (nearest to the door first, expendable soldiers before their commanding officers, a predefined convoy formation, etc.) and adjust pathfinding accordingly (go to the door if they are first, or stop and stay out of the way of the others who should go through the door before them, or follow the leader to ensure they go through the door later).

Audio_Bellum said:
Enemies that get stuck behind open doors when trying to chase down the player

The included code switches behaviours according to distances, which doesn't seem particularly correct in principle.
A potentially more robust basic building block: “bouncing” off a convex obstacle in a direction that ensures progress towards high level pathfinding, which could be applied to the walls around the door opening.

Audio_Bellum said:

I'm just wondering, are their any books that address this problem in-depth for non-professionals? Is their any GDC video/article that address the issues of enemies going through doors?

I can't think of books describing it that are current, mostly because I am not current for beginner books.

There are tons of articles and descriptions of doorway behavior, especially if you are also looking for steering behavior and pathfinding.

Again, they usually have a different logic, it is not usually just a narrow area on the navigation mesh. NPC design can usually have specific, modular logic in terms of building blocks of steps or behaviors with sub-behaviors with associated animations and logic to chain them together. Doors have an area around them used for the effects, for logic and animation and triggers. Avoid the immediate area if not using it, steering either stays outside or passes through, no loitering. Navigation tends to have nodes on either side, when traveling through navigate to the node on one side, then the node on the other side, then continue to nav points. Steering logic can bottleneck if NPCs are going both ways so each door can reserve the nav points with a queue point, in games where NPCs have a queueing sub-behavior. Logic to shoot through them for cover, logic for sniping, logic for hiding, logic for stealth, whatever design needs.

There is a lot of research work on obstacle avoidance for pathfinding that might be useful/interesting. RVO helps agents avoid each other by detetcting collisions using the velocity projected-forward in time.

There is a lot of other crowd simulation research here.

Generally, the framework is to treat agent navigation on a few different levels, where each level depends on the results of the previous:

• High-level goal planning - given target location, plan an optimal path through a navigation graph (navmesh, probabilistic roadmap, etc.). This gives a sequence of nodes which the agent should visit along the path to the goal. Usually this is planned with A* (A-star) or some other path planning algorithm (e.g. fringe search). This phase avoids major collisions with static objects.
• Obstacle avoidance - along the straight-line path to the next node, the agent needs to avoid smaller obstacles and other agents that are not part of the navigation graph. This is where RVO is useful.
• Low-level control - given the current desired movement vector which avoids obstacles and moves toward the next node, apply forces/torques to the agent to make it move in that direction at the desired speed.