Hmmm... two ideas

1. Your owned planets are a graph, let a new planet only be colonized where a path from your home planet to your target planet with only owned planets exists (explain it as safe trading routes whatever).

This although adds the strategically feature, that you can break supply/colonization chains.

2. You can colonize such a planet immediatly, but it needs to grow into a full member of your empire, lets say 10 turns. During this time only limited actions are possible and you can't expand your system from this planet until it has been activated.

This adds the strategically feature, that some aliens can activate a planet in 5 or even 1 turn, or that you can research better activation strategies etc.

1. I have it implemented already. It's a nice mechanic.

2. I have a problem here, it slows down the overall pace. Like, to expand 3 system from your homeworld will *always* take minimum 30 turns (3 planets x10 turns). Such mechanics are always too slow or too fast...

Also, what happens when you conquer enemy planet? You take it over without the need of waiting 10 turns; kind of inconsistent and everything...

Why wouldn't it work? Your insects are just a side that has a locked loyalty trait, so they will never switch loyalties, and would not be counted towards impacting your factories. (Other than possibly degrading output through 'rebel action'.) You add more factors to the process to address what is needed by character types.

Insects can't live alongside other races (they would eat them :D), so any loyalty trait from them is irrelevant (the planet probably first need to be purged before you can send any humanoid colonists).

Anyway, I first need a solution for "each race is insect" case, since that's the most pressing matter for me (and then I could add extras for humanoid races and the like).

You want to stop them from just conquering every planet?

Perhaps an taking into account the ability to adapt to the environment, each planet they conquer changes them to adapt to the planets environment.

if you conquer every type of planet you have no ability to specialize, benefits from one, would offset another, going from a ice planet to a lava one.

if you only stick to cold frozen planets you specialize and get more returns from those planets, stronger defense etc.

perhaps the ability to terraform if you do want them to be able to eventually control all the planets, or only build outposts and not permanent settlements

so they don't gain or lose benefits, but still have territory

You could go as far as tie the environments into skills and tech and make some tech only available if you reached a certain requirement

What happens then after you conquered such a planet? Do you need to colonize it? Get it for free? But if for free why it differs from uncolonized/uninhabited planets (which you have to colonize)?

planets are either occupied by a faction, or unoccupied.

a successful attack on an occupied planet makes to become unoccupied.

unoccupied planets can be colonized.

you can attempt to colonize ANY unoccupied planet. but colonizing a planet deep in enemy territory will lead to things like interception of supply ships and possibly even raids on the colony under construction.

Like, what happens if you conquered an alien planet? In a classic 4X you take it over (so it's basicly a free colonization)

an example of poor simulation modeling. IE they simply don't model colonization of cleared planets - and instead opt for the simpler contrived mechanic of "automatic colonization" of conquered planets.

based on the sheer number of posts you've made about the design of your project, and the depth and care you're putting into the design, i would certainly hope that you would not settle for the easy way out by simply not modeling colonization at all.