Hi! As a tools programmer, this happens to fall into my line of work. Generally, you have 2 main options:

Use any existing engine's UI framework for the outside and render your ‘viewport scene’ to a texture that goes into the spot you want

The main pros of this are that you don't need to build anything ‘gamey’ yourself since the engine will provide all that for you. However, from my experience game UI frameworks are not really ideal for desktop app development. If you expect your simulation to need very complex controls, this approach may become difficult to manage.

Use a desktop app development framework for the UI and connect the renderer to a dedicated viewport control

In other words, write your own renderer (or license middleware) and connect it to a ‘host’ written with something like WPF. This is what we usually do at work, because it makes the application code much easier to manage. Depending on your needs (for instance if you need to be able to run the simulation externally), you can also use a language interop layer or IPC to keep your engine and frontend separate. However, there will be a lot more work involved on the ‘gamey’ parts since you'll have to build or integrate them yourself. It's easy for a company that's already building their own tech to justify, but for a small dev it may be a lot more work!

In your case, it's hard say with 100% certainty what's best since I don't know your exact needs--but I strongly suspect from the fact that you only know Unity that you should take option one.