I'm working on a system to support the in-game UI for my game engine. Below is a list of the scope and key features I intend to provide:

• Rendered on a 2D canvas on top of all gameplay graphics
  • In other words, I want to restrict this to absolute basics. I wouldn't (yet) go into the weeds with “diegetic UI” that needs to be projected into 3D space (e.g what Doom 3 or Dead Space has)
• Uses basic primitives (triangles, quads, etc.) and textures
  • I already have a rendering system to which I can submit draw commands for primitives, so I just need the higher-level logic to turn UI elements into batches of primitives
• Supports basic user interface widgets for things like menus: frames, buttons, labels, etc.
  • This includes both rendering and event handling, potentially with state changes based on events (e.g button having a “hovered" and “clicked” state)
  • Parent-child logic to allow for composite UI elements (e.g window with buttons) that can be modified in bulk (show/hide, etc.)
  • No need for a fully fleshed out window management system! I aim to restrict the UI to “one active window/frame at any one time” (menu pages, etc.). Said windows may have internal child elements (buttons, labels, lists, etc.) but I wouldn't need anything more complex, like having to manage Z ordering and focus for multiple windows.
  • The one exception to the above: modal dialog (i.e popup/notification) logic which is drawn on top and consumes all inputs until the user acknowledges it
• In-game UI elements: HUD, inventory, etc.
  • Besides barebones functionality, I'm hoping to add some features that fit the aesthetic and add to the game experience (animations, sound FX, etc.)
• Convenient client-side API
  • The actual projects using this system have a convenient API to create and manage UI elements. They have interfaces to get/set widget data (e.g getting the text from an input field), they can subscribe and respond to UI events and modify the UI state accordingly

What are some good case studies for an architecture to use to accomplish this, ideally using modern programming paradigms?

Some of the main dilemmas I currently have:

• What does “UI implemented through composition” even look like?
  • If I go down an ECS-like route of widgets being “bundles of components”, the systems that handle events, input, rendering, etc. become much less straightforward.
• How to handle relationships (parent-child, one-to-many, etc.)?
  • This is trivial (if inefficient) in OOP, but less so when it's implemented via IDs and component pools
• What do the systems look like?
  • To pick rendering as an example: I no longer have a virtual render function to call, and my logic may in fact depend on what components are present. I also need to deal with Z order and other considerations.
• What does the user-facing API look like?
  • It's convenient for users if things are encapsulated in a base class pointer, and this convenience should not be ignored. If they have to start looking up individual components by ID, it could make the API much more of a slog to deal with.
• Generally speaking, assuming OOP is not the way to go, how should one break out of that mindset w.r.t UI?
  • UI is the rare case where it seems to work decently enough, and many existing APIs (Qt, wxWidgets) still use this approach, though they are used for building entire GUI applications, not UI within an existing one

I listed below all the approaches I've already considered:

OOP

The most straightforward approach. Have something like a Widget base class, all widget types inherit from this, allowing for parent-child relationships and an interface for all the necessary logic. Client code can subclass the relevant widgets if needed, and they can instantiate and combine widgets to create the UI.

I used this for my initial attempt, where I created a very simplified copy of the widgets in Qt. This does get the job done, but it feels clunky, and I figured I could do a lot better. A big advantage of OOP is how it can hide the “messy bits” from the client, meaning they just get a friendly interface to interact with (show/hide, subscribe to events, etc.), lifetime for them starts and ends with the Widget object, and the backend is left to deal with all the hassle of things like resources and rendering logic.

The downside, however, is that it gets very complicated to achieve modularity through interfaces and inheritance. To give one example, it would be nice to be able to separate the logical aspects of buttons (a thing the user can activate to trigger some functionality) from the visuals (since it can be anything from a colored rectangle to an animated sprite). This gets very messy with OOP, and composition would obviously be better, but I haven't found any good case studies.

Dear ImGui

I already use this for developer/debug UI, and the draw lists would solve the rendering part. The logic for generating the UI is also relatively simple and straightforward.

However, I'm not sure immediate mode is good for in-game UI overall. To get things like fancy layouts and whatnot, it seems like it would add a lot of unwanted complexity, both to the API and the backend. It seems more suited for when the priority is functionality, and is less ideal for when the UI is “part of the game experience”.

Web-UI

Would allow using HTML, the contents of webpages cover pretty much all the requirements I have, and there is plenty of existing material to learn from.

However, it would also introduce a huge load on memory and processing, practically “running an engine within the engine". Feels like overkill.

I'd suggest using OOP for this case. I use it in my GUI system and it has never caused any issues. In some ways, OOP was created as a paradigm to structure GUI systems. It's actually well-suited to the task.

yah-nosh said:
it would be nice to be able to separate the logical aspects of buttons (a thing the user can activate to trigger some functionality) from the visuals (since it can be anything from a colored rectangle to an animated sprite). This gets very messy with OOP

For this case, one option is to use overriding of virtual functions (e.g. you can implement AnimatedSpriteButton by replacing its render() function). It's not that messy in practice, at least in my engine/editor.

Another option is to use delegate functions. This is important in general for clean UI design. Each widget type can have its own type of delegate (a struct containing a std::function or equivalent for each type of event that can be produced). When creating the widget, the delegate gets hooked up to the code that handles the logic. For a button, there might be an “onPress” callback, an “onHover”, and “render” callback. The button calls those functions (if they are not null) inside its base implementation. The delegates allow extending functionality of widgets with less boilerplate code than overriding functions. I'd strongly suggest using this pattern. It's worked very well for my editor.

@NightCreature83 Thank you for your comment, this looks quite promising. Do you know of a codebase which has this kind of system, so I could use it as reference?

I'm not convinced the approach suggested by NightCreature83 is the right one for complex GUI systems. Maybe it's OK for a simple game UI. The benefit of separating out pieces of widgets into behavior components doesn't seem to outweigh the additional complexity this adds. It seems like you would get an exponential explosion of possible combinations of components that would make it hard to sufficiently test all code paths. What happens when you create a widget from components that don't work well together? On the other hand, a monolithic widget can result in clean readable code that is easy to reason about. If you use delegates like I talked about in my first reply, this can also be very extensible with minimal boilerplate.

I also don't like the idea of constantly polling the UI for data. This will break down in performance with large UI with hundreds of widgets. A core advantage of retained-mode GUI is that everything can be event-driven and therefore much faster.

One thing I agree with NightCreature83 on is that there should be no global event handling or looking up widgets by ID. In my GUI, I provide mouse/keyboard/dragdrop events to the top-level widget on the screen. These events then get propagated down through the GUI hierarchy until they are handled by a widget somewhere. Each event handler returns whether or not it handled the event to its parent, so that the parent can decide if the event should be propagated further. Each widget is responsible for passing the events to its children and doing any necessary transformation so that the events are in the child's coordinate system. There are only a handful of non-leaf widgets: GridView, ListView, TreeView, SplitView, ScrollView, CanvasView. Most of the GUI is structured using GridView, which is similar to HTML tables. Leaf widgets can be pretty simple: to click a button all it needs to do is check if mouse position is inside their boundary and then call the delegate function to handle the event.

As for widgets rendering themselves, it works in a similar way to events. I pass a GUIRenderer object into the top-level widget's virtual render() function. The widget then uses high-level functions of GUIRenderer to specify how it is drawn, and calling render() on any children. This might be something like renderer.drawBorderedRectangle(…) or renderer.drawText(...). The renderer then translates these high-level draw operations into low-level primitives and places them into a vertex buffer which is shared between all widgets and rebuilt every frame. The draw calls are placed into a queue which is submitted to the graphics API when the top-level widget returns from its render() function. The base widgets themselves are flexible enough (e.g. with custom style shaders, images, border/background colors and gradients) that I rarely find a need to override the rendering functionality.

Here is an example of an audio recording software I built using this GUI system, implemented entirely in C++. This image has around 1000 widgets visible, to give some idea of the complexity it can handle.

@Aressera If I undestand @nightcreature83 correctly, the principle is “separation of concerns”. All widgets are just “bundles” of various components, and the components themselves are added to their own well-defined systems.

To give a crude example, I add a Clickable component to a widget, which will then add it to a system that only cares about this component, plus the ones it needs for click handling logic (parent/child relationships and the bounding box). The system can then listen for click events, test all the clickable widgets, and run a user-provided callback stored in the component. There's not really that many different ways something can be “clickable”, so the responsibilities and implementation of the system should be fairly straightforward. Same can be done for rendering, where it's either specific primitives (mostly boxes, optionally with textures), or we outsource to the user.

I can see lots of benefits for the backend in this approach, as I know that a modest number of well-defined components should give me the feature set that I need, and it's easier to split the code for different parts of the logic. It also becomes easier to create custom widgets, since features can be added through components, or the widget itself is actually a composite of dedicated child widgets. I think Unity's UI API behaves something like this as well?

While this is promising, there are a couple of hurdles, mainly to do with the user-facing API:

1. Composition is nice, but it will turn iteration into a slog if clients keep having to bundle up all the components they need for every single widget. I could use wrapper classes which do this automatically, but then I risk being back to OOP hell again.
2. Following from above, what would it look like if client code needs to access data stored in specific components? In UI programming, it's hard to deny the convenience of just having a straightforward interface to interact with, rather than having to query components. Then again, I may be stuck in the “Qt mindset”, and there is a way to make this more user-friendly.

I think the main problem is that a lot of existing, tried-and-tested APIs (e.g Qt) are also very opaque. While the API just presents a very straightforward interface, making it seem like a “Button” is a single object that has everything we need, in reality it may be several different objects bundled together, and the API is just hiding this from the client. This makes sense for convenience, but it distorts the perspective on how a UI framework should be designed in the first place, as the frontend is confused with the backend (i.e developers believe the object hierarchy presented to the user is also what the backend uses for event handling, rendering, etc.)

I think the part I can agree on is the separation of concern. We can always have a view that is ready to be mapped with data model behind the scene that is processed by a business logic. This view can really be in any form, even as a separate file, not a code. The easiest example is HTML/CSS. HTML/CSS concerns are really just how things are look and represented. Whether you want it modified; say only part of it to be inserted in certain part of the HTML, or just want to update certain text in that HTML, or even highlight a color by appending new color by modifying/adding a CSS style, that's JavaScript's job (and the browser engine's renderer, internally).

What I'm saying is you can actually combine both opinions just fine. How you want to represent the view as a mere aesthetic data and how your system modify it can be two completely different things. I think UDK 3 (Unreal Engine 3) used Flash at some point, it was called Scaleform, before moving to UMG on UE4. I assume that is more on what nightcreature83 talk about. How the result is mapped into the engine and then rendered is something else entirely, which I assume is what Aressera concerns more.

@xrbtrx Data-driving is definitely a goal, as it would make authoring a lot easier and remove any hardcoded UI creation logic. Scaleform I'm aware of, but Flash is basically extinct, and the method itself feels like the web engine approach (which I already mentioned I would prefer to avoid).

There's another point that I forgot to add to my previous comment: while for certain features, it's enough to have a single component contain all the necessary data, what if something needs to interact with multiple components? For example, click handling might necessitate accessing the Clickable component and the BoundingBox component, and if we go for an ECS-like approach, that means a lot of extra queries from different containers. It gets even more messy with widget hierarchies, since it's rare that things can be accessed in neat sequential loops. More often than not, it will end up requiring a lot of random access.

This is not to say that this approach is bad, just wondering if there exist ways to mitigate this issue (or if the issue is worth worrying about at all). And as mentioned, this also makes the systems less intuitive. OOP is appealing because all the required logic is encapsulated, but this can easily sacrifice modularity.

Put very crudely, my main issue is that I have a hard time fully visualizing the necessary systems, i.e how the API looks on the client side, and how the backend should be structured. Qt appears very friendly on the client side, although the backend is a bit of a nightmare (I spent quite a bit of time diving into the source code). HTML/CSS-like approaches do hint at how things should be data-driven, but there's almost no material on what the backend of that looks like.