designing a game engine from old logs from your profiler

Haven't tried this specific version of it where you use logs to reason about what is going on, but yes, that's exactly what you want. You want to have as much context as possible at all times, so that you can make as many assumptions as you possibly can everywhere. The more you can assume, and the more you know, the faster the code that you can produce will run. You want to write algorithms that do exactly what the actual task is that the application wants to solve. You also want to lay out your data in ways that lets the actual algorithms you use run as fast as possible. If possible, avoid context regathering in hot code via if nestings, and instead remember the context as some enum or something, and implement different versions for different known contexts. A specific bare-minimum implementation is almost always a lot faster than a generic no-assumptions implementation, because it does not have to regather any context, rarely perform any checks, and you can use shortcuts based on assumptions to get more work done with fewer operations.

For me, it helps to think not in terms of programming language features like classes, objects, functions, but in terms of assembly instructions: memory loads, ALU operations, registers, and out-of-order execution, SIMD capabilities, branch misses, cachelines and prefetches, etc. I try to model each task in terms of those fundamentals, and then write code that matches that.

I always do a ballpark estimation: what is the absolute minimum amount of work that a computer would have to do to solve a certain task under ideal conditions, and what is the maximum theoretical performance a computer could achieve for that minimum amount of work? And then I try to write an algorithm that does exactly that, and write the rest of the program to fit to the specific design and requirements of that algorithm.

That way, you open the black box, and look at the whole big picture of the program, and reduce friction at every step. I do this extensively in my compiler at all levels. I usually think about usage context, usage frequency, batching, etc. Recently, I started to think a lot about single-core parallelism via out of order execution, which lets me interleave multiple executions of the algorithm in parallel, even if the algorithm is not necessarily suitable for actual SIMD. For example, I built a global hashmap for my compiler that lets me look up as many values at once as my CPU can handle simultaneous in-flight cache misses. It uses manual prefetches, its buckets are cacheline-aligned, etc. It's a global hashmap for object deduplication (such as data types, constants, AST subtrees, etc.), because then I can batch even lookups of multiple things that would otherwise be in entirely separate hash maps, such as deduplicating a type and an arithmetic expression, while also looking up a function call overload, all in one interleaved operation that amortises latencies.

For example, when I do type checking, I do type checking on N functions at once (still multiple ones on a single core), and whenever I encounter something to look up in the hash map, I queue it up for the next batch of lookups. Whenever the batch is full, I do N lookups at once, which lets me amortise the cache miss latency for the lookups and the cost of chasing pointers. I designed the entire program structure around the constraint that I need to be able to execute multiple instances of all parts of the program branchlessly, so that I can interleave multiple instances of each task on one core.

Normal programs would do a hash map lookup, and then stall for a long time because it's usually a cache miss. It may try to speculate ahead, but CPUs can't really speculate that well through CALL boundaries, and through all the conditional branches that are involved in a hashtable lookup.

All these things are entirely impossible if you do not break open the black box. As soon as you do whitebox programming, you can get insane speedups and eliminate all the useless code that doesn't really progress the actual task at hand. And you can write code that takes shortcuts based on prior knowledge, you can start to reason about memory layout and cache latencies, etc.

P.S.: Only after breaking the black box can you even really identify just what it actually is that your program tries to do, in detail. Black box programming is mostly just fumbling around without any concrete frame of reference. Yeah, an engine needs to do rendering, so you just build something that renders stuff. But there are millions of ways to render stuff, and is the one you did with basically no contextual knowledge any good? You don't know, because you can only evaluate it with a whitebox view.

This article may be interesting to you: Semantic Compression by Casey Muratori. I also recommend looking into Data oriented Design.

P.P.S.: You need to be aware of the statistical distribution of cases in your data, as well as sequences and other patterns that you can optimise for. And then allow the user of the engine to directly provide as much context as you can take when he calls the engine, so that he can leverage all these optimised paths. Know the workload, know the task, know the data layout, know the machine, know the overall program flow. For example, you may have a task that doesn't need to be done immediately, but only sometime soon, or once per frame, or similar. Then you should expose a batched / SIMD-ised version of that task, and maybe restructure surrounding code to be aware of that batching, or create a queue for workloads, etc. Ballpark-estimate the latency of tasks in CPU cycles, and also take note of how many of these a single CPU core could run at once at the same speed, just by leveraging out of order execution and the fact that a single core can schedule 2-4 of most operations in parallel (even without using SIMD), such as multiple bitwise operations in parallel, multiple arithmetic operations in parallel, multiple comparisons in parallel, it can do other work while memory is being fetched, etc. You usually have lots of latency stalls on the critical path of each function, and usually you can execute another instance of that task interleaved with that one, for free, because most code does not fully pump all the execution ports of the machine. Also use perf stat -ddd ./benchmark and look at the instructions per cycle metric. Code that has less than 3 instructions per cycle is something that would benefit from this in-core parallelisation via interleaving multiple instances of a task into one function call.