Benchmarking My Game Engine

It's something I wish I started doing earlier... because profilers drive me crazy.

We’ve all been there, right? You’re writing an app, it feels sluggish, something feels wrong. It’s not a bug, the app works fine, it’s just…there’s no reason typing in a text field should be this laggy on this hardware. You know what I’m talking about, right?

My game UIs feel that way a lot. So does my engine’s editor. I’ve always wanted to figure out why, and fucking fix it. I never could.

My Specs

I’m not just posting my PC specs to brag. Hardware is just as important to understanding what’s going on with slow code as it is understanding what the code is doing. A slow memory leak is going to take longer on a supercomputer with over 1TB of RAM than it would take to cause problems on your phone. CPU and GPU usage is also going to be less of an issue on a game developer’s workstation than it might be on your garden variety cash register.

With that in mind, this PC ain’t slow.

| Hardware | What I have | |:--------:|:----------------------------------------} | CPU | AMD Ryzen 7 3700X | | GPU | AMD Radeon RX 7600XT | | Memory | 64 GB available | | Disks | 2x 1TB NVMe SSDs, 2x 4TB spinning disks |

I’m not on the bleeding edge because I don’t need to be. But this isn’t a slow computer either. It handles any 1080p game I throw at it, and I’m too blind for 4K, HDR or raytracing to matter. I also don’t need to take advantage of AI upscaling because my eyes do a better job of screwing up text readability. Point being, if something is slow, I’m either compiling Plasma or something in the app is poorly optimized.

Having an extremely fast computer is a double-edged sword however. If something feels slow on my PC, it’s slow. But I may also do something that feels perfectly fast on my PC but kills your laptop in minutes.

And I can’t debug what I can’t see.

Profiling with dotTrace

This has been generally unsuccessful for me. I would typically build my engine’s editor for release, get it running on the profiler, and try to do things in the UI that feel slow. Things would either stop feeling slow under the profiler or… I just couldn’t figure out what the problem really was.

That’s not a dock against JetBrains’ tools. I just struggle to navigate them. They don’t display things in a way I know how to read.

Sometimes, I would get lucky and find out that something in my code genuinely is problematic. I’d rewrite it, and be worse off - or not feel a difference.

What I Care About

My engine, Inertia, is written in C#. I will die on that hill. I love C# as a language. I find it fun to write, easy to read, and I do not feel limited by it. I can make it do scary things.

The problem with C# and performance is how the language used to be used. C# started with a lot of inspiration from Java and object-oriented programming. This means that the runtime has a garbage collector because that’s what we did back then. Most objects are stored on the heap because that’s how we did it back then. Newer versions of the language, and of the runtime, allow you to do things differently.

GC Hits

The number one concern I have when tracking down slow code in my engine is GC hits. If you’ve played a Unity game and ever felt it hitch every now and then, every few seconds/minutes, GC hits are very likely what you are feeling.

The runtime knows you have a finite amount of RAM and the garbage collector makes sure things get cleaned up. Every now and then, the GC will decide to clean up any unused memory in the application, freeing space for new heap allocations. Your code cannot execute while the GC is collecting. No matter how fast your code is in other languages, if you trigger GC, it’s moot.

You can’t stop GC hits from happening outright. The only thing you can do is allocate less memory on the heap, and keep references active until you’re genuinely done with them. Keep the memory you need but don’t allocate more than you do.

Frame Time

I find most people who talk about C# are thinking about its use on the web with ASP.NET. If that’s all you’ve ever done, let me introduce you to something really annoying. Frame time is how much time the UI thread in an application spends working on a single frame in one window. We generally want it to be as little time as possible. In some cases, high (or unstable) frame time can make someone feel ill. As someone who cares about accessibility, I do not want that happening.

Knowing frame time is helpful for understanding that there’s a problem. But it’s not very helpful to measure it if your brain is already telling you it’s too high. What’s more important is understanding what is going on in that frame and how long it takes to process each thing going on. If you’re rendering lots of text and the engine feels sluggish when there’s lots of text, you probably want to know how long it takes to render text.

I have a display with 120hz refresh rate. I would prefer my games be able to build frames at that refresh rate. Most of the time I run the display at 60hz, which means I’m willing to accept 60 frames per second. Anything below that in my own game and we’re in trouble.

My engine, especially in editor mode, supports multiple windows. For best results, each window should be able to fully update and repaint in less than 16.7ms. That’s one second divided over 60 frames. This is how much frame time the engine has to work with.

Runtime Performance

This is why I was having trouble with dotTrace. It doesn’t matter how fast my code is, if I’m using the runtime APIs incorrectly or they’re just slow.

Maybe a dictionary isn’t the best way to track inventory? I should probably measure.

It Started With This Video

I’ve been listening to Nick Chapsas videos while trying to sleep.

In this video, he uses BenchmarkDotNet to compare different ways to iterate a collection in .NET. This is where I decided to start in my engine, because I have an entity component system. You will typically be iterating through an ECS every frame, making collection iteration operations a hot path. Prime real estate for performance gains.

Ways to Iterate Lists in C#

Benchmarking is about comparison. If I want to know how fast my code is, I need a frame of reference to measure it against. And there are quite a few ways to iterate collections in C#.

• With foreach loops
• With for and while loops
• With LINQ
• In parallel
• With a Span<T>
• With a ReadOnlySpan<T>
• With a raw pointer
• With fancy unsafe runtime methods

To make matters worse, there are different kinds of collections in C#. Arrays, lists, hash sets, dictionaries, custom collection types… even a Span can be thought of as a collection if it represents a stack-allocated buffer.

So I Wrote A Tool

I wrote a tool for my engine that benchmarks it using BenchmarkDotNet and a series of common tasks my code might perform. I started with collection iterations.

I decided to test with both a standard int[] array and a List<int>. This allows me to compare things like foreach on a list vs. foreach on an array. Each collection is filled with random data, and each test sums up the value of each element in the collection. I also tested with collections with varying sizes, starting with powers of two and then a torture test of 100 thousand items. Gradually, the test would grow the collections larger in size to see how bigger collections affect the measurements.

After my tool runs the benchmark, it prints it out to a fancy-ish Markdown table.

And here’s a report.

If you just scrolled past that and went “jesus fuck that’s a large table,” I agree. Jesus fuck, that’s a large table. But it’s giving me a lot of insight.

Allocations

The Allocations column tells you how much memory has been allocated on the heap during the test (not counting any allocations while preparing the test). None of the measured list iteration methods allocate heap memory, except for one: ListForEachMethod. That would be code that looks like this.

var list = new List<int>();
list.Add(1);
list.Add(2);
list.Add(3);
list.Add(4);

var sum = 0;
list.ForEach(value => sum += value);

I never knew this function existed on List<T> until I measured it. Allegedly, people think it looks elegant to use. But you are paying an 88-byte heap tax when you use it. So…I wouldn’t use this method.

Execution time

The most important columns in my opinion to read are Mean compared with ElementCount. You statistician folks can read the other numbers, I don’t really understand them yet.

Each test in the report was executed several (likely thousands of) times by BenchmarkDotNet. The Mean column is telling me, on average, how long each test run took. It is in fact directly telling me how much frame time I would need to iterate a collection of that size in that way.

My Code’s The Fastest

I can tell you for certain that my code, out of all tests, is the fastest. Here’s how I know.

Enumeration and Iteration (Indexing)

In the chart, Enumeration is any time I use a foreach loop. There is an enumeration test for arrays, lists, and spans.

Iteration is any time I’m using a for loop and a loop counter. There is a test for lists, arrays, and spans.

The Unsafe Span Iteration tests are my code.

If we look at a 4-element array, iteration with a for loop is almost twice as fast on average as enumerating with foreach. If you compare enumeration vs. iteration of List, Array Span, and List Span, you will notice iteration is generally faster on average for a 4-element collection.

We generally don’t iterate over collections that small, so what matters more is if this pattern scales with larger element counts. Even in the torture test, iteration generally wins over enumeration. In most tests, the difference is negligible. Enumeration does win over iteration of an array, but iteration wins by tens of microseconds over enumeration on a list.

Unsafe Iteration

Across all tests, my Unsafe Span Iteration method is generally fastest. It is also faster on array spans than on list spans.

But it looks like this.

[Benchmark]
public int UnsafeSpanIterationOverArray()
{
    var span = array.AsSpan();
    var sum = 0;

    ref var reference = ref span[0];
    for (var i = 0; i < ElementCount; i++)
    {
        sum += reference;
        Unsafe.Add(ref reference, 1);
    }

    return sum;
}

[Benchmark]
public int UnsafeSpanIterationOverList()
{
    var span = CollectionsMarshal.AsSpan(list);
    var sum = 0;

    ref var reference = ref span[0];
    for (var i = 0; i < ElementCount; i++)
    {
        sum += reference;
        Unsafe.Add(ref reference, 1);
    }

    return sum;
}

I’m accessing the span of the collection, taking a reference to its first element, and using Unsafe.Add to add one element worth of offset to the reference every time I iterate. This is effectively like doing this in C.

int length = 100000;
int* array = malloc(sizeof(int) * length);
int* reference = array;
int sum = 0;

for (int i = 0; i < length; i++) {
}
    sum += *reference;
    reference++;
}

…And exactly as dangerous, too. If the array memory is ever moved or freed during iteration, this is going to be problematic. But it is the fastest, and if I can guarantee I won’t lose the memory mid-iteration, I’m inclined to use this in my engine’s ECS to squeexe as much performance out of it as possible.

Should I Be Writing C#?

If I care enough about performance to benchmark different ways of iterating over memory at a low level in the language, why am I not using C, Zig, Rust, or something more low-level for my engine?

The answer is because I love C#. It gives me the freedom to choose how I write my code. It doesn’t fail to compile because I mis-typed a space, nor does it yell at me about borrowing. And besides…

The language ain’t gonna stop you from doing 100,000 matrix multiplications on the CPU every frame and completely neglecting that poor GPU over there just dying to do this instead. And you know what? It’s still decently fast for a one-off.

Inertia Benchmarking Tool: Coming Soon.

What this is missing is fancy graphs and a GUI. I plan on integrating BenchmarkDotNet into the Inertia editor, so you can benchmark your own game code within the editor and get a detailed report of how things are doing.