Rookery C++ Usage

C++ Usage in The Rookery

This post exists to document the specific “coding guidelines” in the Rookery, as well the general principles behind why these guidelines exist. Since code snippets on this blog may often be lifted or paraphrased directly from actual engine code, I wanted to publish this post so there was a “public record” for some of the conventions I’ve adopted.

As with writing style, coding style is very subjective. However, despite coding style being subjective, I think it’s worth approaching it systematically nonetheless. Since I’m a solo developer, I have the luxury of both crafting the set of rules, honoring and enforcing them, and hopefully reaping the benefits also. BUT, much of what’s written here is born from experiences working at much larger studios on fairly sizable code bases, so I don’t want to undersell this post as being strictly useless for non-indie developers also.

That said, even if you are also a solo developer working on a game and game engine, it’s quite possible your opinions diverge from mine. After all, there’s no universal system for evaluating what “good code” looks like.

General Principles

Tooling sympathy

I think it’s common practice for experienced programmers to author code that prioritizes the reader first, and the writer second. I take this a little further, authoring code that also prioritizes:

1. Poor me debugging a nasty crash.
2. Poor me trying to decipher what a crash dump is telling me.
3. Poor me waiting an eternity for the build to finish.
4. Poor me trying to parse an error reported by the compiler or IDE.

This is, of course, in addition to the usual best practices of writing code that is (subjectively) straightforward to read and understand.

In other words, I try to write code that is sympathetic not just to the reader, but also the tooling used by the reader. If the code reads like delightful prose, but produces undecipherable callstacks in a minidump, is difficult to step through in a debugger, or adversely impacts the edit-build-run loop, I would consider this code to be in poor taste.

Carrying this train of thought a bit further, this means that to derive a “good” set of guidelines, we need to have at least a cursory understanding of not just what the tools are, but also how the tools work. I’m going to keep this post focused on the specifics of the various guidelines I’ve settled on, without going much into how a debugger, compiler, Intellisense, or other tools work, but it’s worth bearing this in mind.

Forbidden spells

Aside from the more nuanced guidelines that will follow next, the Rookery forbids:

• C++ exceptions
• C++ RTTI
• C++ new/delete
  • I have a custom allocator with a different API to new and delete that may be discussed in a later post.
• Any 3rd party dependencies that:
  • Create threads internally
  • Allocates without any mechanism for overridding the allocator
• Any GenAI/LLM usage
  • Not for test-writing, prototyping, “boilerplate”, demo code, and certainly not for actual code writing either. I use GenAI/LLMs for absolutely nothing at this time. There may be some things they are good at, but for what I do, this shit sucks (IMO).

C++ Guidelines

No lambdas

Let’s kick off strong with a spicy one. I never use C++11 lambdas. I have used them for years, and also worked with sizable codebases that have used them.

To be quite frank, I hate them. (Did I say that coding style was subjective already?) To me, lambdas are hostile to the debugger, the compiler, and even the reader.

For one thing, lambas make it easy to write code that makes it hard to reason about control flow. Having a lambda nested inside a for-loop that itself contains its own for-loops and branches isn’t just a hypothetical scenario, but an inevitable one. Code structure necessarily changes as the requirements do, and it’s natural that over time, lambdas will intermix with other control structures that confound the reader – a violation of one of the core principles.

But someone said I shouldn’t use raw loops. This school of thought on how C++ “ought to be written” definitely exists, and I can understand the argument to a degree. Everything should just be composable algorithms! The bone I have to pick with this approach is that it feels entirely impractical for “real life” code, whether for performance or due to other constraints.

One of the core problems is that if you are using a lambda, you are using a compiler-generated opaque type, and interfacing with that lambda (aside from direct invocation) requires a template of some sort. This also means that there is at least some impact to compiler throughput when lambdas are involved, given that type instantiation is one of the most expensive responsibilities of the compiler frontend.

Callstacks that contain lambdas are always uglified with extra stackframes that don’t really convey meaningful information. As such, lambda usage also interferes with debugger usability. The step-over, step-in, and step-out operations function intuitively in typical scenarios, but when lambdas are present, the user needs to do a lot more mental gymnastics to understand which operation is needed to steer the debugger correctly.

Going back to the point about template inevitability, I have empirically observed lambda usage (in a very widely used physics engine) contributing considerably to increased code size and compile times, neither of which are things I want. Incidentally, the algorithm and ranges headers are two headers I will likely never use because I’ve found them to vastly increase compiler memory usage (which restricts compiler parallelism), increase code size (debug code size matters also), and increase compile time by an order of magnitude over the equivalent straight-line code.

Lambdas certainly aren’t useless, but I find that the value proposition just isn’t high enough to justify the cost.

Minimize template code

I use templates, and find them to be critical (despite the factors that may make templates unergonomic at times). However, I view them as tools to enforce type constraints. I don’t view them as vehicles for actually implementing actual algorithms, for the most part.

The pattern I will often use is that algorithms and data structures are implemented in type-erased code (the way you would implement them in C, without templates), and there may be a C++ template wrapper that passes data off past the type-erasure boundary. This pattern is used for all the data structures in the engine for example.

I also tend to avoid template nesting wherever possible. Sometimes, it’s just needed obviously, but having less of it saves me time when parsing compiler error message that needs to indicate which template instantiated what. One way to avoid this is leveraging if constexpr as opposed to SFINAE. I also have adopted C++20 concepts as another way to enforce type constraints.

Summarizing, “minimal” template code to me means:

• Authoring code in a type-agnostic form where possible.
• Using templates as-needed to express and enforce type constraints (and not much more).
• Avoiding older techniques like SFINAE that instantiate types in order to function.

On variable naming

Welcome to the bikeshed! I’ve worked with lots of naming schemes over the years, and in the context of a bigger studio, I just adopt whatever convention the codebase has. My own code though, uses these conventions:

• Type names are PascalCase. Variable and function names are snake_case.
  • This may seem arbitrary, but I do this because variables and functions tend to be a fair bit longer than types on average. I find that snake_case is more readable when you have a lot of conjoined tokens, as in register_compute_pipeline as opposed to RegisterComputePipeline due to the extra separation the underscore provides, but maybe that’s just me.
  • Unlike the STL, I still use PascalCase for types because I find it useful to disambiguate types from everything else. Types participate in the “type calculus” of the language after all, and show up in different parts of C++ expressions than variables and functions.
• Member variables have a _ suffix, as in member_variable_. Local variables (including function parameters) don’t (as in local_variable).
  • To me, I want to be sensitive to state access and changes that occur beyond the scope of the current function. Having a trailing underscore suffix makes it really clear that something is being read or mutated outside the function itself. This applies not just to class member variables, but struct member variables also.
• Static variables have a _s suffix, and thread local variables have a _t suffix.
  • Understanding if a variable being accessed or mutuated exists in static or thread local storage is even more critical than understanding if a variable is a member variable or not.
  • I can at a glance be on guard for associated typical problems (issues with stack growth, reentrancy, thread safety, etc).
• Macros use SCREAM_CASE.

Signed and unsigned

If I am:

• Comparing quantities
• Performing arithmetic with some quantities
• Or somehow treating quantities as “numbers” in any other way

then I am storing those quantities in a signed arithmetic type. This means, that counts, sizes, and so on, to me, are always signed, to give a few examples.

Functions that accept quantities that must be positive often still specify a signed parameter. Some people find this uncomfortable, since that means the type system isn’t able to enforce the positive parameter constraint. To me, this is a constraint that should not be enforced by the type system in the first place. An unsigned underflow resulting in a passed argument close to UINT32_MAX is often going to break code all the same, often in less obvious ways.

It’s true that I need to contend with the possibility of a wildly aggressive compiler exploiting UB due to provable signed overflow/underflow, but I have never encountered this in a form that wasn’t extremely obvious about what the issue was (not to mention that I haven’t encountered this in at least a decade now). I view this issue as “a concern” but certainly not a “chief concern”, especially in contrast with other issues like unexpected sign extensions or unsigned overflow/underflow resulting in very real problems that are difficult to catch.

East-const gang

I write char const* instead of const char*. Jon Kalb made a good argument (Youtube link) for this, and I won’t repeat it here.

I think the most popular counterargument to this is that people want to see that something is const “first” because this bit of information is important.

I’m definitely not in that camp personally. That a variable is const or not I find to be quite secondary to what the type even is in the first place, so adopting the east-const convention not only simplifies the ruleset for const application (Foo const* const*), but it also emphasizes the type over the type-decoration.

Almost-never auto

The auto keyword is one of those things that often hurts code comprehension, so I don’t use it if I can help it. In fact, I rarely see it these days because range-based for loops eliminate the need for declared iterators, perhaps one of their more common usages.

One perceived benefit of auto is that you can change the underlying type, recompile the code, and everything will “just work” if the original type and the current type have a matching interface. To me, this is often a sort of anti-feature. When I’m reworking an algorithm or data structure, I often want calling code to break so I can see exhaustively how the interface is used and ensure I’m not violating any existing assumptions in the code. It makes the code less flexible, but also more discoverable.

Minimal STL usage

There are specific headers that I permit:

• <cstdint>
• <type_traits> and <concepts>
  • While I can author many type traits myself, the type traits often use compiler intrinsics that are faster to compile than hand-rolled implementations.
• <cstdio>, <cstdlib>, <cstring>
  • I use C runtime for various things, although not often. Still, including these headers doesn’t cost much, and they are still useful.
• <atomic>
  • I am using this for now because it would be a ton of work to reimplement this for every platform I care about. That said, there are a lot of things I don’t particularly like about the std::atomic interface, so this may or may not change in the future.
• <bit>
  • std::bit_cast and various bit sequence operations are very useful.
• <utility>
  • Included mainly for std::move, std::forward, std::exchange, and std::swap.

There are far more headers that I don’t permit, but perhaps some notable ones are:

• <algorithm>, <format>, <ranges>
  • Dramatic impact on compiler throughput and memory usage, even if a small subset is used.
  • As mentioned above, <algorithm> and <ranges> don’t use a programming style I can get behind due to issues with lambdas.
• <filesystem>
  • Simply too slow to be used reliably but also has a diverging set of goals from a game engine, where the filesystem often needs to abstract a “virtual filesystem” when loading assets from a packed archive.
  • I wrote about my replacement Path abstraction here. Compared to std::filesystem::path, my path is interned in memory, and most operations are O(1) instead of O(n).
• Any of the container headers (<array>, <unordered_map>, etc.).
  • All of the containers needed have implemented replacements that are:
    • More performant
    • More cache-friendly
    • Integrated with the same reflection/serialization system
  • Various aspects of the implementation of these data structures is documented here.

Avoiding the STL for most things ensures that:

• I can step through my code on a Windows PC, Steamdeck, or console and expect identical behavior (modulo platform differences). This eliminates a potential source of non-determinism.
• Facilities for tracking memory usage don’t have blind spots.
• The callstack for a crash on one platform is often the same as the callstack for the same crash on a different platform.
• I don’t run into compiler throughput issues that are somewhat endemic to the design of various parts of the STL.
• I avoid code paths in the STL that are too slow to keep a game interactive in debug builds.

The STL is honestly a monumental achievement, but ultimately is a general purpose solution, ill-suited for my particular domain. Since I’m not shackled with STL constraints, I can sidestep issues I consider defects that others might consider features.

Color functions liberally

What Color is Your Function is a thought provoking post criticizing language design in the context of async functions. Specifically, in many languages like JS, C#, and so on, functions that support await operations must be “colored” with an async attribute in some way. The argument is that this is an additional cognitive tax on the coder, and also prevents code from being as readily composable, since you now have a “barrier” that exists between differently colored functions. An example offered of a language that does this well is Go, which can implicitly suspend the routine once an IO function is encountered. This idea has propagated in the design of languages like Zig as well.

I can see the argument, but don’t really buy it. I want my functions to be colored. I want to know if a function is only supposed to run on a particular thread. I want to know if a function is not threadsafe. I want to know if a function implements a Task that is expected to kick off other tasks on the task thread pool and likely to operate in parallel with other functions.

In the Rookery, most of the ideas expressed in the previous paragraph are expressed either with a specific type signature (in the case of a Task), or with assertions that enforce thread constraints. That is, in addition to the natural coloring that exists with operators like co_await and co_return, I also go out of my way to paint functions manually. Here are some specific “coloring” mechanisms in the engine:

• All work done concurrently is called from a derived Task type which must implement a virtual TaskContext run() function.
  • Tasks are able to dispatch other tasks and await their results.
  • Tasks can suspend mid-execution using C++ coroutines.
  • Tasks are able to call any other function, and can also call the run function of other tasks directly instead of dispatching them.
• A set of macros are used to do various thread annotations. These macros expand to C++ attributes that are ignored by the compiler and don’t “do anything” but are usage hints for the reader.
  • RK_THREAD(ThreadName): This is used to decorate functions or classes that are only meant to be used from a specific thread.
  • RK_THREAD_UNSAFE: This is used to indicate that a function or class requires some form of external synchronization to use safely.
  • RK_THREAD_SAFE: This is added in front of a function to indicate that calling this function from any function at any time is safe. Functions are assumed to be thread safe if undecorated, so I usually use this if I want to emphasize that this function (in contrast to other functions in a file) is thread safe. I will use this also if the function internally manages its own synchronization (i.e. an RK_THREAD_SAFE function probably takes a mutex or implements some lock-free algorithm).

Performance and Memory Guidelines

These guidelines are even more specific to game engines than the style-oriented guidelines.

Try doing less, before being clever

I favor doing less over optimizing the algorithm itself where possible. There are many variations of these theme, but as an example, instead of trying to optimize the allocator to ultra-fine degrees, I’d prefer to focus more on reducing the number of allocations altogether and quickly identifying culprits that allocate gratuitously.

Yet another example is the implementation of a mutex. A lot has been written about how to write speedy mutexes, and while that’s all well and good, I’m actually ok with a non-cutting-edge mutex and reworking my algorithm to avoid the mutex altogether if it ends up being a problem.

This can be extended to choices like deciding to compute something offline and load it at runtime (as opposed to doing all the hard work at runtime). Not always the best choice, but certainly worth consideration if the runtime optimization imposes sufficient complexity to approach real-time throughput.

Selfish optimizations should be weighed as such

Some optimizations are inherently selfish. For example, taking a large subroutine and fully inlining it everywhere it is used might accelerate that routine and surrounding code, but evict useful I$ data elsewhere. Some functions are worth optimization almost regardless of cost to everything else, but I prefer to think of this as an opt-in as opposed to opt-out strategy.

Another example of this is the so-called “small-string optimization” (SSO) where string bytes are interned without the string object instance directly when the string length is below a fixed capacity. This optimization is great if you’re accessing the data frequently, but not so great if every string has a bigger memory footprint. I happened to have a few bytes to spare, so my String implementation does accommodate interning small strings (up to 12 characters in my case, including the null-terminator). However, I wouldn’t add more bytes to the String to accommodate interning strings of a larger size, so my String type sits at 16 bytes as opposed to the std::string which sits at 32 bytes. It’s not hard to imagine an overzealous string implementation expanding this even further, but clearly taking this idea to the limit might help the string implementation, but hurt everything else around it. For me, even 32 bytes is a bit much because a game engine and renderer ideally shouldn’t be doing lots of small-string processing to begin with (in contrast to, say, a compiler or linker).

In other words, optimizations should be done holistically and cooperatively. Micro-benchmarks and derived optimizations can be a useful starting point, but the ultimate goal is a good performance profile of the product as a whole, not any individual part.

Derive usage policy from usage frequency

A common aphorism is to favor SoA over AoS, which I think is a bit too rigid for my tastes. Sometimes, you really do have thousands or millions of things, and it makes sense to group these these in some bespoke way to handle efficiently in bulk. Other times though, you might only have 5 or 6 things and plain-Jane object oriented code is perfectly serviceable.

Blindly applying principles like “composition over inheritance” or principles espoused in “data oriented design” are good ways to waste days/weeks optimizing a problem you don’t have. By the same token, these patterns are sometimes necessary to get a system to have good performance characteristics, and there is a fine line that must be tread when deciding what principles and patterns to apply when.

In a way, I guess I’m sort of saying “don’t prematurely optimize”, but that’s not exactly what I’m saying either. The admittedly vague notion I’m trying to get across with this point is that there is a usability-performance spectrum that exists for each decision, and the side of the spectrum I bias towards depends on usage. In cases where implementing things more optimally doesn’t impose a ton of complexity (and the optimization itself is clear), I don’t think extensive profiling is needed to justify it.

Pack structs responsibly

Not a lot to say here, but unused padding bytes within each data type layout must be intentional. The only times I allow this are cases where:

• Intentional padding is needed to avoid false sharing issues.
• Padding is unavoidable due to subtype alignment and size restrictions.

I don’t move data members within a struct or class to improve locality based on usage, since this is quite difficult to predirect reliably. If I know that some data should be used together (at the exclusion of the rest of the class), that usually tells me that I should model the class with an additional nested subtype.

To parallelize, map first, reduce if needed

Multi-core scalability is non-negotiable for me, so most systems are being designed upfront with this in mind.

The “map first, reduce if needed” guideline is saying that instead of parallelizing work by slapping mutexes on top of resources, it’s better to partition the data set instead. Then, after each partition on is operated on, the work can be merged (reduced) as needed later.

There are plenty of edge cases where even if one tries to do this, a mutex or atomic usage is more practical. In cases where contention is either very unlikely, or is bounded by maybe a couple threads contending for a resource, I don’t sweat it too much. Still though, I’ll try to partition the data so mutexes and atomics aren’t needed in the first place where possible.

Bulk ISPC processing preferred over SIMD intrinsics

While intrinsics are very useful in a pinch, they have a number of drawbacks:

• Debug performance unfortunately suffers when intrinsics are used. This is particularly bad because SIMD intrinsic usage generally occurs in code that is performance sensitive. The poor debug performance of SIMD intrinsics ultimately makes a bad situation worse.
• Usage of intrinsics necessarily shoehorns you into a specific register width. If you intend to ship on a variety of platforms, this is going to leave performance on the table. Console, in particular, generally benefits from wider register widths, but will have lower-frequency CPUs than many desktop CPUs.
• With intrinsics, you’re on the hook for both an x64 implementation and an arm64 version. I think it will be increasingly more difficult to be “x64-only” for engine programmers, not to mention consoles that ship on the arm platform and the uptick in arm laptops.
• Intrinsics are unfortunately a common cesspool for compiler bugs. I’ve run into probably dozens of cases over the years where the compiler optimizer incorrectly handled an intrinsic. This may be much better now than before, but intrinsics are definitely not on the “hot path” of compiler usage.
• Individual SIMD registers are annoying to pass around as function parameters (this is a more minor grievance, but a forgotten __vectorcall attribute can have a fairly negative impact).

The various register types in the Rookery (e.g. float3, int3x3, float3x4, etc.) do not actually have SIMD register storage representations, nor do they have operations expressed with SIMD intrinsics. This means that at rest, these register types are not subject to SIMD register alignment/padding restrictions.

I think a preferable way to leverage SIMD instructions is to adopt a programming model that is register-width agnostic and to leverage SoA style data layouts when data parallelism is needed. Intel’s Implicit SPMD Program Compiler is my preferred tool of choice here, but other options like Google’s Highway exist as well.

Architecture

Module Conventions

The Rookery is subdivided into sets of translation units that constitute modules (module not in the C++20 sense, but the general sense). C++20 modules are not used due to various issues with intellisense providers. Every module must be compilable as both a static library, or a shared library. In debug configurations, modules are compiled as DLLs so that they can be individually loaded by the test harness, and for improved link times.

Platform-specific Code Organization

Each module folder may have a platform/[platform_name]/inc subfolder that is implicitly added to the module’s include path when compiling for a particular platform. Each module folder may also have a platform/[platform_name]/src subfolder containing source files that are compiled when compiling for a particular platform.

A common pattern is to provide different implementations of a platform-agnostic interface for each platform. For example, this is an abbreviated declaration of the File class.

// core/filesystem/File.hpp
#pragma once

// This may be included from any of the following locations:
// - core/platform/windows/inc/core/filesystem/PlatformFile.hpp
// - core/platform/linux/inc/core/filesystem/PlatformFile.hpp
// - ...
// Depending on the platform compiled.
#include <core/filesystem/PlatformFile.hpp>

class File
{
public:
  // ... File interface

private:
  // platform::FileData is defined differently for each platform
  platform::FileData platform_data_;
  
  // ... Other platform-agnostic file data
}

With this, the Windows implementation of File class above would live in core/platform/windows/src/WindowsFile.cpp, and the Linux version would live in core/platform/linux/src/LinuxFile.cpp as an example.

Not all platform-specific functionality is implemented this way. The DX12 backend is, for example, a bit more awkward since much of the PC and Xbox DX12 code can be theoretically shared. Furthermore, it makes sense to support both the Vulkan and DX12 backends on PC. As such, in this specific case, the GPU interface layer is virtualized.

Note: this post may be updated from time-to-time to add/amend various points as the codebase evolves.

Discuss this post on Patreon.