Tell me about a bug you fixed...

This post is a continuation of a series of posts about past interview questions received. In the previous post, we just looked at a few “standard” coding questions and responses. In this post, I’ll review responses I’ve given in the past to the age-old prompt:

Tell me about a tricky bug you encountered in the past. How did you fix it?

Since I’ve had this question several times now in the last few decades, my answer has changed over time. As such, instead of presenting a singular answer in this post, I’ll present several.

The mysterious screen flicker

One of the earliest bugs I was assigned as a graphics programmer was a bug where some users reported that the entire screen would “flashbang” them with a fully white frame occasionally. This is obviously a pretty bad bug, so I was pretty motivated to figure it out. As with most bugs in engineering, step one is data collection. Who was encountering this bug? Under what circumstances?

Looking through the few bug reports that existed, there didn’t appear to be any rhyme or reason as to why this was happening, except that all the users experiencing the issue were using similar Nvidia Kepler GPUs. Not exactly the same, but enough that I noticed a pattern. So the next step was to get my hands on a machine to try and reproduce the problem, since it wasn’t a bug I or anyone near me had experienced.

Eventually, after some keyboard head-banging, I managed to experience the bug once and reproducing it was anything but simple. It took several days of trying all sorts of things before it finally happened. Specifically, I was able to “flashbang” myself when maximizing the window (uh-oh!). So obviously, the next thing to do was to figure out if I could reproduce it more reliably. As it turned out, this wasn’t as easy as just resizing the window. If I dragged the window to do a continuous resize, the flashbang happened, sometimes, but not always.

I ended up writing some code to readback the framebuffer values to see if the center pixel was completely white, and I also hardcoded an ambient light bias to ensure that no matter where in the world I was, a “correct” scene without the flashbang could never produce pure white. This ended up not being necessary because eventually, I found the culprit by accident. It turned out that there was some driver bug that caused the entire contents of the OpenGL buffer to get cleared to white when the source texture size had an odd pixel width. The exact mechanics of what was happening during the resize is now lost in time, and to this day, I still have no idea what was going on here, but I “fixed” things by just ensuring that the window and framebuffer never had an odd width. In hindsight, I wish I had spent more time understanding the exact mechanics of the bug beyond a minimal demo repro, but this was my first “real” graphics programming bug, and I was itching to move onto a new problem, having spent an unexpectedly long amount of time on this one. Since this experience, I’ve lived through many driver bugs, but this one sticks out to me because it was the first driver bug I’d encountered (that I’m sure has since been resolved for some time now).

Takeaways:

  • Driver (and shader compiler) bugs are real and crazy. It’s a good idea to assume the fault is in your code first in either case, but driver/compiler bugs in graphics are frequent enough that you can’t ever fully discount the possibility.
  • The bug reports don’t always tell the full story. Details that you might realize are important aren’t considered by the user, so bug reports may often omit important context. If the bug reports all said “I resized the window and experienced a flashbang,” I probably could have resolved this issue in half the time.

The “random crash”

Just a few weeks before a content release, QA reported a crash that occurred on Xbox 360, where the player would encounter a crash walking through a particular level.

This bug was a nightmare to track down, because neither I nor QA could reliably reproduce the issue for a while. Eventually, I managed to repro it by stumbling on a specific path through the level, and if I walked through that path with the correct timing, I could get a crash. This path was about 30 seconds in duration, but by the time the bug was fixed, I got very good at reliably triggering the crash.

I tried reproducing the same crash on desktop to see if the issue happened there also, but no dice. The only information I had was that there was a GPU crash due to a faulting address so naturally, the thing I suggested was a resource deletion problem. With a bit more trial and error, I learned that if I added a global texture mip bias to the texture streaming system, the problem went away. Ultimately, the issue was fixed by resolving a long-standing bug in the resource deletion system, where resources could be deleted while still in use on the GPU.

Takeaways:

  • Bugs related to the CPU and GPU timeline are always tricky to resolve, because you ultimately have to reason about a distributed system (albeit at a localized scale).
  • Just because a race condition happens on a specific platform (Xbox in this case), it doesn’t mean the issue itself is platform-specific. Here, the bug technically existed on all platforms, but just happened to present itself on Xbox, since the bug was timing-specific.
  • Having knobs to tune in your system (such as the global streaming mip bias in this case) can be very useful to either exacerbate or rule out a root cause of a bug.
  • Fixing a bug without a reliable repro is virtually impossible. I wasted a good chunk of time trying to do “speculative” fixes here, only for QA to ping me later that they managed to encounter the bug again. There wasn’t a substitue for rolling up my sleeves and really hunting for a reliable repro.

Random shader artifacts

Around the time physically-based rendering was taking off, I had the opportunity to work on a “material layer” system, wherein a material would blend between different material layers using a control mask unwrapped to the entire object. The idea was pretty simple: a “control texture” would encode a material ID within each texel, and this ID would be used to lookup material parameters in a material palette. This palette could then be configured at runtime to easily allow for character customization and other features such as displaying status effects.

After the initial implementation, everything seemed to work pretty well, until at some point, artists complained that, occasionally, they noticed what appeared to be “seams” on objects at the boundaries of the layers. This confused me at the time, since for budget reasons, we never allowed more than one layer to be active per-pixel.

The issue was, of course, a classic issue I believe every graphics programmer encounters once (if not many times) in their career. Namely, the shader issued texture samples within a branch, and the uvs used to sample different textures were not always the same. Nowadays, I’d be able to immediately spot the issue by inspection, but at the time, I had a somewhat shaky understanding of the GPU execution model, so it wasn’t clear what was happening. The fix was easy enough: texture samples were fed gradients computed outside the branch.

Takeaways:

  • This bug ended up being fairly instructive for me, since prior to this bug, I more or less treated the GPU’s execution model as somewhat of an opaque box.
  • These days, I pay a lot of attention to branches, not (strictly) because of performance consideration, but because it’s important to recognize issues that can arise when branch directions diverge across a wave. Aside from the usual issues of invoking ddx, ddy, or fwidth on a divergent quantity, the other common issue in a post-SM6.6 world is accessing a non-uniform resource from the global descriptor heap.
  • Traumatic bugs stick with you, and change how you code. Experiences like this one are excellent for improving programming technique in ways that books and articles can’t.

Works in debug, not in release

Hearing that a bug appears to occur only in an optimized build should immediately evoke trepidation in the minds of most engineers. I’ve encountered many different flavors of this bug over the years, so instead of focusing on a specific one, I’ll just list what I look for when investigating a bug of this sort. Each item on this list has constituted hours upon hours of debugging on various disparate issues on different projects.

  • Uninitialized memory
    • This is the usual culprit. Reading from uninitialized memory can be surprisingly deterministic in debug configurations (e.g. you might read zeros), so it’s quite common for this sort of UB to explode in a release build.
  • CRT or general static library mismatch
    • Another issue that can occur is related to the link configuration. It’s common for release builds to perform static links, while debug builds link dynamically for faster iteration times. This unfortunately exposes issues where static library mismatches may cause things to explode only when all modules are linked together.
    • The specific nature of the issue here may be one or several separate issues, such as ODR ambiguity or an actual library version incompatibility.
  • Timing-related issue
    • Obviously, optimized builds (usually) run significantly faster than their unoptimized counterparts, so its possible that release builds expose various concurrency related issues (use-after-free, race condition, double free, etc.).
  • Implementation differences
    • In some cases, the issue at hand is a more direct problem, where code that needed to run was conditionally compiled out in release mode (or vice versa).
  • Numerical determinism
    • Another subtle issue that can occur in optimized builds are numerical issues that arise because of (bad) floating point optimizations. This could result in propagating NaNs or other downstream issues that can take a while to track down. These days, I pretend that “fast math” simply doesn’t exist.

Conclusion

I like the narrative “tell me about a bug you worked on” prompt, because I think almost everyone has a (likely harrowing) recollection of a bug that took a long time to resolve. Most of these bugs end up being blessings in disguise, since subsequent encounters invariably go a lot more smoothly. Bug-squashing also requires a nice blend of skills that go beyond just understanding and implementing an algorithm.


Discuss this post on Patreon.