/Debugging

The 1 Hour Per Year Bug (But Only In Pacific Time!)

- Tomer Aberbach tl;dr: “The date was November 8, 2021 and I was a bug triager on the Google Docs team. That day began like any other. I made myself some coffee and started looking through bug reports from the day before. But then something caught my eye.”

featured in #543

Crazy Debugging Stories - Recursion

- Andrea Bergia tl;dr: “Over a decade ago, I used to work in a large and crazy C++ code base. The system had a feature that allowed our customers to search for a record based on a referred field’s value. For example, if we had an incident table, you could search by an attribute of the incident’s assignee (which referred to a record of user).”

featured in #542

Debugging With Production Neighbors

tl;dr: SLATE is Uber’s E2E testing tool for microservice architectures that allows testing of services alongside production dependencies. It enables developers to generate test requests mimicking production flows while targeting services under test. This blog explores three debugging options in SLATE: remote debugging of deployed instances, local debugging on developer machines, and debugging through filtered monitoring. These features aim to simplify troubleshooting in production-like environments.

featured in #525

The Worst Bug We Faced At Antithesis

- Will Wilson Colin Percival tl;dr: “It was early days at Antithesis, and we had a problem: machines kept crashing. The bug was rare. Frustratingly rare. Like, one crash every couple thousand machine-hours rare. Rare enough to trigger a giant internal debate about whether this bug was even worth fixing.”

featured in #517

Untangling Spaghetti: Debugging Non-Terminating Object Programs

- Kent Beck tl;dr: “I had fun today debugging an infinite loop. Some of the techniques we used build on things I've talked about recently so I thought I'd reflect on the experience a bit. There's no happy ending to my story, though: the defect remained at the end of the session. Still, the techniques are worth thinking about because infinite loops in object programs can be difficult to debug. I wish you better luck with yours.”

featured in #508

Classifying Bug Reports With ChatGPT

- Rafael Quintanilha tl;dr: Rafael creates an automated workflow that: (1) Triggers on new bug reports. (2) Uses ChatGPT to classify the report based on the user description. (3) Updates the priority accordingly. (4) Sends a Slack notification if and only if the bug is deemed as Urgent.

featured in #504

A Very Subtle Bug

- Nelson Elhage tl;dr: "I like to say that complex systems don’t usually fail for complex reasons, but for the simplest, dumbest possible reasons – there are just more available dumb reasons. But sometimes, complex systems do fail for complex reasons, and tracking them down does require understanding across many of the different layers of abstraction we’ve built up. This is a story of such a bug."

featured in #466

67 Weird Debugging Tricks Your Browser Doesn't Want You to Know

- Alan Norbauer tl;dr: "A list of useful, not-obvious hacks to get the most out of your browser’s debugger. Assumes an intermediate-level-or-higher understanding of the developer tools." These include logpoints / tracepoints, changing program behavior, quick and dirty performance profiling, using function arity, using time. And more. 

featured in #465

Lessons From Debugging

- Matt Rickard tl;dr: (1) Reproduce with the smallest example. In the simplest environment. (2) Read and re-read the error statement. Read the stack trace. Add more logging if you don’t know where the error is thrown. (3) Change one thing at a time. (4) Divide and conquer. Sometimes that means binary search on good/bad commits. Other times isolating the problem. (5) Be open to debugging in different environments. (6) State your assumptions. (7) Get a second set of eyes on it. (8) If you're debugging some stateful code, think about \_how\_ you ended up at that state. A recipe that (reproducibly) gets you to that state is often the path to fixing it. (9) Look at the logs, all the logs. (10) When in doubt, start with the most recent changes, especially dependencies changes.

featured in #453

A Systematic Approach To Debugging

- Nicole Tietz-Sokolskaya tl;dr: Nicole’s process involves six steps, each of which she expands on: (1) Identifying the symptoms. (2) Reproducing the bug. (3) Understanding the systems. (4) Forming a hypothesis about the bug's location. Instead of randomly searching through the code, the author suggests forming a hypothesis about where the bug might be located. This narrows down the search and makes the debugging process more efficient. (5) Testing the hypothesis. (6) Fixing the bug. "If you don't understand the bug behavior, you have no hope of knowing if you've fixed it or not."

featured in #448