Hasan's Journal

Stories, lessons, and scars from production.

Mehedi Hasan
Back to blog

How git bisect Saved My Friday Deploy

Production regression, 40 commits since last deploy, and no idea which one broke it. git bisect found the culprit in 5 steps.

#Git#Debugging

The Setup

A feature that had worked correctly on staging turned out to be broken in production, and there were roughly 40 commits between the current state and the last confirmed-good deploy. Reviewing all 40 commits manually to spot the regression would have taken hours of tedious reading with no guarantee of catching it, because the offending change might not look obviously wrong in isolation — a one-line change that breaks a faraway feature through an unexpected interaction looks just as innocent as a one-line change that does nothing. Without a way to narrow the search, I would have been reduced to guessing, which is a terrible debugging methodology when production is broken.

`git bisect` solves exactly this problem by performing a binary search through the commit history to isolate the exact commit that introduced a given bug. The workflow starts with `git bisect start`, followed by marking the current commit as bad and a known-good earlier commit as good. Git then checks out a commit roughly halfway between the two. You test that commit, mark it as good or bad depending on whether the bug is present, and git narrows the range accordingly and checks out the next midpoint. In practice, this converges on the exact offending commit in about 5 to 6 steps even across a range of 40 or more commits, since each step halves the remaining search space.

In this particular case, the offending change turned out to be a single line that looked completely innocent on its own, buried in an otherwise unrelated refactor. The change had moved a side effect from one lifecycle hook to another, and the new hook fired at a slightly different time relative to the rest of the application, which broke a downstream component that had been depending on the old timing without anyone realizing. `git bisect` located it in about 3 minutes of actual testing time, compared to what would likely have been an hour or more of manual code review with no guarantee of catching it.