🔁 Main Branch: The One Where the Retries Made It Worse - Issue #37

Hiya friends,

GitHub’s August 17 outage lasted 7 hours and 47 minutes. The root cause explains about three of them.

Most coverage lands on the Istio misconfiguration. The root cause analysis traces it to a sidecar pod hitting its concurrency limits and failing to autoscale, because the scaling policy watched the host service and not the sidecar limits. Four HAProxy nodes exhausted their flow limits, and the gateway auth path went down with them.

At 16:36 UTC GitHub paused HAProxy on the affected nodes and got immediate broad recovery. That should have been the end of it. Copilot Token Service didn’t fully recover until 21:02, nearly five hours later, because a latent retry bug in VS Code turned every failed token request into a pile of follow-up requests. Traffic went from 7 to 9K RPS to 70 to 100K RPS.

The infra team fixed their part in about three hours. A client-side retry loop extended the outage by another five.

When millions of VS Code instances hammer the same endpoint without backing off, you become the DDoS. GitHub got retry-stormed by its own users trying to use its own product.

Vlad’s post puts the pressure in context: monthly commits have grown from 1.4 billion to 2.9 billion since April. The follow-up work includes consistent retry limits, retry budgets, and variable timeouts across service-to-service interactions. Good. Those guardrails were overdue.

🚢 What shipped

Credential revocation and deauthorization by token type

Kill-switches used to be all or nothing. Hit the button during an incident and every credential a user owned went away, including the ones you still trusted. Now enterprise owners, org admins, and anyone with Manage enterprise credentials can revoke a single credential type: personal access tokens, SSH keys, OAuth app tokens, or GitHub App user access tokens. Delete every PAT for a compromised EMU account without touching their SSH keys. All the enterprise-level bulk actions are now available at the org level too, in the UI and the REST APIs, and every action lands in the audit log with an email to the affected user.

Better tools for managing blocked users

A long blocked-user list has been hard to actually read. This update makes it searchable by username, full name, or email, sortable, paginated, and filterable by block reason. You can attach private notes to preserve context, and org blocks now show who applied them and when they expire. The search field is separated from the block field, so you stop accidentally blocking someone you were only trying to look up. Personal accounts and orgs both, under Settings > Moderation > Blocked users.

📖 What I’m reading

Predictive Autoscaling for Node.js on Kubernetes by Ivan Tymoshenko, Luca Maraschi, and Matteo Collina

Kubernetes autoscaling has a timing problem. HPA waits for CPU to cross a threshold, and CPU can look fine while the event loop is saturated and requests pile up. KEDA adds better triggers but keeps the same reactive loop, so it still detects overload after it starts. The paper forecasts cluster-wide load for the moment new pods will actually be ready, then scales ahead of it. The aggregate matters because adding pods redistributes per-instance metrics and a scaler watching one pod ends up chasing its own changes. The benchmark claim is sharp: 26ms median latency vs 154ms for KEDA and 522ms for HPA on a steady ramp. It’s a research result, not a drop-in controller, but the framing is right. Reactive autoscaling is always late by design.

Worth your time if you run Node.js on Kubernetes and your p99 doesn’t match your CPU graphs.

arxiv.org/abs/2604.19705

🔧 What I’m using

Been running controlled experiments to measure whether AGENTS.md actually changes what coding agents do, rather than assume it helps.

Two conditions, same task, same starting commit. Baseline has no repo instructions. Treatment gets a candidate AGENTS.md. You compare the diffs.

The finding so far: simple tasks are usually neutral. Guardrail tasks, the ones that compile clean but break at runtime, show the clearest signal.

Open source, MIT, dependency-free. Five-minute fixture demo included.

github.com/AndreaGriffiths11/agents-md-benchmark-plugin

✨ This week

Getting ready to travel to Lima for DevOps Days. If you’re going to be there, come find me.

With gratitude, I’ll see you next week.

Andrea

📌 P.S. I’ll be at WeAreDevelopers World Congress North America, September 23 to 25 in San Jose. If a ticket is the thing standing in the way, reply and let me know.

The GitHub Accesscats team is running an access needs survey at gh.io/access-needs. Takes about five minutes.

💛 If Main Branch is useful to you, subscribe or sponsor me on GitHub.