Go's New Goroutine Leak Profiler Traces Deadlock Bugs Found in CockroachDB, etcd, and Kubernetes

Overview

The Go team has published a detailed walkthrough of the language’s new goroutine leak profiler, a diagnostic tool that reached general availability in Go 1.27 and is designed to catch goroutines that will never resume execution, according to the official Go blog. The post, written by Vlad Saioc, opens by noting that “Go’s concurrency features are powerful and easy to use, but that same ease can sometimes lead even seasoned developers to make mistakes,” and that “even existing tools may miss some concurrency bugs, such as the topic of this article, the goroutine leak.”

What We Know

The blog post defines the problem directly: “We may consider a goroutine leaked if it is blocked, but the conditions needed to unblock it can never be met. Over time, an accumulation of leaked goroutines degrades performance through excessive memory usage (by the leaked goroutines themselves or the memory they reference), as well as CPU usage from the garbage collector, especially if GOMEMLIMIT is in use,” according to go.dev.

The feature itself is not brand new to Go 1.27 — it shipped as an experiment behind a flag in the prior release and has now graduated to a stable, on-by-default capability. Per the Go 1.27 release notes, “a new profile type that reports leaked goroutines, previously available as an experiment in Go 1.26, is now generally available. The new profile type, named goroutineleak, is supported in the runtime/pprof package. It is also available as the net/http/pprof endpoint /debug/pprof/goroutineleak.” The release notes add that “the goroutineleakprofile GOEXPERIMENT setting is now deleted,” confirming the flag that gated the earlier experimental version has been removed now that the feature ships by default.

The detection method relies on the runtime’s garbage collector rather than on manual instrumentation. The release notes describe it precisely: “A leaked goroutine is a goroutine blocked on some concurrency primitive (channels, sync.Mutex, sync.Cond, etc) that cannot possibly become unblocked. The runtime detects leaked goroutines using the garbage collector: if a goroutine G is blocked on concurrency primitive P, and P is unreachable from any runnable goroutine or any goroutine that those could unblock, then P cannot be unblocked, so goroutine G can never wake up. While it is impossible to detect permanently blocked goroutines in all cases, this approach detects a large class of such leaks,” according to go.dev. The release notes also flag a known limitation: “the runtime may fail to identify leaks caused by blocking on concurrency primitives reachable through global variables or the local variables of runnable goroutines.”

To illustrate the profiler in practice, the blog post walks through a real bug pulled from CockroachDB, according to the Go blog. “The following example is taken from CockroachDB. It involves acquiring and releasing a lock in a loop, but forgetting to unlock it before executing a break statement,” the post explains, pointing to a specific pull request on the CockroachDB repository. The bug pattern is a bootstrap() method on a Gossip struct that acquires g.mu.Lock() inside a loop, then hits a break on a closed condition without first calling g.mu.Unlock() — so a second call to bootstrap() on an already-closed Gossip blocks forever trying to reacquire the lock. “In such a case, the goroutine will leak when failing to acquire the lock,” the post states, and notes that “adding a call to Unlock before the break addresses the issue.”

Beyond the CockroachDB case, the post’s collection of real-world leak examples names several other open-source projects by section header, including etcd, Kubernetes, and Moby, according to go.dev, alongside generic patterns the post labels things like “Double Send,” “Early Return,” “Timeout with Context,” and “Range Over Unclosed Channel.” The profiler is reachable through the standard runtime/pprof package as a goroutineleak profile type, or automatically over HTTP at /debug/pprof/goroutineleak for any program that already imports net/http/pprof, per the Go blog.

The blog post credits the feature’s origins to outside research: “Goroutine leak detection is the result of a research collaboration between Aarhus University, Washington University in St. Louis, and Uber, as presented in ‘Dynamic Partial Deadlock Detection and Recovery via Garbage Collection’ (Saioc et al., ASPLOS 2025),” the post states. It adds that “the transition from academic prototype to actual Go feature was made possible with the guidance of Michael Knyszek and Michael Pratt on the Go team at Google, and PJ Malloy (@thepudds),” according to go.dev. Separately, the Go 1.27 release notes credit the contribution directly: “Special thanks to Vlad Saioc at Uber for contributing this work,” per go.dev.

As previously reported, the goroutine leak profile was one of several items in the broader Go 1.27 stable release, alongside generic methods and the encoding/json/v2 package. This week’s blog post is the Go team’s dedicated deep dive on the profiler itself, expanding on that one-line release-notes mention with the detection mechanism, usage instructions, and the CockroachDB/etcd/Kubernetes/Moby examples.

What We Don’t Know

The blog post does not disclose how many real-world leaks the technique has caught in production systems beyond the illustrative examples it walks through, nor does it name individual engineers at CockroachDB, etcd, Kubernetes, or Moby who encountered or fixed the specific bugs cited. The post is also explicit that the detector cannot catch every leak: it will miss cases where a blocking channel or lock remains technically reachable through a global variable or a still-running goroutine, even if nothing will ever actually use it again.

Analysis

The choice to publish a dedicated blog post about a feature that already shipped weeks earlier, rather than folding it into release-notes coverage, suggests the Go team sees goroutine leak detection as a durable engineering topic rather than a one-off release bullet point — the examples drawn from CockroachDB, etcd, Kubernetes, and Moby double as a survey of how leak bugs recur across some of the ecosystem’s most widely deployed Go projects.