PyTorch 2.15 Makes a Rewritten NCCL Backend the Default; Release Notes Say Groups Create Communicators Eagerly and Allocate GPU Memory Even if Unused

Overview

PyTorch 2.15.0 was published on October 28, 2026, according to the GitHub release page (a publish timestamp of 2026-10-28 17:09 UTC; the release notes themselves carry no date line). Its release notes list in their highlights a rewritten NCCL backend that becomes the default, with a flight recorder that dumps on hangs and TORCH_DIST_USE_NCCL2=0 or nccl-legacy as rollback paths.

The Register, in an article by Joab Jackson published on November 6, 2026, describes the rewrite as “designed to improve fault diagnosis and recovery when GPUs work together”. It explains that NCCL stands for the Nvidia Collective Communications Library and provides PyTorch with inter-GPU communication primitives. The release notes also list a trade-off: communicators are now created eagerly, “allocating GPU memory even for unused groups”. An open GitHub issue puts numbers on that cost. This article sets the three accounts side by side; it is not a test of the release.

What the release notes say

The 2.15 release notes state that backend="nccl", which is also the default CUDA backend of init_process_group(), now creates the new implementation. The legacy implementation remains available as nccl-legacy “for this release only”, and the notes say it and TORCH_DIST_USE_NCCL2=0 “will be removed in the next release”. The backend was not new in 2.15: the 2.14.0 release notes, published on September 2, 2026, list “A preview of our rewritten NCCL backend for PyTorch”, and the Herald’s coverage of 2.14 noted that preview. In 2.15 it became the default.

Among the differences from the legacy backend that the 2.15 notes list:

• Communicators are created eagerly in init_process_group() and new_group().
• Each group is bound to one CUDA device, and the notes say that using several GPUs per process requires nccl-legacy.
• dist.get_backend_impl(...) returns the new backend class, so an isinstance(..., ProcessGroupNCCL) check returns False.

The notes also describe a variant, nccl-lazy, in which collectives run on one eagerly created communicator while each send/recv peer pair gets its own lazily created communicator and stream. A new option, ProcessGroupNCCL.Options.lazy_init, lets nccl and nccl-lazy groups create their communicator on first use instead of allocating about 200 MiB in new_group.

What The Register reports

The Register says the previous backend, ProcessGroupNCCL, “could spawn dozens of communicators per process group”, and that the new one follows a rule: “One process group gets only one communicator during initialization.” A program that needs parallelism would dispatch more process groups, the article says. It also reports that a third-party orchestrator can exclude a crashed or slow GPU and assign a replacement worker. The article attributes to Tristan Rice, whom it describes as a PyTorch core developer and a Meta software engineer, a livestream remark that the old interface needed a rewrite. The release notes’ description of nccl-lazy shows per-peer-pair communicators in that variant, so the two accounts differ in scope; neither explains the difference.

On timing, The Register says the legacy backend will be removed in the 2.16 release, “planned for early next year”. The release notes say only “the next release” and give no date.

The two accounts also differ on the flight recorder. The Register says a flight recorder “has also been added” to show what went wrong after a stall. The 2.14.0 notes already list “a Flight Recorder that works for any backend rather than only NCCL”, and the 2.15 notes list recording it for backends without built-in support, including the new NCCL backend, and dumping it on health-check failures of the new NCCL backend. No cited source explains the difference in wording.

The Register’s article does not mention the memory behaviour, the one-device binding or any user problem report.

Reported memory cost

GitHub issue #199957, opened on October 6, 2026 by a contributor to the project, was filed against a 2.15 pre-release build (2.15.0.dev20261002). It reports about 230 MiB of GPU memory per new_group() even if the group never runs a collective. On two H100 96GB GPUs and nine groups, the reporter measured an increase of 2080 MiB (231 MiB per group) for the new default and none for the legacy backend. After one all-reduce per group, both showed the same 4672 MiB, so the issue’s argument is that the new backend pays the cost at creation time rather than on first use. The reporter said it broke vLLM’s nightly CI and listed two workarounds: TORCH_DIST_USE_NCCL2=0 and NCCL_COMM_SPLIT_SHARE_RESOURCES=1, which reduced the nine-group increase to 1188 MiB in the reporter’s table.

In the same thread, project member d4l3k first replied that eager allocation is “expected” and suggested nccl-lazy. A later comment on the same day, posted by an AI agent on that member’s behalf, reran the test on a 2.16 development build with two GB200 GPUs: 197 MiB per group for the default, and the same for nccl-lazy, because, the comment says, each group’s primary communicator is still created eagerly. The comment’s table does not include the lazy_init option listed in the 2.15 notes. These figures are the issue participants’ own measurements on pre-release builds, not on the 2.15.0 release.

On November 3, 2026, a commenter on that issue reported that vLLM 0.33.0 failed its startup memory check on an internal PyTorch 2.15 build, with 27.46 GiB free against 28.32 GiB required, and that with TORCH_DIST_USE_NCCL2=0 the two affected workloads completed. That is one user’s report on a self-built version.

A separate issue, #203322, opened on November 4, 2026, reports that on 2.15.0 creating any NCCL process group aborts the process when the cudaMallocAsync allocator backend is set, with TORCH_DIST_USE_NCCL2=0 or the nccl-legacy name as the only workaround, according to the reporter. It has one reporter and no comments in the thread. An open pull request, #203904, opened on November 7, 2026, proposes lazy initialisation by default for single-rank groups when no device is requested.

What is not established

As of November 9, 2026 (GitHub dates in this article are UTC timestamps), issue #199957, issue #203322 and pull request #203904 are all open. None of the cited sources says how many deployments are affected or whether a patch release will change the default. The release notes name the rollback paths and say the legacy backend goes in the next release; only The Register gives a timeframe for that release.