Provenance Record

Well structured News article (1034 words) that keeps release-note facts, the Register's account, and issue-tracker measurements in separate sections and says what each source does and does not establish. Neutral tone, no AI self-reference, no promotional language. Title is long (149 characters) but within the limit and accurate.

Snapshots read from disk (gunzip; all 6 sha256 re-computed and equal to manifest.json; all HTTP 200, no bot-check or JS shell, suspicious_patterns null for all 6 so no scanner override was needed). Because the GitHub pages are large HTML, every GitHub source was also read as raw data through the GitHub API (gh api repos/pytorch/pytorch/releases/tags/<tag>, /issues/<n>, /issues/<n>/comments, /pulls/203904); release-notes bodies: v2.15.0 sha256 b11d0d05e14c90d07abc98e3d88da60db0c7ff90716e9c7f84774122829e4a16, v2.14.0 sha256 770b96748784ac23f40fc7e942c8c7b508829b5325533f34d65930d16edf8fe4. The Register was read as extracted text from source-0.html.gz (sha256 478dc0b1be30ad464e29b1674bef5582a7fe1a97c6cea72c8a63d54f9cab7fd4). Allowlist: theregister.com (line 118) and github.com (line 1162) are listed; pytorch.org and TokenPost are not cited. SOURCE 0 (The Register, Joab Jackson, 'PyTorch 2.15 tackles the one bad GPU problem', 'fri 6 Nov 2026 // 18:36 UTC'): byline, date, and Tristan Rice (PyTorch core developer, Meta software engineer, YouTube livestream 'on Thursday') confirmed. Verbatim: 'designed to improve fault diagnosis and recovery when GPUs work together'; 'could spawn dozens of communicators per process group'; 'One process group gets only one communicator during initialization.'; 'planned for early next year'; 'has also been added' (from 'A flight recorder has also been added'). Orchestrator exclusion/replacement worker, the dispatch-more-process-groups sentence and the 'drowning in technical debt' remark (article paraphrases as 'needed a rewrite') confirmed. Absence claim checked against the full text: the Register body has no mention of memory, device binding or user problem reports. SOURCE 1 (v2.15.0 notes, GitHub API: published_at 2026-10-28T17:09:49Z, created_at 2026-10-28T02:08:20Z, not draft, not prerelease; body has no date line): the article's date basis (publish timestamp, UTC) is stated in the lead and holds; the calendar date is October 28 in the UTC and US time zones. Verbatim: 'allocating GPU memory even for unused groups' (section 'The nccl backend now uses a rewritten, cleaned-up NCCL implementation by default'), 'for this release only', 'will be removed in the next release' (applies to nccl-legacy and TORCH_DIST_USE_NCCL2=0), 'the next release' (no date given anywhere in the notes). Title wording: the notes say communicators 'are created eagerly ... allocating GPU memory even for unused groups'; the title's 'Allocate GPU Memory Even if Unused' is an unquoted paraphrase and 'Eagerly' is the notes' own word. Highlights row confirmed (rewritten backend default, flight recorder that dumps on hangs, TORCH_DIST_USE_NCCL2=0 or nccl-legacy rollback). Details confirmed: backend='nccl' is the default CUDA backend of init_process_group(); one CUDA device per group and 'several GPUs per process requires nccl-legacy'; get_backend_impl returns the new class so isinstance(..., ProcessGroupNCCL) is False; nccl-lazy = collectives on one eagerly created communicator, each send/recv peer pair lazily created communicator and stream; ProcessGroupNCCL.Options.lazy_init so nccl and nccl-lazy groups create their communicator on first use 'instead of allocating ~200 MiB in new_group' (#200020; article says 'about 200 MiB'). The article lists 3 of the 4 differences and introduces them as 'Among the differences', so the quantifier is safe. Flight recorder: the 2.15 notes list recording it for backends without built-in support (the new NCCL backend, MPI, UCC) and dumping it on new-backend health-check failures. SOURCE 2 (v2.14.0 notes, published_at 2026-09-02T17:40:10Z): verbatim 'A preview of our rewritten NCCL backend for PyTorch' and 'a Flight Recorder that works for any backend rather than only NCCL'; the 2.14 notes also add the experimental nccl-lazy and the nccl-legacy name. The Herald link /article/2026-10/17-pytorch-214-adds-nvgemm-kernels-... exists (src/content/articles/2026-10/17-pytorch-214-...md), it does describe a 'preview of a rewritten NCCL backend' and its Distributed paragraph matches; no corrections record exists for it. SOURCE 3 (issue #199957, opened 2026-10-06T19:55:50Z by laithsakka, author_association CONTRIBUTOR; still open, last update 2026-11-03T13:40:43Z, 9 comments): body says '~230 MiB' per new_group, torch 2.15.0.dev20261002+cu134, 2x H100 96GB, 9 groups: nccl2 default +2080 MiB (231 MiB/group), legacy +0 MiB, both +4672 MiB after one all_reduce per group, NCCL_COMM_SPLIT_SHARE_RESOURCES=1 +1188 MiB; 'this broke vLLM's torch-nightly CI'; workarounds TORCH_DIST_USE_NCCL2=0 and NCCL_COMM_SPLIT_SHARE_RESOURCES=1. Comment 1 (d4l3k, MEMBER, 2026-10-06T22:58Z): 'Eager initialization allocating vram earlier is expected' and suggests nccl-lazy. Comment 2 (d4l3k account, 2026-10-06T23:01Z, headed 'AI-generated comment: results collected and posted by an AI agent on behalf of @d4l3k'): torch 2.16.0.dev20261006+cu130, 2x GB200, 197 MiB/group for default and for nccl-lazy, 'each group's primary communicator is still created eagerly'; its table has no lazy_init row (lazy_init arrived later via #200291). The three per-group figures are therefore ~200 MiB (release notes, no build stated), 231 MiB (2.15 pre-release dev build, H100) and 197 MiB (2.16 dev build, GB200); the article reports them side by side with attribution, labels the issue measurements as the participants' own on pre-release builds, and does not reconcile them (policy rule 6). Nov 3 comment (janbernloehr, NONE, 2026-11-03T12:54Z): internal build 'PyTorch 2.15.0a0+fd23299d96', vLLM 0.33.0, reproduced November 2, '27.46 GiB free versus 28.32 GiB required', TORCH_DIST_USE_NCCL2=0 lets both workloads complete; the article's figures and 'one user's report on a self-built version' are accurate, though the commenter's build string is an alpha (2.15.0a0), not the release. Other comments in the thread (a third-party comment with a command line for an external tool, links to test-infra and vLLM PRs) were not cited and not relied on. SOURCE 4 (issue #203322, opened 2026-11-04T04:26:02Z by Balaxxe, author_association NONE; open; 0 comments, updated 2026-11-04T16:31:51Z): the abort under PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync on 2.15.0+cu132 at init_process_group, with 'the only workaround is TORCH_DIST_USE_NCCL2=0 (or the nccl-legacy backend name)', matches the article, which attributes it to the reporter ('according to the reporter') and notes one reporter and no comments. The issue's analysis section is labelled 'AI-assisted, reviewed by the reporter' and names no vendor. SOURCE 5 (PR #203904 by bobrenjc93, CONTRIBUTOR, opened 2026-11-07T03:51:16Z, open, not merged, not draft): title '[c10d] nccl2: lazily init single-rank groups when no device is requested'; body: when no pg_options and no device_id are passed, single-rank nccl2/nccl-lazy groups get lazy_init=True. The article's one-sentence description is accurate (it does not mention the no-pg_options condition, a detail the title also omits). CLAIM-ACCURACY TESTS: (1) the jq/grep quantifier scan was run and every hit checked against all six snapshots. The absence claims hold: 'does not mention the memory behaviour, the one-device binding or any user problem report' (Register), 'only The Register gives a timeframe for that release' (no timeframe in the 2.15 notes, the issues or the PR; the PR and a comment mention 2.16 only as the nightly version), 'None of the cited sources says how many deployments are affected or whether a patch release will change the default' (none does), 'one reporter and no comments' (#203322: 0 comments). 'As of November 9, 2026 ... all open' re-checked against the API on November 9. (2) Headline, summary and lead keep the sources' qualifiers (notes attribute the memory behaviour, 'for this release only', pre-release and single-reporter data are labelled), actor and object not renamed. (3) Dates: all in absolute form with the UTC basis stated; 2.14 September 2, 2.15 October 28, issue October 6, comment dates, November 3/4/6/7/9 in chronological order; no relative dates. (4) Derived figures: 2080/9 = 231 is the issue's own figure; no figure is computed by the Herald; the 1188 MiB figure is labelled as the reporter's table. (5) Disagreements: the Register-versus-notes differences (flight recorder 'added' versus the 2.14 notes' existing Flight Recorder; one communicator per process group versus nccl-lazy per-peer communicators; '2.16 ... early next year' versus 'the next release'; ~200 / 231 / 197 MiB) are reported with attribution and unreconciled as the policy requires, and all 13 quoted strings (12 sentences or phrases plus the code literal "nccl") are verbatim in the raw sources; two minor wording issues are filed as clarifications (see concerns). (6) Independence: 1 independent outlet (The Register, a single article); the other five are the project's own records (two sets of release notes, two issues filed by outside users in the project's tracker, one PR); this is a News article, so no Analysis independence minimum applies; News minimum of 3 sources and 400-1200 words met (6 sources, 1034 words). Anthropic disclosure: not required - the subject is PyTorch/NCCL, Anthropic is not named in the headline, summary or lead, and the article does not report in a passage of its own any finding, advisory or credit line naming Anthropic or 'assisted by Claude'. PR #203904's body ends with the credit line 'Authored with an AI assistant (Claude)' but the article never reports it (it reports only an unnamed 'AI agent' comment in issue #199957, which the comment itself does not attribute to a vendor), so the third trigger is not met and no clarification is filed on this ground.

Every number, date, version, name and quote traced to raw source text; no fabricated or misattributed specific found. Pre-release and single-reporter data are labelled as such. Two subordinate wording issues are recorded as clarifications: the omitted conditional in the Register's removal timeline, and the framing of the one-communicator-per-group versus nccl-lazy comparison as an unexplained difference.

Accurate, well-sourced News article on a default change in a widely used framework, with the memory trade-off and open issues correctly scoped and labelled; publishable with two clarifications. Featured candidacy: 2 - the default NCCL backend of PyTorch changes for most multi-GPU training and inference users, with a documented memory cost and open regressions, but the evidence is release notes and open issues rather than a settled outcome