MLX 0.32.4 Adds Attention and Gated Delta Net Gradients, Metal 1-Bit Affine Quantization and Python 3.15 Wheels
Apple's MLX array framework shipped v0.32.4 on October 14 with training-oriented VJPs, 1-bit affine quantization on CPU and Metal, Python 3.15 support and stft/istft.
Overview
The MLX array framework published v0.32.4 on October 14, 2026, according to the GitHub release page. The release notes list 97 change entries and a “New Contributors” section naming 24 first-time contributors, by a count of the lines in the notes. Headline entries include gradient (VJP) support for scaled dot product attention and for the gated delta net, 1-bit affine quantization on Metal, Python 3.15 support and new short-time Fourier transform functions.
Training-Oriented Gradients
The release notes list “Adding VJP for the scaled dot product attention” among the changes. The pull request description says the implementation is “needed for fine-tuning and training”. It states that the existing fallback “uses a lot of memory since it allocates a full [context_size x context_size] attention matrix”, and that the new approach expresses the VJP as a sequence of tiled matmuls instead.
The author reports benchmarks on an M5 Ultra with 80 cores. According to the same pull request, “For the causal case we have up to 2.8x speedups”, and memory improves in every configuration, “up to 24x at the longest shape: 24.63 GiB to 1 GiB for batch size = 1”. The description also notes an exception: the (256, 24, 4) configuration “can be slower than the fallback”, though its memory use is still better. The description adds an edit saying that later commits improved memory and speedup further.
A second gradient entry, “Adding VJP for gated delta net”, adds two kernels, according to its pull request: “one for the sequential case (pure metal kernel) and one for the chunked case (using neural accelerator)”. The description reports that “The chunked kernel is 1.4x to 6.3x faster than the sequential one and 9x to 79x faster than the fallback”, measured across batch sizes 1, 4 and 8 and sequence lengths from 8 to 4096.
1-Bit Affine Quantization
The release also includes “Add 1-bit affine quantization support (Metal)”. The pull request extends the supported affine bit-widths from {2, 3, 4, 5, 6, 8} to {1, 2, 3, 4, 5, 6, 8} and lists CPU and Metal backend work, including quantize, dequantize and quantized matmul paths. It states that the change “assumes the model has already been quantized externally (e.g. during training)”, so its contribution is packing and inference on Apple Silicon. The description adds that 1-bit is not yet supported on the CUDA backend.
Python 3.15 and Other Additions
Python 3.15 and free-threaded 3.15t are added to the wheel build. The pull request says it adds both to the release wheel matrix and updates nanobind from 3.0.1 to 3.1.0.
The notes also list “Add stft and istft to mlx.core.fft”. According to its pull request, the signature follows torch.stft, except that norm takes the same “backward”/“ortho”/“forward” strings as the rest of mlx.fft.
Other entries in the release notes include a FloorDivide primitive for integer floor division, a minmax method, Metal extension build support, an fp_gather_qmv optimization for gather_qmm on CUDA, an MLX_CONV_WINOGRAD switch to disable Winograd for conv2d, and a fix for cooperative tensor build issues on Mac OS 27. Most of the remaining entries are bug fixes, many of them concerning views, strides and contiguity flags.
What We Don’t Know
The benchmark figures above are self-reported in pull request descriptions and have not been independently reproduced. The release notes do not describe a timeline for CUDA support for 1-bit quantization.