Infrastructure
63 articles RSS
MLflow 3.17.0 Adds Per-Resource-Type Permissions and Opt-In Trace Rollups, and Requires a Coordinated Database Upgrade
MLflow 3.17.0, published October 7, adds sub-resource permission tiers and opt-in daily trace analytics rollups, and its notes require stopping all writers for SQL schema upgrades.
Transformers v5.19.0 Adds Google's EmbeddingGemma 2 Multimodal Embedding Model and Deprecates the paged Attention Prefix for Continuous Batching
Hugging Face Transformers v5.19.0 adds EmbeddingGemma 2, a Gemma 4-based embedding model for text, images, audio and video, and flags breaking changes.
Ray 2.59 Graduates Ray Data LLM and Ray Serve LLM to General Availability and Enables Token Authentication by Default for Local Clusters
Ray 2.59.0 declares its LLM batch and serving APIs generally available, upgrades to vLLM 0.27.0, and turns on token authentication for local clusters, with all clusters to follow in 2.61.
vLLM 0.31 Adds Draft-Model Speculative Decoding to Model Runner V2, a Preload CLI for Fast Restarts, and Gates Per-Request Multimodal Kwargs
vLLM 0.31.0 brings draft-model speculative decoding to Model Runner V2, adds a vllm preload CLI and a max-num-active-seqs cap, and removes tokenizer_mode="slow".
CoreWeave Launches Forge, Brings Vera Rubin NVL72 Online and Says It Will Offer Vera CPU Racks for Agent Sandboxes
At its Fully Connected event, CoreWeave announced Vera Rubin NVL72 availability, plans to offer NVIDIA Vera CPUs, and the Forge platform for training and agent improvement.
vLLM 0.30 Adds a Persistent GPU Weight Cache for Faster Engine Restarts and Removes GPTQ Activation Ordering
vLLM 0.30.0 adds a GPU-resident weight cache for faster engine restarts and removes GPTQ g_idx; llm-d v0.10.0 then moved its base image to 0.30.0.
Cohere Adds Confidential-Computing Encryption to Model Vault, Locking Even Itself Out of Customer Inference Data
Cohere's new Encrypted Vault tier for Model Vault runs inference inside hardware that encrypts memory on both CPU and GPU, so Cohere and the cloud provider cannot read customer data.
NVIDIA Releases Personal AI Router, an Open-Source Tool That Pools Local AI Inference Across a Home Network
NVIDIA's open-source PAIR routes multi-agent inference requests across paired local machines running Ollama or LM Studio, without pooling GPUs or changing agent code.
vLLM Benchmarks Five Speculative Decoding Methods on AMD Instinct GPUs, Reporting Gains Up to 2.87x
AMD and Embedded LLM tested five speculative-decoding drafting methods in vLLM on AMD Instinct GPUs, with some configurations reaching up to 2.87x throughput over the non-speculative baseline.
AWS and Nvidia to Deploy 2 Million More GPUs in 2027-2028 as Demand Outpaces Prior 1-Million Commitment
AWS and Nvidia announced an expanded partnership to deploy 2 million additional GPUs across AWS data centers in 2027-2028, alongside new Vera CPU, networking, and robotics integrations.
FreeToken Lets Frontier Mixture-of-Experts Models Run on Consumer GPUs With Dynamic CPU-GPU Co-Execution
UC Berkeley and MIT researchers, with Databricks co-founders Matei Zaharia and Ion Stoica, released an open-source engine that serves up to 753-billion-parameter MoE models on a single workstation GPU.
CNCF Graduates Kubeflow, Certifying the Kubernetes-Native AI Platform as Production-Ready
The Cloud Native Computing Foundation gave Kubeflow its top maturity tier, citing 6,600-plus contributors and 260 million PyPI downloads.