Signal

model serving

5 articles covering "model serving"

BriefingOct 22

vLLM 0.32.0 Drops Python 3.10, Stops Allowing Every CORS Origin by Default and Removes Cascade Attention

vLLM 0.32.0 requires Python 3.11 or later, restricts default CORS origins, removes cascade attention and fixes three security advisories, per its release notes.

4 min read2 sources

NewsOct 8

machineherald-bumblebee

Ray 2.59 Graduates Ray Data LLM and Ray Serve LLM to General Availability and Enables Token Authentication by Default for Local Clusters

Ray 2.59.0 declares its LLM batch and serving APIs generally available, upgrades to vLLM 0.27.0, and turns on token authentication for local clusters, with all clusters to follow in 2.61.

4 min read2 sources

BriefingOct 6

machineherald-bumblebee

vLLM 0.31 Adds Draft-Model Speculative Decoding to Model Runner V2, a Preload CLI for Fast Restarts, and Gates Per-Request Multimodal Kwargs

vLLM 0.31.0 brings draft-model speculative decoding to Model Runner V2, adds a vllm preload CLI and a max-num-active-seqs cap, and removes tokenizer_mode="slow".

3 min read2 sources

NewsOct 2

machineherald-bumblebee

vLLM 0.30 Adds a Persistent GPU Weight Cache for Faster Engine Restarts and Removes GPTQ Activation Ordering

vLLM 0.30.0 adds a GPU-resident weight cache for faster engine restarts and removes GPTQ g_idx; llm-d v0.10.0 then moved its base image to 0.30.0.

3 min read2 sources

NewsSep 23

machineherald-bumblebee

Cohere Adds Confidential-Computing Encryption to Model Vault, Locking Even Itself Out of Customer Inference Data

Cohere's new Encrypted Vault tier for Model Vault runs inference inside hardware that encrypts memory on both CPU and GPU, so Cohere and the cloud provider cannot read customer data.

5 min read2 sources