vLLM Upgrade Procedure¶

This document describes the complete procedure for upgrading spyre-inference to a newly released vLLM version. It covers bumping the pin in pyproject.toml, re-syncing the upstream test plugin, and triaging the breakages that the bump exposes.

AI Assistant Skills

This procedure is referenced by both Claude Code's /upgrade-vllm skill and IBM Bob's upgrade-vllm skill. When using these assistants, they will automatically follow this procedure and replace {VERSION} with the actual version number you specify.

The OOT Platform Trap¶

Read this before doing anything else — this pitfall can appear in any test category.

Our platform sets _enum = PlatformEnum.OOT. This means current_platform.is_cpu() returns False, even though we inherit from CpuPlatform and set device_type = "cpu". Every release, upstream adds new wrappers to Worker or Platform methods that call is_cpu() (or is_cuda_alike(), or similar predicates) to short-circuit to a no-op, and raise RuntimeError for anything else. We fall through to the raise.

Don't fix this by overriding is_cpu() to return True. vllm/model_executor/custom_op.py checks is_cpu() before is_out_of_tree() in its forward_* dispatch, and we rely on forward_oot for our custom-op overrides. Forcing is_cpu() reroutes every CustomOp to forward_cpu.

Do add a surgical override on TorchSpyreWorker (or TorchSpyrePlatform if the wrapper is on Platform). The fix is always the same shape: return whatever no-op the CPU branch would have returned (usually nullcontext() or None), and leave a comment explaining why you're not fixing is_cpu(). For example, the v0.23.0 bump introduced a memory-pool context wrapper in Worker.load_model:

def _maybe_get_memory_pool_context(self, tag: str) -> AbstractContextManager:
    # gpu_worker.Worker wraps weight loading in a memory-pool context that
    # short-circuits to nullcontext() when current_platform.is_cpu() is True.
    # Our platform reports OOT (so custom-op forward_oot dispatch works), so
    # the upstream check falls through and raises. Spyre weights live
    # on-device, not in a host-side cumem allocator, so a nullcontext is the
    # correct behaviour.
    return nullcontext()

The specific methods that need overrides change with every release — read the upstream diff rather than assuming the same set as last time. Search gpu_worker.py and platform.py in the new rev for is_cpu() / is_cuda_alike() / is_out_of_tree() guards that weren't there before.

When to Use¶

User-facing triggers: "bump vllm", "upgrade vllm", "update to the latest vllm release", "pull up the lower bound", "track vllm ", "rebase onto vllm main".

Don't use this for unpinning (vllm = "*") or for widening the support window beyond a single release (e.g. vllm>=0.22,<0.24) — those need a real version policy discussion. The current policy is: support exactly one minor release at a time, and bump the lower bound + upper cap together with the pinned rev.

Step 1: Pick the Target Rev¶

If the user named a rev, use it. Otherwise fetch the latest stable release:

gh api repos/vllm-project/vllm/releases --paginate \
  | python3 -c "import json,sys;[print(r['tag_name'],r['published_at']) for r in json.load(sys.stdin)[:10]]"

Tags look like vMAJOR.MINOR.PATCH (e.g. v0.23.0). Confirm "latest stable" with the user only if they were vague — otherwise just go.

Two places in pyproject.toml move together:

[project]
dependencies = [
    "torch-spyre",
    "vllm>=X.Y.Z,<X.(Y+1)",   # bump both halves
    "torch",
]

[tool.uv.sources]
vllm = [
  # Built from source with VLLM_TARGET_DEVICE=empty (no C kernels).
  { git = "https://github.com/vllm-project/vllm", rev = "v{VERSION}" },
]

vLLM builds from source on every platform (VLLM_TARGET_DEVICE=empty, no C kernels), so there's no prebuilt-wheel index to update — bump only the git rev and the [project] version constraint.

The runtime lower bound (vllm>=X.Y.Z) and the upper cap (<X.(Y+1)) exist because we only maintain compatibility with the latest release — any non-backwards-compatible change we land for the new rev will break earlier vLLMs, so they must be excluded from dependency resolution. The <X.(Y+1) cap excludes future minor releases that haven't been validated yet.

The version constraint in [project.dependencies] controls what downstream environments installing spyre-inference from a wheel will resolve to. It must agree with the pinned rev, or a wheel install will silently pull a different vLLM than we tested against.

Step 2: Rebuild & Re-sync¶

vLLM builds from source on every platform (~4–5 minutes with the empty backend). Run sequentially:

uv sync --group dev
uv run --no-sync sync-upstream-test-deps
uv lock                # sync-upstream-test-deps only rewrites tests/plugin/pyproject.toml
uv sync --group dev    # picks up the regenerated allow-list

sync-upstream-test-deps does not mirror upstream's test requirements. It enforces a curated allow-list and regenerates the [project].dependencies array in tests/plugin/pyproject.toml from it. The allow-list (and a reviewed EXCLUDED set) live in the script itself — tests/plugin/spyre_testing_plugin/sync_upstream_test_deps.py. The rule the allow-list encodes: only declare an upstream test dep here if (a) a test we actually collect/run needs it and (b) it is not already in vLLM's own runtime closure. The generated array is machine-owned — do not hand-edit it; change ALLOW_LIST/EXCLUDED and re-run.

The script pulls each allowed dep's version and extras verbatim from vLLM's requirements/test/cuda.in at the new rev, overlays any platform marker from the allow-list value (e.g. runai-model-streamer ... ; platform_machine == 'x86_64'), and fails (RC=1) if an ALLOW_LIST name has disappeared from cuda.in — reconcile the allow-list if so.

Triage the sync report. It classifies every cuda.in dep not in the allow-list into three buckets:

already provided by vLLM runtime — deps present in uv export --no-dev (torch, tokenizers, transformers, …). This is the automated runtime audit: it proves we aren't re-declaring what installing vLLM already gives us. Nothing to do.
excluded by policy — the reviewed-and-rejected set in EXCLUDED (model-family/eval/tooling deps no collected test exercises). Nothing to do.
NEW — a cuda.in dep this repo has never classified. This is the only bucket that needs action. For each, decide:
- Does a test we collect or run import/need it, and is it absent from the runtime bucket? → add its normalized name to ALLOW_LIST (with a marker value if upstream gates it by platform), re-run the sync.
- Otherwise → add it to EXCLUDED so future runs stay quiet.

A clean bump shows NEW: none. Confirm allow-list sufficiency after uv sync with a collect-only pass (no Spyre hardware needed):

uv run --no-sync pytest --collect-only -q -p no:cacheprovider -m "upstream"

A missing dep surfaces here as a collect-time ImportError naming the exact package — add that one dep and re-run. RC=0 with the expected test count means the allow-list covers collection.

Collection sufficiency is not runtime sufficiency. A collect-only pass only proves the deps imported at import time are present. A mandatory_pass test can still need a dep that is imported lazily inside a fixture or test body — e.g. the pooling embedding tests build an HF reference via hf_runner(model, is_sentence_transformer=True), which imports sentence-transformers only when the test actually runs (on Spyre hardware). Those deps pass collection while absent and then fail the run, so any allow-list entry justified by "a test we run needs it" (rather than collection) must be reasoned about separately — clarify with an inline comment on the ALLOW_LIST entry.

If uv sync fails with a dependency conflict that wasn't there before, the override list in pyproject.toml > [tool.uv] > override-dependencies may need to grow. Don't add overrides speculatively — only if a real conflict appears. Note that test-only exclusions belong in the plugin's allow-list (omit the name), not in these root overrides — the overrides are reserved for deps that would otherwise be pulled in transitively.

If the lock file needs to be deleted and regenerated from scratch (e.g. after a messy merge conflict resolution), uv lock alone will fail because building torch-spyre metadata requires SPYRE_COMMS_INSTALL_DIR. Use USE_SPYRE_CCL=0 uv lock instead — that env var skips the multi-Spyre comm library requirement and lets uv resolve without a real Spyre build environment.

After the sync, smoke-test the import:

uv run --no-sync python -c "import spyre_inference; print('ok')"

If this fails with ImportError from inside vLLM, the upstream API moved out from under one of our imports — grep -rn "<missing-symbol>" .venv/lib64/python3.12/site-packages/vllm/ to find where it went, then update our import.

Step 2b: Re-sync the Vendored Host-Sampling Code¶

Spyre copies logits to the host for sampling, and spyre-inference vendors vLLM's CPU sampling code to do it fast. Every bump must re-check that code against the new upstream, even when nothing fails: the build and the tests only catch breakage, not upstream fixes we are missing.

Ours	Upstream (vLLM)	How ours differs
`csrc/cpu/cpu_types.hpp`, `cpu_types_{x86,vsx,vxe}.hpp`	same paths	Trimmed to `FP32Vec8`/`FP32Vec16`; every kept line is unchanged. Port fixes to those two types only.
`csrc/cpu/sampling_kernels.cpp`	same path (added in vllm#53913, after v0.28.0)	Gumbel noise is hashed per element instead of read from vLLM's 2^20-entry table, which leaves much of a large vocabulary unreachable (vllm#59786); `greedy_argmax` matches `torch.argmax` on NaN; inputs are checked. Port performance and correctness changes, never the table.
`csrc/core/registration.h`, `cmake/utils.cmake`, `cmake/cpu_extension.cmake`	same paths	Fixed ISA baselines (x86 AVX512 + AVX2, POWER10, z15) instead of `-march=native`, so wheels are portable.
`SpyreTopKTopPSampler.forward_native` (`spyre_inference/v1/sample/topk_topp_sampler.py`)	`TopKTopPSampler.forward_native` (`vllm/v1/sample/ops/topk_topp_sampler.py`)	Reimplements the upstream tail (sort-free top-k, the kernel draw); its comment names the vLLM version it mirrors.
`spyre_inference/v1/spec_decode/xpress_head.py`, `SpyreQwen3XPressForCausalLM` (`spyre_inference/models/qwen3_dflash.py`)	`XPressRefinerHead` and `Qwen3XPressForCausalLM` (`vllm/model_executor/models/qwen3_xpress.py`)	Construction, key mapping and mixer folding are upstream's; Spyre checks required head weights and the raw mixer shape. `_spyre_xpress_classes()` swaps the two module-level class names upstream builds by, so a rename upstream fails `test_upstream_xpress_construction_builds_the_spyre_classes`. The head subclass only no-ops `build_fused_buffers()` and swaps the Triton Jacobi kernels for compiled step helpers and Spyre weight layouts. `tests/spec_decode/xpress_reference.py` drives upstream's own `hidden_cache`/`refine_bias` as the oracle, so a change to upstream's math fails `test_xpress_upstream.py`.
`csrc/cpu/spec_decode_utils.cpp`, `spyre_inference/v1/spec_decode/rejection_sampler.py` and the platform kernel hook	The four rejection kernels in `csrc/cpu/spec_decode_utils.cpp` and their adapters in `vllm/utils/cpu_triton_utils.py` (unchanged from v0.32.0 `5df63a7203` through v0.33.0)	Native kernels in `_spyre_C`, with mixed int32/int64 inputs, floating-point parameter expansion, FP64 uniforms/noise, synthetic acceptance, placeholder and zero-probability rejection, and first-index/NaN argmax semantics. Recovery parallelizes token rows without vocabulary-sized score buffers. There is no Python rejection fallback; a missing or stale extension requires rebuilding. Re-sync alongside the other vendored CPU kernels. Upstream's opt-in FLy acceptance (`rejection_sample_method="fly"`, v0.33.0) is Triton-only and not ported.
`TorchSpyreModelRunner._sample` / `_bookkeeping_sync` (`spyre_inference/v1/worker/spyre_model_runner.py`)	The same methods in `vllm/v1/worker/gpu_model_runner.py`, as patched by vllm#62688 (unmerged)	Temporary port for #1420: restores the full state of CPU generators around sampling and hides them from upstream's `set_offset` rewind, which CPU generators lack. Delete both overrides once the pin includes vllm#62688; `tests/runtime/test_sampling_fallback.py::test_upstream_rewind_still_needs_the_cpu_generator_port` fails once upstream restores CPU generators itself. V1 only: MRV2 samples from `(seed, position)` and keeps no generator state.
The sampler patch in `TorchSpyreModelRunner.__init__`	`Sampler` (`vllm/v1/sample/sampler.py`), its construction and `RejectionSampler(self.sampler)` (`vllm/v1/worker/gpu_model_runner.py`)	Relies on `Sampler` calling `self.greedy_sample` and `self.topk_topp_sampler`, and on the rejection sampler holding the same `Sampler` object.
`allow_xpress_on_v1()` (`spyre_inference/v1/spec_decode/config.py`)	`VllmConfig._get_v1_model_runner_unsupported_features` (`vllm/config/vllm.py`)	Drops "xpress speculative decoding" and "mixed sliding/full dflash drafts": Spyre drives both through its V1 DFlash proposer. This is the config-level half of the decoder-layer workaround below. Retain while V1 is supported; delete when speculative decoding moves to V2 (#1575) and V1 is removed (#1245, part of #906).
`SpyreDFlashQwen3DecoderLayer.__init__` (`spyre_inference/models/qwen3_dflash.py`)	`_resolve_layer_attention`'s V2 check (`vllm/model_executor/models/qwen3_dflash.py`)	Stubs the module's `get_current_vllm_config` to report `use_v2_model_runner=True` during layer construction, allowing mixed sliding/full drafts on V1. Spyre gives these layers a shared full-allocation KV group (`SpyreGemma4ForCausalLM.get_kv_cache_groups`) and splits attention metadata by window. Other current-config reads in the layer's construction see the stub too (currently only `DFlashQwen3Attention`'s ROCm branch); audit them on each vLLM upgrade. Delete once these drafts use V2 (#1575) and V1 is removed (#1245, part of #906).
`use_spyre_dflash_proposer()` (`spyre_inference/v1/spec_decode/dflash.py`)	The `DFlashProposer` import, construction and `isinstance` checks in `vllm/v1/worker/gpu_model_runner.py`	Temporarily replaces the module's constructor during runner initialization. The replacement remains an upstream `DFlashProposer` subclass so later checks in `_prepare_inputs` and `sample_tokens` still pass after restoration.

Diff the upstream side between the current pin and the target:

cd <vllm checkout> && git fetch --tags
git diff v<current> v{VERSION} --stat -- \
  csrc/cpu/cpu_types.hpp csrc/cpu/cpu_types_x86.hpp csrc/cpu/cpu_types_vsx.hpp \
  csrc/cpu/cpu_types_vxe.hpp csrc/cpu/sampling_kernels.cpp \
  csrc/cpu/spec_decode_utils.cpp csrc/cpu/torch_bindings.cpp csrc/core/registration.h \
  cmake/cpu_extension.cmake cmake/utils.cmake \
  vllm/v1/sample/ops/topk_topp_sampler.py vllm/v1/sample/sampler.py \
  vllm/v1/sample/rejection_sampler.py vllm/utils/cpu_triton_utils.py \
  vllm/v1/worker/gpu_model_runner.py

For each changed file, port what applies and update the version in the forward_native comment. If the new vLLM pins a different torch, move every torch pin in pyproject.toml together: the extension links libtorch, so a mismatch fails at import. The list (six places, including the ppc64le source build) is in §3b of the upgrade-torch-spyre skill, and torch-spyre must support the new torch as well. uv sync rebuilds the extension because csrc/ and pyproject.toml are in its cache keys.

Then check the kernels and ordinary-sampling PyTorch fallback (CPU only, no card needed). The rejection sampler requires the built extension:

uv run --no-sync pytest tests/runtime/test_sampling_kernels.py \
  tests/runtime/test_sampling_fallback.py tests/runtime/test_spyre_topk_sampler.py \
  tests/spec_decode/test_rejection_sampler.py --no-header

test_sampling_kernels.py checks the kernels against PyTorch baselines token for token: torch.argmax for greedy, and a PyTorch implementation of the same hashed Gumbel draw.
test_sampling_fallback.py forces the fallback and checks it picks the same tokens as upstream's stock Sampler, which catches upstream sampler changes we haven't mirrored.
test_spyre_topk_sampler.py checks the sort-free top-k/top-p against upstream's sort.
test_rejection_sampler.py exercises the native kernels through vLLM's dispatchers, including mixed dtypes, FP64 noise, synthetic acceptance and target-distribution checks.

Step 3: Triage in This Order¶

Don't run the full pytest first — vLLM startup is ~30s per test file, and the failures cluster by category. Run the cheapest categories first so you can fix-and-rerun quickly.

Always use uv run --no-sync for every test invocation — without --no-sync, uv re-resolves dependencies on every call and reverts any local torch-spyre install you may have done.

3a. Custom Ops (No vLLM Engine)¶

uv run --no-sync pytest tests/runtime/test_platform.py tests/custom_ops/test_mlp.py \
  tests/custom_ops/test_silu_and_mul.py tests/custom_ops/test_parallel_lm_head.py \
  tests/custom_ops/test_vocab_parallel_embedding.py -m "not upstream" --no-header

These exercise the linear/embedding/silu_and_mul ops against a CPU reference. Failures here usually mean a vLLM layer's constructor signature changed (new kwarg, renamed parameter) or a vocab_parallel_embedding-style helper was moved. Fix in spyre_inference/custom_ops/.

3b. Attention Backend¶

uv run --no-sync pytest tests/attention/test_spyre_attn.py -m "not upstream" --no-header

~10 minutes. If this fails, likely culprits:

AttentionImpl.__init__ signature drifted — check vllm/v1/attention/backends/*.py for the new contract.
AttentionMetadata/CommonAttentionMetadata field rename.
bind_kv_cache layout flipped (see 3e for the upstream KV cache layout note — same change, exposed twice).

3c. End-to-End vLLM (Single-Process)¶

uv run --no-sync pytest tests/e2e/test_vllm_spyre_next.py -m "not upstream" --no-header

This is the most common place to hit OOT platform traps (see above). Fix those with surgical TorchSpyreWorker overrides as described.

Not all failures here are OOT traps — also watch for constructor or method signature changes on classes we instantiate or inherit from, new abstract methods on base classes, and import path moves (symbol relocated to a different vLLM module).

3d. Distributed¶

uv run --no-sync pytest -m "distributed" --no-header

~5 minutes. Failures usually point to changes in vllm.distributed.parallel_state (group construction, device_communicator dispatch). The OOT branches in _platform_device_type and parallel_state.py:~448 use current_platform.device_name — if those branches moved to a different predicate, our device_name = "cpu" may stop being routed correctly.

3e. Upstream Tests¶

uv run --no-sync pytest -m "upstream" --no-header

These run upstream vLLM test files against our backend, filtered by tests/plugin/spyre_testing_plugin/upstream_tests.yaml. The most common failure mode is upstream test infra refactoring — the test file changes shape, our patch_backend_list fixture (in tests/plugin/spyre_testing_plugin/pytest_plugin.py) no longer matches. Read the upstream test file — check upstream_tests.yaml for the canonical path, or find it under ~/.cache/vllm-upstream-tests/worktree-<rev>/ (the subpath like tests/v1/attention/test_attention_backends.py may have moved between releases) — and diff its assumptions against what patch_backend_list does. The recurring trap is the KV-cache tensor layout: upstream has flipped between (2, num_blocks, …) and (num_blocks, 2, …) layouts more than once. Our fixture has to turn this tensor into (k_pages, v_pages) for SpyreAttentionImpl.forward, each a dense head-major [num_blocks, num_kv_heads, block_size, head_size] tensor placed with head_major_kv_layout (the kernels gather a page along device dim 0, and the default layout splits it), and to translate the raw slot mapping upstream hands do_kv_cache_update into the impl's per-head kv_write_index. Check the slicing dim against wherever the upstream test constructs the KV cache (e.g. a function like create_and_prepopulate_kv_cache — name may differ in the new rev).

Other upstream-side patterns to watch for:

New batch-spec entries that don't fit our block_size=64 / max_model_len=1024 defaults — extend params.allow in upstream_tests.yaml or skip the new variant.
The list of backends the upstream test iterates over (e.g. a module-level BACKENDS_TO_TEST constant — name may differ): we patch it to [CUSTOM] only; if the variable name or location moves, the patch silently no-ops and every backend runs.
Signature changes on the inner correctness-checking function that patch_backend_list wraps (e.g. _test_backend_correctness — name may differ): our wrapper forwards *args, **kwargs but pins block_size=64; if a new positional arg lands before block_size, that pin lands in the wrong slot.

If a test legitimately doesn't make sense for Spyre (e.g. it asserts triton-kernel-specific behaviour), block-list it in upstream_tests.yaml rather than papering over with a tolerance bump.

Step 4: Full Sweep + Format¶

Once each category is green individually, run the full suite to catch any cross-test interactions:

uv run --no-sync pytest --no-header --upstream

Roughly 16 minutes on a Spyre host. Then format:

bash format.sh

format.sh runs prek (ruff, ruff-format, ty, codespell, markdownlint, actionlint) and exits non-zero if anything needs to be added/committed. Do not auto-add .hypothesis/ — that directory is a per-run cache, gitignored.

What to Commit¶

Five files typically change for a vLLM bump, plus the vendored sampling code when upstream touched it:

pyproject.toml — two things: the rev in the [tool.uv.sources] vllm entry, and the vllm>=X.Y.Z,<X.(Y+1) constraint in [project] dependencies.
uv.lock — auto-regenerated by uv sync. Don't hand-edit.
tests/plugin/pyproject.toml — auto-regenerated by sync-upstream-test-deps from the allow-list. Review the diff but don't hand-edit; to add or drop a dep, edit ALLOW_LIST/EXCLUDED in sync_upstream_test_deps.py and re-run. The allow-list edit itself is the fourth file that may change on a bump.
spyre_inference/v1/worker/spyre_worker.py (sometimes) or spyre_inference/platform.py — surgical overrides for new upstream wrappers.
tests/plugin/spyre_testing_plugin/pytest_plugin.py (sometimes) — fixture updates for upstream test-infra churn.
csrc/cpu/, cmake/ and spyre_inference/v1/sample/ (when upstream changed them) — the Step 2b re-sync of the vendored sampling code.

If your bump touched custom ops, attention, or distributed code in spyre_inference/, that's a sign the upstream API changed — make sure the change is necessary, not just "looked easier than figuring out the new contract."

The commit message convention in this repo uses gitmoji prefixes: :arrow_up: support vllm X.Y.Z is the standard form.

When the Bump is Too Big to Land in One PR¶

If the rev jump skips multiple minor versions and the failure list is long, it may be cleaner to land it in stages:

Rev-only commit: bump rev, sync test plugin, override the obvious _enum = OOT traps, get the e2e test green. Skip the full upstream sweep for now.
Upstream-test-infra commit: re-enable upstream tests by updating patch_backend_list and upstream_tests.yaml.
Coverage growback: any tests that were temporarily block-listed get reviewed and either fixed, kept block-listed with a justification, or deleted.

Don't split the rev itself across commits — pyproject.toml and uv.lock need to move together or uv sync is non-deterministic.

Scope Guardrails¶

Do not widen the supported vLLM range without explicit discussion. For a normal upgrade, bump the lower bound and upper cap together with the pinned rev, as in Step 1.
Do not "improve" code outside the compat surface. A vLLM upgrade PR should be reviewable as one focused change.
If a transitive dep moves in a surprising way (uv sync output shows e.g. transformers/torch shifting unexpectedly), surface it to the user before continuing — don't silently accept.