Skip to content

vLLM Upgrade Procedure

This document describes the complete procedure for upgrading spyre-inference to a newly released vLLM version. It covers bumping the pin in pyproject.toml, re-syncing the upstream test plugin, and triaging the breakages that the bump exposes.

AI Assistant Skills

This procedure is referenced by both Claude Code's /upgrade-vllm skill and IBM Bob's upgrade-vllm skill. When using these assistants, they will automatically follow this procedure and replace {VERSION} with the actual version number you specify.

The OOT Platform Trap

Read this before doing anything else — this pitfall can appear in any test category.

Our platform sets _enum = PlatformEnum.OOT. This means current_platform.is_cpu() returns False, even though we inherit from CpuPlatform and set device_type = "cpu". Every release, upstream adds new wrappers to Worker or Platform methods that call is_cpu() (or is_cuda_alike(), or similar predicates) to short-circuit to a no-op, and raise RuntimeError for anything else. We fall through to the raise.

Don't fix this by overriding is_cpu() to return True. vllm/model_executor/custom_op.py checks is_cpu() before is_out_of_tree() in its forward_* dispatch, and we rely on forward_oot for our custom-op overrides. Forcing is_cpu() reroutes every CustomOp to forward_cpu.

Do add a surgical override on TorchSpyreWorker (or TorchSpyrePlatform if the wrapper is on Platform). The fix is always the same shape: return whatever no-op the CPU branch would have returned (usually nullcontext() or None), and leave a comment explaining why you're not fixing is_cpu(). For example, the v0.23.0 bump introduced a memory-pool context wrapper in Worker.load_model:

def _maybe_get_memory_pool_context(self, tag: str) -> AbstractContextManager:
    # gpu_worker.Worker wraps weight loading in a memory-pool context that
    # short-circuits to nullcontext() when current_platform.is_cpu() is True.
    # Our platform reports OOT (so custom-op forward_oot dispatch works), so
    # the upstream check falls through and raises. Spyre weights live
    # on-device, not in a host-side cumem allocator, so a nullcontext is the
    # correct behaviour.
    return nullcontext()

The specific methods that need overrides change with every release — read the upstream diff rather than assuming the same set as last time. Search gpu_worker.py and platform.py in the new rev for is_cpu() / is_cuda_alike() / is_out_of_tree() guards that weren't there before.

When to Use

User-facing triggers: "bump vllm", "upgrade vllm", "update to the latest vllm release", "pull up the lower bound", "track vllm ", "rebase onto vllm main".

Don't use this for unpinning (vllm = "*") or for widening the support window beyond a single release (e.g. vllm>=0.22,<0.24) — those need a real version policy discussion. The current policy is: support exactly one minor release at a time, and bump the lower bound + upper cap together with the pinned rev.

Step 1: Pick the Target Rev

If the user named a rev, use it. Otherwise fetch the latest stable release:

gh api repos/vllm-project/vllm/releases --paginate \
  | python3 -c "import json,sys;[print(r['tag_name'],r['published_at']) for r in json.load(sys.stdin)[:10]]"

Tags look like vMAJOR.MINOR.PATCH (e.g. v0.23.0). Confirm "latest stable" with the user only if they were vague — otherwise just go.

Two places in pyproject.toml move together:

[project]
dependencies = [
    "torch-spyre",
    "vllm>=X.Y.Z,<X.(Y+1)",   # bump both halves
    "torch",
]

[tool.uv.sources]
vllm = [
  # Built from source with VLLM_TARGET_DEVICE=empty (no C kernels).
  { git = "https://github.com/vllm-project/vllm", rev = "v{VERSION}" },
]

vLLM builds from source on every platform (VLLM_TARGET_DEVICE=empty, no C kernels), so there's no prebuilt-wheel index to update — bump only the git rev and the [project] version constraint.

The runtime lower bound (vllm>=X.Y.Z) and the upper cap (<X.(Y+1)) exist because we only maintain compatibility with the latest release — any non-backwards-compatible change we land for the new rev will break earlier vLLMs, so they must be excluded from dependency resolution. The <X.(Y+1) cap excludes future minor releases that haven't been validated yet.

The version constraint in [project.dependencies] controls what downstream environments installing spyre-inference from a wheel will resolve to. It must agree with the pinned rev, or a wheel install will silently pull a different vLLM than we tested against.

Step 2: Rebuild & Re-sync

vLLM builds from source on every platform (~4–5 minutes with the empty backend). Run sequentially:

uv sync --group dev
uv run --no-sync sync-upstream-test-deps
uv lock                # sync-upstream-test-deps only rewrites tests/plugin/pyproject.toml
uv sync --group dev    # picks up the regenerated allow-list

sync-upstream-test-deps does not mirror upstream's test requirements. It enforces a curated allow-list and regenerates the [project].dependencies array in tests/plugin/pyproject.toml from it. The allow-list (and a reviewed EXCLUDED set) live in the script itself — tests/plugin/spyre_testing_plugin/sync_upstream_test_deps.py. The rule the allow-list encodes: only declare an upstream test dep here if (a) a test we actually collect/run needs it and (b) it is not already in vLLM's own runtime closure. The generated array is machine-owned — do not hand-edit it; change ALLOW_LIST/EXCLUDED and re-run.

The script pulls each allowed dep's version and extras verbatim from vLLM's requirements/test/cuda.in at the new rev, overlays any platform marker from the allow-list value (e.g. runai-model-streamer ... ; platform_machine == 'x86_64'), and fails (RC=1) if an ALLOW_LIST name has disappeared from cuda.in — reconcile the allow-list if so.

Triage the sync report. It classifies every cuda.in dep not in the allow-list into three buckets:

  • already provided by vLLM runtime — deps present in uv export --no-dev (torch, tokenizers, transformers, …). This is the automated runtime audit: it proves we aren't re-declaring what installing vLLM already gives us. Nothing to do.
  • excluded by policy — the reviewed-and-rejected set in EXCLUDED (model-family/eval/tooling deps no collected test exercises). Nothing to do.
  • NEW — a cuda.in dep this repo has never classified. This is the only bucket that needs action. For each, decide:
    • Does a test we collect or run import/need it, and is it absent from the runtime bucket? → add its normalized name to ALLOW_LIST (with a marker value if upstream gates it by platform), re-run the sync.
    • Otherwise → add it to EXCLUDED so future runs stay quiet.

A clean bump shows NEW: none. Confirm allow-list sufficiency after uv sync with a collect-only pass (no Spyre hardware needed):

uv run --no-sync pytest --collect-only -q -p no:cacheprovider -m "upstream"

A missing dep surfaces here as a collect-time ImportError naming the exact package — add that one dep and re-run. RC=0 with the expected test count means the allow-list covers collection.

Collection sufficiency is not runtime sufficiency. A collect-only pass only proves the deps imported at import time are present. A mandatory_pass test can still need a dep that is imported lazily inside a fixture or test body — e.g. the pooling embedding tests build an HF reference via hf_runner(model, is_sentence_transformer=True), which imports sentence-transformers only when the test actually runs (on Spyre hardware). Those deps pass collection while absent and then fail the run, so any allow-list entry justified by "a test we run needs it" (rather than collection) must be reasoned about separately — clarify with an inline comment on the ALLOW_LIST entry.

If uv sync fails with a dependency conflict that wasn't there before, the override list in pyproject.toml > [tool.uv] > override-dependencies may need to grow. Don't add overrides speculatively — only if a real conflict appears. Note that test-only exclusions belong in the plugin's allow-list (omit the name), not in these root overrides — the overrides are reserved for deps that would otherwise be pulled in transitively.

If the lock file needs to be deleted and regenerated from scratch (e.g. after a messy merge conflict resolution), uv lock alone will fail because building torch-spyre metadata requires SPYRE_COMMS_INSTALL_DIR. Use USE_SPYRE_CCL=0 uv lock instead — that env var skips the multi-Spyre comm library requirement and lets uv resolve without a real Spyre build environment.

After the sync, smoke-test the import:

uv run --no-sync python -c "import spyre_inference; print('ok')"

If this fails with ImportError from inside vLLM, the upstream API moved out from under one of our imports — grep -rn "<missing-symbol>" .venv/lib64/python3.12/site-packages/vllm/ to find where it went, then update our import.

Step 3: Triage in This Order

Don't run the full pytest first — vLLM startup is ~30s per test file, and the failures cluster by category. Run the cheapest categories first so you can fix-and-rerun quickly.

Always use uv run --no-sync for every test invocation — without --no-sync, uv re-resolves dependencies on every call and reverts any local torch-spyre install you may have done.

3a. Custom Ops (No vLLM Engine)

uv run --no-sync pytest tests/runtime/test_platform.py tests/custom_ops/test_mlp.py \
  tests/custom_ops/test_silu_and_mul.py tests/custom_ops/test_parallel_lm_head.py \
  tests/custom_ops/test_vocab_parallel_embedding.py -m "not upstream" --no-header

These exercise the linear/embedding/silu_and_mul ops against a CPU reference. Failures here usually mean a vLLM layer's constructor signature changed (new kwarg, renamed parameter) or a vocab_parallel_embedding-style helper was moved. Fix in spyre_inference/custom_ops/.

3b. Attention Backend

uv run --no-sync pytest tests/attention/test_spyre_attn.py -m "not upstream" --no-header

~10 minutes. If this fails, likely culprits:

  • AttentionImpl.__init__ signature drifted — check vllm/v1/attention/backends/*.py for the new contract.
  • AttentionMetadata/CommonAttentionMetadata field rename.
  • bind_kv_cache layout flipped (see 3e for the upstream KV cache layout note — same change, exposed twice).

3c. End-to-End vLLM (Single-Process)

uv run --no-sync pytest tests/e2e/test_vllm_spyre_next.py -m "not upstream" --no-header

This is the most common place to hit OOT platform traps (see above). Fix those with surgical TorchSpyreWorker overrides as described.

Not all failures here are OOT traps — also watch for constructor or method signature changes on classes we instantiate or inherit from, new abstract methods on base classes, and import path moves (symbol relocated to a different vLLM module).

3d. Distributed

uv run --no-sync pytest -m "distributed" --no-header

~5 minutes. Failures usually point to changes in vllm.distributed.parallel_state (group construction, device_communicator dispatch). The OOT branches in _platform_device_type and parallel_state.py:~448 use current_platform.device_name — if those branches moved to a different predicate, our device_name = "cpu" may stop being routed correctly.

3e. Upstream Tests

uv run --no-sync pytest -m "upstream" --no-header

These run upstream vLLM test files against our backend, filtered by tests/plugin/spyre_testing_plugin/upstream_tests.yaml. The most common failure mode is upstream test infra refactoring — the test file changes shape, our patch_backend_list fixture (in tests/plugin/spyre_testing_plugin/pytest_plugin.py) no longer matches. Read the upstream test file — check upstream_tests.yaml for the canonical path, or find it under ~/.cache/vllm-upstream-tests/worktree-<rev>/ (the subpath like tests/v1/attention/test_attention_backends.py may have moved between releases) — and diff its assumptions against what patch_backend_list does. The recurring trap is the KV-cache tensor layout: upstream has flipped between (2, num_blocks, …) and (num_blocks, 2, …) layouts more than once. Our fixture has to turn this tensor into (k_pages, v_pages) for SpyreAttentionImpl.forward, each a dense token-major [num_blocks, block_size, num_kv_heads, head_size] tensor with the slot axis pinned outermost by slot_major_kv_layout (the KV write scatters through a slot-major view, and the default layout silently writes the wrong rows) — which currently also means undoing the head-major transpose the upstream helper applies on the way out. Check the slicing dim against wherever the upstream test constructs the KV cache (e.g. a function like create_and_prepopulate_kv_cache — name may differ in the new rev).

Other upstream-side patterns to watch for:

  • New batch-spec entries that don't fit our block_size=64 / max_model_len=1024 defaults — extend params.allow in upstream_tests.yaml or skip the new variant.
  • The list of backends the upstream test iterates over (e.g. a module-level BACKENDS_TO_TEST constant — name may differ): we patch it to [CUSTOM] only; if the variable name or location moves, the patch silently no-ops and every backend runs.
  • Signature changes on the inner correctness-checking function that patch_backend_list wraps (e.g. _test_backend_correctness — name may differ): our wrapper forwards *args, **kwargs but pins block_size=64; if a new positional arg lands before block_size, that pin lands in the wrong slot.

If a test legitimately doesn't make sense for Spyre (e.g. it asserts triton-kernel-specific behaviour), block-list it in upstream_tests.yaml rather than papering over with a tolerance bump.

Step 4: Full Sweep + Format

Once each category is green individually, run the full suite to catch any cross-test interactions:

uv run --no-sync pytest --no-header --upstream

Roughly 16 minutes on a Spyre host. Then format:

bash format.sh

format.sh runs prek (ruff, ruff-format, ty, codespell, markdownlint, actionlint) and exits non-zero if anything needs to be added/committed. Do not auto-add .hypothesis/ — that directory is a per-run cache, gitignored.

What to Commit

Five files typically change for a vLLM bump:

  1. pyproject.toml — two things: the rev in the [tool.uv.sources] vllm entry, and the vllm>=X.Y.Z,<X.(Y+1) constraint in [project] dependencies.
  2. uv.lock — auto-regenerated by uv sync. Don't hand-edit.
  3. tests/plugin/pyproject.toml — auto-regenerated by sync-upstream-test-deps from the allow-list. Review the diff but don't hand-edit; to add or drop a dep, edit ALLOW_LIST/EXCLUDED in sync_upstream_test_deps.py and re-run. The allow-list edit itself is the fourth file that may change on a bump.
  4. spyre_inference/v1/worker/spyre_worker.py (sometimes) or spyre_inference/platform.py — surgical overrides for new upstream wrappers.
  5. tests/plugin/spyre_testing_plugin/pytest_plugin.py (sometimes) — fixture updates for upstream test-infra churn.

If your bump touched custom ops, attention, or distributed code in spyre_inference/, that's a sign the upstream API changed — make sure the change is necessary, not just "looked easier than figuring out the new contract."

The commit message convention in this repo uses gitmoji prefixes: :arrow_up: support vllm X.Y.Z is the standard form.

When the Bump is Too Big to Land in One PR

If the rev jump skips multiple minor versions and the failure list is long, it may be cleaner to land it in stages:

  1. Rev-only commit: bump rev, sync test plugin, override the obvious _enum = OOT traps, get the e2e test green. Skip the full upstream sweep for now.
  2. Upstream-test-infra commit: re-enable upstream tests by updating patch_backend_list and upstream_tests.yaml.
  3. Coverage growback: any tests that were temporarily block-listed get reviewed and either fixed, kept block-listed with a justification, or deleted.

Don't split the rev itself across commits — pyproject.toml and uv.lock need to move together or uv sync is non-deterministic.

Scope Guardrails

  • Do not bump the lower bound during a vLLM upgrade without explicit discussion. Lower-bound bumps require deleting compat code and pruning the test matrix.
  • Do not "improve" code outside the compat surface. A vLLM upgrade PR should be reviewable as one focused change.
  • If a transitive dep moves in a surprising way (uv sync output shows e.g. transformers/torch shifting unexpectedly), surface it to the user before continuing — don't silently accept.