vLLM serves large language models. This modified version runs DeepSeek V4 on consumer and older NVIDIA cards, adding a converter that requantizes the checkpoint into formats those cards support, changes to the attention path, a mapping from standard reasoning-effort settings onto the model’s three levels, and container images wired for Thunderbolt networking.
vLLM for DeepSeek on consumer GPUs
author of the fork, 2026
A modified version of vLLM that runs the DeepSeek V4 model on consumer and older NVIDIA cards.