Top 100 vLLM Repositories

Back to Home

Ranking

Ranking	Project Name	Stars	Forks	Language	Open Issues	Description	Last Commit
1	vllm	80,529	16,997	Python	1984	A high-throughput and memory-efficient inference and serving engine for LLMs	2026-05-20
2	pi	51,858	6,185	TypeScript	24	AI agent toolkit: coding agent CLI, unified LLM API, TUI & web UI libraries, Slack bot, vLLM pods	2026-05-20
3	litellm	47,637	8,184	Python	1343	Python SDK, Proxy Server (AI Gateway) to call 100+ LLM APIs in OpenAI (or native) format, with cost tracking, guardrails, loadbalancing and logging. [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropi...	2026-05-20
4	llama-cookbook	18,329	2,730	Jupyter Notebook	19	Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We also show you how to solve end to end problems using Llama model f...	2026-05-19
5	nano-vllm	13,520	2,114	Python	26	Nano vLLM	2026-04-26
6	OpenRLHF	9,526	944	Python	295	An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray (PPO & DAPO & REINFORCE++ & VLM & TIS & vLLM & Ray & Async RL)	2026-05-15
7	inference	9,308	826	Python	25	Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready...	2026-05-19
8	ipex-llm	8,806	1,426	Python	1213	Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete...	2026-01-28
9	AI-Research-SKILLs	8,697	665	TeX	8	Comprehensive open-source library of AI research and engineering skills for any AI model. Package the skills and your claude code/codex/gemini agent will be an AI research agent with full horsepower. ...	2026-04-28
10	LMCache	8,310	1,184	Python	102	Supercharge Your LLM with the Fastest KV Cache Layer	2026-05-20
11	UltraRAG	5,550	421	Python	6	A Low-Code MCP Framework for Building Complex and Innovative RAG Pipelines	2026-05-20
12	kserve	5,491	1,490	Go	538	Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes	2026-05-20
13	Mooncake	5,378	764	C++	310	Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.	2026-05-20
14	Awesome-LLM-Inference	5,229	375	Python	0	📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉	2026-04-20
15	sparrow	5,158	516	Python	0	Structured data extraction and instruction calling with ML, LLM and Vision LLM	2026-05-19
16	gpustack	5,032	529	Python	527	A GPU cluster manager that configures and orchestrates inference engines like vLLM and SGLang for high-performance AI model deployment.	2026-05-19
17	cube-studio	5,014	880	Python	33	cube studio开源云原生一站式机器学习/深度学习/大模型AI平台，mlops算法链路全流程，算力租赁平台，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU虚拟化，边缘计算，标注平台自动化标注，deepseek等大模型sft微调/奖励模型/强化学习训练，vllm/ollama/mindie大模型多机推理，私有知识库，AI模型市场，支持...	2026-05-20
18	vllm-omni	4,820	957	Python	444	A framework for efficient model inference with omni-modality models	2026-05-20
19	semantic-router	4,195	674	Go	106	System Level Intelligent Router for Mixture-of-Models at Cloud, Data Center and Edge	2026-05-20
20	tiny-llm	4,193	318	Python	7	A course of learning LLM inference serving on Apple Silicon for systems engineers: build a tiny vLLM + Qwen.	2026-05-18
21	llama-swap	4,165	322	Go	39	Reliable model swapping for any local OpenAI/Anthropic compatible server - llama.cpp, vllm, etc	2026-05-18
22	FastDeploy	3,686	744	Python	287	High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle	2026-05-20
23	llm-compressor	3,261	518	Python	61	Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM	2026-05-20
24	ramalama	2,846	339	Python	68	RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of con...	2026-05-18
25	Model-Optimizer	2,721	403	Python	55	A unified library of SOTA model optimization techniques like quantization, pruning, distillation, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks lik...	2026-05-20
26	production-stack	2,344	404	Python	98	vLLM’s reference system for K8S-native cluster-wide deployment with community-driven performance optimization	2026-05-19
27	InfraTech	2,314	197	Jupyter Notebook	0	分享AI Infra知识&代码练习：PyTorch/vLLM/SGLang框架入门⚡️、性能加速🚀、大模型基础🧠、AI软硬件🔧等	2026-05-08
28	cube-studio	2,184	167	Python	22	cube studio开源云原生一站式机器学习/深度学习/大模型AI平台/MaaS/mlops/人工智能平台/训推平台，算法全链路流程，多租户，算力租赁平台，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务，VGPU虚拟化，云边端协同，边缘计算，自动化标注平台，deepseek等大模型sft微调/奖励模型/强化学习训练，vllm/ollama/mindie大模型多机推理，私...	2026-05-19
29	Aix-DB	2,121	412	JavaScript	3	Aix-DB 基于 LangChain/LangGraph 框架，结合 MCP Skills 多智能体协作架构，实现自然语言到数据洞察的端到端转换。	2026-04-15
30	vllm-ascend	2,105	1,238	C++	1486	Community maintained hardware plugin for vLLM on Ascend	2026-05-20
31	cascadeflow	1,835	444	Python	1	Cascading runtime for AI agents. Optimize cost, latency, quality, and policy decisions inside the agent loop.	2026-05-16
32	turboquant	1,406	173	Python	8	TurboQuant: Near-optimal KV cache quantization for LLM inference (3-bit keys, 2-bit values) with Triton kernels + vLLM integration	2026-03-27
33	auto-round	1,403	134	Python	85	A SOTA quantization algorithm for high-accuracy low-bit LLM inference, seamlessly optimized for CPU/XPU/CUDA, with multi-datatype support and full compatibility with vLLM, SGLang, and Transformers.	2026-05-20
34	spark-vllm-docker	1,397	249	Shell	53	Docker configuration for running VLLM on dual DGX Sparks	2026-05-19
35	vllm-mlx	1,214	173	Python	36	OpenAI and Anthropic compatible server for Apple Silicon. Run LLMs and vision-language models (Llama, Qwen-VL, LLaVA) with continuous batching, MCP tool calling, and multimodal support. Native MLX bac...	2026-05-17
36	BricksLLM	1,204	95	Go	19	🔒 Enterprise-grade API gateway that helps you monitor and impose cost or rate limits per API key. Get fine-grained access control and monitoring per user, application, or environment. Supports OpenAI...	2025-01-05
37	kubeai	1,199	125	Go	82	AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text.	2026-03-31
38	vllm-metal	1,170	134	Python	10	Community maintained hardware plugin for vLLM on Apple Silicon	2026-05-20
39	GPTQModel	1,153	184	Python	42	LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.	2026-05-20
40	index-tts-vllm	1,151	160	Python	151	Added vLLM support to IndexTTS for faster inference.	2026-04-13
41	prometheus-eval	1,083	71	Python	12	Evaluate your LLM's response with Prometheus and GPT4 💯	2025-04-25
42	kvcached	1,043	114	Python	45	Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond	2026-05-16
43	club-3090	1,015	50	Python	10	Community recipes for serving LLMs on RTX 3090/CUDA gpus. Multi-engine (vLLM, llama.cpp, SGLang) and model-agnostic. Currently shipping Qwen3.6-27B Qwen3.6 35B Gemma 4 26B Gemma 4 31B configs for 1× a...	2026-05-20
44	vllm-studio	983	77	TypeScript	0	Control panel for VLLM, Sglang, llama.cpp, exllamav3	2026-05-19
45	InferenceX	981	171	Python	101	Open Source Continuous Inference Benchmarking Qwen3.5, DeepSeek, GPTOSS - GB200 NVL72 vs MI355X vs B200 vs GB300 NVL72 vs H100 & soon™ TPUv6e/v7/Trainium2/3	2026-05-20
46	EduChat	935	105	Jupyter Notebook	4	An open-source educational chat model from ICALK, East China Normal University. 开源中英教育对话大模型。(通用基座模型，GPU部署，数据清理) 致敬: LLaMA, MOSS, BELLE, Ziya, vLLM	2025-07-18
47	llm_note	882	87	Python	0	LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.	2026-05-10
48	recipes	803	276	JavaScript	26	Common recipes to run vLLM	2026-05-20
49	llmcord	800	191	Python	3	Make Discord your LLM frontend - Supports any OpenAI compatible API (Ollama, xAI, Gemini, OpenRouter and more)	2026-02-22
50	BambooAI	777	83	Python	15	A Python library powered by Language Models (LLMs) for conversational data discovery and analysis.	2026-04-01
51	llm-server-docs	770	57	-	2	End-to-end documentation to set up your own local & fully private LLM server on Debian. Equipped with chat, web search, RAG, model management, MCP servers, image generation, and TTS.	2026-03-02
52	MinivLLM	746	101	Python	12	Based on Nano-vLLM, a simple replication of vLLM with self-contained paged attention and flash attention implementation	2026-03-16
53	local_ai_ocr	737	186	Python	0	An local, offline (after initial setup), portable OCR software that can process images and PDF files, using DeepSeek-OCR AI (running directly on your machine).	2026-04-25
54	rvllm	721	67	Rust	4	rvLLM: High-performance LLM inference in Rust. Drop-in vLLM replacement.	2026-05-17
55	LightCompress	715	80	Python	42	[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.	2026-05-14
56	candle-vllm	663	81	Rust	23	Efficent platform for inference and serving local LLMs including an OpenAI compatible API server.	2026-05-19
57	qwen-vllm	646	92	Python	8	通义千问VLLM推理部署DEMO	2024-03-28
58	WeDLM	644	45	Python	4	WeDLM: The fastest diffusion language model with standard causal attention and native KV cache compatibility, delivering real speedups over vLLM-optimized baselines.	2026-03-03
59	vidur	606	115	Python	16	Accurate, large-scale, and extensible simulator for LLM inference Systems	2025-07-25
60	FlashTTS	603	76	Python	47	基于SparkTTS、OrpheusTTS等模型，提供高质量中文语音合成与声音克隆服务。	2025-05-18
61	vllm-turboquant	590	102	Python	2	vLLM TurboQuant	2026-04-16
62	efficientsam3	581	43	Jupyter Notebook	10	EfficientSAM3 compresses SAM3 into lightweight, edge-friendly models via progressive knowledge distillation for fast promptable concept segmentation and tracking.	2026-04-22
63	crater	515	65	TypeScript	46	Crater is a cloud-native AI training & inference platform.	2026-05-19
64	restai	506	101	Python	2	RESTai is an AIaaS (AI as a Service) open-source platform. Supports many public and local LLM suported by Ollama/vLLM/etc. Precise embeddings usage, tuning, analytics etc. Built-in image/audio generat...	2026-05-19
65	vllm-cli	497	28	Python	3	A command-line interface tool for serving LLM using vLLM.	2026-01-25
66	async_cosyvoice	493	66	Jupyter Notebook	99	使用vllm加速cosyvoice2的推理	2025-04-26
67	vllm-playground	458	62	JavaScript	3	A modern web interface for managing and interacting with vLLM servers (www.github.com/vllm-project/vllm). Supports both GPU and CPU modes, with special optimizations for macOS Apple Silicon and enterp...	2026-04-07
68	ome	451	79	Go	42	Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton	2026-05-20
69	speculators	445	85	Python	19	A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM	2026-05-19
70	worker-vllm	443	362	Python	31	The RunPod worker template for serving our large language model endpoints. Powered by vLLM.	2026-05-15
71	ArcticInference	436	61	Python	48	ArcticInference: vLLM plugin for high-throughput, low-latency inference	2026-04-23
72	DeepSeek-OCR-WebUI	428	93	TypeScript	15	🎨 Ready-to-use DeepSeek-OCR Web UI | Modern Interface | 7 Recognition Modes | Batch Processing | Real-time Logging | Fully Responsive	2026-02-20
73	vLLM-Kunlun	416	72	Python	47	vLLM Kunlun (vllm-kunlun) is a community-maintained hardware plugin designed to seamlessly run vLLM on the Kunlun XPU.	2026-05-19
74	vllm-gfx906	414	87	Python	23	vLLM for AMD gfx906 GPUs, e.g. Radeon VII / MI50 / MI60	2026-02-20
75	topicGPT	399	64	Python	3	TopicGPT: A Prompt-Based Framework for Topic Modeling [NAACL'24]	2026-05-15
76	super-json-mode	397	14	Jupyter Notebook	1	Low latency JSON generation using LLMs ⚡️	2024-03-10
77	RoboBrain	391	26	Python	9	[CVPR 2025] RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete. Official Repository.	2025-10-13
78	chatterbox-vllm	378	58	Python	15	VLLM Port of the Chatterbox TTS model	2025-10-18
79	Lvllm	364	34	Python	1	LvLLM is a special NUMA extension of vllm that makes full use of CPU and memory resources, reduces GPU memory requirements, and features an efficient GPU parallel and NUMA parallel architecture, suppo...	2026-04-28
80	OneCompression	363	13	Python	3	Python package for LLM compression	2026-05-20
81	flex-nano-vllm	345	19	Python	1	FlexAttention based, minimal vllm-style inference engine for fast Gemma 2 inference.	2025-11-02
82	vllm_backend	342	45	Python	0		2026-05-08
83	amd-strix-halo-vllm-toolboxes	341	48	Python	29		2026-05-15
84	tpu-inference	335	192	Python	58	TPU inference for vLLM, with unified JAX and PyTorch support.	2026-05-20
85	swiftLLM	330	38	Python	3	A tiny yet powerful LLM inference system tailored for researching purpose. vLLM-equivalent performance with only 2k lines of code (2% of vLLM).	2025-06-10
86	TinyLLM	329	37	JavaScript	7	Setup and run a local LLM and Chatbot using consumer grade hardware.	2025-11-23
87	KVDesk	329	50	TypeScript	20	Versatile Almost Local, Eventually Reasonable Assistant 🔫	2026-05-15
88	llmaz	305	47	Go	39	☸️ Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!	2026-01-26
89	Flash-RL	300	23	Python	7	Implementation for FP8/INT8 Rollout for RL training without performence drop.	2025-11-07
90	1Cat-vLLM	298	60	Python	18	vLLM fork for Tesla V100 (SM70) with AWQ 4-bit support, CUDA 12.8 build flow, and validated Qwen3.5 27B/35B deployment on multi-GPU V100.	2026-05-18
91	openmodelz	282	26	Go	20	Autoscale LLM (vLLM, SGLang, LMDeploy) inferences on Kubernetes (and others)	2023-11-03
92	apex	278	49	TypeScript	66	AI-powered offensive security testing using autonomous agents, directly in your terminal.	2026-05-19
93	unified-cache-management	277	74	Python	17	Persist and reuse KV Cache to speedup your LLM.	2026-05-20
94	smg	269	78	Rust	88	Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across SGLang, vLLM, TRT-LLM, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, ...	2026-05-20
95	vllm-swift	260	17	Python	7	vLLM Metal plugin powered by mlx-swift — high-performance LLM inference on Apple Silicon	2026-05-18
96	llm-benchmark	258	45	Python	3	LLM 并发性能测试工具，支持自动化压力测试和性能报告生成。	2025-12-10
97	Namo-R1	255	25	Python	14	A CPU Realtime VLM in 500M. Surpassed Moondream2 and SmolVLM. Training from scratch with ease.	2025-04-22
98	mimo2codex	254	23	TypeScript	3	让最新版 OpenAI Codex CLI / Codex 桌面端接入主流大模型的本地代理。内置 小米 MiMo V2.5/DeepSeek V4 Pro，并提供通用 provider 机制，OpenAI Chat Completions 兼容（Qwen / GLM / Kimi / 本地 vLLM / Ollama / LM Studio …）或原生 Responses API（...	2026-05-20
99	gpt_server	253	22	Python	3	gpt_server是一个用于生产级部署LLMs、Embedding、Reranker、ASR、TTS、文生图、图片编辑和文生视频的开源框架。	2026-05-09
100	FlashCosyVoice	248	25	Python	3	FlashCosyVoice: A lightweight vLLM implementation built from scratch for CosyVoice.	2026-02-25