Top 100 Serving Repositories

Back to Home

Ranking

Ranking	Project Name	Stars	Forks	Language	Open Issues	Description	Last Commit
1	vllm	80,680	17,082	Python	1988	A high-throughput and memory-efficient inference and serving engine for LLMs	2026-05-22
2	ray	42,624	7,597	Python	2842	Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.	2026-05-22
3	FastChat	39,477	4,792	Python	876	An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.	2026-05-01
4	sglang	28,102	6,052	Python	655	SGLang is a high-performance serving framework for large language models and multimodal models.	2026-05-22
5	dufs	10,172	538	Rust	2	A file server that supports static serving, uploading, searching, accessing control, webdav...	2026-05-20
6	serve	9,852	702	TypeScript	132	Static file serving and directory listing	2026-05-21
7	PowerInfer	9,479	575	C++	123	High-speed Large Language Model Serving for Local Deployment	2026-05-11
8	AppImageLauncher	8,006	338	C++	183	Helper application for Linux distributions serving as a kind of "entry point" for running and integrating AppImages	2026-03-09
9	lmdeploy	7,869	698	Python	526	LMDeploy is a toolkit for compressing, deploying, and serving LLMs.	2026-05-21
10	dynamo	6,932	1,146	Rust	200	A Datacenter Scale Distributed Inference Serving Framework	2026-05-22
11	clearml	6,701	778	Python	542	ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution	2026-05-20
12	serving	6,352	2,199	C++	56	A flexible, high-performance serving system for machine learning models	2026-05-12
13	serving	6,052	1,227	Go	153	Kubernetes-based, scale-to-zero, request-driven compute	2026-05-21
14	flashinfer	5,655	992	Python	357	FlashInfer: Kernel Library for LLM Serving	2026-05-21
15	ToolBench	5,648	485	Python	153	[ICLR'24 spotlight] An open platform for training, serving, and evaluating large language model for tool learning.	2025-05-21
16	volcano	5,589	1,370	Go	420	A Cloud Native Batch System (Project under CNCF)	2026-05-22
17	Mooncake	5,395	770	C++	315	Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.	2026-05-22
18	YCSB	5,211	2,325	Java	270	Yahoo! Cloud Serving Benchmark	2026-04-15
19	SPTAG	4,998	619	C++	119	A distributed approximate nearest neighborhood search (ANN) library which provides a high quality vector index build, search and distributed online serving toolkits for large scale vector search scena...	2026-05-20
20	LLM-engineer-handbook	4,931	696	-	0	A curated list of Large Language Model resources, covering model training, serving, fine-tuning, and building LLM applications.	2025-08-18
21	RouteLLM	4,926	382	Python	28	A framework for serving and evaluating LLM routers - save LLM costs without compromising quality	2024-08-10
22	seldon-core	4,752	866	Go	131	An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models	2026-03-23
23	esProc	4,681	362	Java	44	esProc SPL is a JVM-based programming language designed for structured data computation, serving as both a data analysis tool and an embedded computing engine.	2026-05-21
24	Deep-Learning-in-Production	4,379	690	-	1	In this repository, I will share some useful notes and references about deploying deep learning-based models in production.	2024-11-09
25	serve	4,360	886	Java	386	Serve, optimize and scale PyTorch models in production	2025-08-06
26	mini-sglang	4,227	660	Python	11	A compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems.	2026-05-17
27	tiny-llm	4,197	318	Python	7	A course of learning LLM inference serving on Apple Silicon for systems engineers: build a tiny vLLM + Qwen.	2026-05-21
28	lemonade	4,084	315	C++	234	Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk	2026-05-21
29	LightLLM	4,073	329	Python	85	LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.	2026-05-22
30	FedML	4,044	766	Python	122	FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on a...	2025-10-28
31	LitServe	3,882	285	Python	25	A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.	2026-05-04
32	FastDeploy	3,686	744	Python	287	High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle	2026-05-22
33	lighthouse	3,491	468	PHP	106	A framework for serving GraphQL from Laravel	2026-05-03
34	alpa	3,187	361	Python	68	Training and serving large-scale neural networks with auto parallelization.	2023-12-09
35	MOSS-TTS-Nano	3,137	407	Python	42	MOSS-TTS-Nano is an open-source multilingual tiny speech generation model from MOSI.AI and the OpenMOSS team. With only 0.1B parameters, it is designed for realtime speech generation, can run directly...	2026-05-12
36	ramalama	2,847	340	Python	69	RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of con...	2026-05-21
37	infinity	2,803	190	Python	106	Infinity is a high-throughput, low-latency serving engine for text-embeddings, reranking models, clip, clap and colpali	2026-03-24
38	whitenoise	2,741	153	Python	26	Radically simplified static file serving for Python web apps	2026-05-13
39	evadb	2,678	263	Python	78	Database system for AI-powered apps	2024-05-17
40	blazy	2,599	342	JavaScript	46	Hey, be lazy! bLazy.JS is a lightweight pure JavaScript script for lazy loading and multi-serving images. It's working in all modern browsers including IE7+.	2019-12-11
41	EasyLM	2,518	260	Python	28	Large language models (LLMs) made easy, EasyLM is a one stop solution for pre-training, finetuning, evaluating and serving LLMs in JAX/Flax.	2024-08-13
42	static-web-server	2,247	121	Rust	31	A cross-platform, high-performance and asynchronous web server for static files-serving. ⚡	2026-05-21
43	servant	1,955	422	Haskell	264	Servant is a Haskell DSL for describing, serving, querying, mocking, documenting web applications and more!	2026-05-08
44	S-LoRA	1,913	124	Python	30	S-LoRA: Serving Thousands of Concurrent LoRA Adapters	2024-01-21
45	tensorflow_template_application	1,877	708	Python	16	TensorFlow template application for deep learning	2023-07-05
46	llm-applications	1,855	255	Jupyter Notebook	9	A comprehensive guide to building RAG-based LLM applications for production.	2024-08-02
47	matrixone	1,843	297	Go	564	AI-native HTAP database with Git-for-Data and built-in vector search, serving as the data and memory backbone for intelligent agents and applications.	2026-05-21
48	dingo	1,700	265	Java	2	A multi-modal vector database that supports upserts and vector queries using unified SQL (MySQL-Compatible) on structured and unstructured data, while meeting the requirements of high concurrency and ...	2026-05-08
49	delta	1,605	284	Python	1	DELTA is a deep learning based natural language and speech processing platform. LF AI & DATA Projects: https://lfaidata.foundation/projects/delta/	2025-04-16
50	paddler	1,568	89	Rust	26	Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple...	2026-05-21
51	revive-adserver	1,461	605	PHP	140	The world's most popular free, open source ad serving system. You can download the latest release at:	2026-05-20
52	error-pages	1,426	109	Go	1	🪂 Tiny, zero-dep HTTP server & Docker image serving pretty, themeable, localized HTTP error pages — drop-in for Traefik, Nginx, Kubernetes, and more	2026-05-11
53	clipper	1,422	280	C++	114	A low-latency prediction-serving system	2021-04-26
54	postcodes.io	1,384	280	HTML	7	UK postcode & geolocation API, serving up open data	2026-04-23
55	UCall	1,329	54	C	11	Web Serving and Remote Procedure Calls at 50x lower latency and 70x higher bandwidth than FastAPI, implementing JSON-RPC & REST over io_uring ☎️	2025-09-16
56	parallax	1,295	137	Python	27	Parallax is a distributed model serving framework that lets you build your own AI cluster anywhere	2026-05-18
57	codefuse-chatbot	1,291	145	Python	5	An intelligent assistant serving the entire software development lifecycle, powered by a Multi-Agent Framework, working with DevOps Toolkits, Code&Doc Repo RAG, etc.	2024-07-01
58	sirv	1,168	63	JavaScript	15	An optimized middleware & CLI application for serving static files~!	2025-09-03
59	punica	1,159	62	Python	17	Serving multiple LoRA finetuned LLM as one	2024-05-08
60	club-3090	1,029	51	Python	9	Community recipes for serving LLMs on RTX 3090/CUDA gpus. Multi-engine (vLLM, llama.cpp, ik_llama) and model-agnostic. Currently shipping Qwen3.6-27B Qwen3.6 35B Gemma 4 26B Gemma 4 31B configs for 1×...	2026-05-22
61	multi-model-server	1,026	229	Java	85	Multi Model Server is a tool for serving neural net models for inference	2024-05-20
62	chronon	1,011	95	Scala	34	Chronon is a data platform for serving for AI/ML applications.	2026-05-20
63	cachegrand	1,001	35	C	27	cachegrand - a modern data ingestion, processing and serving platform built for today's hardware	2025-01-02
64	athenz	989	306	Java	39	Open source platform for X.509 certificate based service authentication and fine grained access control in dynamic infrastructures. Athenz supports provisioning and configuration (centralized authoriz...	2026-05-22
65	Nanoflow	961	49	Jupyter Notebook	17	A throughput-oriented high-performance serving framework for LLMs	2026-03-29
66	nodejs-storage	941	395	TypeScript	21	Node.js client for Google Cloud Storage: unified object storage for developers and enterprises, from live data serving to data analytics/ML to data archiving.	2026-03-03
67	langcorn	939	71	Python	12	⛓️ Serving LangChain LLM apps and agents automagically with FastApi. LLMops	2024-07-15
68	Serving	919	246	C++	13	A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）	2026-02-20
69	mosec	900	72	Python	13	A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine	2026-05-20
70	MLServer	888	233	Python	141	An inference server for your machine learning models, including support for multiple frameworks, multi-model serving and more	2026-05-21
71	model_server	876	254	C++	37	A scalable inference server for models optimized with OpenVINO™	2026-05-21
72	redis-inference-optimization	842	106	C	86	A Redis module for serving tensors and executing deep learning graphs	2025-08-20
73	SpecForge	841	233	Python	72	Train speculative decoding models effortlessly and port them smoothly to SGLang serving.	2026-04-02
74	omniserve	838	63	C++	47	[MLSys'25] QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving; [MLSys'25] LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention	2025-03-06
75	DistServe	814	94	Jupyter Notebook	32	Disaggregated serving system for Large Language Models (LLMs).	2025-04-06
76	nginx-gridfs	788	144	C	51	Nginx module for serving files from MongoDB's GridFS	2013-12-10
77	fair	782	28	Go	6	A Go library for serving resources fairly	2026-02-16
78	simple_tensorflow_serving	758	185	JavaScript	28	Generic and easy-to-use serving service for machine learning models	2025-03-20
79	memorizz	738	78	Python	1	MemoRizz: A Python library serving as a memory layer for AI applications. Leverages popular databases and storage solutions to optimize memory usage. Provides utility classes and methods for efficient...	2026-03-10
80	manager	734	401	TypeScript	11	The Akamai Cloud Manager is an open-source, single-page application serving as the official UI for interacting with the Linode API. It is trusted by hundreds of thousands of customers to manage their ...	2026-04-23
81	rust-rpxy	721	61	Rust	13	A simple and ultrafast http reverse proxy serving multiple domain names and terminating TLS for http/1.1, 2 and 3, written in Rust	2026-05-21
82	decision-forests	693	116	Python	34	A collection of state-of-the-art algorithms for the training, serving and interpretation of Decision Forest models in Keras.	2026-05-19
83	ServerlessLLM	682	72	Python	33	Serverless LLM Serving for Everyone.	2026-05-04
84	engintron	678	167	Perl	9	Engintron for cPanel/WHM is the easiest way to integrate Nginx on your cPanel/WHM server. Engintron will improve the performance & web serving capacity of your server, while reducing CPU/RAM load at t...	2026-04-05
85	NiceTab	673	67	TypeScript	43	浏览器插件：OneTab、Toby、N-Tab、SessionBuddy、KepTab 等扩展的升级替代品，功能丰富易用 | A nice, convenient, powerful tab manager serving as an upgraded alternative to OneTab, Toby, SessionBuddy, N-Tab，KepTab...	2026-04-27
86	candle-vllm	663	81	Rust	23	Efficent platform for inference and serving local LLMs including an OpenAI compatible API server.	2026-05-19
87	go-ycsb	642	278	Go	62	A Go port of Yahoo! Cloud Serving Benchmark (YCSB)	2025-12-31
88	serve-favicon	626	74	JavaScript	0	favicon serving middleware	2026-02-01
89	rocket-nginx	618	129	PHP	3	Rocket-Nginx is a Nginx configuration that makes WP-Rocket even faster by serving static pages directly without loading WordPress or PHP.	2025-11-20
90	Awesome-VLA4AD	598	54	-	1	Vision–Language–Action models for Autonomous Driving (VLA4AD) resources, serving as the companion repository to the survey paper “A Survey on Vision–Language–Action Models for Autonomous Driving”.	2025-11-20
91	netlify-lambda	590	113	JavaScript	0	Helps building and serving lambda functions locally and in CI environments	2024-02-06
92	strftime.org	590	306	Mustache	0	A single-serving-site that provided a reference for Python's strftime formatting options from 2010–2026.	2026-04-03
93	Awesome-Video-World-Models-with-AR-Diffusion	550	18	TeX	0	A Curated List of Awesome Video World Models with AR Diffusion: Covering Algorithms, Applications, and Infrastructure, Aimed at Serving as a Comprehensive Resource for Researchers, Practitioners, and ...	2026-05-18
94	llumnix-ray	549	50	Python	6	Efficient and easy multi-instance LLM serving	2026-03-12
95	pinferencia	544	83	Python	12	Python + Inference - Model Deployment library in Python. Simplest model inference server ever.	2023-02-14
96	MetaSpore	543	95	Python	8	A unified end-to-end machine intelligence platform	2024-09-02
97	planktos	536	20	JavaScript	25	Serving websites over bittorrent	2020-05-10
98	angular-login-example	517	166	JavaScript	5	Stand-alone project showing how to make a robust angular application serving access permissions from Server	2015-01-24
99	streamlit-fastapi-model-serving	503	93	Python	3	Simple web app example serving a PyTorch model using streamlit and FastAPI	2026-03-27
100	sarathi-serve	503	63	Python	10	A low-latency & high-throughput serving engine for LLMs	2026-01-08