Top 100 GPU Inference Repositories

Back to Home

Ranking

Ranking	Project Name	Stars	Forks	Language	Open Issues	Description	Last Commit
1	DeepSpeed	42,388	4,837	Python	1163	DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.	2026-05-21
2	airllm	18,194	1,978	Jupyter Notebook	137	AirLLM 70B inference with single 4GB GPU	2026-03-10
3	TensorRT-LLM	13,693	2,400	Python	587	TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT ...	2026-05-22
4	TensorRT	13,004	2,362	C++	548	NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.	2026-04-13
5	server	10,689	1,779	Python	779	The Triton Inference Server provides an optimized cloud and edge inferencing solution.	2026-05-21
6	ipex-llm	8,810	1,427	Python	1213	Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete...	2026-01-28
7	AidLearning-FrameWork	5,772	720	Python	23	🔥🔥🔥AidLearning is a powerful AIOT development platform, AidLearning builds a linux env supporting GUI, deep learning and visual IDE on Android...Now Aid supports CPU+GPU+NPU for inference with high...	2026-04-02
8	DALI	5,693	665	C++	203	A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.	2026-05-21
9	gpustack	5,035	531	Python	521	A GPU cluster manager that configures and orchestrates inference engines like vLLM and SGLang for high-performance AI model deployment.	2026-05-21
10	cube-studio	5,017	882	Python	33	cube studio开源云原生一站式机器学习/深度学习/大模型AI平台，mlops算法链路全流程，算力租赁平台，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU虚拟化，边缘计算，标注平台自动化标注，deepseek等大模型sft微调/奖励模型/强化学习训练，vllm/ollama/mindie大模型多机推理，私有知识库，AI模型市场，支持...	2026-05-20
11	AITemplate	4,720	388	Python	108	AITemplate is a Python framework which renders neural network into high performance CUDA/HIP C++ code. Specialized for FP16 TensorCore (NVIDIA GPU) and MatrixCore (AMD GPU) inference.	2026-04-09
12	exllamav2	4,531	336	Python	137	A fast inference library for running LLMs locally on modern consumer-class GPUs	2026-03-04
13	TransformerEngine	3,349	726	Python	232	A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance wi...	2026-05-22
14	llm-d	3,234	488	Shell	182	Achieve state of the art inference performance with modern accelerators on Kubernetes	2026-05-22
15	chitu	3,126	263	Python	51	High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.	2026-05-21
16	TypeGPU	2,478	66	TypeScript	183	A modular and open-ended toolkit for WebGPU, with advanced type inference and the ability to write shaders in TypeScript	2026-05-21
17	local-dream	2,433	153	Kotlin	73	Run Stable Diffusion on Android Devices with Snapdragon NPU acceleration. Also supports CPU/GPU inference.	2026-05-20
18	dstack	2,142	229	Python	70	Vendor-agnostic orchestration for training, inference and agentic workloads across NVIDIA, AMD, TPU, and Tenstorrent on clouds, Kubernetes, and bare metal.	2026-05-21
19	GPU-Benchmarks-on-LLM-Inference	1,917	75	Jupyter Notebook	15	Multiple NVIDIA GPUs or Apple Silicon for Large Language Model Inference?	2024-05-13
20	transformer-deploy	1,685	152	Python	52	Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗 Hugging Face transformer models 🚀	2024-10-23
21	whichllm	1,670	77	Python	16	Find the local LLM that actually runs and performs best on your hardware. Ranked by real, recency-aware benchmarks, not parameter count. One command, run it instantly.	2026-05-19
22	beta9	1,652	142	Go	9	Ultrafast serverless GPU inference, sandboxes, and background jobs	2026-05-22
23	TurboTransformers	1,549	207	C++	49	a fast and user-friendly runtime for transformer inference (Bert, Albert, GPT2, Decoders, etc) on CPU and GPU.	2025-07-18
24	torchsparse	1,460	189	Cuda	52	[MICRO'23, MLSys'22] TorchSparse: Efficient Training and Inference Framework for Sparse Convolution on GPUs.	2025-02-24
25	stable-fast	1,304	92	Python	58	https://wavespeed.ai/ Best inference performance optimization framework for HuggingFace Diffusers on NVIDIA GPUs.	2025-03-27
26	YOLOs-CPP	975	142	C++	1	Cross-Platform Production-ready C++ inference engine for YOLO models (v5-v12, YOLO26). Unified API for detection, segmentation, pose estimation, OBB, and classification. Built on ONNX Runtime and Open...	2026-04-13
27	Merlin	892	129	Python	201	NVIDIA Merlin is an open source library providing end-to-end GPU-accelerated recommender systems, from feature engineering and preprocessing to training deep learning models and running inference in p...	2024-12-05
28	exllamav3	889	100	Python	71	An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs	2026-05-21
29	pytriton	845	59	Python	11	PyTriton is a Flask/FastAPI-like interface that simplifies Triton's deployment in Python environments.	2025-08-13
30	ai-reference-models	733	222	Python	39	Intel® AI Reference Models: contains Intel optimizations for running deep learning workloads on Intel® Xeon® Scalable processors and Intel® Data Center GPUs	2026-02-11
31	TrainYourOwnYOLO	673	418	Jupyter Notebook	9	Train a state-of-the-art yolov3 object detector from scratch!	2024-08-02
32	tensor_parallel	655	44	Python	33	Automatically split your PyTorch models on multiple GPUs for training & inference	2024-01-02
33	DFloat11	630	37	Python	23	DFloat11 [NeurIPS '25]: Lossless Compression of LLMs and DiTs for Efficient GPU Inference	2025-11-24
34	FastFold	616	89	Python	38	Optimizing AlphaFold Training and Inference on GPU Clusters	2024-07-16
35	BMInf	584	67	Python	16	Efficient Inference for Big Models	2023-01-24
36	LLaMA_MPS	584	46	Python	8	Run LLaMA (and Stanford-Alpaca) inference on Apple Silicon GPUs.	2023-03-25
37	Forward	555	62	C++	0	A library for high performance deep learning inference on NVIDIA GPUs.	2022-01-29
38	qwen600	553	48	Cuda	0	Static suckless single batch CUDA-only qwen3-0.6B mini inference engine	2025-09-08
39	Anakin	536	135	C++	53	High performance Cross-platform Inference-engine, you could run Anakin on x86-cpu,arm, nv-gpu, amd-gpu,bitmain and cambricon devices.	2022-09-23
40	model_analyzer	510	86	Python	3	Triton Model Analyzer is a CLI tool to help with better understanding of the compute and memory requirements of the Triton Inference Server models.	2026-05-13
41	tflite_gles_app	505	132	C	15	GPU accelerated deep learning inference applications for RaspberryPi / JetsonNano / Linux PC using TensorflowLite GPUDelegate / TensorRT	2022-04-22
42	ScaleLLM	501	40	C++	48	A high-performance inference system for large language models, designed for production environments.	2025-12-19
43	VisionMamba	494	23	Python	0	Implementation of Vision Mamba from the paper: "Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model" It's 2.8x faster than DeiT and saves 86.8% GPU memory when ...	2026-05-19
44	ByteTransformer	479	37	C++	5	optimized BERT transformer inference on NVIDIA GPU. https://arxiv.org/abs/2210.03052	2024-03-15
45	isaac_ros_pose_estimation	449	50	C++	36	Deep learned, NVIDIA-accelerated 3D object pose estimation	2026-05-01
46	JetStream	437	64	Python	14	JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).	2026-01-05
47	gpu-rest-engine	423	93	C++	6	A REST API for Caffe using Docker and Go	2018-07-20
48	eLLM	414	43	Rust	1	eLLM can infer LLM on CPUs faster than on GPUs	2026-05-22
49	cuckoo	409	43	TypeScript	0	Cuckoo is a Decentralized AI Model-Serving Platform, starting with GPU-sharing for text-to-image generation and LLM inference.	2025-11-24
50	zinc	394	15	Zig	1	Zig INferenCe Engine — Local LLM inference on AMD GPUs and Apple Silicon	2026-05-22
51	stable-diffusion-deploy	391	39	Python	22	Learn to serve Stable Diffusion models on cloud infrastructure at scale. This Lightning App shows load-balancing, orchestrating, pre-provisioning, dynamic batching, GPU-inference, micro-services worki...	2023-10-03
52	libtorch-yolov5	375	113	C++	18	A LibTorch inference implementation of the yolov5	2021-01-07
53	Lvllm	365	34	Python	1	LvLLM is a special NUMA extension of vllm that makes full use of CPU and memory resources, reduces GPU memory requirements, and features an efficient GPU parallel and NUMA parallel architecture, suppo...	2026-04-28
54	Pedestrian-Detection	355	132	Python	3	Pedestrian detection using the TensorFlow Object Detection API. Includes multi GPU parallel processing inference.	2019-09-04
55	neural-speed	353	37	C++	19	An innovative library for efficient LLM inference via low-bit quantization	2024-08-30
56	mcx	331	16	Python	15	Express & compile probabilistic programs for performant inference on CPU & GPU. Powered by JAX.	2024-03-20
57	InfiniTensor	314	70	C++	24	InfiniTensor is a high-performance inference engine tailored for GPUs and AI accelerators. Its design focuses on effective deployment and swift academic validation.	2026-05-20
58	BMW-TensorFlow-Inference-API-GPU	313	74	Python	0	This is a repository for an object detection inference API using the Tensorflow framework.	2024-11-22
59	fp6_llm	280	24	Cuda	4	An efficient GPU support for LLM inference with x-bit quantization (e.g. FP6,FP5).	2025-07-16
60	BMW-YOLOv4-Inference-API-GPU	279	67	Python	0	This is a repository for an nocode object detection inference API using the Yolov3 and Yolov4 Darknet framework.	2022-06-28
61	hand_detection	275	85	Python	6	using Neural Networks (SSD) on Tensorflow. This repo documents steps and scripts used to train a hand detector using Tensorflow (Object Detection API). As with any DNN based task, the most expensive ...	2022-01-11
62	tflite2tensorflow	272	43	Python	1	Generate saved_model, tfjs, tf-trt, EdgeTPU, CoreML, quantized tflite, ONNX, OpenVINO, Myriad Inference Engine blob and .pb from .tflite. Support for building environments with Docker. It is possible ...	2022-09-04
63	aitune	270	30	Python	2	NVIDIA AITune is an inference toolkit designed for tuning and deploying Deep Learning models with a focus on NVIDIA GPUs.	2026-03-13
64	fiddler	265	35	Python	3	[ICLR'25] Fast Inference of MoE Models with CPU-GPU Orchestration	2024-11-18
65	GPULlama3.java	257	32	Java	17	GPU-accelerated Llama3.java inference in pure Java using TornadoVM.	2026-05-21
66	CLIP-ONNX	234	31	Python	3	It is a simple library to speed up CLIP inference up to 3x (K80 GPU)	2023-07-20
67	amd_inference	233	8	Python	11	Docker-based inference engine for AMD GPUs	2024-10-07
68	InspireFace	225	48	C++	37	InspireFace is a cross-platform face recognition SDK developed in C/C++, supporting multiple operating systems and various backend types for inference, such as CPU, GPU, and NPU.	2025-09-22
69	model_navigator	221	28	Python	3	Triton Model Navigator is an inference toolkit designed for optimizing and deploying Deep Learning models with a focus on NVIDIA GPUs.	2026-02-03
70	llm-server	218	9	Shell	2	Smart launcher for llama.cpp / ik_llama.cpp — auto-detects GPUs, optimizes MoE placement, crash recovery	2026-05-21
71	LightGlue-OnnxRunner	210	31	C++	17	LightGlue-OnnxRunner is a repository hosts the C++ inference code of LightGlue in ONNX format，supporting end-to-end/decouple model inference of SuperPoint/DISK + LightGlue	2024-02-01
72	grove	210	63	Go	39	Kubernetes enhancements for Network Topology Aware Gang Scheduling & Autoscaling	2026-05-20
73	aws-virtual-gpu-device-plugin	203	30	Jupyter Notebook	13	AWS virtual gpu device plugin provides capability to use smaller virtual gpus for your machine learning inference workloads	2023-11-22
74	isaac_ros_object_detection	190	43	C++	12	NVIDIA-accelerated, deep learned model support for image space object detection	2026-05-01
75	rustane	175	22	Rust	3	Rust-native hybrid training & inference engine for Apple Neural Engine + Metal GPU	2026-04-03
76	gpt-oss-amd	173	15	C++	0	implement GPT-OSS 20B & 120B C++ inference from scratch on AMD GPUs	2025-10-25
77	modelbox	160	49	C++	28	A high performance, high expansion, easy to use framework for AI application. 为AI应用的开发者提供一套统一的高性能、易用的编程框架，快速基于AI全栈服务、开发跨端边云的AI行业应用，支持GPU，NPU加速。	2024-06-06
78	grps_trtllm	159	11	Python	7	Higher performance OpenAI LLM service than vLLM serve: A pure C++ high-performance OpenAI LLM service implemented with GPRS+TensorRT-LLM+Tokenizers.cpp, supporting chat and function call, AI agents, d...	2025-12-08
79	-deprecated-NVIDIA-GPU-Tensor-Core-Accelerator-PyTorch-OpenCV	157	23	Jupyter Notebook	0	Computer vision container that includes Jupyter notebooks with built-in code hinting, Anaconda, CUDA 11.8, TensorRT inference accelerator for Tensor cores, CuPy (GPU drop in replacement for Numpy), Py...	2023-10-28
80	anubis-oss	153	8	Swift	3	Local LLM Testing & Benchmarking for Apple Silicon	2026-05-22
81	tensor-fusion	152	31	Go	10	Tensor Fusion is a state-of-the-art GPU virtualization and pooling solution designed to optimize GPU cluster utilization to its fullest potential.	2026-05-22
82	alphafast	151	18	Python	0	AlphaFast: ultra-high-throughput AlphaFold3 inference with MMSeqs2-GPU	2026-05-04
83	asystem-awex	151	17	Python	2	A high-performance RL training-inference weight synchronization framework, designed to enable second-level parameter updates from training to inference in RL workflows	2026-04-22
84	graphlearn-for-pytorch	148	46	Python	8	A GPU-accelerated graph learning library for PyTorch, facilitating the scaling of GNN training and inference.	2025-09-23
85	ros2_yolos_cpp	147	19	C++	1	ROS2 adapters for the Cross-Platform Production-ready C++ inference engine for YOLO models (v5-v12, YOLO26). Unified API for detection, segmentation, pose estimation, OBB, and classification. Built on...	2026-02-06
86	CENT	136	27	Python	1	Artifact for paper "PIM is All You Need: A CXL-Enabled GPU-Free System for LLM Inference", ASPLOS 2025	2025-05-03
87	isaac_ros_dnn_inference	132	18	C++	13	NVIDIA-accelerated DNN model inference ROS 2 packages using NVIDIA Triton/TensorRT for both Jetson and x86_64 with CUDA-capable GPU	2026-05-01
88	comfyui-triton-and-sageattention-installer	132	6	Python	7	Cross-platform installer for Triton and SageAttention on ComfyUI. Simplifies GPU-accelerated inference setup for Windows users with automated dependency management and RTX 5090 support.	2026-03-31
89	llm-d-inference-sim	131	84	Go	27	A lightweight, configurable, and real-time simulator designed to mimic the behavior of vLLM without the need for GPUs or running actual heavy models.	2026-05-20
90	trt-llm-as-openai-windows	128	16	Python	1	This reference can be used with any existing OpenAI integrated apps to run with TRT-LLM inference locally on GeForce GPU on Windows instead of cloud.	2024-02-29
91	optimum-transformers	126	8	Python	0	Accelerated NLP pipelines for fast inference on CPU and GPU. Built with Transformers, Optimum and ONNX Runtime.	2022-04-06
92	dsc	118	5	C++	6	Tensor library & inference framework for machine learning	2025-10-03
93	isaac_ros_image_segmentation	118	10	C++	8	NVIDIA-accelerated, deep learned semantic image segmentation	2026-05-01
94	aikit	117	11	Python	0	Razer AIKit is an AI development toolkit with fast LLM inferencing, fine-tuning, and multi-GPU scaling.	2026-04-27
95	HybriMoE	116	17	Python	3	[DAC'25] Official implement of "HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference"	2025-12-15
96	FunASR-GGML	113	14	C++	2	C++ speech recognition inference engine using GGML — CPU/CUDA GPU, real-time microphone streaming, single GGUF model file, no Python dependency	2026-04-30
97	ai-inference-resources	111	9	-	0	Curated collection of AI inference engineering resources — LLM serving, GPU kernels, quantization, distributed inference, and production deployment. Compiled from the AER Labs community.	2026-02-04
98	llama-dfdx	111	6	Rust	4	LLaMa 7b with CUDA acceleration implemented in rust. Minimal GPU memory needed!	2023-07-27
99	reef	108	11	Cuda	6	REEF is a GPU-accelerated DNN inference serving system that enables instant kernel preemption and biased concurrent execution in GPU scheduling.	2022-12-24
100	MSLK	107	46	Python	0	MSLK (Meta Superintelligence Labs Kernels) is a collection of PyTorch GPU operator libraries that are designed and optimized for GenAI training and inference, such as FP8 row-wise quantization and col...	2026-05-21