Top 100 AI Evaluation Repositories
Ranking
| Ranking | Project Name | Stars | Forks | Language | Open Issues | Description | Last Commit |
|---|---|---|---|---|---|---|---|
| 1 | mlflow | 26,079 | 5,775 | Python | 1522 | The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controllin... | 2026-05-23 |
| 2 | promptfoo | 21,536 | 1,893 | TypeScript | 77 | Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line ... | 2026-05-24 |
| 3 | adk-python | 19,822 | 3,449 | Python | 488 | An open-source, code-first Python toolkit for building, evaluating, and deploying sophisticated AI agents with flexibility and control. | 2026-05-23 |
| 4 | cua | 17,035 | 1,080 | HTML | 120 | Open-source infrastructure for Computer-Use Agents. Sandboxes, SDKs, and benchmarks to train and evaluate AI agents that can control full desktops (macOS, Linux, Windows). | 2026-05-23 |
| 5 | RagaAI-Catalyst | 16,166 | 3,609 | Python | 17 | Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced anal... | 2026-02-11 |
| 6 | WeKnora | 15,433 | 1,960 | Go | 171 | Open-source LLM knowledge platform: turn raw documents into a queryable RAG, an autonomous reasoning agent, and a self-maintaining Wiki. | 2026-05-22 |
| 7 | tensorzero | 11,394 | 832 | Rust | 335 | TensorZero is an open-source LLMOps platform that unifies an LLM gateway, observability, evaluation, optimization, and experimentation. | 2026-05-23 |
| 8 | bisheng | 11,380 | 1,870 | TypeScript | 104 | BISHENG is an open LLM devops platform for next generation Enterprise AI applications. Powerful and comprehensive features include: GenAI workflow, RAG, Agent, Unified model management, Evaluation, SF... | 2026-05-24 |
| 9 | ParlAI | 10,626 | 2,067 | Python | 50 | A framework for training and evaluating AI models on a variety of openly available dialogue datasets. | 2023-11-03 |
| 10 | phoenix | 9,808 | 887 | Python | 468 | AI Observability & Evaluation | 2026-05-24 |
| 11 | adk-go | 7,991 | 683 | Go | 94 | An open-source, code-first Go toolkit for building, evaluating, and deploying sophisticated AI agents with flexibility and control. | 2026-05-22 |
| 12 | evidently | 7,528 | 850 | Jupyter Notebook | 229 | Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics. | 2026-05-02 |
| 13 | agent-starter-pack | 6,433 | 1,477 | Python | 10 | Ship AI Agents to Google Cloud in minutes, not months. Production-ready templates with built-in CI/CD, evaluation, and observability. | 2026-05-23 |
| 14 | EverOS | 5,600 | 593 | Python | 53 | Build, evaluate, and integrate long-term memory for self-evolving agents. | 2026-05-21 |
| 15 | coze-loop | 5,468 | 761 | Go | 42 | Next-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities from development, debugging, and evaluation to... | 2026-05-22 |
| 16 | transformerlab-app | 5,021 | 520 | Python | 18 | The open source research environment for AI researchers to seamlessly train, evaluate, and scale models from local hardware to GPU clusters. | 2026-05-23 |
| 17 | Kiln | 4,844 | 368 | Python | 22 | Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more. | 2026-05-22 |
| 18 | AI-Infra-Guard | 3,763 | 371 | Python | 8 | A full-stack AI Red Teaming platform securing AI ecosystems via OpenClaw Security Scan, Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation. | 2026-05-22 |
| 19 | trulens | 3,338 | 279 | Python | 55 | Evaluation and Tracking for LLM Experiments and AI Agents | 2026-05-23 |
| 20 | langwatch | 3,267 | 320 | TypeScript | 376 | The platform for LLM evaluations and AI agent testing | 2026-05-24 |
| 21 | ChainForge | 2,987 | 254 | TypeScript | 58 | An open-source visual programming environment for battle-testing prompts to LLMs. | 2026-04-06 |
| 22 | lmnr | 2,938 | 199 | TypeScript | 30 | Laminar - open-source observability platform purpose-built for AI agents. YC S24. | 2026-05-23 |
| 23 | agents-cli | 2,544 | 291 | - | 9 | The CLI and skills that turn any coding assistant into an expert at creating, evaluating, and deploying AI agents on Google Cloud. | 2026-05-19 |
| 24 | openlit | 2,463 | 281 | TypeScript | 15 | Open source platform for AI Engineering: OpenTelemetry-native LLM Observability, GPU Monitoring, Guardrails, Evaluations, Prompt Management, Vault, Playground. 🚀💻 Integrates with 50+ LLM Providers, ... | 2026-05-22 |
| 25 | uptrain | 2,348 | 202 | Python | 44 | UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform ro... | 2024-08-18 |
| 26 | inspect_ai | 2,115 | 523 | Python | 103 | Inspect: A framework for large language model evaluations | 2026-05-23 |
| 27 | EvalAI | 2,021 | 982 | Python | 200 | ☁️ 🚀 📊 📈 Evaluating state of the art in AI | 2026-05-22 |
| 28 | WFGY | 1,755 | 162 | Jupyter Notebook | 10 | WFGY is heading toward WFGY 5.0 Polaris Protocol, a major open-source release for AI reasoning, RAG, agents, and real-world workflows. Includes Problem Map, Global Debug Card, WFGY 4.0, and the CFV Ea... | 2026-05-23 |
| 29 | CompressAI | 1,584 | 274 | Python | 67 | A PyTorch library and evaluation platform for end-to-end compression research | 2026-05-09 |
| 30 | evalite | 1,566 | 91 | TypeScript | 43 | Evaluate your LLM-powered apps with TypeScript | 2026-04-28 |
| 31 | adk-java | 1,566 | 350 | Java | 51 | An open-source, code-first Java toolkit for building, evaluating, and deploying sophisticated AI agents with flexibility and control. | 2026-05-24 |
| 32 | pycm | 1,502 | 125 | Python | 11 | Multi-class confusion matrix library in Python | 2026-05-20 |
| 33 | Introduction-to-Quantitative-Finance | 1,408 | 156 | Python | 0 | 入门资料整理：1.多因子股票量化框架开源教程 2.学界和业界的经典资料收录 3.AI + 金融的相关工作，包括LLM, Agent, benchmark(evaluation), etc. | 2026-05-21 |
| 34 | Awesome-Jailbreak-on-LLMs | 1,383 | 110 | - | 0 | Awesome-Jailbreak-on-LLMs is a collection of state-of-the-art, novel, exciting jailbreak methods on LLMs. It contains papers, codes, datasets, evaluations, and analyses. | 2026-03-30 |
| 35 | adk-docs | 1,367 | 1,044 | Shell | 123 | An open-source, code-first toolkit for building, evaluating, and deploying sophisticated AI agents with flexibility and control. | 2026-05-22 |
| 36 | xai | 1,239 | 185 | Python | 0 | XAI - An eXplainability toolbox for machine learning | 2025-11-29 |
| 37 | prompty | 1,214 | 115 | TypeScript | 8 | Prompty makes it easy to create, manage, debug, and evaluate LLM prompts for your AI applications. Prompty is an asset class and format for LLM prompts designed to enhance observability, understandab... | 2026-05-23 |
| 38 | kubetorch | 1,205 | 58 | Python | 1 | Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra. | 2026-04-13 |
| 39 | langtrace | 1,203 | 125 | TypeScript | 0 | Langtrace 🔍 is an open-source, Open Telemetry based end-to-end observability tool for LLM applications, providing real-time tracing, evaluations and metrics for popular LLMs, LLM frameworks, vectorD... | 2025-11-17 |
| 40 | skill | 1,196 | 132 | Python | 17 | PinchBench is a benchmarking system for evaluating LLM models as OpenClaw coding agents. Made with 🦀 by the humans at https://kilo.ai | 2026-05-22 |
| 41 | any-agent | 1,169 | 93 | Python | 16 | A single interface to use and evaluate different agent frameworks | 2026-05-01 |
| 42 | adk-js | 1,146 | 142 | TypeScript | 19 | An open-source, code-first Typescript toolkit for building, evaluating, and deploying sophisticated AI agents with flexibility and control. | 2026-05-18 |
| 43 | JamAIBase | 1,095 | 40 | Python | 1 | The collaborative spreadsheet for AI. Chain cells into powerful pipelines, experiment with prompts and models, and evaluate LLM responses in real-time. Work together seamlessly to build and iterate on... | 2026-05-23 |
| 44 | future-agi | 1,027 | 207 | Python | 54 | Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0. | 2026-05-22 |
| 45 | rogue | 1,024 | 161 | Python | 36 | AI Agent Evaluator & Red Team Platform | 2026-05-04 |
| 46 | Scam-AI-Multi-modal-Evaluation-System | 1,001 | 88 | Python | 0 | 2025-11-05 | |
| 47 | ollama-grid-search | 937 | 56 | TypeScript | 4 | A multi-platform desktop application to evaluate and compare LLM models, written in Rust and React. | 2026-05-21 |
| 48 | autoevals | 902 | 62 | Python | 3 | AutoEvals is a tool for quickly and easily evaluating AI model outputs using best practices. | 2026-04-03 |
| 49 | AIOpsLab | 885 | 159 | Python | 14 | A holistic framework to enable the design, development, and evaluation of autonomous AIOps agents. | 2026-05-19 |
| 50 | deepfabric | 869 | 83 | Python | 13 | Generate High-Quality Synthetics, Train, Measure, and Evaluate in a Single Pipeline | 2026-05-20 |
| 51 | PIXIU | 864 | 114 | Jupyter Notebook | 9 | This repository introduces PIXIU, an open-source resource featuring the first financial large language models (LLMs), instruction tuning data, and evaluation benchmarks to holistically assess financia... | 2025-03-04 |
| 52 | awesome-ai-agent-papers | 858 | 114 | - | 2 | A curated collection of AI agent research papers released in 2026, covering agent engineering, memory, evaluation, workflows, and autonomous systems. | 2026-04-21 |
| 53 | AICGSecEval | 800 | 108 | Python | 5 | A.S.E (AICGSecEval) is a repository-level AI-generated code security evaluation benchmark developed by Tencent Wukong Code Security Team. | 2026-04-21 |
| 54 | prompt-forge | 775 | 76 | Go | 3 | AI prompt engineering workbench for crafting, testing, and systematically evaluating prompts with powerful analysis tools. | 2025-07-16 |
| 55 | MixtralKit | 774 | 75 | Python | 12 | A toolkit for inference and evaluation of 'mixtral-8x7b-32kseqlen' from Mistral AI | 2023-12-15 |
| 56 | CodeGen | 773 | 146 | Python | 36 | Reference implementation of code generation projects from Facebook AI Research. General toolkit to apply machine learning to code, from dataset creation to model training and evaluation. Comes with pr... | 2026-03-12 |
| 57 | SkillNet | 761 | 77 | Python | 0 | Create, Evaluate, and Connect AI Skills | 2026-05-20 |
| 58 | contoso-chat | 761 | 3,993 | Bicep | 14 | This sample has the full End2End process of creating RAG application with Prompty and Azure AI Foundry. It includes GPT-4 LLM application code, evaluations, deployment automation with AZD CLI, GitHub ... | 2025-10-03 |
| 59 | rag-in-action | 731 | 295 | Jupyter Notebook | 3 | End-to-end RAG system design, evaluation, and optimization. 极客时间RAG训练营，RAG 10大组件全面拆解，4个实操项目吃透 RAG 全流程。RAG的落地，往往是面向业务做RAG，而不是反过来面向RAG做业务。这就是为什么我们需要针对不同场景、不同问题做针对性的调整、优化和定制化。魔鬼全在细节中，我们深入进去探究。 | 2025-07-16 |
| 60 | dingo | 702 | 72 | Python | 3 | Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool | 2026-05-22 |
| 61 | Quantus | 663 | 89 | Jupyter Notebook | 55 | Quantus is an eXplainable AI toolkit for responsible evaluation of neural network explanations | 2026-05-04 |
| 62 | genai-compliance-bench | 641 | 55 | Python | 0 | GenAI compliance benchmark is a evaluation benchmarks for generative AI in regulated industries. | 2026-03-27 |
| 63 | Awesome-LLM-Eval | 637 | 67 | - | 5 | Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表，主要面向基础大模型评测，旨在探求生成式AI的技术边界. | 2025-11-24 |
| 64 | TrustLLM | 626 | 67 | Python | 7 | [ICML 2024] TrustLLM: Trustworthiness in Large Language Models | 2025-06-24 |
| 65 | pytensor | 615 | 187 | Python | 335 | PyTensor allows you to define, optimize, and efficiently evaluate mathematical expressions involving multi-dimensional arrays. | 2026-05-22 |
| 66 | RAG-Driven-Generative-AI | 604 | 210 | Jupyter Notebook | 0 | This repository provides programs to build Retrieval Augmented Generation (RAG) code for Generative AI with LlamaIndex, Deep Lake, and Pinecone leveraging the power of OpenAI and Hugging Face models f... | 2025-09-23 |
| 67 | Long-Context | 604 | 46 | Python | 6 | This repository contains code and tooling for the Abacus.AI LLM Context Expansion project. Also included are evaluation scripts and benchmark tasks that evaluate a model’s information retrieval capabi... | 2023-11-17 |
| 68 | can-ai-code | 599 | 35 | Python | 32 | Self-evaluating interview for AI coders | 2025-06-21 |
| 69 | SteptronOss | 570 | 41 | Python | 3 | A lightweight, AI-native training framework for large language models. Designed for fast iteration, reproducible experiments, and modular configuration across SFT, RLVR, and evaluation workflows. | 2026-05-18 |
| 70 | LLM-Agents-Ecosystem-Handbook | 522 | 83 | Python | 0 | One-stop handbook for building, deploying, and understanding LLM agents with 60+ skeletons, tutorials, ecosystem guides, and evaluation tools. | 2026-05-09 |
| 71 | meta-agents-research-environments | 498 | 66 | Python | 14 | Meta Agents Research Environments is a comprehensive platform designed to evaluate AI agents in dynamic, realistic scenarios. Unlike static benchmarks, this platform introduces evolving environments w... | 2026-05-21 |
| 72 | ai-pronunciation-trainer | 479 | 119 | Python | 5 | This tool uses AI to evaluate your pronunciation. | 2025-08-16 |
| 73 | well-architected-iac-analyzer | 474 | 86 | TypeScript | 7 | Sample Generative AI tool for evaluating Infrastructure as Code and architecture diagrams against AWS Well-Architected best practices. | 2026-05-23 |
| 74 | StackRabbit | 463 | 50 | C++ | 21 | An AI for playing NES Tetris at a high level. Based primarily on search & heuristic, with high quality board evaluation through value iteration. | 2026-05-22 |
| 75 | ParseBench | 462 | 54 | Python | 2 | ParseBench - A Document Parsing Benchmark for AI Agents | 2026-05-15 |
| 76 | sound-spaces | 459 | 73 | Python | 56 | A first-of-its-kind acoustic simulation platform for audio-visual embodied AI research. It supports training and evaluating multiple tasks and applications. | 2023-09-29 |
| 77 | phasellm | 458 | 28 | Python | 0 | Large language model evaluation and workflow framework from Phase AI. | 2025-01-21 |
| 78 | ai-resume-analyzer | 438 | 164 | JavaScript | 2 | AI-powered Resume Analyzer built with React & Puter.js. Includes seamless auth, upload and store resumes, and match candidates to jobs using smart AI evaluations. All wrapped in a clean, reusable UI. | 2025-07-18 |
| 79 | ape-papers | 417 | 91 | R | 6 | Can policy evaluation be automated? Or inevitably create hallucinated AI slop? We are trying to find out. | 2026-04-10 |
| 80 | agentguard | 411 | 60 | TypeScript | 5 | Security guard for AI agents — blocks malicious skills, prevents data leaks, protects secrets. 24 detection rules, runtime action evaluation, trust registry. | 2026-05-22 |
| 81 | FAI-PEP | 395 | 87 | Python | 11 | Facebook AI Performance Evaluation Platform | 2026-04-08 |
| 82 | spring-ai-alibaba-admin | 392 | 75 | Java | 19 | Console supporting tracing, prompt engineering and evaluation for Agents developed with Spring AI Alibaba. | 2026-01-11 |
| 83 | yao-meta-skill | 384 | 49 | Python | 0 | YAO = Yielding AI Outcomes. A rigorous engineering, evaluation, governance, and portability system for reusable agent skills. | 2026-05-20 |
| 84 | ExplainaBoard | 367 | 36 | Python | 61 | Interpretable Evaluation for AI Systems | 2023-03-10 |
| 85 | agent-evaluation | 363 | 49 | Python | 15 | A generative AI-powered framework for testing virtual agents. | 2025-12-15 |
| 86 | awesome-foundation-model-leaderboards | 356 | 50 | - | 1 | A curated list of awesome leaderboard-oriented resources for AI domain | 2026-05-24 |
| 87 | cybergym | 355 | 53 | Python | 1 | CyberGym is a large-scale, high-quality cybersecurity evaluation framework designed to rigorously assess the capabilities of AI agents on real-world vulnerability analysis tasks. | 2026-05-18 |
| 88 | xpu-perf | 354 | 127 | Python | 23 | [HPCA 2026] AI Accelerator Benchmark focuses on evaluating AI Accelerators from a practical production perspective, including the ease of use and versatility of software and hardware. | 2026-04-22 |
| 89 | llm | 349 | 85 | Rust | 14 | A powerful Rust library and CLI tool to unify and orchestrate multiple LLM, Agent and voice backends (OpenAI, Claude, Gemini, Ollama, ElevenLabs...) with a single, extensible API. Build, chain, evalua... | 2026-05-05 |
| 90 | career-ops-plugin | 339 | 85 | HTML | 1 | Claude Cowork plugin for job seekers. 9 AI skills: evaluate job postings, generate ATS-optimized resumes, scan company career portals, track applications, draft outreach. Works in any industry. | 2026-04-07 |
| 91 | FlagEval | 337 | 31 | Python | 5 | FlagEval is an evaluation toolkit for AI large foundation models. | 2025-04-24 |
| 92 | adversarial-explainable-ai | 334 | 49 | - | 3 | 💡 Adversarial attacks on explanations and how to defend them | 2024-11-30 |
| 93 | ai-rag-chat-evaluator | 330 | 112 | Python | 15 | Tools for evaluation of RAG Chat Apps using Azure AI Evaluate SDK and OpenAI | 2026-01-26 |
| 94 | bananalyzer | 328 | 39 | Python | 1 | Open source AI Agent evaluation framework for web tasks 🐒🍌 | 2025-01-01 |
| 95 | FinSight-AI | 321 | 27 | Java | 0 | AI equity research agent with resilient workflows, Redis Lua single-flight, pgvector RAG, versioned reports, evidence tracing, and RAG evaluation. | 2026-05-24 |
| 96 | ClawBench | 321 | 20 | Python | 40 | Open-source benchmark for browser AI agents on 153 everyday online tasks across 144 live websites. 5-layer recording + DOM-match + LLM judge. Top score 33.3%. | 2026-05-24 |
| 97 | HackSynth | 310 | 51 | Python | 0 | LLM Agent and Evaluation Framework for Autonomous Penetration Testing | 2025-06-24 |
| 98 | oss-investment-scorecard | 297 | 42 | - | 5 | A structured 5-dimension scoring framework for evaluating open-source AI projects from a VC investment perspective. Maintained by Lucy Chen, EIR at Zoo Capital (Singapore, $2B+ AUM). | 2026-04-25 |
| 99 | beyondllm | 294 | 47 | Jupyter Notebook | 5 | Build, evaluate and observe LLM apps | 2025-01-27 |
| 100 | Evaluator | 283 | 48 | Python | 4 | Open-source library for scalable, reproducible evaluation of AI models and benchmarks. | 2026-05-22 |