Top 100 Prompt Evaluation Repositories
Ranking
| Ranking | Project Name | Stars | Forks | Language | Open Issues | Description | Last Commit |
|---|---|---|---|---|---|---|---|
| 1 | langfuse | 27,695 | 2,829 | TypeScript | 317 | 🪢 Open source LLM engineering platform: LLM Observability, metrics, evals, prompt management, playground, datasets. Integrates with OpenTelemetry, Langchain, OpenAI SDK, LiteLLM, and more. 🍊YC W23 | 2026-05-22 |
| 2 | promptfoo | 21,486 | 1,883 | TypeScript | 75 | Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line ... | 2026-05-22 |
| 3 | Kiln | 4,843 | 368 | Python | 22 | Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more. | 2026-05-22 |
| 4 | agenta | 4,140 | 523 | TypeScript | 66 | The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place. | 2026-05-22 |
| 5 | ChainForge | 2,985 | 253 | TypeScript | 58 | An open-source visual programming environment for battle-testing prompts to LLMs. | 2026-04-06 |
| 6 | promptbench | 2,801 | 220 | Python | 0 | A unified evaluation framework for large language models | 2026-02-20 |
| 7 | openlit | 2,461 | 280 | TypeScript | 15 | Open source platform for AI Engineering: OpenTelemetry-native LLM Observability, GPU Monitoring, Guardrails, Evaluations, Prompt Management, Vault, Playground. 🚀💻 Integrates with 50+ LLM Providers, ... | 2026-05-22 |
| 8 | prompty | 1,213 | 115 | TypeScript | 8 | Prompty makes it easy to create, manage, debug, and evaluate LLM prompts for your AI applications. Prompty is an asset class and format for LLM prompts designed to enhance observability, understandab... | 2026-05-20 |
| 9 | Safety-Prompts | 1,165 | 89 | - | 1 | Chinese safety prompts for evaluating and improving the safety of LLMs. 中文安全prompts，用于评估和提升大模型的安全性。 | 2024-02-27 |
| 10 | JamAIBase | 1,095 | 40 | Python | 1 | The collaborative spreadsheet for AI. Chain cells into powerful pipelines, experiment with prompts and models, and evaluate LLM responses in real-time. Work together seamlessly to build and iterate on... | 2026-05-21 |
| 11 | RepoToTextForLLMs | 791 | 102 | Python | 3 | Automate the analysis of GitHub repositories for LLMs with RepoToTextForLLMs. Fetch READMEs, structure, and non-binary files efficiently. Outputs include analysis prompts to aid in comprehensive repo ... | 2024-05-21 |
| 12 | prompt-forge | 775 | 76 | Go | 3 | AI prompt engineering workbench for crafting, testing, and systematically evaluating prompts with powerful analysis tools. | 2025-07-16 |
| 13 | autoprompt | 640 | 86 | Python | 8 | AutoPrompt: Automatic Prompt Construction for Masked Language Models. | 2024-08-24 |
| 14 | SAMs-CDConcepts-Eval | 575 | 31 | Python | 0 | Inspiring the Next Generation of Segment Anything Models: Comprehensively Evaluate SAM and SAM 2 with Diverse Prompts Towards Context-Dependent Concepts under Different Scenes | 2025-08-27 |
| 15 | promptimize | 494 | 43 | Python | 10 | Promptimize is a prompt engineering evaluation and testing toolkit. | 2026-05-18 |
| 16 | spring-ai-alibaba-admin | 392 | 75 | Java | 19 | Console supporting tracing, prompt engineering and evaluation for Agents developed with Spring AI Alibaba. | 2026-01-11 |
| 17 | genaiops-promptflow-template | 360 | 275 | Python | 28 | GenAIOps with Prompt Flow is a "GenAIOps template and guidance" to help you build LLM-infused apps using Prompt Flow. It offers a range of features including Centralized Code Hosting, Lifecycle Manage... | 2026-04-13 |
| 18 | IntelliNode | 273 | 18 | JavaScript | 3 | Access the latest AI models like ChatGPT, LLaMA, Deepseek, Diffusion, Hugging face, and beyond through a unified prompt layer and performance evaluation | 2025-10-19 |
| 19 | Is-ChatGPT-A-Good-Translator | 249 | 16 | Python | 0 | A preliminary evaluation of ChatGPT/GPT-4 for machine translation. | 2025-04-10 |
| 20 | ISDrama | 236 | 0 | Python | 1 | Dataset and evaluation code of ISDrama(ACM-MM 2025): Immersive Spatial Drama Generation through Multimodal Prompting | 2025-08-20 |
| 21 | agent-systems-handbook | 202 | 31 | MDX | 10 | A practical AI agents handbook covering agent systems, agentic workflows, LangGraph, MCP/A2A, context engineering, agent memory, evaluation, observability, and multi-agent architecture. Current trend ... | 2026-05-21 |
| 22 | spikee | 187 | 41 | Python | 0 | Simple Prompt Injection Kit for Evaluation and Exploitation | 2026-05-20 |
| 23 | artkit | 168 | 38 | Jupyter Notebook | 10 | Automated prompt-based testing and evaluation of Gen AI applications | 2025-03-06 |
| 24 | LLM-RGB | 165 | 16 | TypeScript | 0 | LLM Reasoning and Generation Benchmark. Evaluate LLMs in complex scenarios systematically. | 2025-05-25 |
| 25 | LLM-Evaluation | 161 | 39 | Jupyter Notebook | 0 | Sample notebooks and prompts for LLM evaluation | 2025-11-02 |
| 26 | -L- | 145 | 48 | - | 14 | W3C Strategic Highlights September 2019 This report was prepared for the September 2019 W3C Advisory Committee Meeting (W3C Member link). See the accompanying W3C Fact Sheet — September 2019. For th... | 2021-08-18 |
| 27 | oss-llmops-stack | 140 | 7 | - | 1 | Modular, open source LLMOps stack that separates concerns: LiteLLM unifies LLM APIs, manages routing and cost controls, and ensures high-availability, while Langfuse focuses on detailed observability,... | 2025-02-15 |
| 28 | prompt-atlas | 136 | 8 | Python | 0 | Curated Prompt Card library for LLM trainers, AI PMs, and evaluation teams. | 2026-05-10 |
| 29 | political-neutrality-eval | 133 | 17 | Python | 2 | This repo contains detailed implementation information about Anthropic's paired prompts approach for evaluating political neutrality. | 2025-11-13 |
| 30 | skills | 131 | 13 | Python | 0 | Agent Skills for Langfuse, the open source LLM engineering platform for tracing, prompt management, and evaluation | 2026-05-21 |
| 31 | BIPIA | 130 | 16 | Python | 2 | A benchmark for evaluating the robustness of LLMs and defenses to indirect prompt injection attacks. | 2024-04-15 |
| 32 | FCMBench | 120 | 6 | Python | 0 | Prompts and evaluation scripts of FCMBench | 2026-05-06 |
| 33 | llmkit | 119 | 22 | Rust | 9 | A prompt management, versioning, testing, and evaluation inference server and UI toolkit. Provider agnostic and OpenAI API compatible. | 2025-06-26 |
| 34 | llm-jeopardy | 110 | 4 | JavaScript | 3 | Automated prompting and scoring framework to evaluate LLMs using updated human knowledge prompts | 2023-07-29 |
| 35 | autoarena | 108 | 9 | TypeScript | 0 | Rank LLMs, RAG systems, and prompts using automated head-to-head evaluation | 2024-12-16 |
| 36 | ai-design-skills | 108 | 16 | Python | 1 | AI Design Skills Collection: agentic skills, commands, and plugins for designing AI products — from interaction patterns to alignment, evaluation, agent orchestration, and prompt architecture. | 2026-05-12 |
| 37 | Rephrase-and-Respond | 104 | 11 | Python | 0 | Official repo of Respond-and-Respond: data, code, and evaluation | 2024-08-02 |
| 38 | litmux | 104 | 0 | Python | 0 | ⚡ Unit tests for AI. Test prompts, compare models, save money. | 2026-05-21 |
| 39 | evalstats | 103 | 2 | Python | 0 | Statistical analysis methods for comparing prompt and model performance in LLM evaluations. | 2026-05-13 |
| 40 | promptdesk | 96 | 15 | TypeScript | 0 | Promptdesk is a tool designed for effectively creating, organizing, and evaluating prompts and large language models (LLMs). | 2025-08-29 |
| 41 | ErrorAnalysis_Prompt | 91 | 5 | Ruby | 0 | 🎁[ChatGPT4MTevaluation] ErrorAnalysis Prompt for MT Evaluation in ChatGPT | 2025-10-14 |
| 42 | kereva-scanner | 78 | 7 | Python | 3 | Code scanner to check for issues in prompts and LLM calls | 2025-04-06 |
| 43 | acceptance-bench | 76 | 1 | Python | 0 | A robust LLM evaluation framework measuring acceptance vs refusal across difficulty levels. Features multi-prompt variation testing, temperature sweeping, and LLM-as-judge evaluation. | 2026-04-13 |
| 44 | llm-eval-simple | 68 | 3 | Python | 1 | llm-eval-simple is a simple LLM evaluation framework with intermediate actions and prompt pattern selection | 2026-02-28 |
| 45 | SmartGPT | 68 | 16 | TypeScript | 0 | SmartGPT is a implementation of a dynamic prompting system, inspired by AI Explained on YouTube. This tool generates multiple responses to a prompt and evalu... | 2026-01-22 |
| 46 | qa_metrics | 61 | 6 | Python | 0 | An easy python package to run quick basic QA evaluations. This package includes standardized QA evaluation metrics and semantic evaluation metrics: Black-box and Open-Source large language model promp... | 2025-07-18 |
| 47 | Jailjudge | 60 | 5 | Python | 1 | JAILJUDGE: A comprehensive evaluation benchmark which includes a wide range of risk scenarios with complex malicious prompts (e.g., synthetic, adversarial, in-the-wild, and multi-language scenarios, e... | 2024-12-13 |
| 48 | promptflow-rag-project-template | 58 | 12 | Jupyter Notebook | 2 | An end-to-end sample of RAG showcasing development, evaluation, experimentation, and deployment using Promptflow, search products like CosmosDB, PostgresSQL, and Azure AI Search | 2024-09-06 |
| 49 | SWE-rebench-V2 | 52 | 2 | Python | 3 | Tools and prompt templates used to build and evaluate SWE-rebench-v2 tasks for the paper. | 2026-03-12 |
| 50 | whence | 50 | 1 | JavaScript | 0 | Add context awareness to your apps and frameworks by safely evaluating user-defined conditional expressions. Useful for evaluating expressions in config files, prompts, key bindings, completions, temp... | 2025-02-07 |
| 51 | futureagi-sdk | 48 | 4 | Python | 0 | Production-grade AI evaluation, prompt management & observability SDK. Automated evaluations with sub-100ms guardrails. No human-in-the-loop required. Python + TypeScript. | 2026-05-20 |
| 52 | Prompt-OIRL | 45 | 7 | Python | 0 | code for paper Query-Dependent Prompt Evaluation and Optimization with Offline Inverse Reinforcement Learning | 2024-03-20 |
| 53 | OpenReason | 45 | 2 | TypeScript | 0 | Adaptive Reasoning Engine for Efficient and Context-Aware Intelligence | 2025-11-15 |
| 54 | promptlab | 42 | 11 | Python | 13 | PromptLab is a free, lightweight open-source tool to maintain all stages of prompt lifecycle- development, evaluation and deployment. It simplifies LLMOps, enabling workflows that range from simple p... | 2025-10-29 |
| 55 | NAE_CVPR2024 | 41 | 1 | Python | 6 | [CVPR 2024] Narrative Action Evaluation with Prompt-Guided Multimodal Interaction | 2024-05-16 |
| 56 | poyro | 40 | 2 | TypeScript | 3 | Test your web app LLM integrations using existing testing frameworks. Confidently launch AI-driven webapps to production. | 2025-01-13 |
| 57 | LangChain-SynData-RAG-Eval | 39 | 8 | Jupyter Notebook | 0 | LangChain, Llama2-Chat, and zero- and few-shot prompting are used to generate synthetic datasets for IR and RAG system evaluation | 2023-12-03 |
| 58 | V2P-Bench | 38 | 0 | Python | 0 | V2P-Bench: Evaluating Video-Language Understanding with Visual Prompts for Better Human-Model Interaction | 2026-05-09 |
| 59 | opsml | 37 | 2 | Rust | 71 | AI lifecycle platform for classical ML and agentic systems. Versioned, encrypted registries for data, models, experiments, prompts, agents, MCPs, and skills. Pre/deploy/post-deployment patterns. Rea... | 2026-05-16 |
| 60 | cvevals | 36 | 4 | Python | 1 | Evaluate the performance of computer vision models and prompts for zero-shot models (Grounding DINO, CLIP, BLIP, DINOv2, ImageBind, models hosted on Roboflow) | 2023-10-18 |
| 61 | AI-Agent-Studio | 34 | 5 | TypeScript | 1 | Agent Canvas helps teams describe, prototype, evaluate, and export multi-agent systems that run on the Microsoft Agent Framework. Start from a Lovable-inspired landing page, prompt an initial design, ... | 2025-11-17 |
| 62 | prompteval | 33 | 1 | Jupyter Notebook | 0 | Efficient multi-prompt evaluation of LLMs | 2024-12-06 |
| 63 | CTRLEval | 33 | 2 | Python | 2 | Codes for our paper "CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation" (ACL 2022) | 2022-06-06 |
| 64 | auto-prompting-demo | 32 | 7 | Python | 0 | A demo showing how to build an auto-improving prompt using evaluators | 2025-07-01 |
| 65 | PIArena | 32 | 5 | Python | 0 | [ACL 2026] PIArena: A Platform for Prompt Injection Evaluation | 2026-04-28 |
| 66 | PhD | 32 | 0 | - | 0 | [CVPR25 Highlight] A ChatGPT-Prompted Visual hallucination Evaluation Dataset, featuring over 100,000 data samples and four advanced evaluation modes. The dataset includes extensive contextual descrip... | 2025-04-16 |
| 67 | ralph-loop-optimizer | 31 | 0 | Python | 0 | Ralph Loop Optimizer: an AI-driven framework that turns any evaluatable codebase into a self-improving optimization loop for strategies, models, prompts, and workflows | 2026-04-30 |
| 68 | MultiPriv-PII | 30 | 2 | - | 0 | MultiPriv offers multilingual, multimodal PII entities and prompts for studying privacy risks in LLMs/VLMs. It also supports broader PII-related privacy research in diverse analysis and evaluation set... | 2025-12-10 |
| 69 | EvaluateGPT | 29 | 4 | TypeScript | 0 | Evaluate the effectiveness of a system prompt within seconds! | 2025-12-19 |
| 70 | negotiator-bots | 29 | 1 | Python | 1 | Negotiation Simulator: A tool for testing negotiation strategies using language models. Pit buying and selling bots against each other and evaluate the effectiveness of different prompts | 2023-01-07 |
| 71 | ab_challenge_eval | 28 | 2 | JavaScript | 0 | Evaluator for the A::B Prompting Challenge | 2024-04-10 |
| 72 | promptly | 27 | 2 | Jupyter Notebook | 0 | A prompt collection for testing and evaluation of LLMs. | 2026-02-24 |
| 73 | flip | 27 | 0 | Python | 0 | [NeurIPS 2025 D&B] A comprehensive benchmark suite for large-scale evaluation of federated prompt learning methods | 2025-11-19 |
| 74 | EvalLM | 26 | 8 | TypeScript | 0 | Interactive environment for evaluating LLM prompts on natural language criteria. | 2025-01-09 |
| 75 | SG-Bench | 25 | 3 | Python | 1 | SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types | 2024-11-29 |
| 76 | crispy-winner | 25 | 5 | - | 1 | # Git Credential Manager [ ](https://github.com/GitCredentialManag... | 2022-05-12 |
| 77 | HPT | 24 | 4 | Python | 0 | code for Hierarchical Prompting Taxonomy: A Universal Evaluation Framework for Large Language Models Aligned with Human Cognitive Principles | 2025-07-26 |
| 78 | Rag-Eval-flow | 24 | 2 | Python | 0 | A simple, easy-to-customize pipeline for local RAG evaluation. Starter prompts and metric definitions included. | 2026-01-14 |
| 79 | LLAMA1-Test-Set | 23 | 1 | - | 0 | We introduce the LLAMA1 Test Set, a comprehensive open-domain world knowledge QA dataset for evaluating question-answering systems. We prompted the open-source LLama-7B model for questions and short a... | 2024-03-14 |
| 80 | prompt_engine | 21 | 3 | Python | 0 | Evaluate your model using advanced prompt strategies | 2026-01-30 |
| 81 | CPAD | 21 | 3 | - | 1 | The official dataset of paper "Goal-Oriented Prompt Attack and Safety Evaluation for LLMs". | 2024-02-05 |
| 82 | shellcheck-repl | 20 | 2 | Shell | 1 | Validation of Shell Commands at the Prompt Before Evaluation | 2025-10-16 |
| 83 | BREPS | 20 | 1 | Python | 1 | [AAAI 2026] BREPS: Bounding-Box Robustness Evaluation of Promptable Segmentation | 2026-04-18 |
| 84 | evaluizer | 20 | 0 | TypeScript | 0 | Visualize LLM outputs against datasets, manually annotate results, and run automated evaluations to algorithmically optimize prompts. | 2025-11-22 |
| 85 | researchrubrics | 20 | 1 | Python | 1 | Code repository for ICLR 2026 paper "ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents" (https://www.arxiv.org/abs/2511.07685) | 2026-02-10 |
| 86 | Hypo-Workflow | 19 | 3 | JavaScript | 1 | Serialized prompt execution engine for AI agents — TDD pipeline, self-review, interrupt recovery, multi-dimensional evaluation | 2026-05-21 |
| 87 | dify-promptfoo | 19 | 0 | JavaScript | 0 | Evaluate Dify assistants with promptfoo! | 2024-03-06 |
| 88 | Llmeval-Gaokao2024-Math | 19 | 1 | - | 0 | LLM evaluation on 2024 Chinese Gaokao Mathematics — zero-contamination benchmark with dual prompt formats | 2026-04-15 |
| 89 | llm-evaluator | 18 | 1 | Jupyter Notebook | 0 | Example for Logging LLM Evaluator Prompt Responses | 2023-08-14 |
| 90 | insurance-llm-framework | 18 | 7 | Python | 0 | An interactive framework for experimenting with and evaluating open-source large language models (LLMs) for insurance-related tasks. This application provides a user-friendly interface for prompt engi... | 2026-04-21 |
| 91 | CH-JaDi-Rajput1 | 18 | 4 | - | 1 | # Cmder [ ](https://gitter.im/cmderdev/cmder?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_c... | 2022-08-16 |
| 92 | eval-data | 17 | 3 | TypeScript | 0 | Prompts and evaluation data for LLMs on real world coding and writing tasks | 2025-09-13 |
| 93 | Img2Prompt | 17 | 1 | Python | 1 | Evaluation codes of "From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models". | 2023-05-15 |
| 94 | T2I-RiskyPrompt | 17 | 0 | Python | 1 | The implement of T2I-RiskyPrompt: A Benchmark for Safety Evaluation, Attack, and Defense on Text-to-Image Model | 2025-12-13 |
| 95 | Prompt-Engineering---Infnet-Assessment | 17 | 0 | Jupyter Notebook | 0 | Prompt Engineering studies. This application is focused on evaluating prompts on different LLMs responses on a big variety of prompts. Also, there is a streamlit project attached which brings a dashbo... | 2025-01-19 |
| 96 | sqlmap | 17 | 4 | - | 3 | Usage: python sqlmap.py [options] Options: -h, --help Show basic help message and exit -hh Show advanced help message and exit --version Show program's version number and exit -v VERBOSE Verbosity lev... | 2019-02-07 |
| 97 | Development-Of-Autonomous-Downscaled-Model-Car-Using-Neural-Networks-And-Machine-Learning | 17 | 5 | Python | 1 | Machine learning using convolution neural network Required: raspberry pi pi cam compatibile rc car motor driver l293d Please create the respective files: forward idle left right reverse optimized_... | 2018-12-22 |
| 98 | ZeroPromptSearch | 16 | 4 | Python | 1 | Implementation of an LLM prompting pipeline combined with wrappers for auto-decomposing reasoning steps and for search through the reasoning-step-space (eg. by beam search, MCTS etc.) guided by self-e... | 2024-05-07 |
| 99 | Multi-Prompt-LLM-Evaluation | 16 | 0 | Python | 0 | State of What Art? A Call for Multi-Prompt LLM Evaluation | 2026-04-10 |
| 100 | ai-crypto-deep-research-benchmark | 16 | 1 | JavaScript | 0 | Open benchmark prompts and evaluator workflow for crypto deep research markdown outputs. | 2026-03-17 |