# mcp-llm-eval

Source: https://glimind.com/server/pypi%2Fmcp-llm-eval
Registry id: `pypi/mcp-llm-eval`
Status: unknown

## Install

```json
{
  "mcpServers": {
    "mcp-llm-eval": {
      "command": "uvx",
      "args": [
        "mcp-llm-eval"
      ]
    }
  }
}
```

Claude Code:

```sh
claude mcp add mcp-llm-eval -- uvx mcp-llm-eval
```

## Tools (10)

- `check_retrieval_drift`: Compare two retrieval result files and flag metrics that regressed beyond tolerance
- `check_thresholds`: Validate evaluation results against quality gates (faithfulness, relevance, TTFT, cost, retrieval, RAG)
- `compare_runs`: Compare two evaluation runs and detect regressions beyond configurable tolerance
- `evaluate_rag_end_to_end`: Full RAG pipeline — retrieve, generate, score with context_relevance and citation_faithfulness judges
- `evaluate_retrieval`: Run retrieval metrics (recall@k, precision@k, MRR, nDCG@k) against a labelled chunk dataset; returns per-query metrics, aggregate, p50/p95 latency
- `format_pr_comment`: Generate a markdown PR comment from evaluation results with regression details and threshold status
- `get_evaluation`: Retrieve full details of a specific run (per-question scores, responses, judge reasoning)
- `list_evaluations`: List past evaluation runs with metadata (timestamp, models, cost, pass/fail)
- `run_evaluation`: Load a dataset, query models via streaming, score with LLM-as-judge, return per-question scores and aggregate summary
- `simulate_poisoned_corpus`: Reserved stub; schema is stable today and returns a not-implemented response

Measured by Glimind (https://glimind.com): MCP handshake + tool list only, no tool is ever called. CC BY 4.0.
