# Jcjamet Ia Qa Eval MCP server

Source: https://glimind.com/server/mcp-registry%2Fio.github.jcjamet%2Fia-qa-eval
Registry id: `mcp-registry/io.github.jcjamet/ia-qa-eval`
Status: live (checked 2026-11-22T15:58:26.691Z)

## Measurements (30 days)

- Uptime: 100%
- Checks: 843
- Outages (90 days): 0

## Install

```json
{
  "mcpServers": {
    "jcjamet-ia-qa-eval-mcp-server": {
      "type": "http",
      "url": "https://www.ia-qa.com/mcp/eval"
    }
  }
}
```

Claude Code:

```sh
claude mcp add --transport http jcjamet-ia-qa-eval-mcp-server https://www.ia-qa.com/mcp/eval
```

## Tools (39)

- `ab_test_report`: Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns mean, median, std deviation, improvement percent
- `analyze_responses`: Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pair
- `bm25_score`: Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, Ope
- `build_rag_prompt`: Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, sys
- `compare_models`: Put 2-5 AI models side by side: price per 1k tokens, context window, multimodal, reasoning, provider, pricing page. Facts only: names the cheapest and the large
- `compare_responses`: Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, len
- `consistency_check`: Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, det
- `context_window_check`: Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncatio
- `conversation_analyze`: Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential
- `cot_analyzer`: Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning de
- `count_tokens`: Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Shows whether a text fits the context window of a model and what
- `embedding_similarity`: Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: tru
- `estimate_llm_cost`: Estimate the API cost in USD for a given model and token counts, at the price read daily from the provider's own pricing page (prices_as_of, price_status). Any 
- `extract_json_from_text`: Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks 
- `find_tool`: Search available MCP tools by keyword or category before calling them. Returns names, descriptions and, on request, inputSchemas. Queries are logged 90 days, sc
- `generate_ci_workflow`: Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA: "eval_contract" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (provider 
- `generate_eval_yaml`: Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, n
- `get_testing_guidelines`: Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for 
- `hallucination_check`: Lexical hallucination check: an LLM answer's words, numbers and polarity against the source. Deterministic, no API key. Each answer sentence is aligned to its b
- `list_llm_models`: List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") o
- `llm_format_check`: Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for
- `llm_generate`: Generate text with an open-source model hosted on Groq, on the server's key — no API key needed. Models: openai/gpt-oss-20b, openai/gpt-oss-120b, qwen/qwen3.8-2
- `llm_json_schema_check`: Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-callin
- `llm_output_validator`: Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language
- `metamorphic_check`: Reference-free stability check: does an assistant's answer stay the same when the QUESTION is transformed (case, typo, paraphrase, reorder, translation)? You br
- `model_info`: Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature l
- `needle_haystack_generate`: Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retr
- `prompt_test_suite`: Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as inp
- `rag_relevance_rank`: Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.
- `rate_tool`: Give honest usage feedback on an IA-QA MCP tool. Give a score (1-5) and a comment. Rate low (1-2) if the tool was wrong, irrelevant, or a poor fit; rate high (4
- `rerank_evaluate`: Evaluate RAG retrieval: rank passages against a query, compute Precision@k / Recall@k and a PASS/FAIL verdict from your relevance labels (INCONCLUSIVE without t
- `response_quality_score`: Score an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actua
- `run_eval_contract`: Run an IA-EVAL test suite (.ia-eval.yaml contract): call the model per scenario; return passed (true, false, null if undecided) and a verdict (PASS, FAIL, PARTI
- `run_semantic_tests`: Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embedding
- `run_vlm_test_suite`: Test a vision model (OpenAI, Anthropic or Gemini) on one image: each test case asks a question and asserts on the answer (contains, not_contains, json_format, m
- `similarity_score`: Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key 
- `skill_interview`: Help a person write a SKILL.md they can test. No answers: the questions (why + example), written for the person; invented answers make the result meaningless. W
- `test_skill`: Test a SKILL.md. No model, every call: lint (gate on lint.valid) and review (description lacks what/when, words shared with other_skills, leftovers, no done-che
- `token_budget_calculator`: Plan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds the model's context window (any model f

Measured by Glimind (https://glimind.com): MCP handshake + tool list only, no tool is ever called. CC BY 4.0.
