Skip to content

IA-QA Eval & RAG — 38 tools MCP server

io.github.JcJamet/ia-qa-evalJcJametstreamable-httpv1.0.1Scanned

LLM & RAG evals: hallucination, eval contracts, semantic tests, tokens & model info. Free, no key.

39 tools: 1 can delete or cancel things, 25 change things and 13 only read.

Tools: 1 delete, 0 irreversible, 25 write, 13 read.
  • Reads private data · 1
  • Reads text others write · 7
  • Can send data out · 0
  • 12 findings

Tools listed without sign-in. Remote posture

With Anlyon

1 tool here can delete or cancel things. Govern it.

  • Anlyon lists the tools and adopts none.
  • Each tool you adopt gets approvals and receipts.
  • Your agent never holds the key.
Govern this server →

A tool your agent calls directly is not routed through Anlyon and is not governed by it. Anlyon Radar only lists and labels, and these labels are the same either way.

What the IA-QA Eval & RAG — 38 tools MCP server can do

The IA-QA Eval & RAG — 38 tools MCP server is published by the GitHub account JcJamet. It has 39 tools. It can delete or cancel things (1 tool: rate_tool) and create or change data (25 tools: hallucination_check, run_eval_contract, run_semantic_tests, generate_ci_workflow and 21 more). 13 tools only read.

1 tool reads private data (conversation_analyze); 7 tools bring in text other people can write (run_eval_contract, generate_eval_yaml, test_skill and 4 more).

The fixed checks matched 12 findings (credentials-mention and annotation-conflict). Each one comes with the text it matched.

Its remote endpoint lists its tools without sign-in, and its tools changed once since Anlyon Radar first saw them.

Last scanned Nov 9, 2026, version 1.0.1.

Tools

Grouped by what each tool does, most consequential first. Open one for its reasons and full description; descriptions are the publisher's own words.

39 tools

Deletes or cancels 1Claude Code: deny
  • rate_toolGive honest usage feedback on an IA-QA MCP tool.1 finding

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it destructive (destructiveHint: true).

    Findings

    • Info

      The description mentions "credential".

      Matched text

      description: ".... Comments are screened: one carrying a credential is not stored, personal data becomes pl..."

      What this check looks for
    Description, as the server wrote it
    Give honest usage feedback on an IA-QA MCP tool. Give a score (1-5) and a comment. Rate low (1-2) if the tool was wrong, irrelevant, or a poor fit; rate high (4-5) only if it genuinely solved your need. Maintainers read ratings, one vote per agent and tool a month: comments and per-tool scores are not published, only the overall average and vote count. Comments are screened: one carrying a credential is not stored, personal data becomes placeholders — describe what happened, not what a model should do. Skip routine successes: signal, not praise. Example: rate_tool({ tool_name: "format_json", score: 2, comment: "Tried to pretty-print a JSON5 file, it rejected trailing commas — not usable for my case." })

    Annotations the server declares

    • readOnlyHint: false
    • destructiveHint: true
    • idempotentHint: false
    • openWorldHint: false
Changes things 25Claude Code: ask
  • ab_test_reportGenerate an A/B test report comparing two prompts or model configurations.1 finding

    Why it is in this group

    • The description's first verb is "generate": a word for a change.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.

    Findings

    • Warn

      The server marks "ab_test_report" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

      Matched text

      annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":true,"openWorldHint":false}; The description's first verb is "generate": a word for a change.

      What this check looks for
    Description, as the server wrote it
    Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns mean, median, std deviation, improvement percentage, and a Welch t-test at 95 %: `winner` names a variant only when the difference is significant — otherwise "no_significant_difference", or "insufficient_samples" under 2 scores per variant.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • bm25_scoreCompute BM25 relevance score between a query and one or more documents.No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.
    Description, as the server wrote it
    Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search. Returns ranked results with normalized scores.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • build_rag_promptAssemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query.No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.
    Description, as the server wrote it
    Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, system instruction injection, and source attribution.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • compare_modelsPut 2-5 AI models side by side: price per 1k tokens, context window, multimodal, reasoning, provider, pricing page.1 finding

    Why it is in this group

    • The name "compare_models" says "compare": a word for reading.
    • The description's first verb is "put": a word for a change.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.

    Findings

    • Warn

      The server marks "compare_models" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

      Matched text

      annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":true,"openWorldHint":false}; The name "compare_models" says "compare": a word for reading.

      What this check looks for
    Description, as the server wrote it
    Put 2-5 AI models side by side: price per 1k tokens, context window, multimodal, reasoning, provider, pricing page. Facts only: names the cheapest and the largest context, never a "best model" — run your cases on each to choose.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • context_window_checkGiven an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window.No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.
    Description, as the server wrote it
    Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncation risk.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • embedding_similarityCompute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams).No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.
    Description, as the server wrote it
    Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: true vector embeddings are run_semantic_tests with mode="embeddings" and an OpenAI key. Supports single pair or batch mode with pipe-separated pairs. Useful for RAG retrieval testing, semantic search evaluation, and text deduplication.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • estimate_llm_costEstimate the API cost in USD for a given model and token counts, at the price read daily from the provider's own pricing page (prices_as_of, price_status).No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.
    Description, as the server wrote it
    Estimate the API cost in USD for a given model and token counts, at the price read daily from the provider's own pricing page (prices_as_of, price_status). Any model from list_llm_models.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • extract_json_from_textExtract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text).No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.
    Description, as the server wrote it
    Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks and inline JSON.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • generate_ci_workflowGenerate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA: "eval_contract" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (needs a provider key as a rep…1 finding

    Why it is in this group

    • The name "generate_ci_workflow" says "generate": a word for a change.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.

    Findings

    • Warn

      The server marks "generate_ci_workflow" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

      Matched text

      annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":true,"openWorldHint":false}; The name "generate_ci_workflow" says "generate": a word for a change.

      What this check looks for
    Description, as the server wrote it
    Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA: "eval_contract" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (needs a provider key as a repo secret), "cli_checks" runs deterministic tools via npx @ia-qa/cli and fails the build on their verdict, "selector_drift" gates E2E selectors with @ia-qa/self-healing. Deterministic template, no LLM. Returns the YAML, the secrets to create and the next steps. Pair with generate_eval_yaml for the contract.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • generate_eval_yamlGenerate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do.Reads text others write1 finding

    Why it is in this group

    • The name "generate_eval_yaml" says "generate": a word for a change.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.

    What it can reach

    • untrusted-inputThe server marks it openWorldHint: true: it reaches things outside the server's own data.

    Findings

    • Warn

      The server marks "generate_eval_yaml" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

      Matched text

      annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":false,"openWorldHint":true}; The name "generate_eval_yaml" says "generate": a word for a change.

      What this check looks for
    Description, as the server wrote it
    Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators (cosine_similarity, contains_check, hallucination_check, etc.) based on the task type.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: false
    • openWorldHint: true
  • hallucination_checkLexical hallucination check: verifies an LLM answer's words, numbers and polarity against the provided source/context.No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.
    Description, as the server wrote it
    Lexical hallucination check: verifies an LLM answer's words, numbers and polarity against the provided source/context. Fast, deterministic, no API key needed. Each answer sentence is aligned to its best-matching source sentence, so a number only counts as support when it sits on the SAME statement ("founded in 1998" is not grounded by "sold 1998 units"), and a negation or antonym flip against that sentence returns verdict "contradicted" — the corrupted-fact hallucination that reuses source vocabulary. Limitations: still lexical — it cannot follow a paraphrase, a synonym, or multi-sentence reasoning, so a "well_grounded" verdict means "nothing lexical found", never "verified true". Entailment: run_semantic_tests (NLI/embedding) or a calibrated judge.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • llm_generateGenerate text with an open-source model hosted on Groq, on the server's key — no API key needed.Reads text others write1 finding

    Why it is in this group

    • The name "llm_generate" says "generate": a word for a change.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.

    What it can reach

    • untrusted-inputThe server marks it openWorldHint: true: it reaches things outside the server's own data.

    Findings

    • Warn

      The server marks "llm_generate" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

      Matched text

      annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":false,"openWorldHint":true}; The name "llm_generate" says "generate": a word for a change.

      What this check looks for
    Description, as the server wrote it
    Generate text with an open-source model hosted on Groq, on the server's key — no API key needed. Models: openai/gpt-oss-20b, openai/gpt-oss-120b, qwen/qwen3.8-27b (default "openai/gpt-oss-20b"). HuggingFace models are not served on the server key. A reasoning model that spends max_tokens thinking is retried once at 4096. Rate-limited.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: false
    • openWorldHint: true
  • metamorphic_checkReference-free stability check: does an assistant's answer stay the same when the QUESTION is transformed (case, typo, paraphrase, reorder, translation)?No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.
    Description, as the server wrote it
    Reference-free stability check: does an assistant's answer stay the same when the QUESTION is transformed (case, typo, paraphrase, reorder, translation)? You bring the outputs; no model is called (tfidf mode is deterministic and free). Default thresholds: case .95, typo .90, paraphrase .80, reorder .80, translation .75 (embeddings only), specialization .60 (advisory, never gated). Returns PASS / FAIL / INVALID — INVALID when the BASE answer is a refusal or too short, so a model that refuses everything never scores 1.0. Pair with run_semantic_tests for correctness.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • needle_haystack_generateGenerate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position.1 finding

    Why it is in this group

    • The name "needle_haystack_generate" says "generate": a word for a change.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.

    Findings

    • Warn

      The server marks "needle_haystack_generate" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

      Matched text

      annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":false,"openWorldHint":false}; The name "needle_haystack_generate" says "generate": a word for a change.

      What this check looks for
    Description, as the server wrote it
    Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystack, the question to ask, and metadata. No API key needed.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: false
    • openWorldHint: false
  • prompt_test_suiteDefine a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria.No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.
    Description, as the server wrote it
    Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • rag_relevance_rankRank an array of text chunks by relevance to a query using TF-IDF scoring.No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.
    Description, as the server wrote it
    Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • rerank_evaluateEvaluate RAG retrieval: rank passages against a query, compute Precision@k / Recall@k and a PASS/FAIL verdict from your relevance labels.Reads text others write1 finding

    Why it is in this group

    • The name "rerank_evaluate" says "evaluate": a word for a change.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.

    What it can reach

    • untrusted-inputThe server marks it openWorldHint: true: it reaches things outside the server's own data.

    Findings

    • Warn

      The server marks "rerank_evaluate" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

      Matched text

      annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":false,"openWorldHint":true}; The name "rerank_evaluate" says "evaluate": a word for a change.

      What this check looks for
    Description, as the server wrote it
    Evaluate RAG retrieval: rank passages against a query, compute Precision@k / Recall@k and a PASS/FAIL verdict from your relevance labels. Modes: (1) give every passage the `score` YOUR reranker produced — deterministic, offline, the mode to gate CI on; (2) no scores, no key — a local BM25 keyword baseline, never a judgement of a neural reranker; (3) api_key for an NVIDIA account that still has reranking (NVIDIA retired the hosted endpoints on 2026-05-18).

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: false
    • openWorldHint: true
  • response_quality_scoreScore an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length.No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.
    Description, as the server wrote it
    Score an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actually measured — or total_score null with the reason when none is given, because relevance and correctness cannot be read off word overlap (a correct one-word answer shares no words with its question). Meaning against a reference answer: run_semantic_tests. Also returns unscored signals: question-term overlap, average sentence length, markdown.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • run_eval_contractRun an IA-EVAL test suite (.ia-eval.yaml contract): call the model per scenario; return passed (true, false, null if undecided) and a verdict (PASS, FAIL, PARTIAL).Reads text others write1 finding

    Why it is in this group

    • The name "run_eval_contract" says "run": a word for a change.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.

    What it can reach

    • untrusted-inputThe server marks it openWorldHint: true: it reaches things outside the server's own data.

    Findings

    • Warn

      The server marks "run_eval_contract" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

      Matched text

      annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":false,"openWorldHint":true}; The name "run_eval_contract" says "run": a word for a change.

      What this check looks for
    Description, as the server wrote it
    Run an IA-EVAL test suite (.ia-eval.yaml contract): call the model per scenario; return passed (true, false, null if undecided) and a verdict (PASS, FAIL, PARTIAL). Expectations are assertions that fail a scenario whatever its score; evaluators give the score. Hosted: pass inline_contract. validate_only calls no model.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: false
    • openWorldHint: true
  • run_semantic_testsSemantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L.1 finding

    Why it is in this group

    • The name "run_semantic_tests" says "run": a word for a change.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.

    Findings

    • Warn

      The server marks "run_semantic_tests" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

      Matched text

      annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":true,"openWorldHint":false}; The name "run_semantic_tests" says "run": a word for a change.

      What this check looks for
    Description, as the server wrote it
    Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pipe the JSON verdict field to gate a build.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • run_vlm_test_suiteTest a vision model (OpenAI, Anthropic or Gemini) on one image: each test case asks a question and asserts on the answer (contains, not_contains, json_format, min_length, max_lengt…Reads text others write1 finding

    Why it is in this group

    • The name "run_vlm_test_suite" says "run": a word for a change.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.

    What it can reach

    • untrusted-inputThe server marks it openWorldHint: true: it reaches things outside the server's own data.

    Findings

    • Warn

      The server marks "run_vlm_test_suite" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

      Matched text

      annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":false,"openWorldHint":true}; The name "run_vlm_test_suite" says "run": a word for a change.

      What this check looks for
    Description, as the server wrote it
    Test a vision model (OpenAI, Anthropic or Gemini) on one image: each test case asks a question and asserts on the answer (contains, not_contains, json_format, min_length, max_length, semantic_contains = TF-IDF cosine ≥ 0.4). Returns per-case PASS/FAIL, the pass rate, an overall PASS/WARNING/FAIL against threshold, and latency. BYOK: your key for the model's provider; the image is fetched only from a public address.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: false
    • openWorldHint: true
  • similarity_scoreCompute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU.No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.
    Description, as the server wrote it
    Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key needed. Ideal for LLM eval (expected vs actual), RAG quality checks, and NLG benchmarking. Supports batch mode.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • skill_interviewHelp a person write a SKILL.md they can test.No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.
    Description, as the server wrote it
    Help a person write a SKILL.md they can test. No answers: the questions (why + example), written for the person; invented answers make the result meaningless. With answers: remarks per answer (no model), the SKILL.md, scenarios for test_skill, a starter .ia-eval.yaml.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • test_skillTest a SKILL.md.Reads text others writeNo findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.

    What it can reach

    • untrusted-inputThe server marks it openWorldHint: true: it reaches things outside the server's own data.
    Description, as the server wrote it
    Test a SKILL.md. No model, every call: lint (gate on lint.valid) and review (description lacks what/when, words shared with other_skills, leftovers, no done-check, missing files; why + fix each). Unless lint_only, a model picks a skill per request from names+descriptions: trigger_accuracy PASS ≥80, FIX ≥50, else BLOCK; your scenarios = gate, generated = signal; unanswered = INCONCLUSIVE. Server key 10 runs/day.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: false
    • openWorldHint: true
  • token_budget_calculatorPlan token allocation across system prompt, user input, context/RAG chunks, and expected output.No findings

    Why it is in this group

    • Nothing in its name or description says what it does, so it is counted as a write.
    • The server marks it read-only (readOnlyHint: true), but an annotation never lowers a class.
    Description, as the server wrote it
    Plan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds the model's context window (any model from list_llm_models).

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
Only reads 13Show 13 read-only tools
  • analyze_responsesSemantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox).No findings

    Why it is in this group

    • The name "analyze_responses" says "analyze": a word for reading.
    Description, as the server wrote it
    Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-representative output, and the outlier. With a `reference` (ground truth): also ranks every output by closeness (token cosine + ROUGE-L composite) and names the closest. No LLM, no key, runs locally. The reference mode is gate-able in CI; the consensus/outlier mode is a signal (no reference to fail against). You bring the outputs (2+). A 2-way head-to-head with structural JSON diff is compare_responses.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • compare_responsesCompare two ALREADY-PRODUCED outputs (e.g.No findings

    Why it is in this group

    • The name "compare_responses" says "compare": a word for reading.
    Description, as the server wrote it
    Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a `reference` (ground truth) is given, scores each output against it and picks the closer one. If `model` + `api_key` are given, an LLM judge also picks a qualitative winner for the task. No re-execution — you bring the outputs.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • consistency_checkCompare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison).No findings

    Why it is in this group

    • The description's first verb is "compare": a word for reading.
    Description, as the server wrote it
    Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs "Paris is the French capital" may score low despite semantic equivalence. True semantic consistency: run_semantic_tests in embedding mode. Essential for determinism testing.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • conversation_analyzeAnalyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition.Reads private dataReads text others writeNo findings

    Why it is in this group

    • The name "conversation_analyze" says "analyze": a word for reading.

    What it can reach

    • private-dataThe name "conversation_analyze" says "analyze" and "conversation": it reads messages.
    • private-dataThe description's first sentence says "analyze" and "conversation": it reads messages.
    • untrusted-inputThe name "conversation_analyze" says "analyze" and "conversation": it reads messages, which other people can write.
    • untrusted-inputThe description's first sentence says "analyze" and "conversation": it reads messages, which other people can write.
    Description, as the server wrote it
    Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential for chatbot QA.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • cot_analyzerAnalyze a Chain-of-Thought (CoT) or reasoning trace from an LLM.No findings

    Why it is in this group

    • The description's first verb is "analyze": a word for reading.
    Description, as the server wrote it
    Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth. Useful for o1/o3/DeepSeek-R1 evaluation.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • count_tokensEstimate the token count of a text string using the cl100k_base approximation (~4 chars/token).No findings

    Why it is in this group

    • The name "count_tokens" says "count": a word for reading.
    Description, as the server wrote it
    Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Shows whether a text fits the context window of a model and what it costs. Returns token estimate, character count, and word count.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • find_toolSearch available MCP tools by keyword or category before calling them.1 finding

    Why it is in this group

    • The name "find_tool" says "find": a word for reading.

    Findings

    • Info

      The description mentions "credentials".

      Matched text

      description: "...ueries are logged 90 days, screened for credentials and personal data. Categories: data, de..."

      What this check looks for
    Description, as the server wrote it
    Search available MCP tools by keyword or category before calling them. Returns names, descriptions and, on request, inputSchemas. Queries are logged 90 days, screened for credentials and personal data. Categories: data, dev, encoding, eval, integrations, llm, meta, qa, rag, security, text, web.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • get_testing_guidelinesQuery the IA-QA methodology knowledge base.No findings

    Why it is in this group

    • The name "get_testing_guidelines" says "get": a word for reading.
    Description, as the server wrote it
    Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for a given topic. Call without a topic to list all available topics. Topics: llm-unit-testing, rag-pipeline, prompt-stability, prompt-ab-testing, embedding-quality, eval-framework, semantic-testing, auto-testing, security, api-testing, ci-cd, multimodal, llm-data-security, agent-observability, pro-tips, learning-paths, golden-dataset, selector-drift, qa-recipes, playbooks. Not sure where to start testing an LLM, RAG pipeline or agent? Call without a topic (or with "start-here"): it maps what you are testing to the tools to call and the output field to gate CI on. A plain question such as "how do I test my RAG" also resolves to the right topic.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • list_llm_modelsList all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities.No findings

    Why it is in this group

    • The name "list_llm_models" says "list": a word for reading.
    Description, as the server wrote it
    List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") or return the full catalog. Use this to discover which models are available before calling an LLM API, or to compare providers.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • llm_format_checkValidate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex.No findings

    Why it is in this group

    • The description's first verb is "validate": a word for reading.
    Description, as the server wrote it
    Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for structured output testing.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • llm_json_schema_checkValidate that an LLM JSON output matches a JSON Schema definition.No findings

    Why it is in this group

    • The description's first verb is "validate": a word for reading.
    Description, as the server wrote it
    Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-calling and structured output testing.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • llm_output_validatorValidate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safe…No findings

    Why it is in this group

    • The description's first verb is "validate": a word for reading.
    Description, as the server wrote it
    Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns. Essential for QA testing LLM-powered features.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false
  • model_infoGet detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list.No findings

    Why it is in this group

    • The description's first verb is "get": a word for reading.
    Description, as the server wrote it
    Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list. Covers 30+ models from OpenAI, Anthropic, Google, DeepSeek, Meta, Mistral, Cohere, xAI.

    Annotations the server declares

    • readOnlyHint: true
    • destructiveHint: false
    • idempotentHint: true
    • openWorldHint: false

This is not a safety rating, and no finding is not a safety guarantee. The groups and findings come from fixed checks of the tool definitions the server returned on the date shown, not from its code or what it does when called. How the checks work.

Exfil surface

Which tools read private data, which read text someone else can write, and which can send data out. One of each is a path a prompt injection can use.

No private-data, untrusted-input, outbound path in this server alone: no tool is tagged outbound.

Reads private data1
Reads files, mail, messages, databases, secrets, calendars or repositories.
Reads text others write7
Brings in text someone else can write: web pages, inbound email, messages, issues and comments.
Can send data out0
Can carry data to people or systems: it sends, posts or publishes, or takes a recipient, channel or URL.

Other servers in the same agent add their own tools to these paths. Audit your whole setup to see them together.

Remote posture

What anyone on the internet sees at the remote endpoint: an anonymous initialize and tools/list, and the public OAuth metadata.

Tools listed without sign-in

Looked at Nov 18, 2026, 16:24 UTC, unchanged since Nov 11, 2026. No credentials were sent and no tool was called. What is looked at.

Reachable
yes
HTTP status of an anonymous initialize
200
tools/list served without credentials
yes
Tools that change things, listed without credentials
26
Protocol version
2025-06-18
WWW-Authenticate on 401
not observed
resource_metadata in WWW-Authenticate
not observed
RFC 9728 protected resource metadata
not observed
Authorization servers
not observed
Legacy HTTP+SSE transport
no
Dynamic client registration
not observed
Client ID metadata documents
not observed
  • HighThe whole server

    The endpoint served tools/list without credentials, and the list has 26 tools that change things (1 destructive, 25 write). Whether a call also works without credentials was not tested: tools are never called.

    Matched text

    https://www.ia-qa.com/mcp/eval: tools/list returned 39 tools to an anonymous client, including rate_tool (destructive), hallucination_check (write), run_eval_contract (write), run_semantic_tests (write), generate_ci_workflow (write) and 21 more

    What this check looks for

Posture changes

  1. Nov 11, 2026
    • Tools that change things, listed without credentials: 25 to 26
  2. Nov 4, 2026
    • Tools that change things, listed without credentials: 24 to 25

Findings

Matches of fixed checks, with the text they matched.

10 warn2 info
  • The server marks "run_eval_contract" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

    Matched text

    annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":false,"openWorldHint":true}; The name "run_eval_contract" says "run": a word for a change.

    What this check looks for
  • The server marks "run_semantic_tests" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

    Matched text

    annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":true,"openWorldHint":false}; The name "run_semantic_tests" says "run": a word for a change.

    What this check looks for
  • The server marks "generate_ci_workflow" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

    Matched text

    annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":true,"openWorldHint":false}; The name "generate_ci_workflow" says "generate": a word for a change.

    What this check looks for
  • The server marks "needle_haystack_generate" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

    Matched text

    annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":false,"openWorldHint":false}; The name "needle_haystack_generate" says "generate": a word for a change.

    What this check looks for
  • The server marks "generate_eval_yaml" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

    Matched text

    annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":false,"openWorldHint":true}; The name "generate_eval_yaml" says "generate": a word for a change.

    What this check looks for
  • The server marks "ab_test_report" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

    Matched text

    annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":true,"openWorldHint":false}; The description's first verb is "generate": a word for a change.

    What this check looks for
Show 6 more findings
  • The server marks "run_vlm_test_suite" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

    Matched text

    annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":false,"openWorldHint":true}; The name "run_vlm_test_suite" says "run": a word for a change.

    What this check looks for
  • The server marks "rerank_evaluate" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

    Matched text

    annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":false,"openWorldHint":true}; The name "rerank_evaluate" says "evaluate": a word for a change.

    What this check looks for
  • The server marks "compare_models" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

    Matched text

    annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":true,"openWorldHint":false}; The name "compare_models" says "compare": a word for reading.

    What this check looks for
  • The server marks "llm_generate" read-only (readOnlyHint: true), and its own name, description or input give the class "write".

    Matched text

    annotations: {"readOnlyHint":true,"destructiveHint":false,"idempotentHint":false,"openWorldHint":true}; The name "llm_generate" says "generate": a word for a change.

    What this check looks for
  • The description mentions "credentials".

    Matched text

    description: "...ueries are logged 90 days, screened for credentials and personal data. Categories: data, de..."

    What this check looks for
  • The description mentions "credential".

    Matched text

    description: ".... Comments are screened: one carrying a credential is not stored, personal data becomes pl..."

    What this check looks for

History

Changes to its tool definitions and to its remote endpoint, newest first.

  1. PostureRemote posture changed. Tools that change things, listed without credentials: 25 -> 26.View diff
    • Tools that change things, listed without credentials: 25 to 26
  2. Tools1 added, 38 changed.+1 added~38 changedView diff
  3. PostureRemote posture changed. Tools that change things, listed without credentials: 24 -> 25.View diff
    • Tools that change things, listed without credentials: 24 to 25

Install

Pinned to version 1.0.1. Values in angle brackets are yours to fill in.

claude mcp add --transport http ia-qa-eval https://www.ia-qa.com/mcp/eval
claude mcp add --transport http ia-qa-eval https://www.ia-qa.com/mcp/eval

Run in a terminal.

Claude Code permissions

Allow the reads, ask before a change or a way out, deny the deletes. Paste into .claude/settings.json.

.claude/settings.json
{
  "permissions": {
    "allow": [
      "mcp__ia-qa-eval__get_testing_guidelines",
      "mcp__ia-qa-eval__find_tool",
      "mcp__ia-qa-eval__llm_output_validator",
      "mcp__ia-qa-eval__compare_responses",
      "mcp__ia-qa-eval__analyze_responses",
      "mcp__ia-qa-eval__llm_format_check",
      "mcp__ia-qa-eval__consistency_check",
      "mcp__ia-qa-eval__llm_json_schema_check",
      "mcp__ia-qa-eval__conversation_analyze",
      "mcp__ia-qa-eval__cot_analyzer",
      "mcp__ia-qa-eval__count_tokens",
      "mcp__ia-qa-eval__model_info",
      "mcp__ia-qa-eval__list_llm_models"
    ],
    "ask": [
      "mcp__ia-qa-eval__hallucination_check",
      "mcp__ia-qa-eval__run_eval_contract",
      "mcp__ia-qa-eval__run_semantic_tests",
      "mcp__ia-qa-eval__generate_ci_workflow",
      "mcp__ia-qa-eval__needle_haystack_generate",
      "mcp__ia-qa-eval__generate_eval_yaml",
      "mcp__ia-qa-eval__metamorphic_check",
      "mcp__ia-qa-eval__test_skill",
      "mcp__ia-qa-eval__skill_interview",
      "mcp__ia-qa-eval__prompt_test_suite",
      "mcp__ia-qa-eval__response_quality_score",
      "mcp__ia-qa-eval__ab_test_report",
      "mcp__ia-qa-eval__similarity_score",
      "mcp__ia-qa-eval__run_vlm_test_suite",
      "mcp__ia-qa-eval__embedding_similarity",
      "mcp__ia-qa-eval__rag_relevance_rank",
      "mcp__ia-qa-eval__bm25_score",
      "mcp__ia-qa-eval__build_rag_prompt",
      "mcp__ia-qa-eval__rerank_evaluate",
      "mcp__ia-qa-eval__extract_json_from_text",
      "mcp__ia-qa-eval__estimate_llm_cost",
      "mcp__ia-qa-eval__token_budget_calculator",
      "mcp__ia-qa-eval__context_window_check",
      "mcp__ia-qa-eval__compare_models",
      "mcp__ia-qa-eval__llm_generate"
    ],
    "deny": [
      "mcp__ia-qa-eval__rate_tool"
    ]
  }
}

Tools that only read are allowed, tools that change things ask first, tools that delete or cancel are denied. A read that can send data out (it takes a URL, say) asks first too. It assumes the server is installed as ia-qa-eval. Review it before you use it.

Capability tags behind 7 rules
  • mcp__ia-qa-eval__run_eval_contract untrusted-input
  • mcp__ia-qa-eval__generate_eval_yaml untrusted-input
  • mcp__ia-qa-eval__test_skill untrusted-input
  • mcp__ia-qa-eval__conversation_analyze private-data, untrusted-input
  • mcp__ia-qa-eval__run_vlm_test_suite untrusted-input
  • mcp__ia-qa-eval__rerank_evaluate untrusted-input
  • mcp__ia-qa-eval__llm_generate untrusted-input

Pin

Keep the decision: fail CI when the tools change, follow every change, or show a badge.

Pin in CI

Fail a build when the tools you reviewed are not the tools the server serves today.

curl -s "https://radar.anlyon.com/api/v1/servers/io.github.JcJamet/ia-qa-eval/verify?hash=e959867680441a38558362f6c5dd4e9c1cd86bc4e9158e2b923b54bb863c1f0a" | jq -e '.changed == false'

Pinned hash e95986768044, from Nov 9, 2026, 12:18 UTC.

Watch

Get told when its tools change: a new tool, a new description, a new class.

https://radar.anlyon.com/servers/io.github.JcJamet/ia-qa-eval/changes.xml

An Atom feed. Works in any feed reader or Slack's RSS app.

Badge

Anlyon Radar badge for io.github.JcJamet/ia-qa-eval
[![Anlyon Radar](https://radar.anlyon.com/badge/io.github.JcJamet/ia-qa-eval.svg)](https://radar.anlyon.com/servers/io.github.JcJamet/ia-qa-eval)

Source: MCP Registry · repository. Synced Oct 26, 2026, 22:17 UTC.

IA-QA Eval & RAG — 38 tools MCP server: questions and answers

Answered from the scan above, not from the publisher's description.

Is the IA-QA Eval & RAG — 38 tools MCP server safe?

No scan can promise that. What its definitions show: 39 tools, 26 of which can change, send or delete something, and 12 findings from fixed checks. Allow the reads, ask before each change, and deny the deletes; the permissions block on this page does that.

Can the IA-QA Eval & RAG — 38 tools MCP server delete data?

Yes. 1 tool can delete, cancel or revoke something: rate_tool.

Can the IA-QA Eval & RAG — 38 tools MCP server do things that cannot be undone?

No tool is classed as irreversible: none of them sends, pays, publishes or merges, going by their definitions.

What tools does the IA-QA Eval & RAG — 38 tools MCP server have?

39 tools: 1 destructive, 25 write, 13 read. rate_tool, hallucination_check, run_eval_contract, run_semantic_tests, generate_ci_workflow, needle_haystack_generate, generate_eval_yaml, metamorphic_check and 31 more.

Does the IA-QA Eval & RAG — 38 tools MCP server need sign-in or OAuth?

Its remote endpoint listed its tools to an anonymous client, without sign-in.

Does the IA-QA Eval & RAG — 38 tools MCP server send data to outside services?

None of its tools sends, posts or publishes, or takes a recipient, channel or URL.

Has the IA-QA Eval & RAG — 38 tools MCP server changed its tools?

Yes, once since Anlyon Radar started tracking it, most recently on Nov 9, 2026. Every change is on this page and in its Atom feed.