Evals Stack

What the Evals Stack is

The template deploys one service into a Kubernetes cluster in your AWS account. It reads questions from datasets in the stack’s S3 bucket, sends them to the application you chose, and scores each answer with code checks, Ragas metrics and a rubric judge. Each evaluation is an MLflow run in your MLflow stack, with the score per question kept as a table on the run. Your data, prompts and answers stay in your cloud account.

When to use it

• You change prompts, models, chunking or retrieval and want to know whether answers got better or worse before users see them.
• Each pull request or branch gets its own ephemeral deployment, and you want it scored against main before anyone promotes it.
• You want a sample of real traffic scored continuously, and the answers people dislike turned into test cases.

The wizard estimates about 4 minutes. It needs a native (sgt-k8s) cluster and an MLflow stack on the same cluster. The stack runs on AWS.

How it fits together

Evals stack (one service in your cluster, no ingress)
  |
  |  datasets/<suite>.jsonl  (stack bucket, S3 in your account)
  v
target: RAG stack | LLM Gateway stack | an address
  |  answers (+ retrieved passages)
  v
scorers: code checks | Ragas | rubric judge --> judge model via an LLM Gateway stack
  |
  v
MLflow stack: one run per suite and target, scores per row
  |
  +--> promotion gate --> deployment result in SeaGit --> Promote: warn or block
  +--> GitHub commit status  seagit/evals

RAG / LLM Gateway traces in MLflow --> online scoring, review queue --> dataset rows

The service has no ingress. It reaches MLflow, the target and the judge model by their in-cluster addresses, and reads and writes datasets in the stack bucket through an IAM role, without keys.

Deploy it

1. Deploy an MLflow stack and the application to evaluate, for example a RAG stack or an LLM Gateway, on the same cluster.
2. Open Applications → Templates and choose Evals. Pick the environment and cluster.
3. Choose the MLflow stack and what to evaluate. Pick a judge if you want rubric scores (Amazon Bedrock or an LLM Gateway stack) or the Ragas metrics (an LLM Gateway stack).
4. Choose the promotion gate mode and, for Warn or Block, the instances to watch and the GitHub status.
5. Deploy. With the built-in suite, the first run writes starter questions to datasets/rag-smoke.jsonl (or chat-smoke.jsonl) in the bucket shown under Datasets in the Stack tab. Chat starter questions have one right answer, so contains is measured at once. RAG starter questions fit any documents and have no expected answer, so contains shows not measured until you add rows with your own expected answers. Your rows are never overwritten.

Wizard fields

Targets

A RAG or LLM Gateway stack must run on the same cluster, because the service calls it by its in-cluster address. Its key is handed to the service as a secret at create; it never appears in the Evals stack’s settings or outputs. Each call times out after 60 seconds; a failed call scores 0 on every scorer for that row and counts as an error.

Datasets

A dataset is a JSON Lines file, one question per line. Keep it in the stack bucket under datasets/, or inline it in a suite.

{"input": "What is the refund window?", "expected": "30 days"}
{"input": "Who approves travel over $5,000?", "expected": "the CFO", "tags": {"area": "finance"}}
{"input": "Summarise the security policy.", "contexts": ["Passwords rotate every 90 days."]}

• input is required. expected is needed by exact_match, contains and ragas.context_recall. contexts is optional reference material for the rubric judge. tags are kept with the results.
• An invalid line fails the run and names the line number. A dataset with no rows fails the run.
• Rows added by online scoring and by accepted reviews are written to datasets/<suite>/from-traces.jsonl and datasets/<suite>/reviewed.jsonl. Point a suite at one of them, or merge them into your dataset.

aws s3 cp rag-smoke.jsonl s3://<stack bucket>/datasets/rag-smoke.jsonl

Scorers

Ragas and the rubric judge call the judge model through the LLM Gateway stack you chose. Without a judge model only code checks run: the built-in suite then leaves out the Ragas and judge scorers. A scorer whose inputs are missing, for example faithfulness on a chat answer with no retrieved passages, is reported as skipped, never as 0. The versions of MLflow, Ragas and the judge model are recorded with every run so scores stay comparable.

Judge rubric versions

Instead of writing the rubric in the suite, a judge can name a version of a prompt in the MLflow stack’s Prompt Registry:

scorers:
  - judge:
      rubric_ref: "prompts:/judge-grounded/3"   # a version in the MLflow Prompt Registry

The reference is always a version number, never an alias or @latest, so two runs with the same reference were graded by the same rubric. The rubric is a text prompt without {{variables}}; the judge adds the question, answer and passages itself. Every run records the rubric it was graded with in the tag sg.judge_rubric_ref (an inline rubric as inline@<hash>), and a reference that cannot be loaded stops the run before any question is asked.

Suites

A suite names a target, a dataset, its scorers, and the thresholds the promotion gate checks. With Built-in smoke suite, the stack writes it for you from your choices:

suites:
  - name: rag-smoke                 # chat-smoke for a chat target
    target:
      type: seagit-rag              # openai-chat for a chat target (+ model)
      url: "<the target you chose>"
      api_key_env: TARGET_API_KEY
    dataset:
      path: s3://<stack bucket>/datasets/rag-smoke.jsonl
    scorers:
      - contains
      - latency_ms
      # with a judge model:
      - ragas.faithfulness          # RAG targets only
      - judge:
          rubric: "Is the answer correct and grounded in the retrieved context?"
    thresholds:
      contains: {min: 0.8}
      latency_ms: {max: 10000}
      # with a judge model:
      ragas.faithfulness: {min: 0.7, tolerance: 0.05}
      judge: {min: 0.5, tolerance: 0.1}

With Write your own (YAML) you write one or more suites. The first one is used by the promotion gate and online scoring. Every key:

suites:
  - name: support-bot               # letters, digits, . _ -; unique
    target:
      type: seagit-rag              # seagit-rag | openai-chat | http-json
      url: "${TARGET_URL}"
      api_key_env: TARGET_API_KEY   # optional: the variable holding the key
      # openai-chat: model: "${TARGET_MODEL}"
      # http-json:   request: '{"q": "{{input}}"}'
      #              answer_path: $.answer
      #              contexts_path: $.sources[*].text
    dataset:
      path: s3://${DATASET_BUCKET}/datasets/support-bot.jsonl
      # or inline:  rows: [{input: "What is the refund window?", expected: "30 days"}]
    scorers:
      - contains
      - latency_ms
      - regex: {pattern: "^[A-Z]"}
      - ragas.faithfulness
      - judge:
          rubric: "Is the answer polite, correct and under 100 words?"
          model: nova-pro           # optional; the judge model by default
    thresholds:                     # what the promotion gate checks
      contains: {min: 0.8}
      ragas.faithfulness: {min: 0.7, tolerance: 0.05}
      latency_ms: {max: 5000}
      errors: {max: 0}

• ${TARGET_URL}, ${TARGET_MODEL} (chat targets) and ${DATASET_BUCKET} are filled in with the target, model and stack bucket you chose; the target’s key is in TARGET_API_KEY. A variable that is not set stops the service at start with its name.
• min and max are absolute limits. tolerance is the drop allowed against main; leave it out to compare with the limits only. For a metric with only a max, such as latency, a rise counts as the drop.
• errors and rows can be given thresholds too.

The promotion gate

With Warn or Block, the stack watches every code instance of its organization, or the instances you selected. Nothing polls: the moment a deployment that is not main turns started with a commit — whether SeaGit built it or an external pipeline deployed it — the platform asks the stack, inside its pod, to score that deployment against the main deployment:

1. Main first. It reuses the latest run of the suite for main’s commit from the last 24 hours, or scores main now.
2. Then the deployment. Its scores are compared with the thresholds and with main’s scores.
3. Then the result is reported to SeaGit, to MLflow and, with a token, to GitHub.

The overall verdict is the worst one, with the reasons listed. The deployment’s row in Deployments shows it as pass, warn or fail (or running while it is scored), linked to the MLflow run.

Gate modes

The gate reads the latest result for the deployment’s commit; a short commit id on either side matches the full one. When the service starts scoring a deployment it reports it as running, and Block refuses a running evaluation the same way as a failed one, so a deployment cannot be promoted while its score is still being computed. A deployment with a commit that a gate watches but that has no result (the evaluation was missed or never started) is not evaluated: Warn says so and goes ahead, Block refuses it, and the Promote dialog offers Run evaluation. Deployments without a commit, such as an image or Helm chart deployed directly, are not scored automatically; call POST /v1/gate from your CI to score them. Their result stops counting once the deployment is deployed again.

GitHub commit status

With GitHub status on and a token that may write commit statuses, each result is posted on the commit, in the repository the deployment was built from, with the context seagit/evals: pass and warn as success (a warn mentions its first reason), fail as failure, and a gate that could not run as error. The status links to the MLflow run, so you can require seagit/evals in a branch protection rule.

The stack’s API key

The platform creates an account API key for the stack when it is deployed, named evals-<stack>. It may only list, read and update deployments, and only of the instances the gate watches: a request about any other instance is refused. It is stored encrypted, never shown, listed under API keys, and deleted with the stack. A failed report is retried three times and never changes the verdict. Only an API key can record a result: members see results in Deployments but cannot post them, and each result records the prefix of the key that reported it.

Evals as code

An app can keep its own questions and a stricter bar in its repository, read at the candidate’s commit with the GitHub token (it needs read access to contents):

.seagit/evals/
  evals.yml       # optional: type, model, path, include_shared, scorers, thresholds
  golden.jsonl    # optional: this app's questions, one JSON object per line

The gate asks the stack’s shared questions (unless include_shared: false) plus the app’s, uses the stack’s scorers plus the app’s, and per metric the stricter threshold: a looser value in the repository is ignored and noted in the result. Candidate and main are scored on the same questions, fingerprinted as sg.dataset_sha256 on the MLflow run. golden.jsonl holds at most 500 rows and 1 MB, evals.yml 16 KB; an invalid file fails the gate with its name and the reason. A step-by-step introduction: How to evaluate a RAG or LLM app.

Online scoring and the review queue

Turn on tracing in the RAG or LLM Gateway stack you evaluate (its Trace answers in MLflow or Trace calls in MLflow setting), then choose Live traffic in the Evals stack. The target must trace to the same MLflow stack you chose under Results, because the service reads its traces from that MLflow stack. The Evals stack is refused when the target sends no traces, or sends them to another MLflow stack.

• Online scoring reads new traces on the interval you set and scores the share you chose. The same answer is always in or out of the sample. Only scorers that need no expected answer run, and latency is always measured. Scores are attached to each trace in MLflow, as judge or code feedback. An answer below a min or above a max is added to datasets/<suite>/from-traces.jsonl.
• Review queue. Answers people rate thumbs-down in the RAG web UI are listed by GET /v1/review. Accept one with the answer you expected and it becomes a row in datasets/<suite>/reviewed.jsonl; dismiss it and it leaves the queue.

TOKEN=...   # the token you set on the evals application (see Calling the API)
EVALS=http://<release>-application.<namespace>.svc.cluster.local:8080

curl -s -H "Authorization: Bearer $TOKEN" $EVALS/v1/review
curl -s -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"expected": "30 days from delivery"}' $EVALS/v1/review/<trace_id>/accept

Results in MLflow

Each run of a suite is an MLflow run in the experiment evals/<suite>. It holds the mean of each scorer, the number of rows and errors, and a results table with the question, answer, retrieved passages, scores and latency of every row. Tags say what the run was: sg.role (baseline for main, candidate for a deployment, adhoc for a run you started), sg.verdict, sg.deployment_id and sg.commit_sha. Compare two runs in the MLflow UI to see which questions changed.

Calling the API

The service listens on port 8080 inside the cluster only. Every /v1 call needs Authorization: Bearer and the service’s API token, which is generated when the stack is created and kept as a secret. To call the API from your own pods or CI, set your own value for the evals application’s secret EVALS_API_TOKEN and redeploy it.

When a run or gate names another address, the target’s key is sent only to the target’s own host, to the deployments the gate watches, and to the hosts you listed in the wizard.

Outputs

Limits

• The MLflow stack and the RAG or LLM Gateway stacks it uses must run on the same cluster as the Evals stack.
• One service and one watched instance per stack. Results are kept by the service for its recent runs; MLflow keeps them all.
• Live traffic needs the evaluated stack to trace to the same MLflow stack as the Evals stack.
• AWS only.

Related

• MLOps Platform: how the stacks fit together
• MLflow Stack
• RAG Stacks and LLM Gateway
• Deployments (ephemeral deployments and Promote to main)
• Private MLflow with single sign-on