Build an HR Policy Chatbot with RAG, Step by Step

What you will build

An HR policy chatbot for a made-up company, Brightwater Labs. It uses RAG, retrieval-augmented generation: when someone asks a question, the assistant first finds the passages of your documents that match, then asks a language model to answer using only those passages. The answer cites the passage, so anyone can check it.

Building it takes about fifteen minutes of forms. The more useful half of this guide is what comes after: changing the assistant without breaking it. Every change runs on a copy next to the live one, gets scored on your questions, and only goes live if it passes.

Words you will see

• Stack: A set of apps SeaGit deploys and connects for you from one template, such as the RAG Stack.
• Chunk: A piece of a document. Documents are cut into chunks so the assistant can find the right paragraph.
• Prompt: The instructions sent to the language model with every question: how to answer, and to use only the passages found.
• Golden question: A real question plus the fact its answer must contain. Your definition of “correct”.
• Candidate: A copy of the assistant with your change, running next to the live one so it can be scored first.
• Gate: The rule that decides whether a candidate may go live. With Block, a candidate that scores too low cannot be adopted.
• Trace: A record in MLflow of one question: what was found, the model, the prompt version and the answer.

Before you start

• A SeaGit account on a paid plan. The RAG, MLflow and Evals templates are marked Paid plan.
• An environment with a started cluster on Amazon EKS in your own AWS account. Models come from Amazon Bedrock, so there are no model keys to manage.
• Settings changes and prompt versions are enabled per account. Ask us to turn them on before step 6.
• The sample files: four policies (all linked in step 4), golden.jsonl and suite.yaml.

Step-by-step walkthrough

Step 1: Open the templates

In the SeaGit dashboard, open Applications and then Templates. You will use three cards: MLflow (keeps prompts, traces and scores), RAG Stack (the assistant itself) and Evals (scores answers against your questions). Build them in that order, because each one points at the one before.

Step 2: Create the MLflow stack

Click Deploy on the MLflow card. On Where, name it hr-mlflow and pick your environment; keep Create new Postgres (in this stack). On Configure, choose who can open MLflow in a browser. We picked Use existing Auth Proxy stack because we already run one; New sign-in for this stack works with your own identity provider. Then Review and Deploy.

Step 3: Create the RAG stack

Click Deploy on the RAG Stack card and name it hr-assistant. The form has five steps. For a first assistant, the defaults are right everywhere except the last one:

• Data sources: leave the list empty and keep A new S3 bucket for this stack. You will upload files in the web UI.
• Ingestion: keep chunk size 800 and overlap 100. You can change them later without rebuilding anything by hand.
• Answering: keep the web UI on, then set Trace answers in MLflow to Send traces to an MLflow stack in this environment and pick hr-mlflow. Every question then becomes a trace you can read later.

In our run, all four parts (vector database, rag-api, web UI and sync) were running about a minute after Deploy. Open the application and its Stack tab: you will see the connection details, a Settings changes card, and Prompt versions with Version 1, the built-in prompt, marked Live.

Step 4: Upload the policies and ask a question

Download the four sample policies for a made-up company, Brightwater Labs: parental-leave.md, travel.md, expenses.md and time-off.md. On the Stack tab, click the link next to Web UI access, sign in with the web UI access key, and drag the files into the box. Each one says Stored — searchable within seconds. Then ask: How many weeks of paid parental leave do I get?

Your intranet or Slack bot can ask the same thing in code. Use the RAG API address from the Stack tab and the API key:

curl -s https://<your-rag-api-address>/v1/query \
  -H "Authorization: Bearer $RAG_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"question": "When am I allowed to book business class?"}'

{
  "answer": "You are allowed to book business class only when a single flight is scheduled to be longer than 6 hours. [1]",
  "citations": [{"n": 1, "uri": "s3://<your-bucket>/uploads/travel.md", "source_id": "uploads", "score": 0.4301}],
  "trace_id": "tr-0e0c2ee2ca30847ca5ce1960bcf5ec44",
  "model": "bedrock:us.amazon.nova-lite-v1:0"
}

Step 5: Write golden questions and create the Evals stack

Before you change anything, write down what “correct” means. A golden question is a real question plus the fact the answer must contain. Ours are in golden.jsonl, one per line:

{"input": "How many weeks of paid parental leave do I get?", "expected": "16 weeks"}
{"input": "Can I expense a home-office chair, and how much can it cost?", "expected": "$300"}
{"input": "When am I allowed to book business class?", "expected": "6 hours"}
{"input": "What is the nightly hotel limit in London?", "expected": "$320"}
...six more lines

Click Deploy on the Evals card and name it hr-evals. On What to evaluate, choose A RAG stack, pick hr-assistant, and pick hr-mlflow for results. On Judge, keep None: code checks only: the check is simply whether each answer contains the expected text. On Suites & gate, choose Write your own (YAML) and paste suite.yaml, set Promotion gate to Block, choose Selected instances with none picked (the gate then covers only this RAG stack’s changes), and set GitHub status to Off.

suites:
  - name: hr-golden
    target:
      type: seagit-rag
      url: '${TARGET_URL}'
      api_key_env: TARGET_API_KEY
    dataset:
      rows:
        - {input: "How many weeks of paid parental leave do I get?", expected: "16 weeks"}
        - {input: "What is the nightly hotel limit in London?", expected: "$320"}
        # ...the other eight questions
    scorers:
      - contains
      - latency_ms
    thresholds:
      contains: {min: 0.85}
      latency_ms: {max: 15000}

Step 6: Try a settings change

On the Stack tab, click Change settings. We tried bigger chunks: chunk size 1200 and overlap 200. Both fields say Needs reindex, and the estimate shows the cost before you start (about $0.00 for our four documents). Pick the Evals stack, add golden.jsonl, and click Start change.

Both the live stack and the candidate answered all 10 questions with the expected fact (contains 1.0). Bigger chunks did not help these short policies, but they did not hurt either, so we clicked Adopt. That is a normal result: the point is that you know before your users do.

Step 7: A prompt change the gate refuses

HR asks for friendlier answers, and legal adds: do not quote amounts, point people to the policy. It sounds reasonable. Under Prompt versions, click New version from this on Version 1, change the system message, add a note, and Save version. The user message must keep {{context}} and {{question}}.

You are a friendly HR helper for employees. Answer warmly in one or two
short sentences, using ONLY the provided context. Do not quote exact amounts,
limits or numbers of days; tell the employee which policy covers it so they
can check the details there. Cite sources by their [n] number. If the context
does not contain the answer, say you don't know.

Then click Try this version. It opens Change settings with Version 2 picked; choose the Evals stack and start.

The candidate did exactly what the prompt said, and that is the problem. These are real answers from the run:

Five of ten answers lost the one thing an employee asked for. If you tried three questions by hand, you could easily have picked the ones that still worked. We clicked Discard; nobody ever saw these answers.

Step 8: A prompt change that passes

Keep the friendly tone, but ask for the amounts. Version 3:

You are a friendly HR helper for employees. Answer warmly in one or two
sentences, using ONLY the provided context. Always include the exact amounts,
limits and numbers of days the policy gives, and name the policy. Cite sources
by their [n] number after the sentence they support. If the context does not
contain the answer, say you don't know and cite nothing.

Save it, click Try this version on Version 3, and start the change.

In our run, Version 3 was live about half a minute after Adopt. The RAG API’s /healthz then reports prompt_ref, the version and a fingerprint of its text, and every MLflow trace of a question carries the same value. If someone asks “what did the assistant say last Tuesday, and why?”, you can answer both halves.

Step 9: Roll back

Changed your mind? On the adopted change, click Roll back, type a reason, and confirm. The card records Prompt: version 3 → built-in with your reason. The roll-back window is about 23 hours after you adopt.

What we measured

The score is the share of the 10 golden questions whose answer contained the expected text. The gate needed at least 0.85. Answers took about a second each.

Two lessons carry over to any assistant. A change that sounds harmless can quietly remove the answer people came for. And ten questions you wrote down beforehand catch that in about two minutes, every time, without anyone spot-checking by hand.

Clean up

Delete the stacks in reverse order: hr-evals, then hr-assistant, then hr-mlflow. On each Stack tab, click Delete stack. Tick Also delete my uploaded documents (or Also delete my backups for MLflow) if you want the S3 bucket gone too; it is kept by default so you never lose data by accident.

Frequently asked questions