Agent as Judge as Post-Hook

Using Agent as Judge evaluation as a post-hook for automatic evaluation

This example demonstrates using Agent as Judge evaluation as a post-hook, automatically evaluating every agent response.

Add the following code to your Python file

1from kern.agent import Agent
2from kern.db.sqlite import SqliteDb
3from kern.eval.agent_as_judge import AgentAsJudgeEval
4from kern.models.openai import OpenAIResponses
5
6# Setup database to persist eval results
7db = SqliteDb(db_file="tmp/agent_as_judge_post_hook.db")
8
9# Eval runs as post-hook, results saved to database
10agent_as_judge_eval = AgentAsJudgeEval(
11    name="Response Quality Check",
12    model=OpenAIResponses(id="gpt-5.2"),
13    criteria="Response should be professional, well-structured, and provide balanced perspectives",
14    scoring_strategy="numeric",
15    threshold=7,
16    db=db,
17)
18
19agent = Agent(
20    model=OpenAIResponses(id="gpt-5.2"),
21    instructions="Provide professional and well-reasoned answers.",
22    post_hooks=[agent_as_judge_eval],
23    db=db,
24)
25
26response = agent.run("What are the benefits of renewable energy?")
27print(response.content)
28
29# Query database for eval results
30print("Evaluation Results:")
31eval_runs = db.get_eval_runs()
32if eval_runs:
33    latest = eval_runs[-1]
34    if latest.eval_data and "results" in latest.eval_data:
35        result = latest.eval_data["results"][0]
36        print(f"Score: {result.get('score', 'N/A')}/10")
37        print(f"Status: {'PASSED' if result.get('passed') else 'FAILED'}")
38        print(f"Reason: {result.get('reason', 'N/A')[:200]}...")

Set up your virtual environment

1uv venv --python 3.12
2source .venv/bin/activate

1uv venv --python 3.12
2.venv\Scripts\activate

Install dependencies

1uv pip install -U kern-ai openai

Export your OpenAI API key

1export OPENAI_API_KEY="your_openai_api_key_here"

1$Env:OPENAI_API_KEY="your_openai_api_key_here"

Run the example

1python agent_as_judge_post_hook.py