Policy Testing for AI Agents: Treat Security Rules Like Code

When you write application code, you write tests. When you write infrastructure config, you validate it. But when you write security policies for AI agents, what do you do?

Most teams today are writing security rules and hoping they work. That's not good enough. If your policy is the only thing standing between an AI agent and a DROP TABLE, you should be able to prove it works before deploying it.

Why security policies need tests

Security policies for AI agents have the same properties as code:

• They have logic — match conditions, rule ordering, parameter comparisons
• They have bugs — a typo in a resource name means a rule that never matches
• They have regressions — adding a new permissive rule might accidentally override a deny
• They change over time — new tools, new servers, new team requirements

If policies have the same failure modes as code, they should have the same safety nets.

What a policy test looks like

A policy test defines an action and asserts the expected outcome. Here's what that looks like in practice:

# postgres.test.yaml
policy: "./postgres.yaml"

tests:
  - id: blocks-drop-table
    description: DROP TABLE should always be denied
    action:
      resource: query
      parameters:
        sql: "DROP TABLE users"
    expect: deny
    rule: deny-destructive-ddl

  - id: blocks-truncate
    description: TRUNCATE should always be denied
    action:
      resource: query
      parameters:
        sql: "TRUNCATE TABLE sessions"
    expect: deny
    rule: deny-destructive-ddl

  - id: gates-delete
    description: DELETE should require approval
    action:
      resource: query
      parameters:
        sql: "DELETE FROM users WHERE id = 5"
    expect: require_approval
    rule: approve-data-modification

  - id: allows-select
    description: SELECT queries should be allowed
    action:
      resource: query
      parameters:
        sql: "SELECT * FROM users WHERE active = true"
    expect: allow
    rule: allow-reads

Each test case specifies:

• action — the tool call the agent would make
• expect — the decision the policy should produce (allow, deny, or require_approval)
• rule — the specific rule that should match (catches silent regressions)

Run them with a single command:

$ kvlar test -f postgres.test.yaml
Running 4 tests from postgres.test.yaml...

✓ 4/4 tests passed.

If a test fails, you get a clear error:

✗ blocks-drop-table
  Expected: deny (rule: deny-destructive-ddl)
  Got:      allow (rule: allow-reads)

This tells you exactly what went wrong — your deny rule isn't matching, and the allow rule is catching it instead. Probably a regex issue.

Testing in CI

Policy tests should run in CI alongside your application tests. The --json flag outputs machine-readable results:

kvlar test -f postgres.test.yaml --json

And the exit code is non-zero on failure, so your pipeline fails if a security rule breaks:

# .github/workflows/ci.yml
- name: Test security policies
  run: kvlar test -f policies/postgres.test.yaml

What to test

Good policy test suites cover three categories:

Deny cases — verify that dangerous operations are actually blocked. These are the most important tests. If your policy is supposed to block DROP TABLE, prove it.

Allow cases — verify that legitimate operations still work. A policy that blocks everything is secure but useless. Make sure your agent can still do its job.

Edge cases — test the boundaries. Does drop table (lowercase) match? What about DROP TABLE IF EXISTS? What about a SELECT that contains the word "DROP" in a string literal? These are the cases where policies silently fail.

Building a testing habit

Start small. Write tests for your highest-risk rules first — the ones that block destructive operations. Then expand to cover approval gates, parameter conditions, and glob patterns.

A good rule of thumb: every deny rule should have at least one test. If a rule is important enough to write, it's important enough to verify.

Kvlar ships with test suites for all 8 built-in policy templates — 105 tests covering Postgres, GitHub, Slack, shell, and more. Use them as a starting point for your own test suites.

# Run all built-in tests
kvlar test -f policies/examples/postgres.test.yaml   # 17 tests
kvlar test -f policies/examples/github.test.yaml     # 27 tests
kvlar test -f policies/examples/shell.test.yaml      # 38 tests

Security policies for AI agents are too important to deploy untested. Treat them like code, test them like code, and review them in PRs like code. Your future self will thank you.