Hold the cases constant
Eight authored scenarios run through all three profiles, so the comparison measures architectural behavior instead of changing the task between demos.
04 / Agent reliability lab
Only one architecture stayed useful under attack.
Eight synthetic support cases run through three agent architectures, exposing where safety protects useful work and where it only blocks it.
Product brief
- Product
- BoundaryLab
- Status
- Portfolio demo
- Target user
- An engineer or engineering leader deciding whether an agent architecture can complete useful tool-driven work without turning untrusted context into authority.
- My role
- Product framing, benchmark design, architecture contracts, deterministic adapters, replay UX, verification, and public release.
The problem
A safety result can look strong when an agent simply refuses every action. BoundaryLab holds the cases constant so unsafe completion, unnecessary refusal, and safe completion remain visibly different outcomes.
01 / Workflow
The product, end to end.
Follow the product from first input to a reviewable outcome.
- 01Select one of eight support cases
- 02Run it through three architecture profiles
- 03Inspect context and memory inputs
- 04Follow worker, sentinel, and gateway decisions
- 05Compare the protected judge outcome
- 06Replay unsafe, over-blocked, and safe results
02 / Decisions
Architecture with a reason.
Python 3.12 · Pydantic contracts · LangGraph checkpoints · Deterministic adapters · Dependency-free static replay · Pytest
Authority stays deterministic
Workers and sentinels can propose or advise. Only the capability gateway can authorize a typed mutation inside the synthetic support world.
Protect the conclusion
Expected outcomes remain outside agent context, and the judge scores final state and structured evidence rather than accepting an agent's account of success.
Failure paths
What can go wrong is part of the product.
- Untrusted ticket content is treated as an instruction
- A reviewer blocks a policy-valid customer action
- Stale or cross-case memory contaminates working context
- A proposed action exceeds the caller's capability or refund limit
Verification
What supports the claim.
- Schema-validated 24-run matrix: eight cases by three profiles
- Aggregate assertions for task success, attacks, false refusals, and score
- Guided replay checks for an unsafe completion, false refusal, and safe completion
- Static export, no-network, and security-header checks
Portfolio boundary
Complete enough to review. Honest enough to trust.
- Synthetic support data and authored deterministic adapters only
- Zero model calls; results do not measure production-model accuracy or frequency
- No real customers, refunds, messages, or third-party side effects
- The public replay omits raw prompts, attacker objectives, canary values, and protected ground truth
- Visitors inspect immutable runs and cannot trigger inference