Every scanner improvement we ship could inadvertently create a bypass. A regex change that catches one PII variant might miss another. Unicode normalization that handles math-bold digits might break zero-width character detection. Without automated regression testing, you ship blind.
The problem with manual red-teaming
Before this sprint, Tamga's adversarial tests were run manually:
Start the full stack locally (docker compose up)Wait for everything to be healthyRun pii_bypass.py, check the outputRun injection_bypass.py, check the outputRun secret_bypass.py, check the outputRun policy_bypass.py, check the outputRun k6 load tests, manually check P95 latencyWrite results to a markdown fileTear downTotal time: 20-30 minutes. Frequency: once per sprint (weekly), sometimes skipped. Result: regression protection was essentially manual code review.
The single-command solution
./tests/stress/run_stress_suite.sh
That's it. One command. What happens:
Docker Compose brings up the full Tamga stack (proxy, PostgreSQL, Redis, analyzer)A healthcheck loop waits up to 60 seconds for the proxy to respond 2004 adversarial scripts run sequentially, each with `--json` output3 k6 baseline tests (100, 500, 1000 RPS) measure P95 latency and error rateA workload mix test runs for 3 minutes with realistic traffic patternscheck_regression.py compares results against baseline.jsondocker compose down runs via trap EXIT — guaranteed cleanup even on failureExit code 0 (stable/improved) or 1 (regression detected)The regression check
check_regression.py implements two rules:
Adversarial: Any category with more bypasses than baseline → FAIL. This is strict — if the PII scanner detected 11 bypasses last week and 12 this week, the PR is blocked.:
Load: P95 latency exceeds baseline by more than 20% → FAIL. This gives headroom for normal variance while catching real performance regressions.:
CI integration
The GitHub Actions workflow (adversarial-gate.yml) triggers on every PR to dev or main that touches proxy, analyzer, or stress test files. It builds images, runs the suite, uploads results as a 30-day artifact, and posts a summary comment on the PR.
Lessons learned
**Always teardown.** The trap EXIT pattern (`trap cleanup EXIT`) in bash was critical — without it, a failed test leaves Docker containers running and port 8443 blocked.**JSON output is non-negotiable.** The adversarial scripts originally printed pretty text tables. Adding `--json` made them composable in pipelines.**Baseline is a commitment.** The baseline.json file represents "this is the current state of detection." Updating it should be intentional — never lower the baseline to make a regression disappear.**k6 summary-export is fragile.** The JSON structure changed between k6 versions. We pinned to `--summary-export` with explicit field access wrapped in try/except.The full suite and baseline are public. Clone the repo and run `./tests/stress/run_stress_suite.sh` to see your own numbers.
