Gate CI on Replay Results

A replay can finish sending requests while the application returns the wrong result. Standard replay writes replay-verdict.json in its output directory so you can inspect the overall verdict and each request's result. Choose a gate explicitly when a mismatch should fail CI.

Watch replay verdicts on YouTube.

Run the examples from your application directory, with the application listening at http://localhost:8080. Replace recorded-example with a recording that contains inbound requests for it.

Save a baseline

proxymock replay \
  --in ./proxymock/recorded-example \
  --test-against http://localhost:8080 \
  --out ./proxymock/results/baseline

Keep the output directory, including its observed RRPairs and replay-verdict.json. The verdict includes an aggregate result, per-pair findings, and a gate object recording whether a gate was enabled and the selected exit code. An ungated mismatch does not by itself make the command fail.

By default, the verdict checks response status and stable response-body changes. Fields learned as volatile from repeated observations are excluded from body differences. --ignore-body-changes restricts verdict scoring to status codes; use it only when body correctness is covered separately.

Fail on a new regression

After changing your app, replay the same recording:

proxymock replay \
  --in ./proxymock/recorded-example \
  --test-against http://localhost:8080 \
  --out ./proxymock/results/candidate \
  --baseline ./proxymock/results/baseline \
  --fail-on-new-mismatch

The gate exits 3 when it finds a new mismatch. A request that failed in the baseline is exempt only from that same failure. A different status or a new stable-body change can still be a regression. Preserve request identities by using the same source recording for the comparison.

Verify a bug fix

For a recording containing an error you intend to fix:

proxymock replay \
  --in ./proxymock/recorded-incident \
  --test-against http://localhost:8080 \
  --out ./proxymock/results/fix-check \
  --verify-fix \
  --expect '/checkout'

--expect is a regular expression selecting recorded-error endpoints. It requires --verify-fix. Fix verification interprets a recorded error becoming a successful response as a fix and checks for collateral regressions elsewhere.

Exit	Fix-verification meaning
0	Fix confirmed without detected collateral regressions
2	Fix not confirmed, including a bug that still reproduces
3	Collateral regression detected

Invalid configuration and operational failures can produce other nonzero results. A recording with no selected recorded-error requests is not evidence of a fix. Do not combine --verify-fix with --fail-on-new-mismatch.

Add metric and blueprint requirements

Use --fail-if for aggregate metrics and --require-blueprint for required transform activity:

proxymock replay \
  --in ./proxymock/recorded-example \
  --test-against http://localhost:8080 \
  --require-blueprint 'Field Transforms' \
  --fail-if 'requests.failed > 0' \
  --fail-if 'requests.response-match-pct < 100' \
  --fail-if 'latency.p99 > 500'

requests.response-match-pct is the share of the verdict's pairs that matched, so it fails the run when a response body changes. requests.result-match-pct compares status codes only, so a body change does not move it.

To check the calls your service makes to its dependencies, add an httpOutput assertion to the test config and pass the directory proxymock mock wrote while it served the replay:

proxymock replay \
  --in ./proxymock/recorded-example \
  --test-against http://localhost:8080 \
  --test-config outbound \
  --mocks-out ./proxymock/results/mocked-1

A difference from the recorded calls fails the goal verdict and exits 1, and the differences are written to http-output-diff.json in the replay output. proxymock replay score <replay-dir> --mocks-out <dir> scores a finished replay the same way.

Use the actual blueprint name. A violated --fail-if condition exits 1. These metrics and blueprint checks answer different questions from the per-pair verdict; requiring a blueprint proves that at least one of its chains ran, not that the response was correct.

Output and load-mode limits

Baseline, fix-verification, and semantic comparison need response output. They cannot run with --no-out or --load-test. High-throughput load mode also omits requests.result-match-pct and requests.response-match-pct and cannot verify blueprint activity. --no-out omits requests.response-match-pct too.

Keep CI artifacts from failed runs so you can inspect the request, observed response, and verdict together. Do not use the global --exit-zero option in a job that relies on exit codes. See the CI/CD guide for pipeline setup and the CLI reference for options.