Release verdict · Managed QA
Release 4.12 — "Boxing Day on-sale" build
01Summary
All agreed critical flows are covered and passing on the release candidate. One genuine defect was found by the exploratory session, fixed in commit a3f9c2, and verified by a re-run. The load rehearsal found the payment gateway throttling at 32,000 concurrent users; the fix (connection pool + retry budget) was verified at 40,000 with p95 checkout latency of 1.8 s. No open P1 or P2 issues. Recommendation: ship, with the on-sale standby plan in section 08.
02Scope agreed in writing (week 1)
| Critical flow | Owner | Automated | Exploratory | Load |
|---|---|---|---|---|
| Seat selection & hold (single, group, accessible seating) | Checkout squad | 42 cases | Session 1 | Yes |
| Virtual queue entry & fairness at sale open | Platform | 18 cases | Session 1 | Yes |
| Promo code redemption & stacking rules | Checkout squad | 37 cases | Session 2 | Yes |
| Payment, retry after gateway timeout, 3-D Secure | Payments | 64 cases | Session 2 | Yes |
| Refund & cancellation (post-sale) | Support tooling | 29 cases | Session 2 | — |
| Mobile web (iOS Safari, Android Chrome) | Checkout squad | 126 cases | Session 1 | — |
03Agent pre-validation — PR #2291
The Zenius agent read the three acceptance criteria on the pull request, generated 14 tests from the library, ran them against the branch and classified the results. A senior engineer reviewed every generated test before it joined the suite.
| Acceptance criterion | Generated tests | Result | Triage |
|---|---|---|---|
| A promo code applies at most once per order | 5 | 5 passed | — |
| A refund releases the held seat within 60 seconds | 4 | 4 passed | — |
| A retry after a gateway timeout never double-charges | 5 | 4 passed · 1 failed | P1 genuine defect → dev (fixed a3f9c2, re-run verified) |
04Automated run — 316 cases on build 4127
| Failure | Classification | Evidence | Resolution |
|---|---|---|---|
| checkout/retry-after-timeout-does-not-double-charge | Defect | video · trace · HAR · console | Fixed a3f9c2 · re-run 4128 passed |
| queue/position-updates-under-1s (×3) | Flaky · env | trace · timing logs | staging-2 websocket cap raised · stable across 5 re-runs |
Pass-rate trend, last 30 runs: 96.1% → 98.7%. Escaped defects this quarter: 0.
05Exploratory sessions — senior engineer, 2 × 3 hours
Worked the on-sale the way a frustrated fan, an impatient buyer and a malicious user would: two tabs on the same seat, abandoning at payment and returning, currency switch mid-checkout, promo replay after refund, expired queue tokens.
| Finding | Severity | Reproduction | Status |
|---|---|---|---|
| Promo code re-applies after a refund; second order total goes negative | P1 | Refund order → reuse code → apply twice on new order (video 02:14) | Fixed a3f9c2 · verified |
| Queue position badge shows stale value after tab restore on iOS Safari | P3 | Background tab 90 s → restore (trace attached) | Accepted for 4.13 · cosmetic |
06Load rehearsal — T-minus 17 days
Full on-sale rehearsal at expected concurrency with seat holds, queue fairness and payment retries exercised together. Breaking point found at 32k concurrent (payment gateway throttling → checkout p95 above 6 s). After the fix, 40k concurrent sustained for 20 minutes.
| Stage | Concurrent users | Checkout p95 | Error rate | Outcome |
|---|---|---|---|---|
| Ramp 1 | 10,000 | 0.9 s | 0.02% | Pass |
| Ramp 2 | 25,000 | 1.4 s | 0.05% | Pass |
| Ramp 3 (before fix) | 32,000 | 6.3 s | 4.1% | Gateway throttling — breaking point |
| Ramp 3 (after fix) | 40,000 | 1.8 s | 0.06% | Pass — event-day target met |
07Residual risks & recommendations
1. Third-party fraud-screening latency was mocked at 300 ms; production figures above 800 ms would push p95 over 2.5 s — confirm the vendor's peak-day SLA. 2. Refund flow has no load coverage by design; volumes are low post-sale. 3. iOS Safari stale badge (P3) is scheduled for 4.13.
08On-sale standby plan
A Zenius engineer is on standby from T-minus 60 minutes to T-plus 3 hours with the rehearsal dashboards open, the incident channel joined, and the rollback path pre-agreed with the platform lead. Post-event note delivered within 24 hours.