Page plan · 15 pages

Written for: engineering, operations and incident-review teams

1. 1

   Fare gateway outage: a blameless postmortem

   • This is a blameless review of a fictional fare gateway outage in a city transit system.
   • The trigger was an oversized configuration file, not an individual's mistake.
   • The goal is to learn how the system allowed the failure, not to assign fault.
   • The audience is engineering, operations and incident-review teams.
2. 2

   Incident at a glance: when and how fares stopped working

   • Riders could not pay fares because the fare gateway stopped validating requests.
   • The trigger was a configuration file that grew beyond what the gateway could load.
   • Reaching a first failure took one routine configuration publish.
   • The summary below sets scope for the detailed timeline that follows.
3. 3

   Impact on each transit service

   • Each transit service depended on the gateway differently, so impact varied by service.
   • Services that validate fares online at the gate were hit hardest.
   • Services with offline fallback modes degraded more gracefully.
   • Operations teams had to switch to manual or open-gate procedures where needed.
4. 4

   Two generations of the fare gateway

   • The older gateway generation read its configuration in a fixed-size memory buffer.
   • The newer generation was built to load configuration dynamically and scale with its size.
   • Both generations ran side by side during a migration period.
   • The two generations handled an oversized file differently, which complicated diagnosis.
5. 5

   The failure cycle that kept repeating

   • A gateway instance loaded the new file and failed on startup or during reload.
   • Automatic restarts brought it back up, and it loaded the same file again.
   • Each restart repeated the failure, so instances never stayed healthy.
   • Recovery needed the file to change, not just the process to restart.
6. 6

   Timeline: the first hours of the outage

   • The configuration was published and propagated to gateway instances.
   • Instances began failing, and fare error rates climbed.
   • Early alerts looked like a network or capacity problem rather than a configuration one.
   • Responders were paged and opened an incident channel.
7. 7

   Timeline: rollback and recovery

   • Responders identified the new configuration as the common change across failing instances.
   • The team rolled back to the last known good file.
   • Instances were restarted in stages so they would not overload downstream systems.
   • Fare validation recovered, and the team monitored before declaring the incident resolved.
8. 8

   How the fault was detected and diagnosed

   • Monitoring flagged rising fare errors and unhealthy gateway instances.
   • Responders compared recent changes and matched the failures to the configuration publish.
   • Instance logs showed load errors tied to the size of the file.
   • Early hypotheses such as network faults were ruled out by evidence.
9. 9

   Mechanism: how an oversized configuration file broke the gateway

   • The gateway loads its full configuration into memory when it starts or reloads.
   • The oversized file exceeded the limit the loader could handle.
   • The loader failed instead of rejecting the file and keeping the old configuration.
   • Failure at load meant no instance could serve fares.
10. 10

    Contributing factors behind the oversized file

    • The configuration grew gradually as entries were added over time.
    • Generation tooling did not warn as the file neared the limit.
    • The size limit was not documented where authors would see it.
    • Growth looked routine in review because diffs showed entries, not total size.
11. 11

    The systemic condition that let the file through

    • No validation step checked file size against gateway limits before publishing.
    • Publishing pushed the file to all instances at once, with no staged rollout.
    • Nothing made the gateway keep its last good configuration on a failed load.
    • Any one of these safeguards could have contained the failure.
12. 12

    What went well during the response

    • Responders formed an incident channel quickly and kept clear roles.
    • The team found the link between the change and failures through change review.
    • A last known good configuration was available for rollback.
    • Communication to operations staff enabled manual fare procedures.
13. 13

    Corrective actions and their owners

    • Add a pre-publish check that rejects files above the gateway size limit.
    • Make the gateway keep its previous configuration if a new one fails to load.
    • Roll out configuration changes in stages with automatic halt on errors.
    • Each action has a named owning team and a review date.
14. 14

    Questions the review could not yet answer

    • It is unclear why earlier alerts pointed toward a network problem.
    • The team has not confirmed how close prior files came to the limit.
    • Whether the older gateway generation shares the same weakness is still open.
    • These questions will be tracked as follow-up investigations.
15. 15

    Evidence this review draws on

    • The review draws on gateway logs and fare error metrics from the incident window.
    • Configuration change history shows how the file grew over time.
    • The incident channel record supplies the response timeline.
    • Interviews with responders add context on decisions made.