Building a viral Imax ticketing app that never crashes

Introduction

When a high-demand IMAX film drops, traffic doesn’t ramp up; it detonates. We’re talking about a thundering herd problem where thousands of concurrent requests hammer a finite inventory, demanding strict consistency and sub-second latency. In this scenario, “never crashes” isn’t a marketing slogan. It’s a mathematical constraint defined by Service Level Objectives (SLOs) and error budgets.

Building a system that survives viral traffic requires shifting left on resilience. We stop treating crashes as anomalies and start engineering for failure modes. The architecture must decouple ingestion from processing, enforce admission control at the edge, and guarantee data integrity under race conditions. This article breaks down the DevOps and architectural patterns we use to keep ticketing platforms stable when the load spikes by 10,000x in seconds.

Why This Matters

Most engineering teams design for average load and hope scaling policies catch up during spikes. That approach fails for ticketing. The combination of limited inventory and high concurrency creates two critical failure modes:

1. Database Contention: Synchronous writes to a relational database during a spike exhaust connection pools, causing cascading timeouts that bring down the entire cluster.
2. Double-Booking Races: Without atomic locking mechanisms, race conditions allow the same seat to be allocated to multiple users, resulting in revenue reconciliation nightmares and brand damage.

Solving these problems requires a shift from monolithic request handling to event-driven architectures with strict backpressure mechanisms. Engineers need to understand how to implement admission control, idempotency, and distributed locking to protect the core data layer.

How It Works

The architecture relies on a three-stage defense-in-depth strategy: Edge Filtering, Asynchronous Processing, and Atomic State Management.

1. Edge Filtering and Admission Control

Traffic hits a CDN and WAF, which strip DDoS noise and cache static assets. The API Gateway enforces admission control using a token bucket algorithm. If the downstream system shows signs of strain (high latency or queue depth), the gateway immediately returns 429 Too Many Requests. This prevents slow requests from consuming thread pools and OOM-killing services.

2. Asynchronous Reservation Pipeline

Valid requests are not processed synchronously. The API endpoint validates the request, checks a fast in-memory cache for seat availability, and pushes a reservation event to a message broker. The client receives an immediate acknowledgment with a reservation ID. This decouples the user-facing latency from the heavier database and payment operations.

3. Atomic Inventory Management

Workers consuming from the queue perform the actual seat locking. We use Redis with Lua scripting to execute atomic check-and-set operations. This ensures that even under massive concurrency, a seat is only locked once. If the lock succeeds, the worker persists the order to PostgreSQL and triggers the payment flow.

flowchart TD
    User[End User] --> CDN[CDN & WAF]
    CDN --> LB[Global Load Balancer]
    LB --> GW[API Gateway]
    GW --> RL[Admission Controller]
    RL -->|Pass| TS[Ticket Orchestrator]
    RL -->|Reject| Err[Return 429]
    TS --> IS[Inventory Service]
    IS --> Redis[(Redis Cluster)]
    Redis -->|Lock Result| IS
    IS -->|Async Event| Broker[Message Broker]
    Broker --> Worker[Reservation Worker]
    Worker --> DB[(PostgreSQL)]
    Worker --> PS[Payment Service]
    TS -.-> Traces[Jaeger Traces]
    Worker -.-> Metrics[Prometheus Metrics]

The diagram above illustrates the critical path. Notice the separation between the Ticket Orchestrator and the Reservation Worker. The orchestrator only interacts with Redis and the broker, keeping the user-facing path lightweight. The heavy lifting of database writes and payment gateway calls happens asynchronously, isolated from user latency.

Core Concepts

Admission Control with Backpressure

Admission control isn’t just rate limiting. It’s dynamic throttling based on system health. We monitor the message broker’s queue depth and the p99 latency of the worker services. If the queue depth exceeds a threshold, the admission controller increases the rejection rate. This backpressure mechanism ensures the system degrades gracefully by rejecting excess traffic rather than crashing under load.

Idempotency Keys

Network retries are inevitable. Clients may retry a payment request due to a timeout, even if the request succeeded. To prevent double-charging, every request must include an idempotency key. The API checks this key against a distributed cache before processing. If the key exists and the operation completed, we return the cached result. This makes the API safe to retry without side effects.

Distributed Locking with Lua

Standard Redis commands are not atomic when combined. To check a seat and lock it in one step, we use a Lua script. Redis executes Lua scripts atomically, meaning no other client can run a script or command until the current script finishes. This eliminates TOCTOU (Time-of-Check-to-Time-of-Use) races.

Circuit Breakers for External Dependencies

Payment gateways and SMS providers are external dependencies that can fail. We wrap these calls in circuit breakers. If the payment gateway returns errors above a threshold, the circuit opens, and requests fail fast without waiting for timeouts. This prevents thread pool saturation and keeps the ticketing core responsive.

Examples & Code Walkthrough

Atomic Seat Reservation with Lua

We use a Lua script to atomically check seat status and acquire a lock. This script runs inside Redis, ensuring no other client can intervene between the check and the lock.

-- KEYS[1] = seat_key (e.g., "movie:123:seat:A1")
-- ARGV[1] = user_id
-- ARGV[2] = lock_ttl in seconds

local seat_status = redis.call('GET', KEYS[1])

-- If seat is already locked or sold, return error
if seat_status then
    return { err = "seat_unavailable", status = seat_status }
end

-- Atomically set the lock with the user_id and TTL
-- NX ensures we only set if key doesn't exist
-- EX sets the expiration time
local result = redis.call('SET', KEYS[1], ARGV[1], 'NX', 'EX', ARGV[2])

if result then
    return { err = nil, status = "locked" }
else
    -- Another script ran between GET and SET (should not happen in Lua)
    return { err = "race_condition", status = "unknown" }
end

Idempotency Middleware in Go

This middleware intercepts requests, checks for an idempotency key, and prevents duplicate processing.

package middleware

import (
    "context"
    "crypto/sha256"