The load test that changed everything

We have our first convention event on April 12th. Dozens of players, all on their phones, all hitting the app at once - browsing cards, checking prices, opening repack packs, placing orders, running tournament brackets. Real concurrent load for the first time.

So we ran a stress test. The question was simple: can our server handle 50-100 simultaneous mobile users?

The answer was no.

One process, four cores, and a math problem

CardCore's production server is an Intel i7-7700 with 4 cores and 32GB of RAM. More than enough hardware. But we were running the entire application as a single Node.js process managed by NSSM (a Windows service wrapper). One process means one CPU core. Three cores sitting completely idle while the one active core is maxed out handling every request sequentially.

Node.js is single-threaded by design. It handles I/O well - database queries, file reads, network calls - because those operations are non-blocking. But CPU-bound work like rendering React server components, JSON serialization, and middleware chains still blocks the event loop. Under load, requests start queuing behind each other. Response times climb. At 50 concurrent users, the app would start choking.

The database had its own bottleneck. PostgreSQL was configured with max_connections=100, but our app uses three connection pools of 25 connections each (inventory, cards, customer databases). That's 75 connections reserved, leaving only 25 for overhead. Under load with multiple app instances, we'd hit the wall.

The midnight rebuild

The solution was cluster mode - running multiple instances of the app across all four CPU cores, with a load balancer distributing requests. The tool for this is pm2, the standard Node.js process manager. Simple in theory. The execution was anything but.

Attempt 1: NSSM + pm2-runtime (failed)

Our first idea was to keep NSSM as the Windows service and just change what it runs from next start to pm2-runtime. NSSM would manage the service lifecycle, pm2 would manage the cluster.

It didn't work. NSSM runs as the SYSTEM account, which can't access the user's AppData folder where pm2 is installed. We tried pointing NSSM directly at the pm2-runtime binary. SERVICE_PAUSED every time. We tried changing the NSSM service account. Same result.

The root cause: pm2 daemonizes itself. It spawns a background process and the foreground process exits. NSSM expects a foreground process that stays alive. When pm2's foreground exits, NSSM thinks the service crashed and pauses it. They're architecturally incompatible.

Attempt 2: NSSM + pm2 resurrect --no-daemon (failed)

We tried forcing pm2 into foreground mode. Same SERVICE_PAUSED result. pm2's daemon architecture is too deeply baked in to work inside NSSM's process model.

Attempt 3: PostgreSQL ALTER SYSTEM (failed)

While debugging the app server, we also tried increasing PostgreSQL's connection limit. ALTER SYSTEM SET max_connections = 200 wrote to postgresql.auto.conf inside the Docker container. But our docker-compose file had -c max_connections=100 in the CMD flags, which overrides both postgresql.conf and postgresql.auto.conf. The ALTER SYSTEM command succeeded silently but had zero effect.

We tried sed -i on postgresql.conf inside the container. Same problem - CMD flags always win.

What actually worked

pm2-installer - a community project that creates a proper Windows service for pm2. It handles everything NSSM couldn't: service registration, PM2_HOME at a system-accessible path (C:\ProgramData\pm2\home), proper permissions, log rotation, and auto-resurrection on boot.

We cloned it, ran the configure and setup scripts, created an ecosystem.config.js file defining 4 cluster instances, started it up, and saved the process list. Done. pm2 manages the cluster, the Windows service manager keeps pm2 alive, and all four CPU cores are now working.

Docker compose edit - for PostgreSQL, the fix was editing the docker-compose.yml directly to change -c max_connections=100 to 200, then recreating the container. No amount of runtime configuration changes would override those CMD flags.

What we gained

The numbers tell the story:

4x CPU utilization - four cores working instead of one
Zero-downtime deploys - pm2 reload does rolling restarts across instances, so there's never a moment where the app is down during a deploy
Per-instance crash recovery - if one instance dies, pm2 restarts just that instance while the other three keep serving. Before, a single crash took down the entire app
~200MB per instance - four instances fit comfortably in 32GB RAM with room to spare
125 free database connections - up from 25, enough headroom for the cluster under full convention load

The deploy command changed

This might seem like a small detail but it affects every future deploy. The old command:

nssm restart CardCoreWeb-DV1

Hard restart. App goes down, comes back up. Every user sees a brief outage.

The new command:

pm2 reload ecosystem.config.js

Zero-downtime rolling restart. Instance 1 reloads while 2, 3, 4 serve traffic. Then instance 2 reloads while 1, 3, 4 serve. No user ever sees downtime.

What this means for April 12th

The convention stress test went from "this will probably crash" to "this should handle it comfortably." Four processes sharing the load, double the database headroom, and crash isolation so one bad request doesn't take down the whole platform.

There's still work to do - the in-memory rate limiter and cache don't share state across instances (each instance has its own), which means rate limits are effectively 4x more lenient and cache hit rates are lower. For a convention with 50-100 users, this is an acceptable trade-off. For 500+ users, we'd need Redis. But that's a bridge we'll cross when we get there.

The lesson

Load test before your first real event, not during it. We almost shipped to a convention running on one CPU core with 25 spare database connections. That's the kind of thing that works fine with 5 concurrent users and falls apart catastrophically with 50.

The hardware was there. The software just wasn't using it. Now it is.