@owengretzinger·6dI woke up to an email from AWS: "Your AWS Account may be compromised!"
As head of engineering, that's the email I have nightmares about.
Someone had stolen our AWS keys and was using them to run Claude. AWS blocked over 3,000 of their requests and locked down our whole account, so we couldn't make new deployments.
(The one request that got through was Opus. At least they have taste.)
I checked everything else. No new users, nothing deleted. Ok. Breathe.
Then OpenAI emailed. Then Stripe told us our live key was "publicly available online."
Everyone on the team ran a forensic script on their laptop. My agent went through CI, our servers and git history, and checked for the npm worm going around. All clean. (Which was somehow worse.)
Then we found it. We hadn't used one of the stolen keys since July, and the only place it still existed was an old Railway service that wasn't even running anymore.
Three weeks ago we moved our core infra from Railway to AWS, partly because of all the downtime and partly because we were tired of running up against Railway limits. It took a few months to make the jump, and we'd started wondering if we made the wrong call.
A few hours later, Railway announced it had been breached. AWS told us at 4am. Railway told everyone at 3pm.
They said "rotate your keys". But they also said deployments are frozen. We'd just migrated off, so we dodged that bullet by a hair.
Everyone still on Railway gets to choose: Leave your credentials exposed, or take down your servers?
