Railway, a cloud platform provider, has disclosed that its precautionary request to revoke API keys following a security incident included replacement credentials that were issued after attackers had already lost access. In a clarification posted on October 24, the company noted that AI providers such as OpenAI and Anthropic may have disabled these newer keys as part of the same revocation request, potentially affecting customers who had already rotated their credentials.
Background of the Incident
The initial warning was reported on October 22, urging all users to rotate stored credentials after unauthorized access was detected. Railway's preliminary investigation indicated that the attacker's access ended on October 22 at 03:32 UTC. However, the company later requested another batch of revocations from AI providers on October 23, which included keys that had been generated after the breach was contained. This has created confusion and potential service disruptions for users.
Exposure Scope and Vendor Resets
Railway's preliminary estimate suggests that up to 1.85% of workspaces, or approximately 100,000, could have been exposed. This figure is based on initial forensics and may change as the investigation continues. Importantly, the vendor reset requests covered a broader group than the estimated exposure, as Railway could not filter the batch by the date keys were added. This complicates the essential defense of replacing stolen credentials, as users may have received revocation notices for keys that were never compromised.
Recovery Process and User Impact
As part of the recovery, Railway invalidated user login sessions, requiring users to sign in again. They may also need to reconnect GitHub access and replace CLI or continuous-integration tokens. Each dependency requires a separate recovery step, and updating stored variables requires deploying the change. This has placed a significant burden on users, especially those with complex integrations.
Security Experts Weigh In
GitGuardian's security research team recommends rotating credentials at the upstream service before updating Railway. This guidance also covers sealed variables, which customers cannot read back through normal interfaces. The fact that these variables are sealed does not guarantee they are safe. Symantec analyst Dick O'Brien noted that API keys for AI models are valuable because anyone holding one can run inference at the owner's expense. He also stated that there was no evidence linking Railway to separate attacks his team had observed, but the motive for the attack appears to be obtaining model access for distillation, though this remains preliminary.
Timeline of Events
- October 22, 03:32 UTC: Attacker access ended.
- October 23: Another batch of revocation requests sent to AI providers.
- October 24, 13:57 UTC: Clarification posted.
- October 25, 06:26 UTC: Notice dismissed.
The elapsed time between access ending and clarification was 58 hours and 25 minutes.
Future Outlook
Railway has accelerated its transition to single-tenant environments and rebuilt internal tooling as part of its security enhancements. Customer deployments remained online during the build suspension. The company's latest update states that working keys do not need to be rotated again under that notice, but the number of replacement keys revoked remains undisclosed. As the investigation concludes, further remediation may be required for some customers, and the exposure estimate could be revised.
