We tell how excessive faith in neural networks turned a cloudy dream into a digital ash.
AI tools for development are increasingly taking on routine tasks, but the recent failure has shown how expensive an additional level of trust can be. In PocketOSPockets, a car rental service, the AI-agent launched in Cursor, in just nine seconds removed the entire working database along with backups, and the recovery turned into a real crisis.
PocketOS founder Jer Crane said that the agent worked in a test environment and faced access problems. Instead of stopping and requesting assistance, the system began to look for the necessary API tokens, found it in a third-party file and completed the command to delete the data volume in the Railway, where the startup infrastructure was located.
According to Crane, the dangerous operation was stopped by a request for confirmation, no encircling checking, no warning of risk to working data. The request went immediately, and the backups were stored in the same volume, so they disappeared along with the main base. The most recent merchandise copy was three months ago.
Crane claims that the agent later recognized the violation of his own safety rules. The system, he said, acted on the basis of assumptions, performed a destructive team without permission and did not figure out to the end of which infrastructure it interacts with.
The founder of Pockets considers the incident not a one-time error, but a sign of a deeper problem. The paper did not use experimental assembly, but Cursor with the Anthropic Claude Opus model and prescribed safety rules. However, such restrictions did not prevent the agent from gaining access to a critical operation.
Separate claims Crane made to the Railway. According to him, API tokens were not sufficiently limited by rights, so the key for a simple task could perform actions at the level of critical infrastructure. Storing backups near working data also deprived the company of a normal recovery route.
30 hours after the incident, the service was restored, but on the old backup. Pockets customers have temporarily lost access to fresh bookings, customer data and payment information. Part of the recordings had to be restored manually by letters, calendars and payment systems. Although now the main functionality of the service is available, the gap in the data will have to be closed for weeks.
The incident strengthened the dispute over the introduction of AI agents into real systems. Crane urged developers to introduce mandatory confirmations for dangerous commands, strictly restrict the rights of tokens, store backup copies separately and not count system instructions for AI with full protection.
AI tools for development are increasingly taking on routine tasks, but the recent failure has shown how expensive an additional level of trust can be. In PocketOSPockets, a car rental service, the AI-agent launched in Cursor, in just nine seconds removed the entire working database along with backups, and the recovery turned into a real crisis.
PocketOS founder Jer Crane said that the agent worked in a test environment and faced access problems. Instead of stopping and requesting assistance, the system began to look for the necessary API tokens, found it in a third-party file and completed the command to delete the data volume in the Railway, where the startup infrastructure was located.
According to Crane, the dangerous operation was stopped by a request for confirmation, no encircling checking, no warning of risk to working data. The request went immediately, and the backups were stored in the same volume, so they disappeared along with the main base. The most recent merchandise copy was three months ago.
Crane claims that the agent later recognized the violation of his own safety rules. The system, he said, acted on the basis of assumptions, performed a destructive team without permission and did not figure out to the end of which infrastructure it interacts with.
The founder of Pockets considers the incident not a one-time error, but a sign of a deeper problem. The paper did not use experimental assembly, but Cursor with the Anthropic Claude Opus model and prescribed safety rules. However, such restrictions did not prevent the agent from gaining access to a critical operation.
Separate claims Crane made to the Railway. According to him, API tokens were not sufficiently limited by rights, so the key for a simple task could perform actions at the level of critical infrastructure. Storing backups near working data also deprived the company of a normal recovery route.
30 hours after the incident, the service was restored, but on the old backup. Pockets customers have temporarily lost access to fresh bookings, customer data and payment information. Part of the recordings had to be restored manually by letters, calendars and payment systems. Although now the main functionality of the service is available, the gap in the data will have to be closed for weeks.
The incident strengthened the dispute over the introduction of AI agents into real systems. Crane urged developers to introduce mandatory confirmations for dangerous commands, strictly restrict the rights of tokens, store backup copies separately and not count system instructions for AI with full protection.
