The day our sysadmin went full Thanos on our databases (and why it should never have been possible)
At 9:02 AM on a perfectly normal Tuesday, our systems engineer—let’s call him Dave (because that’s his name)—did the unthinkable:
DROP DATABASE production;
And just like that, our customer data, billing records, logs, analytics—gone. Not in a “check-the-trash-bin” kind of way, but in a “call-your-lawyer” kind of way.Dave later explained:
“I was cleaning up old dev environments and… you know what? Never mind.”
What Went Wrong (Besides Dave)
1. Root Access for Dave
Dave had root access to everything: development, staging, and production. Like a digital god with no one watching.
Lesson: The principle of least privilege should not be optional. No one should have full access to production without oversight.
2. No Multi-Factor Authentication
Dave used SSH keys saved in a folder named keys/ on his desktop. His laptop password was literally dave123.
Lesson: If someone can guess your password faster than they can microwave a burrito, it’s time to enable MFA.
3. No Command Approval or Auditing
No peer review, no change ticket, no “are you sure?” confirmation. Dave typed the command and hit enter.
Lesson: Destructive actions in production should require multiple approvals and logging. It shouldn’t be this easy to ruin everything.
4. Backups That Didn’t Help
We had backups. But they weren’t recent. And we never tested them.
Lesson: A backup isn’t a backup until you’ve restored from it and verified it works.
The Fallout
Our CISO cried.
- Our customers noticed.
- Our engineers pulled 72-hour shifts trying to reconstruct data from logs and memory.
- Dave was laterally promoted to “Cloud Strategy” — which, for the record, involves no direct server access.
What Should Have Happened
In a world where security practices are taken seriously:
- Dave would only have access to development, and maybe read-only logs from production.
- Every change to production would go through a proper change management system with multiple approvals.
- MFA would be enforced for every access point.
- Backups would be automated, versioned, offsite, and regularly tested.
- Dave would be too busy submitting access requests to run wild in production.
This isn’t just a story about Dave. This is a story about every organisation that hasn’t locked down its environments, enforced access control, or assumed that internal users might cause accidental,or catastrophic harm !
While external attackers try for weeks to breach your systems, Dave can destroy them in 12 keystrokes.
Takeaway: Your biggest security threat might already be inside your network—with an access badge and a good performance review.