Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Transparently Deleting (and Restoring) a Production Database at GitLab

Transparently Deleting (and Restoring) a Production Database at GitLab

At the end of January 2017, we accidentally deleted the production database for GitLab.com. This presentation focuses on how we communicated with our users to make this mistake as transparent as possible.

This was presented in April 2017 for the Colorado State University ACM meetup.

Connor Shea

April 12, 2017
Tweet

More Decks by Connor Shea

Other Decks in Technology

Transcript

  1. Transparently Deleting (and Restoring) a Production Database at GitLab Connor

    Shea, Community Advocate and Frontend Engineer at GitLab (and also a Q&A at the end)
  2. What is GitLab? • Source code management • Issue tracking

    • Code review • Continuous Integration (running the test suite) • Deployment
  3. Background • GitLab is remote-only, team members in 36 countries.

    • Almost all our products are either open source or source-visible. • Transparency is a core value of GitLab.
  4. Background • On January 31, 2017 an engineer accidentally deleted

    the production database for GitLab.com. • We ended up losing six hours of data at the database layer, no code was lost but some issues, comments, etc. were.
  5. The Google Doc • The infrastructure team started putting together

    a Google Doc to track progress. • Sid (CEO of GitLab) made the decision to tweet this out. • It ended up at the top of Hacker News within about 15 minutes.
  6. The Live Stream • A few hours later we started

    a live stream as we recovered the database from a backup. • This turned into a Q&A session with viewers once everything was over.
  7. What we learned • Audit your backups and procedures for

    running commands on production servers. • Communication with users, even in really bad situations, leads to better outcomes. • Users really value transparency.
  8. Why do this at all? • It’s the right thing

    to do. • Users value transparency. • Pushing issues under the rug will hurt you in the long run.
  9. Results • Outpouring of support on Twitter (3000+ tweets) •

    ~5000 viewers of the live stream on YouTube • Top of Hacker News • Top of some programming-related subreddits
  10. Q&A • Questions about: • The presentation • Working at

    GitLab • Using Git / Version Control in general • Open Source • Working remotely • Web development • The Legend of Zelda: Breath of the Wild • Something else?