Slide 1
Slide 1 text
IT-Tage Frankfurt 2015
Lasst uns die Datenbank
in der Cloud hosten...
Philipp Krenn̴̴̴̴̴̴@xeraa
Slide 2
Slide 2 text
Wien
Slide 3
Slide 3 text
Wien
Slide 4
Slide 4 text
Wien
Slide 5
Slide 5 text
ViennaDB
Papers We Love Vienna
Slide 6
Slide 6 text
Schulungen
MongoDB
NoSQL Überblick
DevOps
AWS
Slide 7
Slide 7 text
Electronic Data Interchange (EDI)
Slide 8
Slide 8 text
No content
Slide 9
Slide 9 text
Anwendungsfall
Metadaten
Slide 10
Slide 10 text
Es war einmal...
PaaS (DBaaS) vs Eigenhosting — AWS
Slide 11
Slide 11 text
No content
Slide 12
Slide 12 text
No content
Slide 13
Slide 13 text
Wir haben nie
Daten verloren
sichtbare Downtime gehabt
Slide 14
Slide 14 text
ABER
Slide 15
Slide 15 text
No content
Slide 16
Slide 16 text
Erster Versuch
Shared Replica
Slide 17
Slide 17 text
#9351 xxx.member0.mongohq.com:10000 down
(Feb 27, 2013)
"Yes, xxx.member0:10000 was down for a about
16 minutes. I was upgrading other instances on
the environment to 2.2.3, and mistakenly stop
your instance on the server."
Slide 18
Slide 18 text
No content
Slide 19
Slide 19 text
Generelle Limitierung
Keine Firewall-Regeln
Slide 20
Slide 20 text
Zweiter Versuch
Dedicated Replica
Slide 21
Slide 21 text
No content
Slide 22
Slide 22 text
No content
Slide 23
Slide 23 text
#10861 Broken S3 Backup a!er Instance Upgrade
(Apr 21, 2013)
"Failed to backup xxx: Error with S3
permissions."
"We are working on better backup solutions for
our new dedicated deployments."
Slide 24
Slide 24 text
#11453 Wrong Billing for April 2013 (May 15,
2013)
Abgebucht: $1297.73
Erwartet: $258.70
Erste hilfreiche Antwort nach 14 Tagen
Slide 25
Slide 25 text
#11454 Follow Up: Broken Backup to S3 on
Dedicated Instances (May 15, 2013)
"If they run they can be restored, but I would say
over 75% of the ones that we have tried to run
hang or never run at all."
Bessere Lösung in Aussicht gestellt
Slide 26
Slide 26 text
No content
Slide 27
Slide 27 text
#11513 Old Replica Member: not master and
slaveOk=false (May 17, 2013)
"each day we're getting a warning about a
connection problem from a replica member
which has been removed a month ago"
"I've cleaned up the old replica that was trying to
connect. Sorry this wasn't done earlier."
Slide 28
Slide 28 text
#11844 Firewall Rule (Jun 01, 2013)
Erstmalig nach GUI für Selbstservice gefragt
Slide 29
Slide 29 text
#11976 URGENT: Database Update Stuck (Jun 08,
2013)
"We basically need to open a ticket for every
single operation we need to do (backup,
compaction, upgrade plus a billing issue). Could
you (or someone else) comment on this, please?
Are we just 'unlucky' or is this simply not working
too well at the moment?"
Slide 30
Slide 30 text
#12146 Billing May 2013 (Jun 16, 2013)
"But we've also got an entry for the same
database on the shared infrastructure, which we
are not using any more"
Slide 31
Slide 31 text
#12659 Billing and Backups (Jul 07, 2013)
"We have been working hard on the backups and
gotten things working for most of our setups.
That is why you see it working again. Though we
are still working on a new backup system that
will deal with big data much more efficiently."
Slide 32
Slide 32 text
No content
Slide 33
Slide 33 text
#13628 Additional database on dedicated replica
set (Aug 19, 2013)
"However, it doesn't seem to work - the database
(https://app.mongohq.com/xxx/mongo/billing)
seems to use a different port than the webapp.
Please see the screenshot I've attached."
Slide 34
Slide 34 text
No content
Slide 35
Slide 35 text
#15493 Backups broken on dedicated instances?
(Oct 26, 2013)
"The whole backup has about 500 bytes (not KB
or MB, just B)."
"I've corrected the issue. A!er the migration to
the dedicated hosts, our S3 backup system still
had your database located at the [shared] host."
Slide 36
Slide 36 text
Sicherheit
bufferapp.com
Slide 37
Slide 37 text
"On October 28, 2013, we detected unauthorized
access to an internal support application using a
password that was shared with a compromised
personal account."
http://security.mongohq.com/notice#oct-31-
update (Oct 28, 2013)
Slide 38
Slide 38 text
Zutaten
1. Spear Phishing gegen MongoHQ Angestellte
2. Unbeschränkt erreichbares sudo über interne
Support-Applikation
Slide 39
Slide 39 text
No content
Slide 40
Slide 40 text
No content
Slide 41
Slide 41 text
No content
Slide 42
Slide 42 text
"[...] we became aware that a SendGrid
employee’s account had been compromised by a
cyber criminal and used to access several of our
internal systems on three separate dates in
February and March 2015."
https://sendgrid.com/blog/update-on-security-
incident-and-additional-security-measures/
(Apr 27, 2015)
Slide 43
Slide 43 text
No content
Slide 44
Slide 44 text
"[...] suffered a major extended outage. This
outage was the result of an attack on our systems
using a compromised API key."
http://status.bonsai.io/incidents/qt70mqtjbf0s
(Jul 03, 2014)
Slide 45
Slide 45 text
No content
Slide 46
Slide 46 text
No content
Slide 47
Slide 47 text
3 Monate und 2 Tickets später
2FA
https://www.compose.io/articles/two-factor-authentication-and-
security-auditing-now-available-for-all-mongohq-accounts/
(Jan 28, 2014)
Slide 48
Slide 48 text
#18695 DB Upgrade Failed (Jan 25, 2014)
"we just tried to upgrade our database from 2.4.4
to 2.4.8, but the operation failed"
"our system had the wrong auth information
stored to make that change for your servers"
Slide 49
Slide 49 text
#36374767 Re: DB Backup errors (Jul 4, 2014)
"'Error retrieving S3 files. Please check your
backup settings.' on the Admin/Backups page."
"Our ops team is looking into issues whey our
backup servers cannot connect to your database
host:port."
Slide 50
Slide 50 text
"The backup I've started manually seems to be
stuck with: 'Backing up xxx. Updated 2 hours
ago'"
"You will need to create a new S3 bucket without
periods in the bucket name."
Slide 51
Slide 51 text
#48928851 Legacy Backups? (Oct 7, 2014)
'Legacy Backups: the S3 backup system has been
deprecated. [...] please upgrade to a new
MongoDB Deployment.'
"This latest S3 backup problem that caused
several days of outages appears to be a problem
on S3's side, which is not an unusual occurrence."
Slide 52
Slide 52 text
No content
Slide 53
Slide 53 text
#48928476 Additional subnet (Oct 7, 2014)
Falschen Port geöffnet — schon wieder!
Und es hat länger als 50 Stunden gedauert
Slide 54
Slide 54 text
#66790506 Unknown database 'nagios' (Jan 1,
2015)
"No worries at all. This is likely a remnant from
our own legacy monitoring very, very long ago. In
our old data browser, we only displayed
databases created within the browser."
Slide 55
Slide 55 text
No content
Slide 56
Slide 56 text
#102547608 Stuck on deprovision (Jul 5, 2015)
"I took a look at it and pushed the deprovision [of
the Disque queue] through"
Slide 57
Slide 57 text
No content
Slide 58
Slide 58 text
Migration
AWS Ireland EC2 Classic
AWS Frankfurt VPC
Slide 59
Slide 59 text
Dritter Versuch
Eigenhosting
Slide 60
Slide 60 text
No content
Slide 61
Slide 61 text
Erfahrung
Automatisierung
Slide 62
Slide 62 text
Relativ einfach zu konfigurieren
JavaScript Shell problematisch
Details: Logrotate, Chaining deaktivieren, Linux Transparent Huge
Pages deaktivieren,...
Slide 63
Slide 63 text
No content
Slide 64
Slide 64 text
Erfahrung
Monitoring
Slide 65
Slide 65 text
cloud.mongodb.com
newrelic.com
datadoghq.com
Slide 66
Slide 66 text
No content
Slide 67
Slide 67 text
Erfahrung
Backups
Slide 68
Slide 68 text
1. mongodump
2. Komprimieren und verschlüsseln
3. Auf AWS S3 hochladen
Slide 69
Slide 69 text
Falle
backup User kann Profiling Daten
nicht lesen — dbAdmin zusätzlich
notwendig
Slide 70
Slide 70 text
Tipps
deadmanssnitch.com
S3: Write-Only, Versioning, Cross-
Region, Infriquently Accessed
Slide 71
Slide 71 text
Schrödingers Backup
Der Zustand eines Backups ist
unbekannt bis zum Versuch der
Wiederherstellung
Slide 72
Slide 72 text
Zusammenfassung
Slide 73
Slide 73 text
No content
Slide 74
Slide 74 text
No content
Slide 75
Slide 75 text
There are a few great providers available, and
today we use a Compose MongoDB instance for
some of our non-critical BI data.
http://blog.sendwithus.com/from-postgresql-to-dynamodb/
Slide 76
Slide 76 text
Wir sind trotzdem überzeugt, dass Compose
große Fortschritte macht
sich immer bemüht hat
nicht schlechter als die Konkurrenz ist
Slide 77
Slide 77 text
Zumindest in der Vergangenheit
Dedicated Installationen
schwierig
Slide 78
Slide 78 text
You shall not
PaaS
Slide 79
Slide 79 text
No content
Slide 80
Slide 80 text
Danke!
Fragen?!
@xeraa
Slide 81
Slide 81 text
Bilder
· Schnitzel https://flic.kr/p/9m27wm
· Architektur https://flic.kr/p/6dwCAe
· Conchita https://flic.kr/p/nBqSHT
· Papier http://www.freeimages.com/photo/
432276