Ceph @ Medallia - CEPH DAY BA 2017

A journey on our usage of CEPH @ scale in Medallia and uses for upcoming versions

Transcript

1. Medallia © Copyright 2017. Confidential. 1 CEPH Day Buenos Aires - November 2017 marcelo@medallia.com @marceloklein
2. Medallia © Copyright 2017. Confidential. 2 What do you need the scale for? Medallia
3. Medallia © Copyright 2017. Confidential. 3 Next Gen Datacenter Platform Atlas • Commodity Components • Open Standards • Cheap & Scalable • No specific-HW-tied services • Redundancy • Self-healing • Servers as cattle Commodity Products No special machines
4. Medallia © Copyright 2017. Confidential. 4 Our Requirements Ceph in Atlas • Storage Requirements • Docker runs everywhere… ephemerally or locally. • Support Dev and Production uses • Multi-tenancy on shared infra • We ran PG DBs < 9.6 (Latency + IO) • 4 hour SLA vs “don’t touch me” servers
5. Medallia © Copyright 2017. Confidential. 5 The Contenders Ceph in Atlas iScsi +RAID Scale Performance Management Native
6. Medallia © Copyright 2017. Confidential. 6 Why did we choose CEPH? Ceph in Atlas • All-in-one solution (Block, Share, Object) • Designed for scale • Compatible with public cloud • Open Source backed by community • Self-Healing • Giant was “mature enough” - Kernel mod • Cheaper than other proprietary solutions
7. Medallia © Copyright 2017. Confidential. 7 Atlas DC Spec compute servers controller storage servers tor switches compute servers controller storage servers tor switches compute servers controller storage servers tor switches compute servers controller storage servers tor switches compute servers controller storage servers tor switches
8. Medallia © Copyright 2017. Confidential. 8 Atlas Rack Spec 0.9 Linux (Ubuntu) 2xIntel E5-v4 256GB Memory 50GbE Network 240GB SSD 22 x Compute Node 3 x Networking Linux (Cumulus) 1xIntel 64-bit 16 GB Memory 32x40/100 GbE Network 8 x Storage Node Linux (Ubuntu) 1xIntel E5-v4 64 GB Memory 50GbE Network 8x800GB SSD* PCIe NVRAM
9. Medallia © Copyright 2017. Confidential. 9 • GREAT Use Cases • Relocating non-movable services using containers • PG Databases + Snapshots, Monolith mounts • Reexporting NFS + Snapshots, Replace HDFS • RadosGW: → Replacing CephFS ◦ → KV Store ◦ → S3 compatible tooling • GREAT Performance • < 5 - 30ms reads for DBs • MDS as local shared NFS Yay we chose the best solution!
10. Medallia © Copyright 2017. Confidential. 10 • GREAT upgrading process and management • GREAT User community: Large, friendly and engaged! • AWESOME version names The Good
11. Medallia © Copyright 2017. Confidential. 11 Next Gen Datacenter Platform Atlas • Proudly serving traffic since 2015 • 5 Datacenters + 2 AWS Zones • ~ 3000 containers running in almost 1k nodes in Aurora/Mesos • More than 3 PB of CEPH storage + 1.2 TB Ephemeral / Node • DC Deployment every 2 weeks • Storage Team: 6 people*
12. Medallia © Copyright 2017. Confidential. 12 Atlas Rack Spec 1.1 Linux (Ubuntu) 2xIntel Skylake 384GB Memory 100GbE Network 2x1TB SSD 28 x Compute Node 3 x Networking Linux (Cumulus) 1xIntel 1047UE 64-bit 16 GB Memory 32x100 GbE Network 12 x Storage Node Linux (Ubuntu) 1xIntel Skylake 96 GB Memory 100GbE Network 10x4TB SSD
13. Medallia © Copyright 2017. Confidential. 13 Did we chose the best solution? Storage Team Jan ‘17
14. Medallia © Copyright 2017. Confidential. 14 • The “Stuck OSD” issue + “The Slow OSD” issue • Orphans in RadosGW • The MDS Integer • Mixed/Cold Storage tier • The Kraken project
15. Medallia © Copyright 2017. Confidential. 15 • RBD Wrapper • RadosGW is S3… NOT yet • Replacing SSDs • Trimming and flattening images every night -- and pool renames Automation is your best friend
16. Medallia © Copyright 2017. Confidential. 16 Getting ready for production Next Stop: Luminous • rbd mirror! • rbd lock • DC federation • Bluestore • Multi-MDS • RGW Shards • RGW Aborted uploads • RGW Policies • Ceph MGR
17. Medallia © Copyright 2017. Confidential. 17 Q&A
18. Medallia © Copyright 2017. Confidential. 18 Stuck OSD • sc2-r11-u31 - [540251.257113] NMI watchdog: BUG: soft lockup - CPU#13 stuck for 3s! [tp_fstore_op:11960] • [540251.257115] NMI watchdog: BUG: soft lockup - CPU#6 stuck for 3s! [ceph-osd:11266] • [540251.257117] Modules linked in: ipmi_watchdog stap_bb3e8ebe62703ce4b2c5acc6a1297a93_8511(OE) bridge stp llc xfrm_user xfrm_algo overlay xfs ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_conntrack ipt_REJECT nf_reject_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack binfmt_misc iptable_filter ip_tables x_tables intel_rapl sb_edac edac_core x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate intel_rapl_perf ipmi_ssif lpc_ich input_leds joydev pl2303 usbserial mei_me mei ioatdma shpchp ipmi_si acpi_power_meter acpi_pad mac_hid sunrpc ib_iser rdma_cm iw_cm ib_cm ib_core configfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipmi_devintf ipmi_msghandler autofs4 btrfs raid10 raid456 async_raid6_recov • [540251.257131] async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ses enclosure bcache hid_generic usbhid hid igb ast ttm mlx5_core drm_kms_helper syscopyarea dca devlink sysfillrect aesni_intel sysimgblt aes_x86_64 fb_sys_fops crypto_simd ptp ahci mpt3sas cryptd raid_class nvme drm libahci i2c_algo_bit glue_helper scsi_transport_sas nvme_core pps_core wmi fjes • [540251.257141] CPU: 6 PID: 11266 Comm: ceph-osd Tainted: G D OEL 4.10.0-21-generic #23~16.04.1-Ubuntu • [540251.257141] Hardware name: Racklive SYS-1018R-WC0R/X10SRW-F, BIOS 2.0 12/17/2015 A real nightmare!