Premday #3 - Disaggregating the network with SONiC

Criteo presents how they disaggregated their network with SONiC

Transcript

1. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. Slaying Disaggregating the network Piece by piece 05 MAY 2026
2. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. MATHIEU LAUNAY 2 • Started as SE in 2012 • Moved to networking in 2014 • Lead since 2018
3. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. Ownership Open-source and standards Vendor support CONCLUSIONS 3
4. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. Thank you!
5. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. 5 A brief story of our designs 01 Designs 02 SONiC 03 Challenges A focus on the NOS A list of engineering issues
6. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. Network designs 6
7. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. The prehistory 7 •L2 Network •Mono vendor •All was installed by hand Pre-2010 era
8. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. DCv2 8 • TRILL-like network • Gigantic L2s • Still mono-vendor • Still a lot of contention • Rudimentary automation (Perl/Python/Shell scripts) • Nagios + Centreon 2010-2015
9. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. DCv3 9 • Multi-vendor (duopole) • L3 network : BGP • Still vertically integrated: HW, SW, licenses, support • Better contention 1:3 at ToR level • Automation of provisioning (Ansible) • Monitoring: SNMP LibreNMS to Prometheus, blackbox monitoring (netprobify open-source Criteo GH) 2015-2020
10. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. DCv4 10 •Break the vertical integration •Break the monolithic DC •Improve automation provisioning •Automate operations: listen to Kevin prez for more infos (17h) •Small monitoring updates 2020-2025
11. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. DCv5 11 •Densify: support dense CPU + Liquid Cooling setups •Kill the last SPOF: dual ToR •Start studies on Network for GPUs (packet spraying, packet trimming, QoS) 2025+ 👯
12. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. A Focus on SONiC 12
13. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. OCP Summit 13 2017
14. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. Phase 1 - evaluation 14 • Business model • Hardware market evaluation • Mellanox, Quanta, Edgecore (wedge 100, 40, Tofino, asxxxx), supermicro(celestica), Agema, Arista, Juniper, Dell, HP • Software market evaluation • ip infusion, ocNOS, ONL, picaOS, Openswitch, pluribus, Nokia SROS, VyOS, snaproute, Fboss, SONiC 2018Q1
15. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. Phase 2 - lab 15 • Understand how it works and make it work! • No doc: at that time, code was doc • Lab building • Testing 2018Q2
16. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. Phase 3 – test deployment 16 2018H2
17. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. Phase 4 17 2019+
18. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. Phase 4 18
19. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. Phase 4 19
20. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. SONiC: where are we now? (1/2) 20 • ~1000 ToR on all our data centres • ~600 SP/SS/HS on all our DCs • Four HW vendors, four ASICs, seven SKUs • Three SONiC versions (202205, 202211, 202305)
21. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. SONiC: where are we now? (2/2) 21 •Provisionning •Running •Monitoring and alerting
22. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. SONiC: transitioning your team(s) 22 •From CLI riders to DevOps •Broader scope: HW to NOS •Broader skillset: technical, vendor, ecosystem
23. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. Integration workload Same than any OS (monitoring, automation, training) Some SONiC specificities 23
24. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. People Teams in devops mode Mix profiles 24
25. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. Design and Automation SONiC might not fit any need Consider preparing your design and your eco-system first 25
26. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. Contracts & Support Anticipate reluctance and/or resistance from some vendors Negotiation is "off track" 26
27. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. Our challenges 27
28. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. DCv5 Dual ToR 28 • Kill latest SPOF • ~18 months to prod • Several issues • IPv6 boot • /31 support on old NICs • FRR bugs • rp_filter • BGP convergence time • default routes fight (BGP vs DHCP) Since 2024
29. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. Futures evolutions 29 • Network for GPU cluster • Don’t miss Anisse’s presentation tomorrow morning! • DCI on SONiC • Routed MNET on SONiC • Kill legacy tooling (Ansible) • ORv3 • (Direct) Liquid Cooling What’s next?
30. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. A word to vendors 30 -Community SONiC is the way! -Open-source -Redistributable SAI -Standard protocols
31. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. A word to vendors (cont.) 31 -We need vendor support on low-level drivers / firmwares! -Open-source them! -Functional IPv6 (PXE, UEFI) … please?
32. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. Thank you!
33. PROPRIETARY & CONFIDENTIAL. COPYRIGHT © CRITEO 2026. ALL RIGHTS RESERVED. Questions?