WinOps London 2017

A brief history of Ticketmaster's journey and some of the bumps in the road that affected our collaboration between Engineering and Operations . . . and what we're doing about it next.

International evolved as a franchise of a US company into a large group in it's own right, expanded by the merger in 2010 for Ticketmaster to become part of LiveNation Entertainment. Over the years as teams and products expanded and contracted we have been faced with different barriers - timezones, culture, compliance, politics & technology.

As we have started to migrate to AWS, our latest 'DevOps' changes started to highlight some gaps in our thinking we wanted to share what we are doing next to help us prepare better for the future.

Transcript

1. EVOLUTION OF TICKETMASTER’S JOURNEY TO DEVOPS WINOPS LDN SEPTEMBER 2017 #WinOps @connonm @tmTech
2. WHO AM I & WHY AM I HERE? Connon MacRae VP, Technical Operations (International) #WinOps @connonm @tmTech HISTORY IS USEFUL CONTEXT IS EVERYTHING DON’T BE AFRAID
3. MICROSOFT VS LINUX MICROSOFT LINUX NO MORE CYGWIN ON WINDOWS #WinOps @connonm @tmTech
4. #WinOps @connonm @tmTech WHO IS TICKETMASTER?
5. LIVE NATION ENTERTAINMENT – 42 MARKETS Live Nation & Ticketmaster Live Nation ONLY Ticketmaster ONLY #WinOps @connonm @tmTech
6. #WinOps @connonm @tmTech THE HOST 8 HOURS 5-9 HOURS 1975 – TM started with Albert Leffler, Peter Gadwaw, Gordon Gunn . . . . . . Also Bill Gates and Paul Allen develop a BASIC program for the Altair 8800, Microsoft becomes a registered trademark. BetaMax is introduced. Motorola obtains patent for mobile phone! 1976 – TM Starts . . . . IBM Launch the first laser printer. 1977 – ELO is first ticketed event at Johnson Gymnasium, University of New Mexico 1978 – Norway is the first international client with Oslo Nye Theatre 1981 – Ticketmaster UK Opens above the Albery Theatre 1983 – Ticketmaster UK starts to work with Arsenal 1988 – Ticketmaster Australia operations begin
7. THE THING YOU REALLY NEED TO KNOW #WinOps @connonm @tmTech I WANT THE BEST SEAT(S) ONSALE . . . 9AM . . .
8. #WinOps @connonm @tmTech GROWING FAST 8-9 HOURS 5-9 HOURS 5-6 HOURS 10 HOURS 1-4 HOURS 5-9 HOURS 1991 – Ticket Tron in North America acquired 1996 – UEFA Euros with Ticketmaster United Kingdom 1996 – First Ticketmaster internet sale! 1997 – Archtics (Sports & Arts) integrated with and later ‘DSA’ acquired. 1997 – Synchro Systems (later Ticketmaster Sport) 1997 – Ticketmaster Ireland as a joint venture 1998 – Mexcio starts operations 1998 – ticketmaster.co.uk goes online, Windows NT 4 1999 – Joint venture in Australia 1999 - Rugby World Cup in Wales 2000 – TicketWeb, Vista and Admission Canada 2001 – First deal with Clear Channel – later to become Live Nation 2001 – Norway Billetservice acquired 2002 – Manchester Commonwealth Games 2002 – Netherlands & Denmark acquired 2004 – Athens Olympics 2004 – Sweden and Finland acquired 2005 – New Zealand established and Kartenhaus in Germany acquired 2006 – Acquires TickTackTickets in Spain and Biletix in Turkey 2007 – TM China joint venture 2008 – Beijing Olympics 2008 – TM acquires TicketsNow & GetMeIn, 2008 - Paperless tickets debut in North America 2008 – Paperless debuts in UK with 18k people attending at the O2 Arena 2008 – Ticketmaster Spun off from IAC 2008 - FrontLine Entertainment management acquired 2009 – TM UK breaks records with Michael Jackson, 50 dates at the O2. 2010 – LN and TM Merge to become Live Nation Entertainment 2010 - Ticnet in France acquired 2011 – Expansion in Spain with ServiCaixa 2012 – Ticketing for London2012, over 10.9 million tickets sold and processed. Unprecedented at the time. 2013 - Launches in the middle east with UAE 2014 – Glasgow Commonwealth Games. 2014 - TM acquires Seatwave across Europe. 2014 - Middle east expands to Qatar 2014 – Relaunch DE, ES, acquires PL and Sherpa in Belgium. 2015 – Universe and Frontgate join the company 2015 – England Rugby World Cup 2016 – TM acquires Tickethour 2017 – TM acquires TicketPro in Czech Republic and launches Ticketmaster Israel 14 CORE TICKETING PRODUCTS
9. #WinOps @connonm @tmTech THE INTERNET - 1998 WWW NT4 SQL THE ‘HOSTS’ - VAXEN 1999 8%? 20% GULP
10. #WinOps @connonm @tmTech REBUILD v1 – 2000 WWW LINUX ORACLE THE ‘HOSTS’ - VAXEN MySQL
11. #WinOps @connonm @tmTech THEN THESE . . . . . SOX - 2002 PCI v1 - 2004
12. #WinOps @connonm @tmTech REMEMBER THESE?
13. #WinOps @connonm @tmTech WE BUILT THIS . . . DEV QA CAP STAGE 5-8 HOURS PRODUCTION . . . WE WERE PROUD OF IT!
14. LESS THAN . . . OPTIMAL? #WinOps @connonm @tmTech
15. Group A WINDOW OPS Service Delivery WINDOWS SYSTEMS DBA TEAM LINUX OPS LINUX SYSTEMS Group B Group C PLATFORM Product Support CLOUD SYSTEMS TOOLING NETWORKING DATACENTRE INTERNATIONAL TECHNOLOGY TEAMS IN 2016 #WinOps @connonm @tmTech SECURITY CORP IT ARCHITECTURE NA TEAMS Product Group 1 Product Group 2 Product Group 1 Product Group 2 Product Group 1 Product Group 2 Product Group 3 Product Group 3
16. THE MORE YOU TIGHTEN YOUR GRIP, THE MORE STAR SYSTEMS WILL SLIP THROUGH YOUR FINGERS #WinOps @connonm @tmTech
17. WHERE HAVE WE BEEN? Engineering Ops Syseng Database Engineering Ops Syseng Database Engineering Ops Syseng Database #WinOps @connonm @tmTech
18. WHERE HAVE WE BEEN? #WinOps @connonm @tmTech Engineering Ops Syseng Database Platform
19. PUSH ME, PULL YOU #WinOps @connonm @tmTech
20. PUSH ME, PULL YOU #WinOps @connonm @tmTech
21. Ops & Reliablity Service Management Ops & Reliablity Ops & Reliablity PLATFORM ENGINEERING Product Support Runtime Services & Data Tooling Infrastructure (including DC) Network Engineering Technical Service Delivery SCALING THE SILOS, LOWERING THE WALL PLATFORM INFRASTRUCTURE #WinOps @connonm @tmTech Group A Group B Group C Product Group 1 Product Group 2 Product Group 1 Product Group 2 Product Group 1 Product Group 2 Product Group 3 Product Group 3 Group B Product Group 1 Product Group 2 Ops & Reliablity
22. HOW CAN WE EVOLVE? #WinOps @connonm @tmTech
23. CAT = LOVE = PRODUCT! #WinOps @connonm @tmTech
24. TOO LONG, EXPENSIVE #WinOps @connonm @tmTech
25. DOES NOT FIT ANYMORE #WinOps @connonm @tmTech
26. PLAY (RE)NICE? KILL -9 #WinOps @connonm @tmTech UNACCEPTABLE!!!!!!
27. TICKETMASTER CLOUD #WinOps @connonm @tmTech
28. SOMEBODY ELSES BOX TRANSPARENT COSTS #WinOps @connonm @tmTech
29. CONTAINERS – PART I #WinOps @connonm @tmTech
30. CONTAINERS – PART II #WinOps @connonm @tmTech
31. 31 (MORE) MODERN TECH CHANCE TO CLEAN UP WHAT ARE WE USING? #WinOps @connonm @tmTech
32. TECH MATURITY Open Source @ https://github.com/Ticketmaster/techmaturity
33. 1. CONTAINERISE your application with Docker or AUTOMATE your Operating Systems with CHEF 2. IMPLEMENTATE BETTER INSTRUMENTATION - no SSH or RDP in production! Log aggregation and rotation, metrics 3. Use SYNTHETIC MONITORING to understand the health of your product Define SLAs, track & alert on availability and performance 4. AUTOMATE YOUR BUILD PIPELINE by reusing common tools Terraform & GitLab CI are the products of choice 5. GET SECURE Pass pen/static testing No passwords/keys in configs 6. DESIGN A SHARED NOTHING ARCHITECTURE No shared filesystems (e.g. NFS, CIFS) 7. BUILD FOR AVAILABLITY- NO SINGLE POINTS OF FAILURE Use auto-scaling groups and without hard-coded IPs OUR PRINCIPLES FOR MODERNISATION #WinOps @connonm @tmTech
34. SUMMARY #WinOps @connonm @tmTech HISTORY IS USEFUL CONTEXT IS EVERYTHING DON’T BE AFRAID
35. Q & A #WinOps @connonm @tmTech