Messy data != Messy code

Messy data != Messy code

The role of our API in Switzerland is to consume a lot of data that was not meant for a digital age and to transform it into beautiful output, for one of the biggest retailers in Switzerland. This is a journey of consuming a lot of data and APIs from different sources and in different formats. Some of them made us laugh, some of us got migraines. We built a smooth architecture to consume and output data. I am proud of our architecture that we seamlessly upgraded to keep the latest versions, now Symfony 4 along the way. I want to share with you how we managed to keep this API up to date for over 5 years and the architecture that we use to make it happen.

01da6d807a29ad6d49801c0157518148?s=128

Michelle Sanver

February 20, 2020
Tweet

Transcript

  1. Messy Data != Messy Code An API built in Symfony,

    for one of the biggest retailers of Switzerland PHPUK 2020 @michellesanver
  2. @michellesanver WIIIIE \o/ “Learn the most by sharing Your knowledge

    with others” - @coderabbi
  3. None
  4. @michellesanver Michelle Sanver Colour & Code addict

  5. @michellesanver Accent!?!?!?

  6. @michellesanver

  7. @michellesanver “We build a product that improves the way that

    Swiss people do shopping” - Michelle Sanver
  8. @michellesanver Disclaimer: 
 I do a lot of “ranting” in

    this talk. Retailer data is a complex business. We have nothing against our data providers.
  9. @michellesanver Talk for everyone Some concepts may be confusing

  10. @michellesanver Agenda – The project: A retailer of Switzerland –

    Challenges – Big API: Solving the serializer bottleneck – Importing: When your 3rd party data provider “lies” to you – Mapping: Contain the mess! – Evolving with Symfony in a long term project
  11. @michellesanver The Project

  12. @michellesanver The Project It started as a small API

  13. @michellesanver Huge Technology Stack

  14. @michellesanver API Platform I know you’ll ask, no we don’t

    use it
  15. DEV: Rae Knowler DEV: Tobias Schultze DEV: Christian Riesen DEV:

    Thereza Scherrer DEV: Martin Janser DEV: Emanuele Panzeri DEV: Michelle Sanver “Cloud Tamer”: Chregu PO: Timur Erdag PO: Colin Frei SM: Léo Davesne DEV: David Buchmann Team: 8 developers, 2 PO’s 1 SM, and… Chregu @michellesanver
  16. @michellesanver REST Controllers ElasticSearch MySQL The Data Provider Serializing Importing

    Mapping Frontend
  17. @michellesanver REST Controllers ElasticSearch MySQL The Data Provider Serializing Importing

    Mapping Frontend
  18. @michellesanver Challenges

  19. @michellesanver Our API is HUGE Code & Data

  20. @michellesanver /src 8.7 MB: 2’067 items

  21. @michellesanver /tests 10.3 MB: 1’089 items

  22. @michellesanver /config 1.8 MB: 329 items

  23. @michellesanver /vendor 163.6 MB: 21’763 items

  24. @michellesanver /src /Api /Client /Infrastructure /Migration / … A few

    lose things that makes sense, like Serializer Structure & Naming
  25. @michellesanver Importing a lot of data From several sources

  26. @michellesanver REST Controllers ElasticSearch MySQL The Data Provider Serializing Importing

    Mapping Frontend
  27. @michellesanver REST Controllers ElasticSearch MySQL The Data Provider Serializing Importing

    Mapping Frontend
  28. @michellesanver REST Controllers ElasticSearch MySQL The Data ProviderS Serializing Importing

    A LOT Mapping Frontend
  29. @michellesanver Importing Data With import commands

  30. @michellesanver Storing Original Data Importing into MySQL

  31. @michellesanver Importing a lot of data

  32. @michellesanver Queues & Workers With Autoscaling

  33. @michellesanver Switching to Symfony Messenger

  34. @michellesanver

  35. @michellesanver

  36. @michellesanver Switching to messenger simplified our code A LOT

  37. @michellesanver It forced us to use more value objects

  38. @michellesanver bin/console messenger:consume From a crazy amount of commands, making

    bin/console difficult to overview without grep to… This:
  39. @michellesanver

  40. @michellesanver Switching to Messenger was well worth the time

  41. @michellesanver And I love that we could give back to

    the Symfony community with it
  42. @michellesanver I can exit the worker with ctrl+c now

  43. @michellesanver Consuming “bad” API’s Without crying or becoming an alcoholic

  44. @michellesanver Soapish

  45. @michellesanver Restful Soap

  46. @michellesanver The “Flexible” Api

  47. @michellesanver Humor Write songs, laugh

  48. @michellesanver Thank you By the way, we’re hiring ;) Michelle

    Sanver michelle@liip.ch
  49. @michellesanver Pairing is caring When we suffer, we can suffer

    together
  50. @michellesanver We’re super heroes Our consumers are shielded from the

    “pain” we have
  51. @michellesanver Makes me feel good about our API

  52. @michellesanver 3rd party data providers Lies. All lies!!

  53. @michellesanver Pairing is caring When we suffer, we can suffer

    together
  54. @michellesanver JSON Schema We can scream at them now ;)

  55. @michellesanver JSON Schema

  56. JSON Schema We can scream at them now ;)

  57. @michellesanver

  58. JSON Schema We can scream at them now ;)

  59. @michellesanver Defensive Programming There’s no such thing as “This won’t

    happen”
  60. @michellesanver Extremely Defensive PHP - Marco Pivetta

  61. @michellesanver Switching to several data sources

  62. @michellesanver “Decider Service” Ooh, that ID? You’re from API X

  63. @michellesanver Importing is “easy” But mapping…

  64. @michellesanver Data Quality…

  65. @michellesanver • Missing spaces • String instead of int •

    Array instead of object • Object instead of string • Differently named fields • Required data missing • … And more
  66. @michellesanver • Missing spaces • String instead of int •

    Array instead of object • Object instead of string • Differently named fields • Required data missing • … And more
  67. @michellesanver Mappers From MySQL to ElasticSearch

  68. @michellesanver ProductMapper Product on ElasticSearch /products.json

  69. @michellesanver ProductMapper Name Category Brand Price Description

  70. @michellesanver ProductMapper N B C P D

  71. @michellesanver ProductMapper

  72. @michellesanver ProductMapper

  73. @michellesanver ProductMapper

  74. @michellesanver

  75. @michellesanver MapperInterface

  76. @michellesanver ProductFactory Data From MySQL Clean Data To store in

    ES
  77. @michellesanver Config of all the mappers, in order. 7 depends

    on 3 4 depends on 1 25 depends on basically everything 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 ProductFactory
  78. @michellesanver ! " # $

  79. @michellesanver Data From MySQL Clean Data To store in ES

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 ProductFactory * 3
  80. @michellesanver Mapper Dependencies With a compiler pass!

  81. @michellesanver

  82. @michellesanver

  83. @michellesanver

  84. @michellesanver

  85. @michellesanver Dealing with languages Only when you have to!

  86. @michellesanver

  87. @michellesanver

  88. @michellesanver Ensuring Quality Tests, Logging, Monitoring

  89. @michellesanver Log a lot “Debugging you” will love you

  90. @michellesanver Monitoring Queues, uptime, etc.

  91. @michellesanver Acceptance Tests Test for critical data, often

  92. @michellesanver Project Challenges Big API responses

  93. @michellesanver

  94. @michellesanver Serializing, Versioning & Groups

  95. @michellesanver Our needs Versioning & Groups

  96. @michellesanver We tried it all And it “sucks".

  97. @michellesanver Plain JSON decode json_decode, json_encode

  98. @michellesanver Symfony Serializer It’s cool and all

  99. @michellesanver Better Serializer Maybe it’d be better, if we could

    make it work
  100. @michellesanver JMS What 99% of PHP developers need for serialization

  101. @michellesanver JMS Annotations

  102. @michellesanver JMS Version Support @until, @since

  103. @michellesanver JMS Virtual Properties

  104. @michellesanver JMS Works like magic with most frameworks, including Symfony

    which we use
  105. @michellesanver JMS

  106. @michellesanver JMS Read The Docs ;) https://jmsyst.com/libs/serializer

  107. @michellesanver We called “visitProperty” over 60 000 times!!!

  108. @michellesanver The Liip/Serializer It’s more of a generator, really.

  109. @michellesanver Model Parser SerializerGenerator DeserializerGenerator

  110. @michellesanver An overall performance gain of 55% over JMS for

    our use-case 390 ms => 175 ms
 CPU and I/O wait both down by ~50%. Memory gain: 21%, 6.5 MB => 5.15 MB
  111. @michellesanver Curious about our Serializer? https://www.liip.ch/en/blog/fast-serialization-with-liip-serializer
 https://github.com/liip/serializer

  112. @michellesanver Project Challenges Communication is hard

  113. @michellesanver Feedback Because we have to work together

  114. @michellesanver Retrospective Let’s look back and improve

  115. @michellesanver Code Reviews Respectfully improving code together

  116. @michellesanver Team Events Keeps moral high

  117. @michellesanver An amazing customer They listen to us, it’s important

  118. @michellesanver Evolving With Symfony
 In a long term project

  119. @michellesanver Prioritise upgrades Upgrade ASAP!

  120. @michellesanver Prioritise upgrades Upgrade minor versions

  121. @michellesanver Prioritise upgrades Fix deprecation warnings

  122. @michellesanver Refactor Often It’s not optional

  123. @michellesanver Use new components

  124. @michellesanver Contribute to Open Source Feel good to give something

    back And… Have some control over our tools
  125. @michellesanver Tests!! Lots of tests (Do I still need to

    emphasise this?)
  126. @michellesanver An amazing customer

  127. @michellesanver Final Words

  128. @michellesanver Write dev docs Don’t repeat our mistake

  129. @michellesanver Defensive programming Refactoring often Working as a Team

  130. @michellesanver Messy Data !== Messy Code

  131. @michellesanver Thank you By the way, we’re hiring ;) Michelle

    Sanver michelle@liip.ch