Messy data != Messy code

Messy Data != Messy Code An API built in Symfony,
for one of the biggest retailers of Switzerland PHPUK 2020 @michellesanver

@michellesanver WIIIIE \o/ “Learn the most by sharing Your knowledge
with others” - @coderabbi

@michellesanver Michelle Sanver Colour & Code addict

@michellesanver Accent!?!?!?

@michellesanver

@michellesanver “We build a product that improves the way that
Swiss people do shopping” - Michelle Sanver

@michellesanver Disclaimer:   I do a lot of “ranting” in
this talk. Retailer data is a complex business. We have nothing against our data providers.

@michellesanver Talk for everyone Some concepts may be confusing

@michellesanver Agenda – The project: A retailer of Switzerland –
Challenges – Big API: Solving the serializer bottleneck – Importing: When your 3rd party data provider “lies” to you – Mapping: Contain the mess! – Evolving with Symfony in a long term project

@michellesanver The Project

@michellesanver The Project It started as a small API

@michellesanver Huge Technology Stack

@michellesanver API Platform I know you’ll ask, no we don’t
use it

DEV: Rae Knowler DEV: Tobias Schultze DEV: Christian Riesen DEV:
Thereza Scherrer DEV: Martin Janser DEV: Emanuele Panzeri DEV: Michelle Sanver “Cloud Tamer”: Chregu PO: Timur Erdag PO: Colin Frei SM: Léo Davesne DEV: David Buchmann Team: 8 developers, 2 PO’s 1 SM, and… Chregu @michellesanver

@michellesanver REST Controllers ElasticSearch MySQL The Data Provider Serializing Importing
Mapping Frontend

@michellesanver Challenges

@michellesanver Our API is HUGE Code & Data

@michellesanver /src 8.7 MB: 2’067 items

@michellesanver /tests 10.3 MB: 1’089 items

@michellesanver /conﬁg 1.8 MB: 329 items

@michellesanver /vendor 163.6 MB: 21’763 items

@michellesanver /src /Api /Client /Infrastructure /Migration / … A few
lose things that makes sense, like Serializer Structure & Naming

@michellesanver Importing a lot of data From several sources

@michellesanver REST Controllers ElasticSearch MySQL The Data Provider Serializing Importing
Mapping Frontend

@michellesanver REST Controllers ElasticSearch MySQL The Data ProviderS Serializing Importing
A LOT Mapping Frontend

@michellesanver Importing Data With import commands

@michellesanver Storing Original Data Importing into MySQL

@michellesanver Importing a lot of data

@michellesanver Queues & Workers With Autoscaling

@michellesanver Switching to Symfony Messenger

@michellesanver

@michellesanver Switching to messenger simpliﬁed our code A LOT

@michellesanver It forced us to use more value objects

@michellesanver bin/console messenger:consume From a crazy amount of commands, making
bin/console diﬃcult to overview without grep to… This:

@michellesanver

@michellesanver Switching to Messenger was well worth the time

@michellesanver And I love that we could give back to
the Symfony community with it

@michellesanver I can exit the worker with ctrl+c now

@michellesanver Consuming “bad” API’s Without crying or becoming an alcoholic

@michellesanver Soapish

@michellesanver Restful Soap

@michellesanver The “Flexible” Api

@michellesanver Humor Write songs, laugh

@michellesanver Thank you By the way, we’re hiring ;) Michelle
Sanver [email protected]

@michellesanver Pairing is caring When we suffer, we can suffer
together

@michellesanver We’re super heroes Our consumers are shielded from the
“pain” we have

@michellesanver Makes me feel good about our API

@michellesanver 3rd party data providers Lies. All lies!!

@michellesanver Pairing is caring When we suffer, we can suffer
together

@michellesanver JSON Schema We can scream at them now ;)

@michellesanver JSON Schema

JSON Schema We can scream at them now ;)

@michellesanver

JSON Schema We can scream at them now ;)

@michellesanver Defensive Programming There’s no such thing as “This won’t
happen”

@michellesanver Extremely Defensive PHP - Marco Pivetta

@michellesanver Switching to several data sources

@michellesanver “Decider Service” Ooh, that ID? You’re from API X

@michellesanver Importing is “easy” But mapping…

@michellesanver Data Quality…

@michellesanver • Missing spaces • String instead of int •
Array instead of object • Object instead of string • Differently named ﬁelds • Required data missing • … And more

@michellesanver Mappers From MySQL to ElasticSearch

@michellesanver ProductMapper Product on ElasticSearch /products.json

@michellesanver ProductMapper Name Category Brand Price Description

@michellesanver ProductMapper N B C P D

@michellesanver ProductMapper

@michellesanver

@michellesanver MapperInterface

@michellesanver ProductFactory Data From MySQL Clean Data To store in
ES

@michellesanver Conﬁg of all the mappers, in order. 7 depends
on 3 4 depends on 1 25 depends on basically everything 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 ProductFactory

@michellesanver ! " # $

@michellesanver Data From MySQL Clean Data To store in ES
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 ProductFactory * 3

@michellesanver Mapper Dependencies With a compiler pass!

@michellesanver

@michellesanver Dealing with languages Only when you have to!

@michellesanver

@michellesanver Ensuring Quality Tests, Logging, Monitoring

@michellesanver Log a lot “Debugging you” will love you

@michellesanver Monitoring Queues, uptime, etc.

@michellesanver Acceptance Tests Test for critical data, often

@michellesanver Project Challenges Big API responses

@michellesanver

@michellesanver Serializing, Versioning & Groups

@michellesanver Our needs Versioning & Groups

@michellesanver We tried it all And it “sucks".

@michellesanver Plain JSON decode json_decode, json_encode

@michellesanver Symfony Serializer It’s cool and all

@michellesanver Better Serializer Maybe it’d be better, if we could
make it work

@michellesanver JMS What 99% of PHP developers need for serialization

@michellesanver JMS Annotations

@michellesanver JMS Version Support @until, @since

@michellesanver JMS Virtual Properties

@michellesanver JMS Works like magic with most frameworks, including Symfony
which we use

@michellesanver JMS

@michellesanver JMS Read The Docs ;) https://jmsyst.com/libs/serializer

@michellesanver We called “visitProperty” over 60 000 times!!!

@michellesanver The Liip/Serializer It’s more of a generator, really.

@michellesanver Model Parser SerializerGenerator DeserializerGenerator

@michellesanver An overall performance gain of 55% over JMS for
our use-case 390 ms => 175 ms  CPU and I/O wait both down by ~50%. Memory gain: 21%, 6.5 MB => 5.15 MB

@michellesanver Curious about our Serializer? https://www.liip.ch/en/blog/fast-serialization-with-liip-serializer  https://github.com/liip/serializer

@michellesanver Project Challenges Communication is hard

@michellesanver Feedback Because we have to work together

@michellesanver Retrospective Let’s look back and improve

@michellesanver Code Reviews Respectfully improving code together

@michellesanver Team Events Keeps moral high

@michellesanver An amazing customer They listen to us, it’s important

@michellesanver Evolving With Symfony  In a long term project

@michellesanver Prioritise upgrades Upgrade ASAP!

@michellesanver Prioritise upgrades Upgrade minor versions

@michellesanver Prioritise upgrades Fix deprecation warnings

@michellesanver Refactor Often It’s not optional

@michellesanver Use new components

@michellesanver Contribute to Open Source Feel good to give something
back And… Have some control over our tools

@michellesanver Tests!! Lots of tests (Do I still need to
emphasise this?)

@michellesanver An amazing customer

@michellesanver Final Words

@michellesanver Write dev docs Don’t repeat our mistake

@michellesanver Defensive programming Refactoring often Working as a Team

@michellesanver Messy Data !== Messy Code

@michellesanver Thank you By the way, we’re hiring ;) Michelle
Sanver [email protected]

Messy data != Messy code

Messy data != Messy code

More Decks by Michelle Sanver

Other Decks in Programming

Featured

Transcript