About me Name: Alexander Schranz Workplace: Sulu (sulu.io) Tools: PHP, Symfony, Twig, Elasticsearch, Redis ReactJS, MobX, React Native (Expo) Experience: Web Developer since 2012 Certified Symfony 5 Expert X / Twitter: @alex_s_ Github: @alexander-schranz Mastodon: [email protected] 

Efficient JSON API 

The Problem 4K images JSON API with more than 100 items 

Example Object for the API: class Article { #[ORM\Id] #[ORM\GeneratedValue] #[ORM\Column(type: Types::INTEGER)] public readonly int $id; #[ORM\Column(type: Types::STRING)] public string $title = ‘’; #[ORM\Column(type: Types::TEXT)] public string $description = ‘’; #[ORM\Column(type: Types::TEXT)] #[Serializer\Ignore()] public string $note = ‘’; } 

Our JSON (JSON-HAL) { "_embedded": { "articles": [ { "id": 1, "title": "Article 1", "description": "Description 1\nMore description text ..." }, ... ] } } 

Typical Controller 

Finding Bottlenecks What want we reduce: ❏ I/O usage ❏ CPU usage ❏ Memory usage 

The Basic Bottlenecks - ORM Hydration / Serializer Object Hydration Serialization 

Understanding Hydration The Hydration process of the ORM will convert the SQL result (associative array) to an Object (Article) Understanding Serialization The Serialization process of the Serializer will convert (normalize) the Object (Article) into an associative array and encode it then to JSON https://symfony.com/doc/current/components/serializer.html 

Reduce CPU and Memory usage by avoid Hydration and Normalization process 

Reduce I/O usage by avoid overfetching and select only required columns 

Is that all? Here most devs stops with the optimization and to avoid that many articles need to be kept in memory limit there APIs to never return more as 100 items. How we can go further with the optimization? 

Static Image / File Streaming 

Current State our API 

Generators A generator allows you to write code that uses foreach to iterate over a set of data without needing to build an array in memory, which may cause you to exceed a memory limit, or require a considerable amount of processing time to generate. Instead, you can write a generator function, which is the same as a normal function, except that instead of returning once, a generator can yield as many times as it needs to in order to provide the values to be iterated over. https://www.php.net/manual/en/language.generators.overview.php 

Generators in PHP 1 10 1_ ---- 2 20 2_ ---- 3 30 3_ ---- 

Generators in JavaScript 1 10 1_ ---- 2 20 2_ ---- 3 30 3_ ---- 

Use Generators to read from Database 

Use Generators with Doctrine 

Split JSON in Static and Dynamic Part { "_embedded": { "articles": [ {"id": 1, "title": "Article 1", …}, {"id": 2, "title": "Article 2", …}, {"id": 3, "title": "Article 3", …}, … ] } } 

Replace JsonResponse with StreamedResponse Get Static Parts 

Output Static Before Dynamic Part Output Static After 

Whole Code 

Make it reusable 

Efficient JSON API 

Contribute to Symfony 124 Comments from 11 different people over 3 month we improved it to even a better version where the Generator can directly be set in the Structure. https://github.com/symfony/symfony/pull/47709 

Adopted by Laravel Part of Laravel since v10.43: https://github.com/laravel/framework/pull/49873 

Efficient JSON API 

Stats Old: New: 

Buuuuut What about the client? 

Alternatives: Line-delimited JSON {"id": 1, "title": "Article 1", …} {"id": 2, "title": "Article 2", …} {"id": 3, "title": "Article 3", …} Use HTTP Headers to provide more data like total or other numbers. Line-delimited JSON Streams can be even ready efficiently by clients while structured nested JSON objects are harder to read object by object. https://en.wikipedia.org/wiki/JSON_streaming 

Time for your questions https://github.com/alexander-schranz/efficient-json-streaming-with-symfony-doctrine X / Twitter: @alex_s_ Github: @alexander-schranz Mastodon: [email protected]