Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Manage Large Data Sets with Streams

Manage Large Data Sets with Streams

Working with streams sounds scary and complicated but we’ll show you how to leverage streams to process large data imports without having to sell your house to buy RAM. Streams are a way to generalize data for easier processing in a linear way including the ability to seek around the stream. We’ll cover our own tips and tricks we’ve learned along the way to help you dive deep into processing streams.

Streams have been in PHP since back in the 4.x days however we continually see developers trying to iterate on huge data sets and often run out of memory. We’ll show you a better solution instead of “ ini_set(‘memory_limit’,’16GB’);”

Joe Ferguson

May 23, 2019

More Decks by Joe Ferguson

Other Decks in Programming


  1. Who Am I? Joe Ferguson Senior Full Stack Developer @

    Preteckt Twitter: @JoePFerguson OSMI Board Member The Joindin Foundation & Joindin Leadership Team
  2. Agenda Streams: What they are and why you shouldn’t cross

    them Searching a 5 million line CSV Guzzling Streams with Guzzle
  3. When to use Streams Reading files that may not fit

    in memory Downloading files from a remote system Fetching data from APIs