Slide 1

Slide 1 text

Manage Large Data Sets with Streams Joe Ferguson

Slide 2

Slide 2 text

Who Am I? Joe Ferguson Senior Full Stack Developer @ Preteckt Twitter: @JoePFerguson OSMI Board Member The Joindin Foundation & Joindin Leadership Team

Slide 3

Slide 3 text

Agenda Streams: What they are and why you shouldn’t cross them Searching a 5 million line CSV Guzzling Streams with Guzzle

Slide 4

Slide 4 text

https://www.php.net/manual/en/intro.stream.php a stream is a resource object which exhibits streamable behavior

Slide 5

Slide 5 text

Linear Data 0 17.5 35 52.5 70 Record 1 Record 2 Record 3 Record 4 Age

Slide 6

Slide 6 text

scheme://target

Slide 7

Slide 7 text

You’re already using them file() open() fwrite() fclose() file_get_contents() file_put_contents()

Slide 8

Slide 8 text

Stream Transports

Slide 9

Slide 9 text

Stream Wrappers

Slide 10

Slide 10 text

Stream Filters

Slide 11

Slide 11 text

Stream Context

Slide 12

Slide 12 text

Size Matters

Slide 13

Slide 13 text

Size Matters

Slide 14

Slide 14 text

“I know how to fix it”

Slide 15

Slide 15 text

“Job’s Done Boss!”

Slide 16

Slide 16 text

Wait a minute… 2G 2G 2G 2G 2G 2G

Slide 17

Slide 17 text

What’s Burning?

Slide 18

Slide 18 text

Save the memory…

Slide 19

Slide 19 text

Now I have this ISO

Slide 20

Slide 20 text

Streaming People (Don’t worry, they’re not real)

Slide 21

Slide 21 text

file_get_contents()

Slide 22

Slide 22 text

Reading from pointers

Slide 23

Slide 23 text

What’s a pointer?

Slide 24

Slide 24 text

Reading from pointers

Slide 25

Slide 25 text

Memory Usage

Slide 26

Slide 26 text

Memory Usage

Slide 27

Slide 27 text

Double Down

Slide 28

Slide 28 text

Memory Usage

Slide 29

Slide 29 text

Rewinding Streams

Slide 30

Slide 30 text

Rewinding Streams

Slide 31

Slide 31 text

Rewinding Streams

Slide 32

Slide 32 text

Rewinding Streams

Slide 33

Slide 33 text

Rewinding Streams

Slide 34

Slide 34 text

Rewinding Streams

Slide 35

Slide 35 text

Seeking Around Streams

Slide 36

Slide 36 text

Seeking Around Streams

Slide 37

Slide 37 text

Guzzling Streams

Slide 38

Slide 38 text

Guzzling Streams

Slide 39

Slide 39 text

Guzzling Streams

Slide 40

Slide 40 text

Testing Our Response

Slide 41

Slide 41 text

Guzzling Streams

Slide 42

Slide 42 text

Read 100 bytes

Slide 43

Slide 43 text

Read 100 bytes

Slide 44

Slide 44 text

Content Type

Slide 45

Slide 45 text

Content Type

Slide 46

Slide 46 text

Content Type

Slide 47

Slide 47 text

Drilling Down

Slide 48

Slide 48 text

Drilling Down

Slide 49

Slide 49 text

Inspecting a Post

Slide 50

Slide 50 text

Interesting!

Slide 51

Slide 51 text

Interesting!

Slide 52

Slide 52 text

When to use Streams Reading files that may not fit in memory Downloading files from a remote system Fetching data from APIs

Slide 53

Slide 53 text

Resources

Slide 54

Slide 54 text

Joe Ferguson Twitter: @JoePFerguson Email: [email protected] Contact Info: