String Theory

9/6/16, 10:57 AM String Theory Page 1 of 93 http://localhost:9090/print

String Theory Artwork & photo by Mahmoud Al-Qammari

James Edward Gray II graysoftinc.com @jeg2

Nathan Long NathanMLong.com @sleeplessgeek

Q&A Post on elixirforum.com Ping us

Overview of Stringy Types Charlist Atom String "iolist"

Charlist (Single-quoted Strings) List of integers that represent Unicode codepoints 'abc' == [97, 98, 99] > i 'abc' Term: 'abc' Data type: List Raw representation: [97, 98, 99]

Atom Fast pattern matching {:ok, contents} = File.read( "some_file.txt" ) Module names File == :"Elixir.File"

Atom Never garbage collected ✗String.to_atom(param) ✓String.to_existing_atom(param)

Stringy Things Bitstrings Binaries Strings

Bitstrings A sequence of bits, contiguous in memory <<1::size(1), 0::size(1)>>

Binaries A bitstring containing a sequence of bytes <<0, 255>> == <<0::size(8), 255::size(8)>>

Strings (Double-quoted Stings) A binary whose bytes can be interpreted as characters ✓ <<97, 98, 99>> => "abc" ✓ <<195, 161, 114, 98, 111, 108>> => "árbol" ✗ <<255, 0>> => <<255, 0>>

All Together Now bitstring if series of bits via <<…>> binary if byte count is an integer string if bytes are UTF-8 encoded

"iolists"

String Concatenation iex(1)> {james, n, nathan} = {"James", " and ", "Nathan"} {"James", " and ", "Nathan"} iex(2)> IO.puts james <> n <> nathan James and Nathan :ok Requires allocating an extra string Involves memory copying into that string

String Interpolation iex(3)> IO.puts "#{james}#{n}#{nathan}" James and Nathan :ok This is just good looking concatenation Under the hood it's similar

Can we just use lists? iex(4)> IO.puts [james, n, nathan] James and Nathan :ok Allocates a few small lists They point at the binaries No memory copying is needed

Improper Nesting iex(5)> IO.puts [[james | n] | nathan] James and Nathan :ok Now we're down to just two lists This format is also faster to build by appending data

Quick List Appending Slow: ["a"] ++ ["b"] ++ ["c"] ++ ["d"] Fast: output = "a" output = [output | "b"] output = [output | "c"] output = [output | "d"] # [[["a" | "b"] | "c"] | "d"]

Naming is Hard

Expert Help

Elixir's Deﬁnitions bitstring: Anything expressed as <<…>> binary: A bitstring with a length divisible by eight string: A binary that contains UTF-8 codepoints charlist: A proper list of integers representing codepoints iolist: A proper or improper list of numeric bytes, binaries, and/or nested iolists iodata: An iolist or a binary chardata: A proper or improper list of UTF-8 codepoints, strings, and/or nested chardata lists or a string

Elixir and Unicode

ASCII: A Mapping

Converting to Base 2 base_2 = fn (i) -> Integer.to_string(i, 2) end

ASCII Fits in 1 Byte ?a == 97 base_2.(?a) == "1100001" 01100001

Unicode á ç ü ñ ż

Unicode λ φ θ Ω

Unicode 韴

Unicode 韴=!+"

Unicode #$%&'

Unicode

Unicode https://www.tofugu.com/japan/japanese-internet/

Unicode

Unicode: Also A Mapping A = 65 λ = 923 (= 128,372

~1 million codepoints! ?韴 == 132_878 base_2.(?韴) == "100000011100001110" Some codepoints need more bytes than others... )

UTF-8 Templates 0xxxxxxx 110xxxxx 10xxxxxx 1110xxxx 10xxxxxx 10xxxxxx 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

The Bits For ⏰ ?⏰ == 9200 base_2.(?⏰) == "10001111110000"

Encoding ⏰ Into UTF-8 10001111110000 1110xxxx 10xxxxxx 10xxxxxx 11100010 10001111 10110000 =

Is That What Elixir Does? i "⏰" .... Raw representation <<226, 143, 176>> [226, 143, 176] |> Enum.map(base_2) == ["11100010", "10001111", "10110000"]

Three Kinds of Bytes Starts With Kind 0 Solo 10 Continuation 110 or 1110 or 11110 First of N (count the 1s)

Three Kinds of Bytes Character UTF-8 bytes a 01100001 + 11110000 10011111 10001101 10100000

String Reversal Solo First of 3 Continuation Continuation

String Reversal: Wrong Solo First of 3 Continuation Continuation ꔅ Continuation Continuation First of 3 Solo

String Reversal: Right Solo First of 3 Continuation Continuation ꔅ First of 3 Continuation Continuation Solo

Combining Diacritical Marks Multiple codepoints, one "grapheme" ë = e + ̈

Combining Diacritical Marks noel = "noe\u0308l" "noël" String.codepoints(noel) ["n", "o", "e", "̈", "l"] String.graphemes(noel) ["n", "o", "ë", "l"]

Combining Diacritical Marks

Tricky Traversal - O(N) some_string |> String.length some_string |> String.slice(2,3)

Tricky Length # Length in bytes or graphemes? byte_size("noël") == 6 String.length("noël") == 4

Tricky Reversal Elixir: String.reverse("noe\u0308l") => "lëon" Ruby: "noe\u0308l".reverse => "l̈eon"

Tricky Equality {combined, single} = {"e\u0308", "\u00EB"} => {"ë", "ë"} combined == single => false String.equivalent?(combined, single) => true

Tricky Casing Elixir String.upcase("mañana") == "MAÑANA" Ruby < 2.4.0 "mañana".upcase => "MAñANA"

Even Elixir Is Not Perfect String.downcase("ΛΣ") # should be "λς" => "λσ"

The Rules of the BEAM Processes are isolated Data is immutable Message sends are memory copies

The Large Binary Space Binaries under 64 bytes are stored in process This is the same as other data types "Large" binaries (called Refc Binaries) are stored outside processes The actual binary is in the "Large Binary Space" A small reference to the binary (called a ProcBin) is used in process

So What??? This transparent … and AMAZING … until it isn't … and it tries to kill your code

The Major Win Passing large binaries between processes is cheap Small references are copied over This potentially very helpful since we store encoded data in binaries HTTP responses Rendered sections of Phoenix templates

The Tradeoff Refc Binaries are "reference counted" When a reference is made in a new process, the count goes up When a reference is garbage collected, the count goes down When the count hits zero, the binary is removed But what if a process doesn't hit garbage collection for a while?

Mysteries This problem sometimes surfaces as mysterious Memory leaks Crashes It has affected Heroku Splunk Avdi Grimm It can be a hard scenario to debug

Detecting the Leak This may help you see the memory used: :erlang.system_info(:allocated_areas) |> Enum.find(fn category -> elem(category, 0) == :binary end)

Possible Solutions Manually force the GC of processes periodically Make processes short running (There's no GC like exit!)

Look Ma, No Concatenation! IO.puts "Hi " <> "James" # => "Hi James" IO.puts ["Hi ", "James"] # => "Hi James" IO.puts ["Hi, ", 9731] # => "Hi, ‚" IO.puts ["H",["e",["l",["l",["o"]]]]] # => "Hello"

String Reuse users = [%{name: "Joe"}, %{name: "Amy"}] [start_li, end_li] = ["<li>", "</li>"] response = Enum.map(users, fn (user) -> [start_li, user.name, end_li] end) IO.puts response

Beneﬁts of String Reuse Skip the work of concatenation Less string allocation = less RAM used Less string allocation = less GC work

"iolists" are for I/O Your process talking with the outside world: Printing to standard output Writing to a ﬁle Sending data over the network

System Calls

Different System Calls The write system call writes out a binary to ﬁles or sockets But writev writes a vector (list) of binaries Addresses can be reused Concatenation isn't needed

Different System Calls {:ok, file} = :file.open("/tmp/tmp.txt", [:write, :raw]) foo = "foo" bar = "bar" output = [foo, bar, foo] output = Enum.join(output) :file.write(file, output) System Call: write:return Write data (9 bytes): \ 0x00000000146007e2 foobarfoo

Different System Calls {:ok, file} = :file.open("/tmp/tmp.txt", [:write, :raw]) foo = "foo" bar = "bar" output = [foo, bar, foo] # output = Enum.join(output) :file.write(file, output) System Call: writev:return Writev data 1/3: (3 bytes): \ 0x0000000014600430 foo writev:return Writev data 2/3: (3 bytes): \ 0x0000000014600120 bar writev:return Writev data 3/3: (3 bytes): \ 0x0000000014600430 foo

What IO has Repetitive Strings? Web responses! Snippets repeated in page: <li class="product"> Sections repeated across requests: <footer>...</footer>

Simple Caching

Caching via function compilation foo.html.eex is found Phoenix uses EEx to compile it Phoenix speciﬁes that rendered chunks should be added to an iolist Compiles to MyView.render("foo.html", assigns) That function builds iolists

Resulting iolist

Simple Caching All static strings from template are reused - "cached" No dynamic parts of template are cached Cache is invalidated only if you change the template

Phoenix Using writev

Moral: Use iolists When Doing I/O (With caveats) When passing iolists across processes (eg to Cowboy), small (non-refc) strings will get combined You have to use raw mode when writing ﬁles to get writev I'll elaborate in a blog post on https://www.bignerdranch.com/blog/ soon But: it won't hurt to use iolists, and may help There's no point in joining strings yourself

James graysoftinc.com Nathan NathanMLong.com THANKS FOR LISTENING!!! ,✌-

String Theory

String Theory

Other Decks in Programming

Featured

Transcript