Of representation and interpretation A unified theory @arnoutboks Arnout Boks #odeaandecode 25-04-2019

@arnoutboks #odeaandecode Hard problems in programming • Cache invalidation • Naming things • Off-by-one errors

@arnoutboks #odeaandecode Hard problems in programming • String escaping • Timezones • Character encoding

@arnoutboks #odeaandecode Many difficulties with these topics are related

@arnoutboks #odeaandecode This talk is… • My personal view • Not an absolute truth • Meant to make you think

Data And its meaning

@arnoutboks #odeaandecode A number

@arnoutboks #odeaandecode A byte

@arnoutboks #odeaandecode Some JSON data { "income":100000 }

@arnoutboks #odeaandecode A word

@arnoutboks #odeaandecode A prime number 4931083597028501900275777672390764957284907772150208632080750184097926278850976588645578020 1366007328679544734112831735367831201557535981978545054811571939345877330038009932619505876 4525023820408110189885042615176579941704250889037029119015870030479432826073821469541570330 2279875576818956016240300641115169008728798381942582716745647748166843479284645809291315318 6007001004335318936319343912948604450370991980047709462921558180711169153031876288477878354 1575932891093295447350881882465495060005019006274705305381164278294267474853496525745368151 1706550281905552656221353146310421008662867971144467063669219825861581112515556504813420768 6732340765505485910826956266693066236799702104812396562518006818323653959348395675357557532 4619023481064700987753027956186892925380693305204238149969945456945774138335689906005870832 1812704861133682026515905166351874029X18197693937677852928722109550412925792573818660584501 5055250274994771883129310457698090915304613359419030258813205932277444385255046677902451869 7062627788891979580423065750615669834695617797879659201644051939960716981112615195610276283 2339825791423321726961443744381056485529348876349210309887028787453233132532122678633283702 7925099749969488775936915917644588032718384740235933020374888506755706587919461134193230781 4854436454375113207098606390746417564121635042388002967808558670370387509410769821183765499 2052043682558546422885024299633226853691246485500075591664024729240716450725319674499952944 8434741902107729606820558130923626837987951966199798285525887161096136561780745661592488660 8898164568541721362920846656279131478466791550965154310113538586208196875836883595577893914 5453935681996098808540476590735897289898342504712891841626587896821853808795627903997862944 9397605467534821256750121517082737107646270712467532102483678159400087505452543537

@arnoutboks #odeaandecode Data has a different meaning under different interpretations

Functions A bit of theory

@arnoutboks #odeaandecode Functions x y y = f(x) f

@arnoutboks #odeaandecode Functions D R x y f: D → R y = f(x)

@arnoutboks #odeaandecode Functions

@arnoutboks #odeaandecode Functions float int 3.84 4 round: float → int 4 = round(3.84)

@arnoutboks #odeaandecode Functions D R x1 y f: D → R x2

@arnoutboks #odeaandecode Functions float int 3.84 4 round: float → int 4.35

@arnoutboks #odeaandecode Functions D R x y1 f: D → R y2

@arnoutboks #odeaandecode Pure functions D R x y1 f: D → R y2

@arnoutboks #odeaandecode Impure functions • Randomness • State (global or local) • IO • Filesystem • Network • System clock

@arnoutboks #odeaandecode Functions D R x ? f: D → R

@arnoutboks #odeaandecode Functions (in mathematics) D R x f: D → R

@arnoutboks #odeaandecode Functions (in programming) D R x Exception! f: D → R

Representation & Interpretation

@arnoutboks #odeaandecode Level of abstraction float int round: abstraction

@arnoutboks #odeaandecode Level of abstraction Money string serialize_money: abstraction

@arnoutboks #odeaandecode Level of abstraction string byte[] encode_as_utf16: abstraction

@arnoutboks #odeaandecode Representation • Translation of a high-level concept to a lower-level representation • All input values supported • Lossless (ideally) • Same data, just in a different form

@arnoutboks #odeaandecode Level of abstraction byte[] string decode_from_utf16: abstraction

@arnoutboks #odeaandecode Level of abstraction string Money parse_money: abstraction

@arnoutboks #odeaandecode Interpretation • Translation of a low-level representation (back) to a higher-level concept • Usually not all input values supported • Same data, just in a different form

@arnoutboks #odeaandecode Why do we do this?

No content

No content

@arnoutboks #odeaandecode Transmission of data Money string abstraction byte[] light pulses, magnetic grains, etc. …

Escaping A different view

@arnoutboks #odeaandecode Escaping for SQL

@arnoutboks #odeaandecode Escaping for SQL

@arnoutboks #odeaandecode Escaping for SQL

@arnoutboks #odeaandecode Escaping for SQL

@arnoutboks #odeaandecode Escaping for SQL

@arnoutboks #odeaandecode String escaping is NOT a security measure

@arnoutboks #odeaandecode String escaping is just properly representing data

@arnoutboks #odeaandecode Escaping as representation string SQLFragment (?) db_escape_string: "O'Shea" "O\'Shea"

@arnoutboks #odeaandecode Escaping as representation "John O\'Shea" "'John O\'Shea'" Better:

@arnoutboks #odeaandecode Escaping as representation string SQLStringLiteral as_sql_string_literal: "O'Shea" "'O\'Shea'"

@arnoutboks #odeaandecode Chained escaping as representation Show greeting"; $sql = "INSERT INTO html_examples VALUES(" . as_sql_string_literal($html_tag) . ")";

@arnoutboks #odeaandecode Premature representation $value) { $_REQUEST[$key] = htmlentities( db_escape_string($value) ); } // ‘Now the input data is safe’

@arnoutboks #odeaandecode Premature representation $value) { $_REQUEST[$key] = htmlentities( db_escape_string($value) ); } // ‘Now the input data is safe’

@arnoutboks #odeaandecode Premature representation $value) { if ( strpos($value, "DROP TABLE") !== false ) { die("SQL injection!"); } }

@arnoutboks #odeaandecode Premature representation $value) { if ( strpos($value, "DROP TABLE") !== false ) { die("SQL injection!"); } }

What goes wrong and how to prevent that

@arnoutboks #odeaandecode Misrepresentation Money string serialize_money_ok: (€ 10) "€ 10"

@arnoutboks #odeaandecode Misrepresentation Money string serialize_money_bad: (€ 10) "10"

@arnoutboks #odeaandecode Misrepresentation Money string serialize_money_bad: (€ 10) "10" ($ 10)

@arnoutboks #odeaandecode Misrepresentation Money string (€ 10) "10" ($ 10) Money ?

@arnoutboks #odeaandecode That doesn’t happen to me! DateTime string 2022-03-14 06:28:00 Europe/Amsterdam 2022-03-14T\ 06:28:00+01:00

@arnoutboks #odeaandecode That doesn’t happen to me! fraction float 1/3 0.33333333333333

@arnoutboks #odeaandecode That doesn’t happen to me! string byte[] ⛽☂️ ? encode_as_iso8859_1

@arnoutboks #odeaandecode Lossy representations cannot be re-interpreted

@arnoutboks #odeaandecode Misinterpretation byte[] […] string "café"

@arnoutboks #odeaandecode Misinterpretation byte[] […] string "café" utf8 decode_from_utf8 Exception!

@arnoutboks #odeaandecode Misinterpretation byte[] […] string "café" iso8859-1 utf8

@arnoutboks #odeaandecode Misinterpretation byte[] […] string "café" iso8859-1 utf8 "café"

@arnoutboks #odeaandecode Duck typing byte[] […] iso8859-1 utf8

@arnoutboks #odeaandecode Duck typing byte[] […] iso8859-1 utf8 HOW?

@arnoutboks #odeaandecode The meaning of data comes from our interpretation of it…

@arnoutboks #odeaandecode …which may very well be wrong

@arnoutboks #odeaandecode Interpretation hints Store/transmit desired interpretation along with data Examples: • HTTP headers • Content-Type • Content-Encoding • Content-Language • UTF8 byte order mark

@arnoutboks #odeaandecode Interpretation hints Store/transmit desired interpretation along with data Examples: • Music staves

@arnoutboks #odeaandecode Interpretation hints utf16 LE utf16 BE […]

@arnoutboks #odeaandecode Interpretation hints utf16 LE with BOM utf16 BE with BOM […]

@arnoutboks #odeaandecode Hints by variable naming Include intended interpretation in variable name: • $message_utf8 • $username_sql • $title_html

@arnoutboks #odeaandecode Hints by variable naming

@arnoutboks #odeaandecode Hints by variable naming query("SELECT * FROM users WHERE username = " . $username); Username is not properly represented for SQL

@arnoutboks #odeaandecode Hints by variable naming " . htmlentities($title_html) . ""; Double escaping, title is already represented as HTML

PHP’s string type A strange hybrid

@arnoutboks #odeaandecode Role overloading of string string byte[] encode_as_utf16: abstraction

@arnoutboks #odeaandecode PHP’s string type is actually a byte array

@arnoutboks #odeaandecode Role overloading of string string = byte[] encode_as_utf16: abstraction

@arnoutboks #odeaandecode Role overloading of string string = byte[] abstraction ‘real’ string use as if utf16

@arnoutboks #odeaandecode Role overloading of string string = byte[] abstraction ‘real’ string use as if utf16 use as if utf8

@arnoutboks #odeaandecode More accurate types • A better type system makes mis-interpretation more difficult • If our programming language does not provide the types, we have to do it ourselves • Value Objects!

@arnoutboks #odeaandecode More accurate types bytes = $bytes; } public static function fromPHPString(string $bytes) { return new self($bytes); } public function toPHPString(): string { return $this->bytes; } // ... }

@arnoutboks #odeaandecode More accurate types bytes = $bytes; } public static function fromPHPString(string $bytes) { return new self($bytes); } public function toPHPString(): string { return $this->bytes; } // ... }

@arnoutboks #odeaandecode More accurate types bytes = $bytes; } public static function fromPHPString(string $bytes) { return new self($bytes); } public function toPHPString(): string { return $this->bytes; } // ... }

@arnoutboks #odeaandecode More accurate types value_utf16 = $value_utf16; } public static function fromUTF8(UTF8Bytes $utf8_bytes) { $value_utf16 = mb_convert_encoding( $utf8_bytes->toPHPString(), 'UTF-16', 'UTF-8'); return new self($value_utf16); } public static function fromISO88591(ISO88591Bytes $iso88591_bytes) { // ... } // ... }

@arnoutboks #odeaandecode More accurate types value_utf16, 'UTF-8', 'UTF-16'); return UTF8Bytes::fromPHPString($value_utf8); } public function toISO88591(): ISO88591Bytes { $value_iso88591 = mb_convert_encoding( $this->value_utf16, 'ISO-8859-1', 'UTF-16'); return ISO88591Bytes::fromPHPString($value_iso88591); } // ... }

@arnoutboks #odeaandecode More accurate types value_utf16 . $other->value_utf16); } public function substring(int $start, int $length): RealString { $substr_utf16 = mb_substr($this->value_utf16, $start, $length, 'UTF-16'); return new RealString($substr_utf16); } // ... }

@arnoutboks #odeaandecode More accurate types (usage) concat($string2)->substring(7, 42); header("Content-Type: text/plain; charset=utf-8"); print $new_string->toUTF8()->toPHPString();

@arnoutboks #odeaandecode Without value objects iso8859-1 utf8 string = byte[] abstraction

@arnoutboks #odeaandecode With value objects abstraction UTF8Bytes ISO88591Bytes RealString

@arnoutboks #odeaandecode Value objects • Achieve higher levels of abstraction • Avoid misinterpretation • No silver bullet! • Comes with additional overhead

@arnoutboks #odeaandecode Recap • Data • Functions • Interpretation and representation • String escaping • Misrepresentation and –interpretation • The string type • Value objects

@arnoutboks #odeaandecode The meaning of data is defined by how we interpret it

@arnoutboks #odeaandecode Feedback & Questions @arnoutboks @arnoutboks @aboks Arnout Boks

@arnoutboks #odeaandecode Image Credits • https://pixabay.com/en/reading-relaxation-glasses-sight- 3088491/ • https://fr.m.wikipedia.org/wiki/Fichier:DARPA_Big_Data.jpg • https://www.flickr.com/photos/elefevre/3936916711 • https://www.flickr.com/photos/perspective/9045532603 • https://www.flickr.com/photos/wwarby/11644168395/ • https://www.flickr.com/photos/opengridscheduler/16480450157 • https://www.flickr.com/photos/cogdog/14401469262 • https://www.flickr.com/photos/paulsimpson1976/3998279762 • https://www.flickr.com/photos/pewari/3499963407