Tweet
Tweet

Slide 1

Slide 1 text

HTTP Can Do That?! A collection of bad ideas by Sumana Harihareswara @brainwane Changeset Consulting

Slide 2

Slide 2 text

@brainwane HTTP Hypertext Transfer Protocol

Slide 3

Slide 3 text

@brainwane Diagrams! – Internet Engineering Task Force (IETF) RFC 7230 Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing

Slide 4

Slide 4 text

@brainwane HTTP: crash course Client Server

Slide 5

Slide 5 text

@brainwane Client Server Request HTTP: crash course

Slide 6

Slide 6 text

@brainwane Client Server Request Response HTTP: crash course

Slide 7

Slide 7 text

@brainwane An HTTP Message (Request or Response) START-LINE HTTP version (1.1) Request method (GET, POST) Response status code (200, 404, 500)

Slide 8

Slide 8 text

@brainwane An HTTP Message (Request or Response) HEADERS Content­Type Content­Length …... START-LINE HTTP version (1.1) Request method (GET, POST) Response status code (200, 404, 500)

Slide 9

Slide 9 text

@brainwane An HTTP Message (Request or Response) HEADERS Content­Type Content­Length …... BODY START-LINE HTTP version (1.1) Request method (GET, POST) Response status code (200, 404, 500)

Slide 10

Slide 10 text

@brainwane Example Request HEADERS Host: www.sumana.biz Accept: text/html User­Agent: ScraperBot BODY START-LINE GET / HTTP/1.1

Slide 11

Slide 11 text

@brainwane Example Response HEADERS Content­Type: text/html Content­Length: 203 Date: Tue, 16 Jun 2015 16:21:56 GMT Last­Modified: Tue, 16 Jun 2015 13:27:14 GMT BODY Welcome to Sumanaville

Ro ckin'

This is a pretty START-LINE HTTP/1.1 200 OK

Slide 12

Slide 12 text

@brainwane Methods

Slide 13

Slide 13 text

@brainwane ● GET gimme ● POST here you go Popular request methods (“verbs”)

Slide 14

Slide 14 text

@brainwane First bad idea: POST but not GET more: https://gitlab.com/http-can-do-that/secureapi

Slide 15

Slide 15 text

@brainwane POST but not GET: use cases letters to Santa Claus

Slide 16

Slide 16 text

@brainwane POST but not GET: use cases employee suggestion box

Slide 17

Slide 17 text

@brainwane POST but not GET: use cases extremely moderated blog comments

Slide 18

Slide 18 text

@brainwane (a logistical note)

Slide 19

Slide 19 text

@brainwane

Slide 20

Slide 20 text

@brainwane

Slide 21

Slide 21 text

@brainwane Bad Idea Scale

Slide 22

Slide 22 text

@brainwane Giving client no way to GET – bad idea

Slide 23

Slide 23 text

@brainwane Remember “CRUD”? Create Read Delete Update

Slide 24

Slide 24 text

@brainwane Remember “CRUD”? Create POST Read GET Delete POST Update POST

Slide 25

Slide 25 text

@brainwane Remember “CRUD”? Create POST Read GET Delete POST Update POST

Slide 26

Slide 26 text

@brainwane Remember “CRUD”? Create POST Read GET Delete POST Update POST INELEGANT! INELEGANT!

Slide 27

Slide 27 text

@brainwane DELETE delete a resource! Underappreciated methods

Slide 28

Slide 28 text

@brainwane Implementing DELETE

Slide 29

Slide 29 text

@brainwane Implementing DELETE

Slide 30

Slide 30 text

@brainwane Implementing DELETE

Slide 31

Slide 31 text

@brainwane Implementing DELETE

Slide 32

Slide 32 text

@brainwane Implementing DELETE

Slide 33

Slide 33 text

@brainwane Implementing DELETE

Slide 34

Slide 34 text

@brainwane Implementing DELETE

Slide 35

Slide 35 text

@brainwane Implementing DELETE

Slide 36

Slide 36 text

@brainwane Implementing DELETE

Slide 37

Slide 37 text

@brainwane Implementing DELETE

Slide 38

Slide 38 text

@brainwane DELETE – good idea?

Slide 39

Slide 39 text

@brainwane PUT “here you go” Underappreciated methods

Slide 40

Slide 40 text

@brainwane I thought POST meant “here you go” Wait

Slide 41

Slide 41 text

@brainwane So what is POST, anyway? The standard says it means: “Above our pay grade; take this to the boss” a.k.a. Overloaded POST

Slide 42

Slide 42 text

@brainwane So what is POST, anyway? Often, we use it for: “Create a new item in this set” a.k.a. POST-to-append

Slide 43

Slide 43 text

@brainwane

Slide 44

Slide 44 text

@brainwane PUT vs. POST PUT /cards/5 Body: Means: “Put this picture at /cards/5 .” POST /cards/5 Body: Means: “Tell the webapp that this picture applies to /cards/5 somehow – figure it out.”

Slide 45

Slide 45 text

@brainwane “CRUD” & HTTP verbs Create PUT Read GET Delete DELETE Update PUT

Slide 46

Slide 46 text

@brainwane PUT – good idea?

Slide 47

Slide 47 text

@brainwane ● PATCH update just part of this document/resource More underused methods

Slide 48

Slide 48 text

@brainwane PATCH – good idea?

Slide 49

Slide 49 text

@brainwane ● PATCH update just part of this document/resource ● OPTIONS ask what verbs the client’s allowed to use (for a specific path, or server-wide) More underused methods

Slide 50

Slide 50 text

@brainwane OPTIONS – good idea?

Slide 51

Slide 51 text

@brainwane HEAD like GET, but just for metadata A super-cool method

Slide 52

Slide 52 text

@brainwane GET vs. HEAD Request: GET / HTTP/1.1 Response: ● Start-line ● Headers ● Body Request: HEAD / HTTP/1.1 Response: ● Start-line ● Headers

Slide 53

Slide 53 text

@brainwane HEAD saves time

Slide 54

Slide 54 text

@brainwane HEAD saves time

Slide 55

Slide 55 text

@brainwane HEAD saves time

Slide 56

Slide 56 text

@brainwane You don’t need the body to check: Does it exist? Do I have permission to GET it? Content­Length Last­Modified Content­Type ETag Retry­After

Slide 57

Slide 57 text

@brainwane HEAD – good idea?

Slide 58

Slide 58 text

@brainwane Headers

Slide 59

Slide 59 text

@brainwane Popular headers include: Content­Type Content­Length

Slide 60

Slide 60 text

@brainwane Popular headers include: Content­Type Content­Length Also known as MIME or Mime

Slide 61

Slide 61 text

@brainwane Popular headers include: Content­Type Content­Length text/*

Slide 62

Slide 62 text

@brainwane Popular headers include: Content­Type Content­Length application/*

Slide 63

Slide 63 text

@brainwane Popular headers include: Content­Type Content­Length chemical/*

Slide 64

Slide 64 text

@brainwane Popular headers include: Content­Encoding Accept­Encoding Content­Language Accept­Language

Slide 65

Slide 65 text

@brainwane More headers ETag If­Match If­None­Match

Slide 66

Slide 66 text

@brainwane More headers If­Modified­Since If­Unmodified­Since Last­Modified Cache­Control

Slide 67

Slide 67 text

@brainwane A popular header User­Agent

Slide 68

Slide 68 text

@brainwane An unpopular header From The email address of the person making the request

Slide 69

Slide 69 text

@brainwane Uses for From Really bad auth

Slide 70

Slide 70 text

@brainwane Uses for From “Yes, I saw your site launch”

Slide 71

Slide 71 text

@brainwane Uses for From Coded messages meant for network surveillor

Slide 72

Slide 72 text

@brainwane From – bad idea

Slide 73

Slide 73 text

@brainwane Another spy trick “Each header field consists of a case-insensitive field name followed by a colon (":")...” So: vary the case of the headers you send!!! – Internet Engineering Task Force (IETF) RFC 7230 Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing

Slide 74

Slide 74 text

@brainwane Header casing as secret info channel – bad idea

Slide 75

Slide 75 text

@brainwane A popular header Host

Slide 76

Slide 76 text

@brainwane A required header Host required in request messages

Slide 77

Slide 77 text

@brainwane $ netcat myhostname.tld 80 GET /bicycle HTTP/1.1 Host: myhostname.tld Host & path work together

Slide 78

Slide 78 text

@brainwane Host & path work together

Slide 79

Slide 79 text

@brainwane Host & path work together

Slide 80

Slide 80 text

@brainwane Host & path work together

Slide 81

Slide 81 text

@brainwane Host & path work together

Slide 82

Slide 82 text

@brainwane A popular header Host (wait – why do we need to repeat this? It's in the URL! right?)

Slide 83

Slide 83 text

@brainwane How Host helps HTTP is separate from the Domain Name System

Slide 84

Slide 84 text

@brainwane How Host helps Host helps route requests among different domains that sit on the same server

Slide 85

Slide 85 text

@brainwane Examples of virtual hosts www.debian.org

Slide 86

Slide 86 text

@brainwane Examples of virtual hosts bugs.debian.org

Slide 87

Slide 87 text

@brainwane Examples of virtual hosts lists.debian.org

Slide 88

Slide 88 text

@brainwane Examples of virtual hosts wiki.debian.org

Slide 89

Slide 89 text

@brainwane But watch out...

Slide 90

Slide 90 text

@brainwane But watch out...

Slide 91

Slide 91 text

@brainwane A spam story

Slide 92

Slide 92 text

@brainwane A spam story My 404 logs (Drupal admin console): TYPE page not found DATE Thursday, October 9, 2014 - 10:46 USER Anonymous (not verified) LOCATION http://myphishingsite.biz/http://myphishingsite.biz REFERRER MESSAGE ttp://myphishingsite.biz SEVERITY warning HOSTNAME [IP address]

Slide 93

Slide 93 text

@brainwane A spam story My 404 logs (Drupal admin console): TYPE page not found DATE Thursday, October 9, 2014 - 10:46 USER Anonymous (not verified) LOCATION http://myphishingsite.biz/http://myphishingsite.biz REFERRER MESSAGE ttp://myphishingsite.biz SEVERITY warning HOSTNAME [IP address]

Slide 94

Slide 94 text

@brainwane A spam story My access logs: [IP address] ­ ­ [09/Oct/2014:10:46:09 ­0400] "GET http://myphishingsite.biz HTTP/1.1" 404 7574 "­" [User­Agent]

Slide 95

Slide 95 text

@brainwane A spam story Legit mistakes would look like: [IP address] ­ ­ [09/Oct/2014:10:46:09 ­0400] "GET /http://berkeley.edu HTTP/1.1" 404 7574 "­" [User­Agent]

Slide 96

Slide 96 text

@brainwane A spam story Intentionally malform your request! $ netcat myhostname.tld 80 GET http://spam.com HTTP/1.1 Host: spam.com

Slide 97

Slide 97 text

@brainwane A spam story Intentionally malform your request! $ netcat myhostname.tld 80 GET /viagra­bitcoin HTTP/1.1 Host: spam.com

Slide 98

Slide 98 text

@brainwane 404 spamming – bad idea

Slide 99

Slide 99 text

@brainwane Define your own header! “Header fields are fully extensible: there is no limit on the introduction of new field names, each presumably defining new semantics, nor on the number of header fields used in a given message.” -(RFC 7230)

Slide 100

Slide 100 text

@brainwane Define your own header! X­blah­blah­blah

Slide 101

Slide 101 text

@brainwane Define your own header! X­Wikimedia­Debug

Slide 102

Slide 102 text

@brainwane Define your own header! X­Wikimedia­Debug an HTTP request header ● Backend selection (Varnish) ● Caching behavior ● Request profiling (record a trace) ● Debug logs ● Read-only mode ● Browser extensions More: https://wikitech.wikimedia.org/wiki/X- Wikimedia-Debug

Slide 103

Slide 103 text

@brainwane Define your own header!

Slide 104

Slide 104 text

@brainwane Define your own header!

Slide 105

Slide 105 text

@brainwane Define your own header!

Slide 106

Slide 106 text

@brainwane Define your own header!

Slide 107

Slide 107 text

@brainwane Define your own header! https://gitlab.com/http-can-do-that/novel-titles

Slide 108

Slide 108 text

@brainwane Define your own header!

Slide 109

Slide 109 text

@brainwane Defining your own headers – good idea?

Slide 110

Slide 110 text

@brainwane Status codes

Slide 111

Slide 111 text

@brainwane Status codes 100 & 101: Informational 2xx: Successful 3xx: Redirection 4xx: Client error 5xx: Server error

Slide 112

Slide 112 text

@brainwane Status (response) codes 404 Not Found Code Reason-phrase

Slide 113

Slide 113 text

@brainwane Status (response) codes “A client SHOULD ignore the reason-phrase content.”

Slide 114

Slide 114 text

@brainwane Heard of these? ● 410 Gone It was here, but now it’s not. ● 304 Not Modified You said, ‘GET this, if it’s been modified since [date]’. It hasn’t been.

Slide 115

Slide 115 text

@brainwane 451 Unavailable For Legal Reasons Server is legally required to reject client’s request

Slide 116

Slide 116 text

@brainwane 451 Unavailable For Legal Reasons Can’t let you see that; it’s censored.

Slide 117

Slide 117 text

@brainwane 451 Unavailable For Legal Reasons “This is considered a client-side error even though the request is well formed and the legal requirement exists on the server side. After all, that representation was censored for a reason. There must be something wrong with you, citizen.” -RESTful Web APIs, Leonard Richardson & Mike Amundsen

Slide 118

Slide 118 text

@brainwane 451 Unavailable For Legal Reasons

Slide 119

Slide 119 text

@brainwane 451 – good idea?

Slide 120

Slide 120 text

@brainwane WTF responses All of these were found in the wild

Slide 121

Slide 121 text

@brainwane WTF responses Code: 126 Reason: Incorrect key file for table '/tmp/mysqltmp/#sql_13fb_2.MYI'; try to repair it SQL=SHOW FULL COLUMNS FROM `y4dnu_extensions`

Slide 122

Slide 122 text

@brainwane WTF responses Code: 301 Reason: explicit_header_response_code

Slide 123

Slide 123 text

@brainwane WTF responses Code: 403 Reason: You've got to ask yourself one question: Do I feel lucky?

Slide 124

Slide 124 text

@brainwane WTF responses Code: 403 Reason: can't put wasabi in bed

Slide 125

Slide 125 text

@brainwane WTF responses Code: 404 Reason: HTTP/1.1 404

Slide 126

Slide 126 text

@brainwane WTF responses Code: 404 Reason: Not Found"); ?>

Slide 127

Slide 127 text

@brainwane WTF responses Code: 200 Reason: Forbidden

Slide 128

Slide 128 text

@brainwane WTF responses Code: 404 Reason: Apple WebObjects

Slide 129

Slide 129 text

@brainwane WTF responses Code: 404 Reason: forbidden

Slide 130

Slide 130 text

@brainwane WTF responses Code: 434 Reason: HTTP/1.1 434

Slide 131

Slide 131 text

@brainwane WTF responses Code: 451 Reason: Unknown Reason-Phrase

Slide 132

Slide 132 text

@brainwane WTF responses Code: 503 Reason: Backend is unhealthy

Slide 133

Slide 133 text

@brainwane WTF responses Code: 520 Reason: Origin Error

Slide 134

Slide 134 text

@brainwane WTF responses Code: 525 Reason: Origin SSL Handshake Error

Slide 135

Slide 135 text

@brainwane WTF responses Code: 533 Reason: mtd::http: Unknown: Banned

Slide 136

Slide 136 text

@brainwane WTF responses Code: 732 Reason: http://www.[hostname].com/intro/copyright.php

Slide 137

Slide 137 text

@brainwane WTF responses Code: 999 Reason: Request denied

Slide 138

Slide 138 text

@brainwane Changing Reason-phrases more at https://gitlab.com/http-can-do-that

Slide 139

Slide 139 text

@brainwane

Slide 140

Slide 140 text

@brainwane

Slide 141

Slide 141 text

@brainwane Bespoke status codes/reasons – good idea?

Slide 142

Slide 142 text

@brainwane Conclusion

Slide 143

Slide 143 text

@brainwane There’s so much more ● “Don’t cache this” ● Pragma – pass instructions to server/client ● CONNECT, TRACE, LINK, & UNLINK methods ● 409 Conflict ● Look-before-you-leap requests ● Resources at HTTPS vs. HTTP URLs can differ ● “q” and preference ranking in the Accept header ● Content-Disposition (e.g. “attachment”)

Slide 144

Slide 144 text

@brainwane The feeling of power The sense of wonder

Slide 145

Slide 145 text

@brainwane What might the web have been? What might it still be?

Slide 146

Slide 146 text

@brainwane Read & play ● RFCs 7230-7235 ● requests ● netcat, wget, netstat, telnet ● basic HTTP servers (in your favorite language) ● https://gitlab.com/http-can-do-that

Slide 147

Slide 147 text

@brainwane Thanks Leonard Richardson Greg Hendershott Zack Weinberg The Recurse Center Clay Hallock Paul Tagliamonte Open Source Bridge Julia Evans, Allison Kaptur, Amy Hanlon, and Katie Silverio

Slide 148

Slide 148 text

Thank you Sumana Harihareswara http://changeset.nyc @brainwane https://gitlab.com/http-can-do-that sumanah@panix.com