.NET Day 2022 - Anatomy of a Web Request

What happens after you type the web address before the page renders? Is it a black box to you? It doesn't need to be. We'll peel back the onion and look at DNS, HTTP, TLS, and briefly discuss the client render pipeline and windows metaphors. Ultimately, we'll see the internet is not a black box. Open the lid and look inside.

Transcript

1. @rob_rich Anatomy of a Web Anatomy of a Web Request Request by Rob Richardson @rob_rich https://robrich.org/
2. About Me About Me Rob Richardson is a so ware cra sman building web properties in ASP.NET and Node, Angular and React. He's a frequent speaker at conferences, user groups, and community events, and a diligent teacher and student of high quality so ware development. You can find this and other talks on and follow him on twitter at . https://robrich.org/presentations @rob_rich
3. Typically we look at the server pipeline Typically we look at the server pipeline
4. http://karthikk.com/blog/wp-content/uploads/2011/01/asp_lifecycle-e1296387796559.jpg
5. https://dotnettrickscloud.blob.core.windows.net/img/mvc/mvcrequestcycle.png
6. We're not going to do that We're not going to do that
7. What happens when What happens when I type an ip I type an ip into the browser? into the browser?
8. What happens when What happens when I type an ip I type an ip into the browser? into the browser?
9. What happens when What happens when I type an ip I type an ip into the browser? into the browser? opens a socket sends data server replies browser renders it
10. What data is sent? What data is sent? telerik.com/fiddler
11. What data is sent? What data is sent? http method url http headers http body accept header (content negotiation) user agent
12. What data is returned? What data is returned? telerik.com/fiddler
13. What data is returned? What data is returned? status code headers body content length content type
14. HTTP Status Codes HTTP Status Codes 2xx: it worked 3xx: go look over there 4xx: error, user / client's fault 5xx: error, server's fault 1xx: not done yet, but still here https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
15. HTTP Status Codes HTTP Status Codes 302: "found" (moved temporarily) 302: "found" (moved temporarily) browser redirects to location header url many libraries and command line tools automatically redirect next request, they'll come back to the original url 301: moved permanently 301: moved permanently browser redirects to location header url browser will cache the result search engines will forward link love
16. Let's look deeper Let's look deeper
17. OSI Model OSI Model Open Systems Interconnection model 7. Application layer 6. Presentation layer 5. Session layer 4. Transport layer 3. Network layer 2. Data link layer 1. Physical layer https://en.wikipedia.org/wiki/OSI_model
18. OSI Model OSI Model Open Systems Interconnection model 7. HTTP 6. 5. 4. TCP 3. IP 2. 1. Wires https://en.wikipedia.org/wiki/OSI_model
19. Packets Packets http://books.msspace.net/mirrorbooks/snortids/0596006616/snortids-CHP-2-SECT-2.html
20. TCP Packet TCP Packet TCP packet max size is 64k TCP packets are usually under 1500 bytes Ethernet frames are 1500 bytes stackoverflow.com/questions/2613734/maximum-packet-size-for-a-tcp-connection
21. TCP connection handshake TCP connection handshake https://static.lwn.net/images/2012/tfo/foc_use.png
22. TCP slow-start TCP slow-start Purpose: Avoid over-saturating the connection Start by sending a few packets Check latency of responses Assume lack of response is due to congestion Send more packets next time Infer how fast it can go See also https://en.wikipedia.org/wiki/Slow-start
23. TCP slow-start TCP slow-start Result: Connection starts slow and speeds up as you use it See also https://en.wikipedia.org/wiki/Slow-start
24. Resolving URLS Resolving URLS
25. What happens when What happens when I type a url? I type a url?
26. What happens when What happens when I type a url? I type a url?
27. What happens when What happens when I type a url? I type a url? resolve url to ip opens a socket to ip sends http packet over tcp over ethernet server replies browser renders it
28. DNS (Domain Name System) DNS (Domain Name System) NS record: "that guy is responsible for domain" A record: "here's the IP" CNAME record: "go ask that guy" MX record: mail server TXT record SPF record other: AAAA, SRV, PTR, etc TTL: how long should DNS servers cache this
29. How does the How does the browser resolve DNS? browser resolve DNS?
30. http://www.hill2dot0.com/wiki/index.php?title=DNS
31. How does the How does the browser resolve DNS? browser resolve DNS? browser asks it's DNS server it asks it's DNS server recursively until someone knows or ... until one of the 13 root name servers answers > ipconfig IPv4 Address. . . . . . . : 192.168.1.203 DNS Servers . . . . . . . : 192.168.1.1 https://en.wikipedia.org/wiki/DNS_root_zone
32. How does the How does the DNS server answer? DNS server answer? A: "here's the IP" CNAME: "go ask that guy" consuming server caches the answer so it can answer faster next request
33. DNS Example DNS Example browser asks for foo.com to my DNS server my DNS server doesn't know asks it's DNS server that server answers "CNAME: ask bar.com" my DNS server caches the answer then passes it on browser caches the answer browser asks for bar.com my DNS server knows answers with "A: 123.45.67.89" browser opens socket to 123.45.67.89 browser sends http packet to http://123.45.67.89/ includes http header: "hostname: foo.com"
34. DNS Example DNS Example
35. secure transmission secure transmission
36. What happens when What happens when I type a url? I type a url?
37. What happens when What happens when I type a url? I type a url?
38. What happens when What happens when I type a url? I type a url? resolve url to ip opens a socket to ip security stuff sends http packet over tcp over ethernet server replies browser renders it
39. https (TLS) handshake https (TLS) handshake
41. http://technet.microso .com/en-us/library/Cc767139.f14-6_big(l=en-us).gif
42. Diffie-Hellman key exchange Diffie-Hellman key exchange 1. Alice and Bob agree: mod p = 23, base g = 5 2. Alice chose: a = 4, sends Bob A = ga mod p A = 54 mod 23 = 4 3. Bob chose: b = 3, sends Alice B = gb mod p B = 53 mod 23 = 10 4. Alice computes s = Ba mod p s = 104 mod 23 = 18 5. Bob computes s = Ab mod p s = 43 mod 23 = 18 6. Alice and Bob now share a secret: 18 https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange
43. What's the purpose What's the purpose of https urls? of https urls? encrypt data in transit client verifies server server doesn't verify client (that's client certificates)
44. How does How does client verify server? client verify server? client download certificate client walks the trust chain until it gets to something it pre-trusts https://superuser.com/questions/347588/how-do-ssl-chains-work
45. Server certificate trust chain Server certificate trust chain
46. Client's trusted certificates Client's trusted certificates
47. What is encrypted? What is encrypted? Everything apart from the hostname is encrypted so a server hosting multiple domains can respond with the correct certificate TLS extension called Server Name Indicator or SNI https://stackoverflow.com/questions/8277323/what-information-is-visible-to-a-packet-sniffer-which- intercepted-a-https-packet
48. https startup https startup TCP slow start TLS handshake certificate download including trust chain
49. https startup https startup best practices best practices Minimize certificate size 2048 bit, sha256 ssl certificate is fine Don't include a lot of extra domains
50. browser suggest browser suggest
51. What happens when What happens when I type a url in chrome? I type a url in chrome?
52. What happens when What happens when I type a url in chrome? I type a url in chrome?
53. What happens when What happens when I type a url in chrome? I type a url in chrome? resolve Google url to ip opens a socket to Google sends "is this a url or a query" Google answers and returns suggestions resolve url to ip opens a socket to ip over tcp over ethernet server replies browser renders it
54. client page lifecycle client page lifecycle
55. Client page lifecycle Client page lifecycle https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/timing-overview.png
56. Client concerns Client concerns stops at <script>, <link>, <style> downloads linked resource parses resource applies changes (re-flow page if necessary) ... need to get the cascade order right
57. Client concerns Client concerns only two six simultaneous connections to the same subdomain passes cookies automatically
58. Client best practices for HTTP/1 Client best practices for HTTP/1 styles at the top scripts at the bottom use a CDN for static content (at least a different subdomain) minimize web requests https for anything even remotely sensitive https://developer.yahoo.com/performance/rules.html
59. Client best practices for HTTP/2 Client best practices for HTTP/2 styles at the top scripts at the bottom minimize web requests use a CDN for static content (at least a different subdomain) https all the time https://developer.yahoo.com/performance/rules.html
60. browser runs on windows browser runs on windows
61. WIN32 message pump WIN32 message pump http://gemsres.com/story/nov06/299077/arthur-fig-1.gif
62. WIN32 message pump WIN32 message pump things write messages into the queue the message pump runs on the GUI thread the message pump reads the queue and takes action raises events in listening applications paints on the screen with GDI, OpenGL, DirectX, etc
63. WIN32 message WIN32 message best practices best practices If you compute on the GUI thread you'll freeze the process for a time. Get to a background thread as soon as possible.
64. What happens? What happens? So what happens when you type "https://www.somesite.com/" in Google on Windows?
65. What happens? What happens? key down, key up, key press events, message pump reads them DNS lookup of Google suggest servers Google suggestions says "go there" DNS lookup of the url, resolve CNAME a er CNAME until you get to an A record open socket to ip ethernet wraps tcp wraps http tcp slow-start handshake https handshake ...
66. What happens? What happens? ... client verifies trust chain http method, headers, accept, hostname server parses request and forms response how many tcp packets does it take to arrive? browser parses response, loads DOM browser stops at <script>, <link>, <style> onload, render writes win32 messages to paint the screen
67. The internet is not a black box. The internet is not a black box. Open the lid and look inside. Open the lid and look inside.
68. Questions? Questions? @rob_rich @rob_rich https://robrich.org/ https://robrich.org/