Curling with Julia JuliaCon 2024 Philip Tellis / Akamai

Philip Tellis Principal RUM Distiller @ Akamai @bluesmoon Julia since v0.2

Quick start https://github.com/bluesmoon/CurlHTTP.jl ● Available in the Package Registry ● Pkg.add("CurlHTTP") ● Current version v0.1.3 ● Docs: https://bluesmoon.github.io/CurlHTTP.jl/ ● Julia 1.6+

Ways to HTTP in Julia ● LibCURL.jl - low level wrapper around libcurl C library. ○ Good for short recipes, but quickly gets cumbersome with complex use cases. ○ Does more than just HTTP (eg: FTP, SMTP, SSH, Telnet) ● HTTP.jl - higher level Julia API ○ Supports client & server mode as well as websockets ○ Complicated to do mutual TLS (client certificates) ○ Doesn’t support single-threaded parallel requests ● Downloads.jl ● CurlHTTP.jl - higher level wrapper around libcurl C library ○ This is what the rest of this talk is about!

CurlHTTP at a glance ✔ Two interfaces - CurlEasy and CurlMulti for single and multi connection usage. ✔ Same API across both interfaces. ✔ Read/write data from/to HTTP servers in blocks or as streams. ✔ Support for TLS client certificates and CA certificates. ❌ No websocket support ❌ No cookie handling yet (you have to maintain your own cookie jar)

Our use-case ● Make GET/POST/PUT/DELETE requests to a server over HTTPS ● Use mutual TLS for authentication ● Use a custom CA (Certificate Authority) certificate to validate the server ● Process response data (JSON stream data) as a stream ● Occasionally make multiple parallel requests and combine the post-processed responses

A quick note about TLS ● Transport Layer Security. Used by HTTPS and a bunch of other protocols to encrypt traffic over the wire. ● Uses asymmetric key cryptography. Typically the server has a certificate and a private key. The certificate is signed by a certificate authority that the client trusts. This allows the client to trust that the server is who they say they are. ● mTLS is Mutual TLS. The client also has a certificate & key, but this is typically signed by the server. This is used for authentication. ● Certificates contain a common name. This is used for authorization.

Simple GET request using CurlHTTP curl = CurlEasy( url = "https://postman-echo.com/get?foo=bar", method = CurlHTTP.GET, verbose = true ) res, http_status, errormessage = curl_execute(curl) # curl.userdata[:databuffer] is a Vector{UInt8} containing the bytes of the response responseBody = String(curl.userdata[:databuffer]) # curl.userdata[:responseHeaders] is a Vector{String} containing the response headers responseHeaders = curl.userdata[:responseHeaders]

POST with streaming response using CurlHTTP curl = CurlEasy( url = "https://postman-echo.com/post", method = CurlHTTP.POST ) requestBody = """{"testName":"test_writeCB"}""" headers = ["Content-Type: application/json"] databuffer = UInt8[] res, http_status, errormessage = curl_execute(curl, requestBody, headers) do d if isa(d, Array{UInt8}) append!(databuffer, d) end end responseBody = String(databuffer)

mTLS curl = CurlEasy(; url = "https://your-https-server/url/", method = CurlHTTP.OPTIONS, cacertpath = "/path/to/custom-ca-cert.crt", # optional certpath = "/path/to/client-tls-cert.crt", keypath = "/path/to/client-tls-key.key" ) ● The certificate may contain a complete certificate chain. ● All three of these files may be combined into a single file: ○ Certificate + certificate chain goes first ○ Private key goes second ○ CA Certificate goes last ● By default CurlHTTP will validate certificates

Just TLS ● You don’t need anything special to do TLS. ● If your URL is over https, then CurlHTTP will use TLS and use the most secure options available 1. ● The default CA certificate uses LibCURL.cacert (which comes from Mozilla) ● Pass in cacertpath to override it 1. Security also depends on the version of libcurl compiled in. On Julia 1.6, there are known vulnerabilities: https://github.com/JuliaLang/julia/pull/46116

● FOLLOWLOCATION: Follows 30x ● SSL_VERIFYPEER: Verify the authenticity of the peer’s certificate ● SSL_VERIFYHOST: Verify that the certificate matches the host ● SSL_VERSION (highest possible up to TLS 1.3) ● HTTP_VERSION (H2 over TLS or HTTP/1.1) ● TCP_FASTOPEN disabled due to tracking vuln and to allow TLS session cache ● TCP_KEEPALIVE: use TCP KeepAlive probes ● ACCEPT_ENCODING best supported compression ● TRANSFER_ENCODING ● DNS_CACHE_TIMEOUT disabled because your DNS server should handle this Default Options

Parallel requests pool = map(1:3) do i curl = CurlEasy(url="https://postman-echo.com/post?val=$i", method=CurlHTTP.POST) requestBody = """{"testName":"test_multiPOST","value":$i}""" curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, length(requestBody)) curl_easy_setopt(curl, CURLOPT_COPYPOSTFIELDS, requestBody) curl_add_headers(curl, [ "Content-Type: application/json", "Content-Length: $(length(requestBody))" ]) return curl end multi = CurlMulti(pool) res = curl_execute(multi)

Parallel requests ● Requests are made when we call curl_execute ● All requests run in parallel ● Responses are streamed back in parallel to the response handler for each CurlEasy handle pool = map(1:3) do i curl = CurlEasy(url="https://postman-echo.com/post?val=$i", method=CurlHTTP.POST) requestBody = """{"testName":"test_multiPOST","value":$i}""" curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, length(requestBody)) curl_easy_setopt(curl, CURLOPT_COPYPOSTFIELDS, requestBody) curl_add_headers(curl, [ "Content-Type: application/json", "Content-Length: $(length(requestBody))" ]) return curl end multi = CurlMulti(pool) res = curl_execute(multi)

Streaming parallel responses pool = map(1:3) do i curl = CurlEasy(url="https://postman-echo.com/post?val=$i", method=CurlHTTP.POST) requestBody = """{"testName":"test_multiPOST","value":$i}""" curl.userdata[:index] = i # userdata is a Dict to store anything you want curl.userdata[:channel] = Channel() CurlHTTP.curl_setup_request(curl, requestBody, ["Content-Type: application/json"]; data_channel = curl.userdata[:channel] ) return curl end # Set up tasks to handle the data channels before calling curl_execute multi = CurlMulti(pool) res = curl_execute(multi)

Demo

Summary ● CurlHTTP offers some enhancements over existing ways of doing HTTP in Julia ● mTLS and single task parallel requests are easy ● https://github.com/bluesmoon/CurlHTTP.jl ● https://github.com/bluesmoon/CurlHTTP.jl/issues