Curling with Julia

Julia already has LibCURL packaged with it, and has all the packages behind JuliaWeb to do HTTP, so why did we need another HTTP package built on LibCURL?

In this talk I go over our motivation for writing CurlHTTP, and talk about some of the cool features that differentiate it from other libraries available in Jula. We’ll look at the CurlEasy and CurlMulti interfaces and show how easy it is to do things like mutual TLS (mTLS) authentication.

CurlHTTP is in the Package Registry and is available on github at https://github.com/bluesmoon/CurlHTTP.jl

Transcript

1. Curling with Julia Cambridge Area Julia Users Network Philip Tellis / Akamai
2. Philip Tellis Principal RUM Distiller @ Akamai @bluesmoon Julia since v0.2
3. Quick start https://github.com/bluesmoon/CurlHTTP.jl • Available in the Package Registry • Pkg.add("CurlHTTP") • Current version v0.1.3 • Docs: https://bluesmoon.github.io/CurlHTTP.jl/ • Julia 1.6+
4. Ways to HTTP in Julia • LibCURL.jl - low level wrapper around libcurl C library. ◦ Good for short recipes, but quickly gets cumbersome with complex use cases. ◦ Does more than just HTTP (eg: FTP, SMTP, SSH, Telnet) • HTTP.jl - higher level Julia API ◦ Supports client & server mode as well as websockets ◦ Complicated to do mutual TLS (client certificates) ◦ Doesn’t support single-threaded parallel requests • Downloads.jl • CurlHTTP.jl - higher level wrapper around libcurl C library ◦ This is what the rest of this talk is about!
5. CurlHTTP at a glance ✔ Two interfaces - CurlEasy and CurlMulti for single and multi connection usage. ✔ Same API across both interfaces. ✔ Read/write data from/to HTTP servers in blocks or as streams. ✔ Support for TLS client certificates and CA certificates. ❌ No websocket support ❌ No cookie handling yet (you have to maintain your own cookie jar)
6. Our use-case • Make GET/POST/PUT/DELETE requests to a server over HTTPS • Use mutual TLS for authentication • Use a custom CA (Certificate Authority) certificate to validate the server • Process response data (JSON stream data) as a stream • Occasionally make multiple parallel requests and combine the post-processed responses
7. Simple GET request using CurlHTTP curl = CurlEasy( url = "https://postman-echo.com/get?foo=bar", method = CurlHTTP.GET, verbose = true ) res, http_status, errormessage = curl_execute(curl) # curl.userdata[:databuffer] is a Vector{UInt8} containing the bytes of the response responseBody = String(curl.userdata[:databuffer]) # curl.userdata[:responseHeaders] is a Vector{String} containing the response headers responseHeaders = curl.userdata[:responseHeaders]
8. POST with streaming response using CurlHTTP curl = CurlEasy( url = "https://postman-echo.com/post", method = CurlHTTP.POST ) requestBody = """{"testName":"test_writeCB"}""" headers = ["Content-Type: application/json"] databuffer = UInt8[] res, http_status, errormessage = curl_execute(curl, requestBody, headers) do d if isa(d, Array{UInt8}) append!(databuffer, d) end end responseBody = String(databuffer)
9. mTLS curl = CurlEasy(; url = "https://your-https-server/url/", method = CurlHTTP.OPTIONS, cacertpath = "/path/to/custom-ca-cert.crt", # optional certpath = "/path/to/client-tls-cert.crt", keypath = "/path/to/client-tls-key.key" ) • The certificate may contain a complete certificate chain. • All three of these files may be combined into a single file: ◦ Certificate + certificate chain goes first ◦ Private key goes second ◦ CA Certificate goes last • By default CurlHTTP will validate certificates
10. Just TLS • You don’t need anything special to do TLS. • If your URL is over https, then CurlHTTP will use TLS and use the most secure options available 1. • The default CA certificate uses LibCURL.cacert (which comes from Mozilla) • Pass in cacertpath to override it 1. Security also depends on the version of libcurl compiled in. On Julia 1.6, there are known vulnerabilities.
11. • FOLLOWLOCATION • SSL_VERIFYPEER • SSL_VERIFYHOST • SSL_VERSION (highest possible up to TLS 1.3) • HTTP_VERSION (H2 over TLS or HTTP/1.1) • TCP_FASTOPEN disabled (due to tracking vuln) • TCP_KEEPALIVE • ACCEPT_ENCODING best supported • TRANSFER_ENCODING • DNS_CACHE_TIMEOUT disabled Default Options
12. Parallel requests pool = map(1:3) do i curl = CurlEasy(url="https://postman-echo.com/post?val=$i", method=CurlHTTP.POST) requestBody = """{"testName":"test_multiPOST","value":$i}""" curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, length(requestBody)) curl_easy_setopt(curl, CURLOPT_COPYPOSTFIELDS, requestBody) curl_add_headers(curl, [ "Content-Type: application/json", "Content-Length: $(length(requestBody))" ]) return curl end multi = CurlMulti(pool) res = curl_execute(multi)
13. Parallel requests • Requests are made when we call curl_execute • All requests run in parallel • Responses are streamed back in parallel to the response handler for each CurlEasy handle pool = map(1:3) do i curl = CurlEasy(url="https://postman-echo.com/post?val=$i", method=CurlHTTP.POST) requestBody = """{"testName":"test_multiPOST","value":$i}""" curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, length(requestBody)) curl_easy_setopt(curl, CURLOPT_COPYPOSTFIELDS, requestBody) curl_add_headers(curl, [ "Content-Type: application/json", "Content-Length: $(length(requestBody))" ]) return curl end multi = CurlMulti(pool) res = curl_execute(multi)
14. Streaming parallel responses pool = map(1:3) do i curl = CurlEasy(url="https://postman-echo.com/post?val=$i", method=CurlHTTP.POST) requestBody = """{"testName":"test_multiPOST","value":$i}""" curl.userdata[:index] = i # userdata is a Dict to store anything you want curl.userdata[:channel] = Channel() CurlHTTP.curl_setup_request(curl, requestBody, ["Content-Type: application/json"]; data_channel = curl.userdata[:channel] ) return curl end # Set up tasks to handle the data channels before calling curl_execute multi = CurlMulti(pool) res = curl_execute(multi)
15. Demo
16. Summary • CurlHTTP offers some enhancements over existing ways of doing HTTP in Julia • mTLS and single task parallel requests are easy • https://github.com/bluesmoon/CurlHTTP.jl • https://github.com/bluesmoon/CurlHTTP.jl/issues