PHPUK 2017 - Websockets and Torrents - A match made with PHP

PHPUK 2017 - Websockets and Torrents - A match made with PHP

(and not NodeJS)


James Mallison

February 16, 2017


  1. @J7mbo

  2. Two words about myself @J7mbo James Mallison

  3. @J7mbo

  4. Websockets and Torrents A match made with PHP! (and not

    nodeJS) @J7mbo • The story of my interest in websockets • Implementing the technology for arbitrary (torrent) data • Problems I encountered and solved • A Demo!! (this completely failed at PHPNW) • What I learned
  5. Legit speeds @J7mbo

  6. @J7mbo

  7. Transmission - Worst Torrent GUI Ever @J7mbo

  8. (Can I get this in Yaml?) @J7mbo transmission-remote --debug

  9. • Browser makes AJAX request • Server performs CLI call

    • Server returns Json Response AJAX Request Browser Server Command line Json Response Browser Server AJAX to Server @J7mbo
  10. @J7mbo

  11. @J7mbo

  12. • Browser makes AJAX request • Server SSHs to VPS

    over TOR • SSH over TOR using Torify • VPS performs CLI call • VPS returns Json Response • Server returns Json Response Browser My Server SSH to VPS Json Response Browser My Server Command line AJAX Request Ajax with SSH to VPS @J7mbo TOR (torify)
  13. @J7mbo

  14. @J7mbo

  15. • Browser sends ws:// or wss:// • Server responds with

    HTTP/ 1.1 Switching Protocols • Open and persistent connection • Connection can be closed by either server or client at any time. Websockets @J7mbo
  16. Request to Server Response to Client END Http VS Websockets

    • Send a request • HALF DUPLEX • Wait for a response • Rinse, repeat Http Event Loop WS Connection Started Send response Send response Send another response Receive message Websockets • Initialise connection • FULL DUPLEX • Send / receive at any time • Close connection @J7mbo
  17. Duplex wtf? @J7mbo Half Duplex In a conversation, only one

    person can talk at once The other person has to listen and wait to give a response Full Duplex In a conversation, everyone’s able to talk and listen at once Everyone is able to understand what each other is saying (Like PHP Internals) (Not like PHP Internals)
  18. @J7mbo Event Loop

  19. • Not just while(){} loops • Use interrupt driven I/O

    instead of polling • IDI/O flag is set on CPU, process suspended and buffer filled • Flag changes, callback executed (Still requires flag checks, not magic) • Useful for IPC • PHP’s PCNTL similar to this (signal based) @J7mbo Event Loops
  20. PCNTL Extension @J7mbo

  21. @J7mbo Ratchet

  22. • Create “Event Handler” class that implements Ratchet’s interface that

    has methods “onSubscribe()”, “onOpen()” etc @J7mbo Using Ratchet @J7mbo • Tell Ratchet that your class is the thing that will handle events and is ready to be run() • In development, run terminal, type php index.php, hit enter and leave the event loop (and event handler class) terminal window open, running, and ready to handle events
  23. Using Ratchet (Code) @J7mbo The above code allows: Javascript to

    call .subscribe() over websockets OurEventHandlerClass::subscribe() to be executed (by event loop)
  24. Implementation Time @J7mbo @J7mbo WAMP Spec (Methods you have to

    implement in PHP) JavaScript library calls to make events happen
  25. The Web Application Messaging Protocol @J7mbo WAMP Events

  26. onOpen() @J7mbo

  27. onOpen() @J7mbo

  28. onSubscribe() @J7mbo

  29. onSubscribe() @J7mbo

  30. Websocket Event Handler + Loop HTTP or Websocket? ProxyPass Http

    Ws Apache + PHP Client makes a request Enable mod_proxy • ProxyPass /wss/ wss:// @J7mbo
  31. Event Loop .subscribe(func, topic) 1 onSubscribe($conn, $topic) • store connection

    in array • start timer for connection • store timer against connection 2 High level functionality + events Timer hit • Get torrent data via CLI • This takes ages… … … 3 ‘Broadcast’ data back to topic • $topic->broadcast($data) 4 Client calls .unsubscribe(func, topic) • Event handler onUnsubscribe() called • Stop timer for connection • Remove from internal connections array 5 @J7mbo
  32. Event Loop Problem 1 - Topic ‘Broadcast’ data back to

    topic • $topic->broadcast($data) 4 Option 1: $topic = ‘torrents’ • - Not on a per-user or per-connection basis • - Everyone subscribed to ‘torrents’ gets the data pushed to them • - Open same page, data gets sent to both pages twice Option 2: $topic = { ‘topic’ => ‘torrents’, ‘userId’ => 245 } • Every subscription is now unique per user id • - Same user can open same page and start getting duplicate data like before @J7mbo
  33. Problem 2: Authentication and Duplicate data The Problem • Pass

    user id from PHP to JS variable • Subscribe to torrents with topic containing user id • Client could have just changed user id JS variable before subscribing One Solution • Create unique token on each standard HTTP request in your application • Use PHP 7’s CSPRNG rather than open_ssl_random_pseudo_bytes() • Pass token with user id in topic and check against database • Recreate token in database so that a second check will fail $topic = { ‘topic’ => ‘torrents’, ‘userId’ => 245, ‘token’ => ‘385e33f741’ // bin2hex(random_bytes(5)) in PHP 7.1 }; @J7mbo
  34. Problem 3: Blocking!! The Problem • Whilst command line call

    is being made (5 seconds), nothing else can happen • No .subscribe ——> onSubscribe() : Nobody else can do anything • This is blocking! Same goes for authentication (checking database) A few solutions • Use another process to do the work - on the same machine • Use a job queue - requires a way to get data back into event loop once done • Requires good architecture decisions for low coupling + high cohesion for event naming, job naming, whether worker knows who’s job it is etc @J7mbo
  35. Blocking - Option 1 - Child Process The Result •

    Event loop can continue, fine for small projects or just as a proof of concept • Process is only on one machine - not scalable @J7mbo
  36. Blocking - Option 2 - Job Queue (and ZMQ) In

    onSubscribe() - add to queue ZMQ listening - When setting up event handler @J7mbo
  37. Blocking - Option 2 - Job Queue (and ZMQ) Worker

    sends data over :5555 when done to onZmqResponse() @J7mbo
  38. Event Loop .subscribe(func, topic) 1 High level architecture onSubscribe($conn, $topic)

    • store connection in array • add job to queue (fire and forget) 2 Queue Workers pick up job • Perform job • Send result to :5555 3 Workers Anything sent to port 5555 goes to onZmqRequest() Broadcasted to client 4 • Event loop is just an intermediary for data retrieval and sending • Nobody is blocked from registering their interest in receiving data • Workers can be in Java, C*, Erlang, Python, Ruby, JS etc @J7mbo
  39. Demo? (Last time this failed completely) @J7mbo

  40. What about scaling? (One way without a load balancer) @J7mbo

    Client API DB / Redis Workers Queue Call API Event Loop Event Loop Event Loop Call chosen ws:// server Round Robin? ZMQ Scalable Scalable Broadcast data to client
  41. But we’re still polling on the server for torrent data!!

    (Obviously) @J7mbo For ‘realtime’ data, you have to program a ‘daemon’, which requires (yet again) a different mindset Monitor CPU Usage example: The crap way • Python script runs top • Parses output • SUPER inefficient The proper way • C++ Binary • Queries HOST-RESOURCES MIB • Sleeps on pthread_mutex in-between • (Involves threading or something?)
  42. • Using an event loop != non-blocking or asynchronous •

    Easiest way to not block, use threads, processes or an external library • All connections between client / ws server over SSH with TOR • Could accomplish with NodeJS (but didn’t want to sell my soul) • Could have just made RPC calls to the server anyway instead of cmd-line (I found this out right at the end of it all) • Didn’t have to broadcast on a $topic, as $connection had methods of sending data to specific connections • Server-side real-time requires a daemon, threads, clever stuff What I learned @J7mbo
  43. • Event loops and reactors in PHP: React, amphp •

    Websocket server-side libraries: ratchet, aerys • WAMP-specced client-side libraries: Autobahn.js, AngularWAMP, Minion • Event-loop underlying libs: LibEv, LibEvent, LibUV (windows) Your Choices @J7mbo
  44. Where YOU can go next @J7mbo • Get a simple

    example working (slides up) • Use processes first to offload work • Get ZMQ working and use terminal to send something into your event loop! • Watch Justin Carmony’s talk from PHPNW15 on job queues and writing them in a clean and methodical OO way, and put that knowledge together with the stuff from my talk: • Have fun and be awesome because you now know a simple architecture for using websockets with PHP (and not nodejs)
  45. • @j7mbo (PHP6 jokes, php, oop, software engineering) • PHPNW15:

    Justin Carmony - Scaling & Managing Asynchronous Workers (and staying sane!) - Watch this for queues etc - it’s an awesome talk! Thanks @J7mbo
  46. DEMO (if it works)