Gyokuro Bike-Sharing Control System Mario Camou Twitter: @thedoc, @abstracc Lessons learned with Tuesday, November 20, 2012 

Bike Station Bike Station Bike Station Gyokuro System Message gateway Scala / Akka Process engine ruote (JRuby) Business rules Administration web app REST API Access-control engine End-user web app End Users End-user Web Administrators Service Administrators Operations Team System Architecture Third-party Applications Third-party Applications Third-party Applications JSON / AMQP Binary protocol JSON / HTTP RabbitMQ JSON / HTTP JSON / HTTP Tuesday, November 20, 2012 

Why Actors / Akka? • Concurrency management • Asynchronous operation • Easy integration with AMQP • STM Tuesday, November 20, 2012 

Gateway (v1) • Fully actor-based • Asynchronous message encoding/decoding • Uses Akka-AMQP for communication Tuesday, November 20, 2012 

Gateway (v1) • StationConnectionActor • Handles Socket and SocketOutputStream • Manages Station lifecycle (connect / ID / disconnect) and Station -> Actor table • Stores last incoming/outgoing messages (for JMX monitoring) • StationReadActor • Handles the SocketInputStream • Depacketizes messages and sends them to the MessageDecoder Tuesday, November 20, 2012 

Gateway (v1) • MessageEncoder • Receives a single binary encoded message from the StationReadActor and converts to JSON • Sends to Akka AMQP Sender actor • MessageDecoder • Receives a single JSON-encoded message from the Akka AMQP Receiver actor and converts to binary • Locates the proper StationConnectionActor (depending on the Station ID) and sends the message to it Tuesday, November 20, 2012 

Gateway (v1) Akka AMQP Receiver Akka AMQP Sender actor MessageEncoder actor MessageDecoder actor StationRead actor StationConnection actor connection ID lastIncoming lastOutgoing status Station ID -> Actor mapping (using STM) From station To station To ruote via AMQP From ruote via AMQP Per Station Tuesday, November 20, 2012 

• Connection cannot be assigned to a Station until first message is received • Many “unidentified station” open connections • Can’t use Station ID as Actor ID in ActorRegistry • Solved by: • Using a short TTL on connection open and changing TTL after first message received • Sending a “Request status” message to station upon connection • Use STM to create a Station ID -> Actor UUID table Problems found Tuesday, November 20, 2012 

Problems found • Random gateway hangs • Deadlock in message passing between actors • Actors cannot call each other directly - must be through messages • Actor A receives a message, sends to actor B which then sends back to Actor A • Solved by moving station ID logic out of the actor into an object shared by all other Station actors, and using STM • This also helps with JMX monitoring Tuesday, November 20, 2012 

Problems found • Gateway is a “black box” • Cannot tell which stations are connected and which sockets are unidentified • No easy way of sending messages directly to station bypassing AMQP • Solved by: JMX Tuesday, November 20, 2012 

Problems found • Too many threads? Is this really necessary? • Originally used a single Thread pool for all actors • For finer tuning, moved to one Thread pool per Station with 4-8 Threads each • Thread dump became very hard to debug • Solved by: • Refactoring so shared info is outside of Actor • Judicious use of Actor/thread IDs (for identifying in Thread dumps) • Change message decoding to be a method call instead of an Actor message (called in the same thread as the message receipt) - reduced Thread pool to 2-4 Threads per Station • You CAN go overboard with Actors! Tuesday, November 20, 2012 

Gateway (v2) Akka AMQP Receiver Akka AMQP Sender actor MessageEncoder actor StationRead actor (also decodes) StationWrite actor Station ID -> StationConnection table From station To station To ruote via AMQP From ruote via AMQP Per Station StationConnection token lastInbound lastOutbound status JMX agent Tuesday, November 20, 2012 

Future development • Migration to Akka 2.x • Akka receiver / participant for Ruote (using Akka remoting) Tuesday, November 20, 2012 

Thank you Mario Camou Twitter: @thedoc, @abstracc Tuesday, November 20, 2012