Extending Scalability with JBoss Stack (Infinispan)

RedHat DevConf2013 in Brno

Transcript

1. www.ysoft.com RedHat Developer Conference Brno, Feb 23rd 2013 Ondrej “Ondra” Krajíček CHIEF RESEARCH OFFICER EXTENDING SCALABILITY
2. A STORY... ▪ To share our experience with designing new, more scalable generation of our products... ▪ ...using JBoss technologies.
3. ▪ Ondřej Krajíček ▪ about.me/ondrej.krajicek ▪ 2005: Graduated from FI MU ▪ 2007: Y Soft ▪ Other professional cooperations: Microsoft, VMWare and others
4. ▪ Jan Slezak ▪ Senior Developer / Team Lead ▪ 2008: Y Soft ▪ The main Infinispan magician at Y Soft
5. ABOUT Y SOFT ▪ Czech global company ▪ Student startup (started in 2000) ▪ No venture capital ▪ Our vision is to build Czech global company without venture funding
6. THE ORIGINAL GOAL ▪Extend the throughput performance of the system to facilitate deployments for Enterprise Customers ▪Retain the properties of centrally managed solution
7. SYSTEM OVERVIEW - SAFEQ • Printing solution • Pre-processing of print jobs • Authentication and authorization of users • Auditing and cost-allocation (charge back) • Failover and convenience
8. ORIGINAL SYSTEM ARCHITECTURE ▪ SafeQ servers in application level cluster (up to 6 nodes) ▪ Approx. 200 MFPs per node.
9. ▪ Extended spooler ▪ Authentication ▪ Authorisation ▪ Accounting ▪ Auditing ▪ 3rd party connectivity ▪ web based management
10. INTEGRATION WITH OTHER KEY CUSTOMER SYSTEMS SafeQ is usually tightly integrated into customer infrastructure
11. SafeQ RS/ORS (Offline Remote Spooler) Additional layer of business logic to increase throughput
12. PRIMITIVE REMOTING SAFEQ 3.2 – REMOTE SPOOLER TECHNOLOGY ▪ Application logic decoupled into two layers with transparent remoting using facades ▪ Most operations still remained synchronous accross remoting boundaries PROPRIETARY MESSAGING ▪ Proprietary message queing technology (XML-based) ▪ Low-latency ▪ In-memory only ▪ Location and migration transparency
13. LIMITED SCALABILITY PROBLEMS WITH SAFEQ RS / LESSONS LEARNED ▪ Synchronous dependencies across architectural layers ▪ Each user operation execution spanned both layers DURABILITY ▪ RS failure resulted in data-loss due to lack of persistence ▪ Network failures caused inconsistencies in application data SINGLE POINT OF FAILURE ▪ Failures of CML (Central Components) caused immediate service failure of the entire system
14. The tough decision was made and we have concluded complete system redesign.
15. AUTONOMOUS OPERATION BASIC PRINCIPLES ▪ ORS system shall be able to operate autonomously, without connection to the CML PERSISTENCE ▪ ORS system shall store data and remove data which were successfully replicated to the CML ASYNCHRONOUS OPERATION ▪ All operations which span the decoupling of the application logic shall be asynchronous
16. DATA CACHE SAFEQ ORS (OFFLINE REMOTE SPOOLER) COMPONENTS ▪ PERSISTENT ▪ SEARCHABLE ▪ ZERO MAINTENANCE ASYNCHRONOUS MESSAGING ▪ Replication buffers and service channels ▪ Async replication of bulk data ▪ Async remote service invocation LOAD BALANCING ▪ Proprietary load balancer ▪ Selective service denial (based on est. operation cost and server load) DEPLOYMENT ▪ Completely unattended ORS deployment, initialization and configuration
17. PERSISTENCE JBOSS CACHE MAINTENANCE AND UPDATE ▪ JDBM Backend ▪ Cache Loaders SEARCHING ▪ Based on bundled Hibernate Search and Lucene ▪ DAO objects reimplemented using Lucene queries LIFECYCLE ▪ Eviction ▪ Extinction (controlled by application logic) ▪ Caching Strategies (depend on types of entities) REPLICATIONS ▪ Update Protocols ▪ Persistent Replication Buffers (bulk data) ▪ Replication Buffers are stored in the cache in separate namespace
18. PERSISTENCE JBOSS CACHE MAINTENANCE AND UPDATE ▪ JDBM Backend ▪ Cache Loaders SEARCHING ▪ Based on bundled Hibernate Search and Lucene ▪ DAO objects reimplemented using Lucene queries LIFECYCLE ▪ Eviction ▪ Extinction (controlled by application logic) ▪ Caching Strategies (depend on types of entities) REPLICATIONS ▪ Update Protocols ▪ Persistent Replication Buffers (bulk data) ▪ Replication Buffers are stored in the cache in separate namespace
19. PRODUCER/CONSUMER COMMUNICATION PATTERNS ▪ Consumer-based mechanism (indirect messaging) SYNCHRONOUS SERVICE CHANNELS ▪ Synchronous service invocation is built on top of the async messaging DATA PRESENTATION ▪ JBoss Marshalling ▪ River (serialization protocol) MESSAGE-LEVEL BALANCING ▪ For stateless consumers, the load is balanced on message- level ASYNCHRONOUS MESSAGING ▪ Entire system designed around asynchronous messages ▪ No synchronous dependencies PUBLISH/SUBSCRIBE ▪ Topic based publish/subscribe model for selected data ▪ On top of the messaging system
20. SCALABILITY IMPLICATIONS
21. SCALABILITY IMPLICATIONS
22. ORS SERVERS DEPLOYED CUSTOMER REFERENCE ▪ 652 ORS servers ▪ 4 nodes in application cluster in HQ data centers MULTIFUNCTION DEVICES ▪ 2063 devices in the system THROUGHPUT ▪ 1808 GB of print jobs processed (per month) ▪ = 3 217 225 print jobs (per month) JBoss Stack enabled technology allowed us to grow the size of this installation 10x.
23. W/O ORS Print jobs travel between branches and central location. WITH ORS Only metadata are synced between ORS and CML. Based on true customer data. BANDWIDTH/COST CONSIDERATIONS BASED ON A TRUE STORY 24973 $5 994 580 $174 0 5000 10000 15000 20000 25000 30000 Upstream data transfer w/o ORS [GB] Est. Price (Amazon S3, lowest pricing option) Upstream data transfer with ORS [GB] Est. Price (Amazon S3, lowest pricing option)
24. PRINT JOB ROAMING SCALING OUT FURTHER ▪ Seemingly crazy idea. ▪ Who wants to print and then travel to pick the printed documents? EVERYTHING IS POSSIBLE ▪ A customer in Germany has thousands of users roaming among 4000 thousand printers. Every day. LATENCY REQUIREMENTS ▪ Need to deliver print jobs meta-data and data in seconds between ORS servers.
27. DISCOVERY INFINISPAN CLUSTERS ▪ TCP Ping NEAR ROAMING GROUPS BACKEND ▪ Still JDBM DATA TRANSPORT ▪ TCP Multicast ▪ up to 20 nodes ▪ (~ 10 recommended) ▪ UDP Multicast ▪ hundreds of nodes LAYERED CACHES ▪ ORS-local Data Caches (Infinispan-based) ▪ Distributed Caches (Infinispan-based) ▪ Decoupled for lower replication overhead DISTRIBUTED SEARCH ▪ Distributed Caches with “emulated” Searching ▪ Buckets with distributed partial meta-data
29. RULES FLEXIBLE PRINT JOB PRE-PROCESSING ▪ Restrict printing of certain documents ▪ Give priority to certain users ▪ Add watermarks or embed user identity FAILOVER ▪ Redirect jobs between printers (MFDs) ▪ Redirect jobs when printing would be too expensive MONITORING ▪ Monitor printer status and health
30. RULE BASED ENGINE WITH JBOSS DROOLS ▪ Setup rules which enhance pre-processing of print jobs ▪ Trigger/Condition/Action model 1. Rules apply when user prints in his application 2. Rules apply when job is received by the server 3. Rules apply before the job is actually printed 4. Rules apply when user tries to authenticate with the MFD
31. SAFEQ RULE SET ▪ User interface to enable users create conditions/actions and define triggers ▪ Incorporated into SafeQ web interface ▪ Stored in a native format (Drools) ▪ Generated from the frontend whenever user saves the rule set ▪ Parsed from the native format whenever user loads the rule set
32. WIZARD – ACTIONS
34. EXAMPLE
35. SCALE OUT WITH JBOSS CACHE CONCLUSIONS ▪ Decouple application logic to two layers ▪ ORS (print job processing) and CML (management and reporting) SCALE OUT EVEN MORE WITH INFINISPAN ▪ ORS Servers in “near roaming groups” RULE BASED PROCESSING ▪ DROOLS is a powerful rule evaluation system ▪ Nice and easy to use end- user frontend was required ▪ Users can still write DROOLS code directly
36. Questions?