Ruby on Rails/Sinatra for Enterprise Smart Energy Cloud Service

RubyConfChina 2012

Transcript

1. Ruby on Rails/Sinatra for Enterprise Smart Energy Cloud Service Nov 17, 2012
2. About me •  蔡望勤 Jesse Cai http://caiwangqin.com http://twitter.com/caiwangqin •  Ruby Developer since 2006 •  The Founder of China on Rails Community http://chinaonrails.com
3. Projects
4. Video
6. Overview Web portal - Ruby on Rails - Nodejs - Highcharts Mobile app - API with Sinatra - Sencha - PhoneGap - Google Chart Tools
7. Cloud Service Advantages •  Reading energy usage in anytime anywhere •  Remote control •  Flexible policy to reduce energy and save money •  Be able to communication with 3rd parties •  Forecast monthly energy bills •  Remote firmware upgrade for device •  Remote monitoring and management of entire network
8. Getting start
9. •  Team build •  Process •  Tools •  Communication
10. Team build •  Keep team size small, 4-6 developers is ideal •  Have at least 2-3 ruby/rails/front-end expert on the team •  Auto test from the beginning
11. Process •  Paired Programming if needed •  Daily morning meeting •  Weekly sprint plan •  Dedicated graphic designer, and a Product Manager
12. Tools
13. Communication •  Continuous Integration server runs all automated tests (Hudson, Jenkins are good) •  Use Chat (Skype) to notify team about check-ins, deploys or failed builds
14. Evolve Project
15. •  Prototype & Design •  Project management & Communication •  Implementation •  Deployment •  Monitoring
16. Prototype & Design
17. Axure
18. Keynote
19. Sketches
20. Project management & Communication
21. Redmine •  Multiple projects support •  Flexible role based access control •  Flexible issue tracking system •  Gantt chart and calendar •  Feeds & email notifications •  Time tracking •  Custom fields for issues, time-entries, projects and users •  SCM integration (SVN, CVS, Git, Mercurial, Bazaar and Darcs)
23. Common Libraries •  Rspec for testing •  Devise for authentication •  Twitter Bootstrap for early UI •  Capistrano for deploy
24. Database MongoDB •  almost table-like •  full index support •  replication & high availability •  high write performance
25. Implementation
26. New Rails Project rails new my-app cd my-app rails s
27. TDD •  Rspec - controllers - models - mailers - routing - views •  Code review •  Continuous Integration
28. RubyGems •  gem - Package manager for Ruby •  Ruby community's gem hosting - http://rubygems.org/ - http://ruby.taobao.org/
30. Deployment
31. Version management Release and deploy continually
32. Architecture •  Varnish - web application accelerator •  Nginx - static assets •  Passenger - application server for Ruby (Rack) •  REE - Ruby Enterprise Edition •  Cloud - Amazon EC2 - Amazon CloudFront
34. Cloud
35. Monitoring
37. Cacti & Plugins & Templates Plugins Monitor Threshold Templates Memcached MongoDB Redis Nginx Varnish Template IO/Memory/ Process
38. New Relic
39. Growing Up •  Could be all in one box in the beginning •  Site usage grows, Responses get slow.
40. Scaling Out •  Split into multiple App servers •  HAProxy to distribute load
41. Scaling Up •  Memcached •  Redis •  Action Cache •  Upgrade EC2 Instance to achieve - High Memory - High ECU - High I/O Performance
42. Deployments and Downtime •  DNS round robin with two balancers and two public IP addresses to achieve almost Zero downtime. (because passenger restart can be several minutes for big application) •  MongoDB is schema less, do not need migrate •  Config MongoDB replica sets support auto failover
44. Performance tuning Passenger: - Set instance number depends on server memory MongoDB: - At least 3 replSet instances for auto failover - Huge difference without index on query (when disk I/O in high load, probably you are querying without index in big collections)
45. More Problems •  Huge concurrent write data •  Real time power consumption data reading •  Data integrity between cloud server and local devices •  Control device behind router •  Security •  License •  Global customers
47. Bulk Insert •  Dedicated MongoDB for Devices upload usage data •  Minimize data package in json, e.g: - {"device_serial": "xxx"} -> {"s": "xxx"} •  Replica set secondary use for web portal and API - Write on Primary, Read on Secondary - Note there is latency between Primary to Secondary
48. Real time •  Device post minutely power consumption data to server every 5 minutes - for power consumption data, almost real time
49. Data Integrity •  Device connect to server through ethernet/ wifi/3G/LTE, the network may unstable •  Device may powered off sometimes - Local device store data for a month - Background ruby fork job scan data missing, and request data from local device when it become online. - Cron task monitor this process every minute, start it when not exist in processes.
50. Control device behind router •  UDP heartbeat from device to server, it benefit on: - Notify device activity and network activity - Make server be able to send UDP message to device any time •  Delayed_job in background for command queue •  Retry when no feedback from end device
51. Security •  Physical Security •  Data Protection and API Protection •  Authentication, Authorization and Accounting •  Secure Coding •  Threat defense - CRSF - XSS - SQL Injection •  Real-time monitoring, management and correlation
52. License •  License Rule •  Encrypt Code - rubyencoder (http://rubyencoder.com/) - JRuby?
53. Globalize •  Dyn - Managed DNS, Outsourced DNS & Anycast DNS
54. The End Thanks!