Beyond Language Cloud: Technology and Lessons

Creating a cloud platform it’s a complex task, it’s about dealing with rapid changes, scalability, API’s, security, discovery, infrastructure and many more. In summary lots of moving parts that must hang together.
This session will uncover details about what it takes to build a cloud platform looking at key technical concepts, lessoned learned and real data taken while building language cloud.

Transcript

1. Beyond Language Cloud Technology and Lessons Romulus Crisan June 2015
2. 2 Image placeholder Click on image icon Browse to image you want to add to slide Evangelize ◦ Engage with developers ◦ Speaking/ Presentations / Demonstrations ◦ Design & Develop ◦ Connect with SDL Frederic Harper – Success in Programing: How to Gain Recognition, Power, and Influence Through Personal Branding Someone who gives love to developers
3. 3 Image placeholder Click on image icon Browse to image you want to add to slide Agenda ◦ What is Language Cloud? ◦ Lessons learned building Language Cloud ◦ Demo ◦ Outro
4. What is Language Cloud?
5. 5 Language Cloud ◦ Translation services ◦ Machine translation ◦ API’s The only platform for all your translation needs
6. 6 Language is key in the Internet 70% of internet traffic comes from non-English native countries Only 27% of Internet users speak English 75% of Facebook users outside of US 34% of Twitter messages are in English
7. 7 Machine translation ◦ Available for several decades ◦ Set off in the 50s during cold war ◦ Linguistic approaches ◦ Statistical approaches ◦ Development in academia, government and industry
8. 8 Machine translation usage Gisting Post-editing Speech Translation Search and Discovery Sentiment analysis
9. 9 Language cloud in action 2.3 billion words per month 8 million translations per day 221million translations per month
10. Lessons learned building Language Cloud
11. 11 Service Discovery
12. 12 Service discovery ◦ Custom solution ◦ No resiliency ◦ No instance level granularity at the application level ◦ Discovery concern held in two systems ◦ Standardization Finding the physical location of a running service instance
13. 13 Service discovery - Solution ◦ Eureka Server – Discovery registry management – Detects unexpected client failures – Offers a GUI dashboard and Rest info ◦ Eureka Client – Register/unregister service instance with Eureka server – Client side caching of registry information ◦ DNS – Server list management – Enables discovery of Eureka servers – The only static resource Finding the physical location of a running service instance
14. 14 Service Discovery - Architecture
15. 15 Telemetry
16. 16 Telemetry ◦ Not enough information to bill based on usage ◦ No event notifications ◦ Not enough information for autoscalling The basics for instrumentation
17. 17 Telemetry - solution ◦ Dropwizzard metrics for code insights ◦ Graphite dashboard The basics for instrumentation
18. 18 Telemetry- View
19. 19 Centralized logging
20. 20 Centralized logging ◦ Not able to search ◦ Not able to visualize ◦ Not able to build statistics Get the most out of your loggs
21. 21 Centralized logging - Solution ◦ Rsyslog sender – On each service machine – Read logs – Push to central log machine – Mark each log with a tag ◦ Rsyslog receiver – On the centralized log machine ◦ Logstash – Reads and parses the service logs – Define parsing rules – Structured output sent to Graylog ◦ Graylog web interface to view, query and build statisctics Get the most out of your loggs
22. 22 Centralized logging- Architecture
23. 23 Demo
24. 24 Resources ◦ Netflix Eureka - http://bit.ly/1KYMcPl ◦ Dropwizzard Metrics - http://bit.ly/1dZHt2w ◦ Graphite - http://bit.ly/1cIjvaz ◦ Rsyslog - http://bit.ly/1JBvg0V ◦ Logstash - http://bit.ly/1GwqtgA ◦ Graylog - http://bit.ly/1cIjLGz
25. 25 Questions? https://community.sdl.com/develo pers/language-developers/ @cromica_82