Index your Salesforce data in Elasticsearch using the Streaming API

In this talk I show how to integrate Salesforce and Elasticsearch using heroku and the Streaming API.

Transcript

1. Index your Salesforce data in ElasticSearch using the Streaming API Vicente Mundim <vicente.mundim@dtmtec.com.br> quarta-feira, 30 de abril de 14
2. Why ElasticSearch? quarta-feira, 30 de abril de 14
3. Why ElasticSearch? Fast quarta-feira, 30 de abril de 14
4. Why ElasticSearch? Scalable Fast quarta-feira, 30 de abril de 14
5. Why ElasticSearch? Scalable Schema Free Fast quarta-feira, 30 de abril de 14
6. Why ElasticSearch? Scalable Schema Free Easy to setup Fast quarta-feira, 30 de abril de 14
7. Why Streaming API? quarta-feira, 30 de abril de 14
8. Why Streaming API? Less API calls quarta-feira, 30 de abril de 14
9. Why Streaming API? Near Real Time Less API calls quarta-feira, 30 de abril de 14
10. Why Streaming API? Near Real Time No Apex Code! Less API calls quarta-feira, 30 de abril de 14
11. Why Streaming API? Near Real Time No Apex Code! Integration via Ruby client on Heroku Less API calls quarta-feira, 30 de abril de 14
12. Restforce quarta-feira, 30 de abril de 14
13. Restforce Ruby client for Salesforce API quarta-feira, 30 de abril de 14
14. Restforce Supports Realtime API Ruby client for Salesforce API quarta-feira, 30 de abril de 14
15. Restforce Supports Realtime API Ruby Way Ruby client for Salesforce API quarta-feira, 30 de abril de 14
16. Restforce Supports Realtime API Ruby Way OAuth Authentication Ruby client for Salesforce API quarta-feira, 30 de abril de 14
17. Create a PushTopic require './environment' client = Restforce.new client.create! 'PushTopic', { ApiVersion: Restforce.configuration.api_version, Name: "AllAccounts", NotifyForOperationCreate: 'true', NotifyForOperationDelete: 'true', NotifyForOperationUpdate: 'true', NotifyForOperationUndelete: 'true', NotifyForFields: 'All', Query: "select Id, Name from Account" } create_push_topic.rb quarta-feira, 30 de abril de 14
18. Listen to PushTopic notifications require './environment' client = Restforce.new elastic_search_client = Elasticsearch::Client.new({ hosts: ENV['BONSAI_URL'] || "localhost:9200" }) EM.run do client.faye.set_header 'Authorization', "OAuth #{client.authenticate!.access_token}" client.subscribe 'AllAccounts' do |message| elastic_search_client.index( index: 'salesforce', type: 'Account', id: message['sobject']['Id'], body: message['sobject'] ) end end indexer.rb quarta-feira, 30 de abril de 14
19. Setup Ruby App indexer: bundle exec ruby indexer.rb Procfile source 'https://rubygems.org' ruby '2.1.1' gem 'restforce', git: 'https://github.com/ejholmes/restforce.git' gem 'elasticsearch' gem 'eventmachine' gem 'faye', '0.8.9' gem 'debugger' Gemfile require 'bundler/setup' Bundler.require :default Restforce.configure do |config| config.compress = true config.instance_url = ENV['SALESFORCE_INSTANCE_URL'] || 'https://na10.salesforce.com' config.api_version = '29.0' end Environment quarta-feira, 30 de abril de 14
20. Setup Heroku APP Console > heroku create my-streaming-app-name --addons bonsai > heroku config:add SALESFORCE_CLIENT_ID=<client-id> SALESFORCE_CLIENT_SECRET=<A-SECRET> SALESFORCE_SECURITY_TOKEN=<security-token> SALESFORCE_USERNAME=<your-username> SALESFORCE_PASSWORD=<your-user-password> > git push heroku master > heroku ps:scale web=0 indexer=1 quarta-feira, 30 de abril de 14
21. Create or Update an Account on Salesforce now quarta-feira, 30 de abril de 14
22. Check results Console > curl http://<YOUR_BONSAI_URL>/salesforce/Account/_search?pretty { "took" : 1, "timed_out" : false, "_shards" : { ... }, "hits" : { "total" : 1, "max_score" : 0.5, "hits" : [ { "_index" : "salesforce", "_type" : "Account", "_id" : "001F0000015ea7TIAQ", "_score" : 0.5, "_source" : {"Name":"Some Account","Id":"001F0000015ea7TIAQ"} } ] } } quarta-feira, 30 de abril de 14
23. Conclusion quarta-feira, 30 de abril de 14
24. Conclusion No Apex coding! quarta-feira, 30 de abril de 14
25. Conclusion Easy setup with Restforce No Apex coding! quarta-feira, 30 de abril de 14
26. Conclusion Easy setup with Restforce Happy developers :) No Apex coding! quarta-feira, 30 de abril de 14
27. ElasticSearch + Streaming API sample project http://github.com/dtmtec/streaming-api-indexer quarta-feira, 30 de abril de 14