Fun with 3rd Party Services

Third party integrations can supercharge any application. In this talk we’ll walk through EE third party service integration best practices, how to test locally, and how everything functions together. Real world examples will be shown with integration with Elasticsearch, SQS, and a throttling library all on one project. We’ll talk about how to keep these integrations isolated and lessons learned all the way.

Transcript

1. Fun with 3rd Party Services in EE Khaliq Gant Friday, 2018/10/26 Vector Media Group
2. • Khaliq (Ka-leak) • Head of Development, 
 Europe • Parlez-vous français? How do you pronounce that?
3. Definition of “Fun” • Fun === working • Debugging + Tools
4. Why Services? • Battled tested • Save development time
5. Services SQS Custom Throttling Library / Redis Elasticsearch
6. Overview • Testing • Configuring • Play nicely with EE • Debugging • Complexities
7. Overarching Principles • Logging everywhere • Localized dependency management (composer) • Libraries for services • Service specific config • Locally testable
8. • Serving ES • Permanent or ephemeral? • Indexing vs. Querying
9. Our Setup { "require": { "elasticsearch/elasticsearch": "~1.3", "vmg/vmg-es-manager": "^2.1" }, "repositories": [ { "type": "vcs", "url": "git@vmg.git.beanstalkapp.com:/vmg/vmg-es-manager.git" } ], "scripts": { "post-install-cmd": "find vendor/. -name '.gi*' -type f -delete", "pre-update-cmd": "find vendor/. -name '.gi*' -type f -delete", "post-update-cmd": "find vendor/. -name '.gi*' -type f -delete" } }
10. Getting ES Results $config['resource_router'] = array( 'api/:any' => 'api/index', ); {exp:vmg_api:get_response}
11. Getting ES Results public function get_response() { if( $this->req_method === 'GET' ) { ee()->vmg_api_lib->set_input($_GET); } $this->output = $this->route_request(); $host = "http" . ((isset($_SERVER['HTTPS']) && $_SERVER['HTTPS'] != 'off') ? "s" : "") . "://". $_SERVER["HTTP_HOST"]; @header('Content-type: application/json'); echo trim($this->output); exit(); }
12. WHY?
13. Specific Config $config['elasticsearch'] = [ 'host' => [ getenv('ES_HOST') ], 'auth' => [ 'user' => getenv('ES_USERNAME') ?: NULL, 'pass' => getenv('ES_PASSWORD') ?: NULL, 'type' => getenv('ES_AUTHTYPE') ?: NULL, ], 'logging' => getenv('ES_LOGGING') ?: false, 'debug' => getenv('ES_DEBUG') ?: false, 'index_name' => getenv('ES_INDEX') ?: "vmg", 'protocol' => getenv('ES_PROTOCOL') ?: "http" ];
14. class Properties_model extends CI_Model { public function __construct() { parent::__construct(); } public function sort_options() { return [ 'size', 'from', 'sort', 'orderby' ]; } public function allowed_sorts() { return [ 'asc', 'desc' ]; } }
15. version: '2' services: elasticsearch: container_name: vmg-es mem_limit: 25GB image: ${ECR_REPO}/elasticsearch ports: - "9200:9200" - "9300:9300" environment: ES_JAVA_OPTS: "-Xmx10g -Xms10g" ES_ENDPOINT: ${ES_ENDPOINT} ES_CLUSTER_NAME: ${ES_CLUSTER_NAME} ES_AVAILABILITY_ZONES: ${ES_AVAILABILITY_ZONES} ES_EC2_GROUP: ${ES_EC2_GROUP} ES_HOST: ${ES_HOST} ES_MASTER_NODES: ${ES_MASTER_NODES} ES_IS_SEARCH_NODE: "true" volumes: - esdata:/usr/share/elasticsearch/data ulimits: nofile: soft: 65536 hard: 65536 memlock: soft: -1 hard: -1 volumes: esdata: driver: local
16. Monitoring • Kibana • Status checks and notifications
18. ES Lessons Learned • Config values into ENV variables • Logic abstraction • Essential to be to able to run locally
19. SQS Usage • Nightly and daily syncs • Pipeline uses SQS to pass data and IDs • Constantly items on the queue • What are our EE needs?
20. Solution • Supervisor [program:sync_queue] command=/usr/bin/php /var/www/vhosts/_app/system/user/addons/vmg_api/scripts/ process_sync_queue.php -h staging-ee4.vmg.com 2>&1 | /usr/bin/logger -t process_sync_queue stdout_logfile=/var/www/vhosts/_app/system/user/addons/vmg_api/logs/supervisord-stdout.log directory=/var/www/vhosts/_app/system/user/addons/vmg_api/scripts autorestart=true autostart=true process_name=%(program_name)s%(process_num)s numprocs=3 umask=022 startsecs=0 user=vmg
21. Solution • Bootstrap EE to have the convenience of 
 ee()-> • Queue handler
23. $config['sqs_region'] = getenv('SQS_REGION'); $config['sqs_version'] = getenv('SQS_VERSION'); $config['sqs_key'] = getenv('SQS_KEY'); $config['sqs_secret'] = getenv('SQS_SECRET'); $config['sqs_queue_url'] = getenv('SQS_QUEUE_URL');
27. /** * Send Message * @use add message to the queue * @see http://docs.aws.amazon.com/aws-sdk-php/v3/api/api- sqs-2012-11-05.html#sendmessage */ public function sendMessage($message, $delay=0) { try{ $result = $this->client->sendMessage([ 'MessageBody' => json_encode($message), 'QueueUrl' => ee()->config->item('sqs_queue_url'), 'DelaySeconds' => $delay ]); } catch(Exception $e) { ee()->vmg_rollbar_lib->report_message($e, 'E_ERROR'); } }
28. Consistencies • Logging at every step • Error reporting everywhere • Separation of responsibilities • Let the service do what it does best
29. Local Development
31. Lessons Learned With SQS • Simulate and test end to end • Silent errors are evil • Code abstraction for easy tweaks
32. Throttling Needs • Prevent API abuse • Logic past the built in EE throttling • API library calls throttle library before response
33. Library
34. { "autoload": { "psr-4": { "VmgApi\\": "src/" } }, "require-dev": {}, "minimum-stability": "dev", "require": { "aws/aws-sdk-php": "3.19.0", "guzzlehttp/psr7": "1.3.1", "davedevelopment/stiphle": "^0.9.2", "predis/predis": "^1.1", "illuminate/support": "^5.4" } }
35. private function route_request() { // set the routes for the segments $method = ee()->uri->segment(2); $action = ee()->uri->segment(3); // make sure this route is method if (!in_array($method, $this->routes)) { return $this->outputError(404, 'Resource not found'); } $throttle = ee()->vmg_throttle_lib->throttle($method); if ($throttle->isThrottled()) { header("HTTP/1.1 429 Too Many Requests", true, 429); header('Content-Type: application/json'); exit(json_encode(array( 'success' => false, 'error' => $throttle->toArray(), ))); } // call the api libe to handle this request return ee()->vmg_api_lib->$method(); }
38. Local • Local redis • brew install redis • Swap out config
39. Getting Started With Services • Local • Config • Wrappers
40. Final Lessons • Debugging with right tools • Monitoring • Logging and error reporting
41. Final Lessons • Tools++ == Complexity++ • Testable and isolated components • When something breaks…
42. Something broke.. • What changed recently? • Error reporting and logs • Services up • Services -> EE OK? • EE method problem? • Bad data?
43. SQS Debugging….
44. Isolated Code • Testability • Dependency management, allows us to take
 advantage of PHP ecosystem • Easier development iteration cycle
45. Fun…right? • Untangled code • Process and checklist for debugging • Fun === no worries and working system • How could this be expanded upon?
46. Thank You! Please leave feedback: https://eeconf.com/speaker-feedback @khaliqgant github.com/khaliqgant