Building Elasticsearch Plugins for Fun and Profit

Elastic Los Angeles User Group - February 21, 2018
Ivan Brusic

Transcript

1. Building Elasticsearch Plugins for Fun and Profit Elastic Los Angeles User Group - February 21, 2018 Ivan Brusic ivan@brusic.com linkedin | github @brusic
2. Elasticsearch Plugins Plugins extend Elasticsearch's features and capabilities First commit in version v0.6.0 on Mar 27, 2010 Code operates with the same server process • same JVM • different classloaders
3. In The Beginning Analysis Site (removed in 5.0) River (deprecated in 1.5, removed in 2.x) Transport Scripting Misc
4. Plugins Today Analysis Discovery Ingest Management Security Snapshot/Restore Store Mapper Alerting API Extension
5. Official Plugins Built as part of the server in the main repository https://github.com/elastic/elasticsearch/tree/master/plugins Examples: • analysis-icu • discovery-ec2 • ingest-attachment
6. Community Plugins Analysis: Hebrew, Vietnamese, Emoji, Network Address Discovery: Kubernetes Ingest: CSV Security: Search Guard, Readonly REST Snapshot/Restore: Openstack Swift Integrations: Drupal, Wordpress
7. Technical Aspects Plugins can be built in any JVM language Requires Java 8+ Use any tools you want
8. Plugin Packaging
9. Plugin Structure Generally a zip file Plugin Structure All plugin files must be contained in a directory called elasticsearch. Plugin descriptor file All plugins must contain a file called plugin-descriptor.properties in the folder named elasticsearch. Optional plugin security file Plugin elasticsearch version must match elasticsearch server version
10. plugin-descriptor.properties ### example plugin for "foo" # # foo.zip <-- zip file for the plugin, with this structure: #|____elasticsearch/ #| |____ <arbitrary name1>.jar <-- classes, resources, dependencies #| |____ <arbitrary nameN>.jar <-- any number of jars #| |____ plugin-descriptor.properties <-- example contents below: # # classname=foo.bar.BazPlugin # description=My cool plugin # version=2.0 # elasticsearch.version=2.0 # java.version=1.7
11. Security SpecialPermission.check(); AccessController.doPrivileged((PrivilegedAction<Void>) () -> { template.execute(writer, params); return null; }); grant { // needed to do crazy reflection permission java.lang.RuntimePermission "accessDeclaredMembers"; }; plugin-security.policy
12. The Code
13. Plugin Interface package org.elasticsearch.plugins; public abstract class Plugin implements Closeable { public get...() }
14. Concrete Plugin Example public class ExampleRescorePlugin extends Plugin implements SearchPlugin { @Override public List<RescorerSpec<?>> getRescorers() { return singletonList( new RescorerSpec<>(ExampleRescoreBuilder.NAME, ExampleRescoreBuilder::new, ExampleRescoreBuilder::fromXContent)); } }
15. Plugin Types ActionPlugin AnalysisPlugin ClusterPlugin DiscoveryPlugin ExtensiblePlugin IngestPlugin MapperPlugin NetworkPlugin RepositoryPlugin ScriptPlugin SearchPlugin
16. Plugin Interfaces - Simple Example public interface IngestPlugin { default Map<String, Processor.Factory> getProcessors(Processor.Parameters parameters) { return Collections.emptyMap(); } }
17. Plugin Interfaces - Complex Example public interface ActionPlugin { default List<ActionHandler<? extends ActionRequest, ? extends ActionResponse>> getActions() {...} default List<GenericAction> getClientActions() {...} default List<ActionFilter> getActionFilters() {...} default List<RestHandler> getRestHandlers(...) {...} default Collection<String> getRestHeaders() {...} default Collection<String> getTaskHeaders() {...} default UnaryOperator<RestHandler> getRestHandlerWrapper(ThreadContext threadContext) {...} }
18. Plugin Discovery Elasticsearch automatically handles plugins via the PluginService Changes from 2.x public final void onModule(SettingsModule settingsModule) {} public final void onModule(ScriptModule module) {} public final void onModule(AnalysisModule module) {} public final void onModule(ActionModule module) {} ...
19. The Build Process
20. The Plugin Plugin Make your life easier: use the gradle plugin! Introduced with the gradle build in 5.x Did I mention you should use the gradle plugin?
21. Example build.gradle apply plugin: 'elasticsearch.esplugin' esplugin { name 'example-rescore' description 'An example plugin implementing rescore and verifying that plugins *can* implement rescore' classname 'org.elasticsearch.example.rescore.ExampleRescorePlugin' }
22. Example build.gradle - Standalone buildscript { repositories { jcenter() } dependencies { classpath "org.elasticsearch.gradle:build-tools:${elasticsearch_version}" } } group 'com.brusic.elasticsearch.plugin.ingest' version '1.0' apply plugin: 'elasticsearch.esplugin' apply plugin: 'idea' esplugin { name 'ingest-aws-rekognition' description 'Ingest processor that uses AWS Rekognition for image analysis' classname 'com.brusic.elasticsearch.ingest.rekognition.IngestAwsRekognitionPlugin' }
23. Pre-commit Tasks test.enabled = false integTest.enabled = false licenseHeaders.enabled = false dependencyLicenses.enabled = false thirdPartyAudit.enabled = false jarHell.enabled = false thirdPartyAudit.enabled = false forbiddenPatterns.enabled = false Useful to disable all at first, and then re-enabling
24. JAR HELL
25. Precommit Tasks - More Than Exclude forbiddenPatterns { exclude '**/*.jpg' } thirdPartyAudit.excludes = [ // joni has AsmCompilerSupport, but that isn't being used: 'org.objectweb.asm.ClassWriter', 'org.objectweb.asm.MethodVisitor', 'org.objectweb.asm.Opcodes', ]
26. my_plugin.foo = qwerty my_plugin.bar = ["xyz", "abc"] elasticsearch.yml Custom Settings public class MyPlugin extends Plugin { @Override public List<Setting<?>> getSettings() { return Arrays.asList( Setting.simpleString(" my_plugin.foo", Property.NodeScope), Setting.listSetting(" my_plugin.bar", Collections.emptyList(), String::valueOf, Property.NodeScope)); } }
27. bin/elasticsearch-keystore add custom.secured Setting<SecureString> secureSetting = SecureSetting.secureString("custom.secured", null); Secure Settings integTestCluster { // Adds a setting in the Elasticsearch keystore // before running the integration tests keystoreSetting 'custom.secured', 'password' } Fixed in Elasticsearch 5.6
28. Fixtures Fixtures are external processes that are used for integration testing • AWS (EC2, S3) • HDFS • Web Servers Separate processes - even other than Java
29. apply plugin: 'elasticsearch.esplugin' esplugin { description 'Example plugin with fixture' classname 'org.elasticsearch.plugin.example.ExamplePlugin' } configurations { exampleFixture } dependencies { exampleFixture project(':test:fixtures:example-fixture') } task exampleFixture(type: org.elasticsearch.gradle.test.AntFixture) { dependsOn project.configurations.exampleFixture executable = new File(project.javaHome, 'bin/java') args '-cp', "${ -> project.configurations.exampleFixture.asPath }", 'example.ExampleFixture', baseDir } integTestCluster { dependsOn exampleFixture }
30. Example Fixture public class ExampleFixture { public static void main(String[] args) throws Exception { // do stuff Thread.sleep(Long.MAX_VALUE); } }
31. Detailed Fixture Example https://github.com/brusic/plugin-with-fixture-skeleton
32. REST Integration Testing public class MyClientYamlTestSuiteIT extends ESClientYamlSuiteTestCase { public MyClientYamlTestSuiteIT(@Name("yaml")ClientYamlTestCandidate testCandidate) { super(testCandidate); } @ParametersFactory public static Iterable<Object[]> parameters() throws Exception { return ESClientYamlSuiteTestCase.createParameters(); } @Override protected ClientYamlTestExecutionContext getAdminExecutionContext() { return super.getAdminExecutionContext(); } }
33. REST YAML Example - do: Indices.create: index: test - do: Indices.put_alias: index: test name: test_alias - do: cat.aliases: {} Headers: Accept: application/yaml - match: {0.alias: test_alias} - match: {0.index: test} - match: {0.filter: "-"} - match: {0.routing\.index: "-"} - match: {0.routing\.search: "-"}
34. Random Gotchas Required files project.licenseFile = project.rootProject.file('LICENSE') project.noticeFile = project.rootProject.file('NOTICE') Bug in 5.x project.ext.projectSubstitutions = ["": ""] HeartBeatEvent NullPointerException Fixed upstream in 6.x Dependencies are not transitive
35. More Gradle run { // Whitelist reindexing from the local node so we can test reindex-from-remote. setting 'reindex.remote.whitelist', '127.0.0.1:*' } test { systemProperty 'es.set.netty.runtime.available.processors', 'false' } test { exclude '**/*CredentialsTests.class' } integTestRunner { systemProperty 'external.address', "${ -> exampleFixture.addressAndPort }" }
36. Plugin changes in 6.x Gradle build process stable since 5.0 Meta plugins: bundling plugins Plugin extensions: build upon existing plugins Painless: add additional whitelisted methods and classes X-Pack: add custom Realms Change in directory structure in upcoming 6.3
37. Should Plugins Be Used? Deployment requires node restart Potential drain on resources Version match: potential jarhell
38. Special thanks to Ryan Ernst @rjernst
39. Questions? Thank you!