Building Elasticsearch Plugins for Fun and Profit Elastic Los Angeles User Group - February 21, 2018 Ivan Brusic [email protected] linkedin | github @brusic 

Elasticsearch Plugins Plugins extend Elasticsearch's features and capabilities First commit in version v0.6.0 on Mar 27, 2010 Code operates with the same server process ● same JVM ● different classloaders 

In The Beginning Analysis Site (removed in 5.0) River (deprecated in 1.5, removed in 2.x) Transport Scripting Misc 

Plugins Today Analysis Discovery Ingest Management Security Snapshot/Restore Store Mapper Alerting API Extension 

Official Plugins Built as part of the server in the main repository https://github.com/elastic/elasticsearch/tree/master/plugins Examples: ● analysis-icu ● discovery-ec2 ● ingest-attachment 

Community Plugins Analysis: Hebrew, Vietnamese, Emoji, Network Address Discovery: Kubernetes Ingest: CSV Security: Search Guard, Readonly REST Snapshot/Restore: Openstack Swift Integrations: Drupal, Wordpress 

Technical Aspects Plugins can be built in any JVM language Requires Java 8+ Use any tools you want 

Plugin Packaging 

Plugin Structure Generally a zip file Plugin Structure All plugin files must be contained in a directory called elasticsearch. Plugin descriptor file All plugins must contain a file called plugin-descriptor.properties in the folder named elasticsearch. Optional plugin security file Plugin elasticsearch version must match elasticsearch server version 

plugin-descriptor.properties ### example plugin for "foo" # # foo.zip <-- zip file for the plugin, with this structure: #|____elasticsearch/ #| |____ .jar <-- classes, resources, dependencies #| |____ .jar <-- any number of jars #| |____ plugin-descriptor.properties <-- example contents below: # # classname=foo.bar.BazPlugin # description=My cool plugin # version=2.0 # elasticsearch.version=2.0 # java.version=1.7 

Security SpecialPermission.check(); AccessController.doPrivileged((PrivilegedAction) () -> { template.execute(writer, params); return null; }); grant { // needed to do crazy reflection permission java.lang.RuntimePermission "accessDeclaredMembers"; }; plugin-security.policy 

The Code 

Plugin Interface package org.elasticsearch.plugins; public abstract class Plugin implements Closeable { public get...() } 

Concrete Plugin Example public class ExampleRescorePlugin extends Plugin implements SearchPlugin { @Override public List> getRescorers() { return singletonList( new RescorerSpec<>(ExampleRescoreBuilder.NAME, ExampleRescoreBuilder::new, ExampleRescoreBuilder::fromXContent)); } } 

Plugin Types ActionPlugin AnalysisPlugin ClusterPlugin DiscoveryPlugin ExtensiblePlugin IngestPlugin MapperPlugin NetworkPlugin RepositoryPlugin ScriptPlugin SearchPlugin 

Plugin Interfaces - Simple Example public interface IngestPlugin { default Map getProcessors(Processor.Parameters parameters) { return Collections.emptyMap(); } } 

Plugin Interfaces - Complex Example public interface ActionPlugin { default List> getActions() {...} default List getClientActions() {...} default List getActionFilters() {...} default List getRestHandlers(...) {...} default Collection getRestHeaders() {...} default Collection getTaskHeaders() {...} default UnaryOperator getRestHandlerWrapper(ThreadContext threadContext) {...} } 

Plugin Discovery Elasticsearch automatically handles plugins via the PluginService Changes from 2.x public final void onModule(SettingsModule settingsModule) {} public final void onModule(ScriptModule module) {} public final void onModule(AnalysisModule module) {} public final void onModule(ActionModule module) {} ... 

The Build Process 

The Plugin Plugin Make your life easier: use the gradle plugin! Introduced with the gradle build in 5.x Did I mention you should use the gradle plugin? 

Example build.gradle apply plugin: 'elasticsearch.esplugin' esplugin { name 'example-rescore' description 'An example plugin implementing rescore and verifying that plugins *can* implement rescore' classname 'org.elasticsearch.example.rescore.ExampleRescorePlugin' } 

Example build.gradle - Standalone buildscript { repositories { jcenter() } dependencies { classpath "org.elasticsearch.gradle:build-tools:${elasticsearch_version}" } } group 'com.brusic.elasticsearch.plugin.ingest' version '1.0' apply plugin: 'elasticsearch.esplugin' apply plugin: 'idea' esplugin { name 'ingest-aws-rekognition' description 'Ingest processor that uses AWS Rekognition for image analysis' classname 'com.brusic.elasticsearch.ingest.rekognition.IngestAwsRekognitionPlugin' } 

Pre-commit Tasks test.enabled = false integTest.enabled = false licenseHeaders.enabled = false dependencyLicenses.enabled = false thirdPartyAudit.enabled = false jarHell.enabled = false thirdPartyAudit.enabled = false forbiddenPatterns.enabled = false Useful to disable all at first, and then re-enabling 

JAR HELL 

Precommit Tasks - More Than Exclude forbiddenPatterns { exclude '**/*.jpg' } thirdPartyAudit.excludes = [ // joni has AsmCompilerSupport, but that isn't being used: 'org.objectweb.asm.ClassWriter', 'org.objectweb.asm.MethodVisitor', 'org.objectweb.asm.Opcodes', ] 

my_plugin.foo = qwerty my_plugin.bar = ["xyz", "abc"] elasticsearch.yml Custom Settings public class MyPlugin extends Plugin { @Override public List> getSettings() { return Arrays.asList( Setting.simpleString(" my_plugin.foo", Property.NodeScope), Setting.listSetting(" my_plugin.bar", Collections.emptyList(), String::valueOf, Property.NodeScope)); } } 

bin/elasticsearch-keystore add custom.secured Setting secureSetting = SecureSetting.secureString("custom.secured", null); Secure Settings integTestCluster { // Adds a setting in the Elasticsearch keystore // before running the integration tests keystoreSetting 'custom.secured', 'password' } Fixed in Elasticsearch 5.6 

Fixtures Fixtures are external processes that are used for integration testing ● AWS (EC2, S3) ● HDFS ● Web Servers Separate processes - even other than Java 

apply plugin: 'elasticsearch.esplugin' esplugin { description 'Example plugin with fixture' classname 'org.elasticsearch.plugin.example.ExamplePlugin' } configurations { exampleFixture } dependencies { exampleFixture project(':test:fixtures:example-fixture') } task exampleFixture(type: org.elasticsearch.gradle.test.AntFixture) { dependsOn project.configurations.exampleFixture executable = new File(project.javaHome, 'bin/java') args '-cp', "${ -> project.configurations.exampleFixture.asPath }", 'example.ExampleFixture', baseDir } integTestCluster { dependsOn exampleFixture } 

Example Fixture public class ExampleFixture { public static void main(String[] args) throws Exception { // do stuff Thread.sleep(Long.MAX_VALUE); } } 

Detailed Fixture Example https://github.com/brusic/plugin-with-fixture-skeleton 

REST Integration Testing public class MyClientYamlTestSuiteIT extends ESClientYamlSuiteTestCase { public MyClientYamlTestSuiteIT(@Name("yaml")ClientYamlTestCandidate testCandidate) { super(testCandidate); } @ParametersFactory public static Iterable parameters() throws Exception { return ESClientYamlSuiteTestCase.createParameters(); } @Override protected ClientYamlTestExecutionContext getAdminExecutionContext() { return super.getAdminExecutionContext(); } } 

REST YAML Example - do: Indices.create: index: test - do: Indices.put_alias: index: test name: test_alias - do: cat.aliases: {} Headers: Accept: application/yaml - match: {0.alias: test_alias} - match: {0.index: test} - match: {0.filter: "-"} - match: {0.routing\.index: "-"} - match: {0.routing\.search: "-"} 

Random Gotchas Required files project.licenseFile = project.rootProject.file('LICENSE') project.noticeFile = project.rootProject.file('NOTICE') Bug in 5.x project.ext.projectSubstitutions = ["": ""] HeartBeatEvent NullPointerException Fixed upstream in 6.x Dependencies are not transitive 

More Gradle run { // Whitelist reindexing from the local node so we can test reindex-from-remote. setting 'reindex.remote.whitelist', '127.0.0.1:*' } test { systemProperty 'es.set.netty.runtime.available.processors', 'false' } test { exclude '**/*CredentialsTests.class' } integTestRunner { systemProperty 'external.address', "${ -> exampleFixture.addressAndPort }" } 

Plugin changes in 6.x Gradle build process stable since 5.0 Meta plugins: bundling plugins Plugin extensions: build upon existing plugins Painless: add additional whitelisted methods and classes X-Pack: add custom Realms Change in directory structure in upcoming 6.3 

Should Plugins Be Used? Deployment requires node restart Potential drain on resources Version match: potential jarhell 

Special thanks to Ryan Ernst @rjernst 

Questions? Thank you!