Plugin development for elasticsearch

Slides are also available here http://slides.com/salyh/plugin_development_elasticsearch

Transcript

1. Plugin development for Plugin development for elasticsearch elasticsearch Hendrik Saly
2. Simply Simply explained explained by geek & poke
3. Why plugins Why plugins Getting started with a simple plugin Getting started with a simple plugin How they fit into Elasticsearch architecture How they fit into Elasticsearch architecture Examples walkthrough Examples walkthrough Development Development Testing Testing Distribution Distribution Limitations Limitations Not covered Not covered
4. Why plugins Why plugins
5. If there is no configuration option for your need If there is no configuration option for your need If you need unexposed (but public) API If you need unexposed (but public) API To enforce specific behaviour To enforce specific behaviour If scripting is simply not enough If scripting is simply not enough Site plugin vs JVM Plugin Site plugin vs JVM Plugin Here Here Plugin == JVM Plugin Plugin == JVM Plugin
6. For example: For example: Creating a new REST endpoint Creating a new REST endpoint Custom mapping Custom mapping Custom analyzer Custom analyzer Transport layer manipulation Transport layer manipulation HTTP layer manipulation HTTP layer manipulation
7. Getting started with a Getting started with a simple plugin simple plugin
8. Plugins are written in Java (1.7+) Plugins are written in Java (1.7+) They live within a node (or transport client) They live within a node (or transport client) They can access all internals They can access all internals They can make a node stop working (or They can make a node stop working (or even worse) even worse)
9. public class MyPlugin extends AbstractPlugin { private final Settings settings; public MyPlugin(Settings settings) { this.settings = settings; } @Override public String name() { return "MyPlugin"; } @Override public String description() { return "This is the description for my plugin"; } } Extend AbstractPlugin class Extend AbstractPlugin class ... but it does not anything useful yet ... but it does not anything useful yet
10. public class MyPlugin extends AbstractPlugin { ... //Define own modules @Override public Collection<Class<? extends Module>> modules() { return ImmutableList.of(MyModule.class); } //Define own services @Override public Collection<Class<? extends LifecycleComponent>> services() { return ImmutableList.of(MyService.class); } } So lets define our own modules So lets define our own modules or services which do something meaningful or services which do something meaningful
11. public class MyPlugin extends AbstractPlugin { ... //Define own modules @Override public void processModule(Module module) { if ((module instanceof RestModule)) { ((RestModule)module).addRestAction(MyRestAction.class); } if ((module instanceof ???)) { ((???)module).doModuleSpecificStuffHere(); } } } or do something with core modules or do something with core modules to accomplish your needs to accomplish your needs
12. plugin=org.company.es.plugins.MyPlugin Add es-plugin.properties to src/main/resources Add es-plugin.properties to src/main/resources
13. Jar it, then zip it with all dependencies included Jar it, then zip it with all dependencies included Maven can do that for you Maven can do that for you Main artefact Your code Dependecies
14. How they fit into How they fit into Elasticsearch Elasticsearch architecture architecture
15. Elasticsearch is build upon Elasticsearch is build upon 2.0 2.0 pronounced "juice" pronounced "juice" Dependency injection Dependency injection Inversion of control Inversion of control Guice code is copied in ES codebase (slightly Guice code is copied in ES codebase (slightly modified for lower memory footprint) modified for lower memory footprint) So elasticsearch is build upon guice modules So elasticsearch is build upon guice modules Google guice Google guice
16. ... and services ... and services Modules start up, but they don't shut down Modules start up, but they don't shut down Modules should be tested Modules should be tested Modules can be overridden Modules can be overridden Services have a lifecycle Services have a lifecycle because of because of
17. Interesting core modules (for plugins) Interesting core modules (for plugins) ActionModule ActionModule AnalysisModule AnalysisModule HttpServerModule HttpServerModule RestModule RestModule ScriptModule ScriptModule TransportModule TransportModule
18. Interesting core services (for plugins) Interesting core services (for plugins) ClusterService ClusterService TransportService TransportService NettyTransport NettyTransport IndicesService IndicesService Read more https://www.found.no/foundation/elasticsearch-internals/
19. Examples walkthrough Examples walkthrough https://github.com/salyh/elasticsearch-plugin-examples
20. Audit Plugin Audit Plugin Capture index changes Capture index changes Write them into elasticsearch Write them into elasticsearch (Make them visible through kibana) (Make them visible through kibana) Transport SSL/TLS Plugin Transport SSL/TLS Plugin Implement and enforce SSL/TLS encryption Implement and enforce SSL/TLS encryption for transport protocol for transport protocol
21. Development Development
22. by geek & poke
23. Audit Plugin Audit Plugin AuditService registers listener on AuditService registers listener on ShardIndexingService ShardIndexingService Write changes into elasticsearch via Write changes into elasticsearch via BulkProcessor BulkProcessor
24. public class AuditService extends AbstractLifecycleComponent<AuditService>{ ... @Inject public AuditService(Settings settings, IndicesService indicesService, Client client, ClusterService clusterService, TransportFlushAction tfa) { super(settings); this.indicesService = indicesService; this.clusterService = clusterService; ... } @Override protected void doStart() throws ElasticsearchException { ... this.indicesService.indicesLifecycle().addListener(auditIndicesLsListener); } Define a new service called AuditService Define a new service called AuditService Register a indices lifecycle listener Register a indices lifecycle listener Dependency Injection by guice
25. IndicesLifecycle.Listener auditIndicesLsListener = new IndicesLifecycle.Listener() { @Override public void afterIndexShardStarted(final IndexShard indexShard) { if (indexShard.routingEntry().primary() && !indexShard.indexService().index().name().equals(auditIndexName)) { AuditIndexOpListener auditListener = new AuditIndexOpListener(indexShard); indexShard.indexingService().addListener(auditListener); } } If a shard starts, register If a shard starts, register an IndexingOperationListener an IndexingOperationListener
26. class AuditIndexOpListener extends IndexingOperationListener { private final IndexShard indexShard; public AuditIndexOpListener(IndexShard indexShard) { this.indexShard = indexShard; } @Override public void postIndex(Index index) { String nodeName = indexShard.nodeName(); String indexName = indexShard.indexService().index().name(); Change change = new Change(nodeName, indexName, ...); //store it in elasticsearch (or anywhere else) IndexRequest ir = new IndexRequest().source(change.sourceAsMap()); addToBulkIndex(ir); } } Let the IndexingOperationListener store the Let the IndexingOperationListener store the change in elasticsearch (or anywhere else) change in elasticsearch (or anywhere else)
27. public class AuditModule extends AbstractModule { @Override protected void configure() { //nothing to bind here } @Override public void processModule(Module module) { if ((module instanceof ActionModule)) { ((ActionModule)module).registerAction(FlushAction.INSTANCE, TransportFlushAction.class); } if ((module instanceof RestModule)) { ((RestModule)module).addRestAction(AuditRestAction.class); } } } Define the AuditModule (not covered today) Define the AuditModule (not covered today) API for flushing outstanding bulk requests API for flushing outstanding bulk requests
28. public class AuditPlugin extends AbstractPlugin { ... public String name() { return "AuditPlugin"; } public String description() { return "This is the description for the AuditPlugin"; } @Override public Collection<Class<? extends Module>> modules() { return ImmutableList.of(AuditModule.class); } @Override public Collection<Class<? extends LifecycleComponent>> services() { return ImmutableList.of(AuditService.class); } } Last but not least define the AuditPlugin itself Last but not least define the AuditPlugin itself
29. plugin=de.saly.es.example.audit.plugin.AuditPlugin Add es-plugin.properties Add es-plugin.properties
30. Demo Demo
31. Transport SSL Plugin Transport SSL Plugin Elasticsearch uses Netty for tcp Elasticsearch uses Netty for tcp communication communication Extend NettyTransport and inject SslHandler Extend NettyTransport and inject SslHandler into pipeline into pipeline Replace original Netty transport in Replace original Netty transport in TransportModule TransportModule Expose some SSL/TLS related informations Expose some SSL/TLS related informations through a new REST API through a new REST API
32. public class SSLNettyTransport extends NettyTransport { @Override public ChannelPipelineFactory configureServerChannelPipelineFactory(String name, Settings settings) { return new SSLServerChannelPipelineFactory(this, name, settings, this.settings); } protected class SSLServerChannelPipelineFactory extends SecureServerChannelPipelineFactory { public SSLServerChannelPipelineFactory(NettyTransport nettyTransport, String name, Settings sslsettings, Settings essettings) { super(nettyTransport, name, sslsettings); } @Override public ChannelPipeline getPipeline() throws Exception { ChannelPipeline pipeline = super.getPipeline(); SSLEngine engine = ... SslHandler sslHandler = new SslHandler(engine); pipeline.addFirst("ssl_server", sslHandler); return pipeline; } } } Extend NettyTransport and do some netty Extend NettyTransport and do some netty pipeline magic pipeline magic
33. public class TSslPlugin extends AbstractPlugin { ... public void onModule(TransportModule transportModule) { transportModule.setTransport(SSLNettyTransport.class, name()); } public void onModule(RestModule restModule) { restModule.addRestAction(TSslRestAction.class); } } Replace transport in TransportModule Replace transport in TransportModule Register a custom rest action for ssl infos Register a custom rest action for ssl infos
34. public class TSslRestAction extends BaseRestHandler{ @Inject public TSslRestAction(Settings settings, Client client, RestController controller) { super(settings, controller, client); controller.registerHandler(Method.GET, "/_tssl/state", this); controller.registerHandler(Method.POST, "/_tssl/state", this); } @Override protected void handleRequest(RestRequest request, RestChannel channel, Client client) throws Exception { XContentBuilder builder = JsonXContent.contentBuilder(); builder.startObject(); builder.field("enabled_protocols", SecurityUtil.ENABLED_SSL_PROTOCOLS); builder.field("enabled_chipers", SecurityUtil.ENABLED_SSL_CIPHERS); builder.endObject(); channel.sendResponse(new BytesRestResponse(RestStatus.OK, builder)); } } Register a handler for an endpoint Register a handler for an endpoint Send a JSON response Send a JSON response
35. plugin=de.saly.es.example.tssl.plugin.TSslPlugin version=${project.version} Add es-plugin.properties Add es-plugin.properties
36. Call the new REST endpoint Call the new REST endpoint
37. Testing Testing
38. Unittests Unittests Randomized testing by Randomized testing by extending ElasticsearchIntegrationTest extending ElasticsearchIntegrationTest Extend Extend Start nodes directly in the test method Start nodes directly in the test method Test in real elasticsearch node/cluster Test in real elasticsearch node/cluster https://github.com/tlrx/elasticsearch-test https://github.com/tlrx/elasticsearch-test MultiJvmUnitTest MultiJvmUnitTest DevOps magic does help here DevOps magic does help here Always test with multiple nodes (cluster scenario)
39. If you want use the randomized testing suite, If you want use the randomized testing suite, include test dependencies in pom include test dependencies in pom <dependency> <groupId>org.apache.lucene</groupId> <artifactId>lucene-test-framework</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.elasticsearch</groupId> <artifactId>elasticsearch</artifactId> <type>test-jar</type> <scope>test</scope> </dependency>
40. Distribution Distribution
41. Use Maven for building plugins Use Maven for building plugins because ES is using Maven because ES is using Maven Plugins are best published in Plugins are best published in Or distributed as .zip file Or distributed as .zip file maven central maven central # Install from maven central bin/plugin --install de.saly/elasticsearch-sample-plugin-tssl/1.1 # Install from a .zip file bin/plugin --url file:///Users/.../elasticsearch-sample-plugin-tssl-1.1.zip \ --install elasticsearch-sample-plugin-tssl
42. src/main/assemblies/plugin.xml src/main/assemblies/plugin.xml <?xml version="1.0"?> <assembly> <id>plugin</id> <formats> <format>zip</format> </formats> <includeBaseDirectory>false</includeBaseDirectory> <dependencySets> <dependencySet> <outputDirectory>/</outputDirectory> <useProjectArtifact>true</useProjectArtifact> <useTransitiveFiltering>true</useTransitiveFiltering> <excludes> <exclude>org.elasticsearch:elasticsearch</exclude> </excludes> </dependencySet> </dependencySets> </assembly>
43. Use maven-assembly-plugin Use maven-assembly-plugin <plugin> <artifactId>maven-assembly-plugin</artifactId> <configuration> <appendAssemblyId>false</appendAssemblyId> <outputDirectory>${project.build.directory}/releases/</outputDirectory> <descriptors> <descriptor>${basedir}/src/main/assemblies/plugin.xml</descriptor> </descriptors> </configuration> <executions> <execution> <phase>package</phase> <goals> <goal>single</goal> </goals> </execution> </executions> </plugin> !! !!
44. Plugin related configuration options in Plugin related configuration options in elasticsearch.yml elasticsearch.yml # Load this plugin from the classpath # (does only makes sense if plugins.load_classpath_plugins is false) # Order is respected plugin.types: org.company.MyPlugin,com.guhgle.FantasticPlugin # If a plugin listed here is not installed for current node, the node will not start. plugin.mandatory: mapper-attachments,lang-groovy # If its true (which is the default) load all plugins which are in the classpath # No order guarantee plugins.load_classpath_plugins: true
45. Limitations Limitations
46. No trust model (yet) No trust model (yet) ​ Plugins are allowed to do anything ​ Plugins are allowed to do anything With ES 2.0 maybe Java SecurityManager With ES 2.0 maybe Java SecurityManager will be will be utilised utilised No isolation No isolation Same classloader for ES and all plugins Same classloader for ES and all plugins Plugins can interfere with others Plugins can interfere with others your dependencies your dependencies Install plugins on as few nodes as possible Install plugins on as few nodes as possible Consider using dedicated plugin nodes Consider using dedicated plugin nodes Shade Shade
47. Not covered Not covered today
48. Creating a custom transport request/response Creating a custom transport request/response Manipulating the HTTP layer Manipulating the HTTP layer Plugin scopes Plugin scopes (Rest)ActionFilter (Rest)ActionFilter Custom analyzers Custom analyzers Custom mapping types Custom mapping types Site plugins Site plugins ... maybe there will be a Part II of this talk ... maybe there will be a Part II of this talk
49. @hendrikdev22 @hendrikdev22 by geek & poke System.exit(0);