Slide 1

Slide 1 text

Plugin development for Plugin development for elasticsearch elasticsearch Hendrik Saly

Slide 2

Slide 2 text

Simply Simply explained explained by geek & poke

Slide 3

Slide 3 text

Why plugins Why plugins Getting started with a simple plugin Getting started with a simple plugin How they fit into Elasticsearch architecture How they fit into Elasticsearch architecture Examples walkthrough Examples walkthrough Development Development Testing Testing Distribution Distribution Limitations Limitations Not covered Not covered

Slide 4

Slide 4 text

Why plugins Why plugins

Slide 5

Slide 5 text

If there is no configuration option for your need If there is no configuration option for your need If you need unexposed (but public) API If you need unexposed (but public) API To enforce specific behaviour To enforce specific behaviour If scripting is simply not enough If scripting is simply not enough Site plugin vs JVM Plugin Site plugin vs JVM Plugin Here Here Plugin == JVM Plugin Plugin == JVM Plugin

Slide 6

Slide 6 text

For example: For example: Creating a new REST endpoint Creating a new REST endpoint Custom mapping Custom mapping Custom analyzer Custom analyzer Transport layer manipulation Transport layer manipulation HTTP layer manipulation HTTP layer manipulation

Slide 7

Slide 7 text

Getting started with a Getting started with a simple plugin simple plugin

Slide 8

Slide 8 text

Plugins are written in Java (1.7+) Plugins are written in Java (1.7+) They live within a node (or transport client) They live within a node (or transport client) They can access all internals They can access all internals They can make a node stop working (or They can make a node stop working (or even worse) even worse)

Slide 9

Slide 9 text

public class MyPlugin extends AbstractPlugin { private final Settings settings; public MyPlugin(Settings settings) { this.settings = settings; } @Override public String name() { return "MyPlugin"; } @Override public String description() { return "This is the description for my plugin"; } } Extend AbstractPlugin class Extend AbstractPlugin class ... but it does not anything useful yet ... but it does not anything useful yet

Slide 10

Slide 10 text

public class MyPlugin extends AbstractPlugin { ... //Define own modules @Override public Collection> modules() { return ImmutableList.of(MyModule.class); } //Define own services @Override public Collection> services() { return ImmutableList.of(MyService.class); } } So lets define our own modules So lets define our own modules or services which do something meaningful or services which do something meaningful

Slide 11

Slide 11 text

public class MyPlugin extends AbstractPlugin { ... //Define own modules @Override public void processModule(Module module) { if ((module instanceof RestModule)) { ((RestModule)module).addRestAction(MyRestAction.class); } if ((module instanceof ???)) { ((???)module).doModuleSpecificStuffHere(); } } } or do something with core modules or do something with core modules to accomplish your needs to accomplish your needs

Slide 12

Slide 12 text

plugin=org.company.es.plugins.MyPlugin Add es-plugin.properties to src/main/resources Add es-plugin.properties to src/main/resources

Slide 13

Slide 13 text

Jar it, then zip it with all dependencies included Jar it, then zip it with all dependencies included Maven can do that for you Maven can do that for you Main artefact Your code Dependecies

Slide 14

Slide 14 text

How they fit into How they fit into Elasticsearch Elasticsearch architecture architecture

Slide 15

Slide 15 text

Elasticsearch is build upon Elasticsearch is build upon 2.0 2.0 pronounced "juice" pronounced "juice" Dependency injection Dependency injection Inversion of control Inversion of control Guice code is copied in ES codebase (slightly Guice code is copied in ES codebase (slightly modified for lower memory footprint) modified for lower memory footprint) So elasticsearch is build upon guice modules So elasticsearch is build upon guice modules Google guice Google guice

Slide 16

Slide 16 text

... and services ... and services Modules start up, but they don't shut down Modules start up, but they don't shut down Modules should be tested Modules should be tested Modules can be overridden Modules can be overridden Services have a lifecycle Services have a lifecycle because of because of

Slide 17

Slide 17 text

Interesting core modules (for plugins) Interesting core modules (for plugins) ActionModule ActionModule AnalysisModule AnalysisModule HttpServerModule HttpServerModule RestModule RestModule ScriptModule ScriptModule TransportModule TransportModule

Slide 18

Slide 18 text

Interesting core services (for plugins) Interesting core services (for plugins) ClusterService ClusterService TransportService TransportService NettyTransport NettyTransport IndicesService IndicesService Read more https://www.found.no/foundation/elasticsearch-internals/

Slide 19

Slide 19 text

Examples walkthrough Examples walkthrough https://github.com/salyh/elasticsearch-plugin-examples

Slide 20

Slide 20 text

Audit Plugin Audit Plugin Capture index changes Capture index changes Write them into elasticsearch Write them into elasticsearch (Make them visible through kibana) (Make them visible through kibana) Transport SSL/TLS Plugin Transport SSL/TLS Plugin Implement and enforce SSL/TLS encryption Implement and enforce SSL/TLS encryption for transport protocol for transport protocol

Slide 21

Slide 21 text

Development Development

Slide 22

Slide 22 text

by geek & poke

Slide 23

Slide 23 text

Audit Plugin Audit Plugin AuditService registers listener on AuditService registers listener on ShardIndexingService ShardIndexingService Write changes into elasticsearch via Write changes into elasticsearch via BulkProcessor BulkProcessor

Slide 24

Slide 24 text

public class AuditService extends AbstractLifecycleComponent{ ... @Inject public AuditService(Settings settings, IndicesService indicesService, Client client, ClusterService clusterService, TransportFlushAction tfa) { super(settings); this.indicesService = indicesService; this.clusterService = clusterService; ... } @Override protected void doStart() throws ElasticsearchException { ... this.indicesService.indicesLifecycle().addListener(auditIndicesLsListener); } Define a new service called AuditService Define a new service called AuditService Register a indices lifecycle listener Register a indices lifecycle listener Dependency Injection by guice

Slide 25

Slide 25 text

IndicesLifecycle.Listener auditIndicesLsListener = new IndicesLifecycle.Listener() { @Override public void afterIndexShardStarted(final IndexShard indexShard) { if (indexShard.routingEntry().primary() && !indexShard.indexService().index().name().equals(auditIndexName)) { AuditIndexOpListener auditListener = new AuditIndexOpListener(indexShard); indexShard.indexingService().addListener(auditListener); } } If a shard starts, register If a shard starts, register an IndexingOperationListener an IndexingOperationListener

Slide 26

Slide 26 text

class AuditIndexOpListener extends IndexingOperationListener { private final IndexShard indexShard; public AuditIndexOpListener(IndexShard indexShard) { this.indexShard = indexShard; } @Override public void postIndex(Index index) { String nodeName = indexShard.nodeName(); String indexName = indexShard.indexService().index().name(); Change change = new Change(nodeName, indexName, ...); //store it in elasticsearch (or anywhere else) IndexRequest ir = new IndexRequest().source(change.sourceAsMap()); addToBulkIndex(ir); } } Let the IndexingOperationListener store the Let the IndexingOperationListener store the change in elasticsearch (or anywhere else) change in elasticsearch (or anywhere else)

Slide 27

Slide 27 text

public class AuditModule extends AbstractModule { @Override protected void configure() { //nothing to bind here } @Override public void processModule(Module module) { if ((module instanceof ActionModule)) { ((ActionModule)module).registerAction(FlushAction.INSTANCE, TransportFlushAction.class); } if ((module instanceof RestModule)) { ((RestModule)module).addRestAction(AuditRestAction.class); } } } Define the AuditModule (not covered today) Define the AuditModule (not covered today) API for flushing outstanding bulk requests API for flushing outstanding bulk requests

Slide 28

Slide 28 text

public class AuditPlugin extends AbstractPlugin { ... public String name() { return "AuditPlugin"; } public String description() { return "This is the description for the AuditPlugin"; } @Override public Collection> modules() { return ImmutableList.of(AuditModule.class); } @Override public Collection> services() { return ImmutableList.of(AuditService.class); } } Last but not least define the AuditPlugin itself Last but not least define the AuditPlugin itself

Slide 29

Slide 29 text

plugin=de.saly.es.example.audit.plugin.AuditPlugin Add es-plugin.properties Add es-plugin.properties

Slide 30

Slide 30 text

Demo Demo

Slide 31

Slide 31 text

Transport SSL Plugin Transport SSL Plugin Elasticsearch uses Netty for tcp Elasticsearch uses Netty for tcp communication communication Extend NettyTransport and inject SslHandler Extend NettyTransport and inject SslHandler into pipeline into pipeline Replace original Netty transport in Replace original Netty transport in TransportModule TransportModule Expose some SSL/TLS related informations Expose some SSL/TLS related informations through a new REST API through a new REST API

Slide 32

Slide 32 text

public class SSLNettyTransport extends NettyTransport { @Override public ChannelPipelineFactory configureServerChannelPipelineFactory(String name, Settings settings) { return new SSLServerChannelPipelineFactory(this, name, settings, this.settings); } protected class SSLServerChannelPipelineFactory extends SecureServerChannelPipelineFactory { public SSLServerChannelPipelineFactory(NettyTransport nettyTransport, String name, Settings sslsettings, Settings essettings) { super(nettyTransport, name, sslsettings); } @Override public ChannelPipeline getPipeline() throws Exception { ChannelPipeline pipeline = super.getPipeline(); SSLEngine engine = ... SslHandler sslHandler = new SslHandler(engine); pipeline.addFirst("ssl_server", sslHandler); return pipeline; } } } Extend NettyTransport and do some netty Extend NettyTransport and do some netty pipeline magic pipeline magic

Slide 33

Slide 33 text

public class TSslPlugin extends AbstractPlugin { ... public void onModule(TransportModule transportModule) { transportModule.setTransport(SSLNettyTransport.class, name()); } public void onModule(RestModule restModule) { restModule.addRestAction(TSslRestAction.class); } } Replace transport in TransportModule Replace transport in TransportModule Register a custom rest action for ssl infos Register a custom rest action for ssl infos

Slide 34

Slide 34 text

public class TSslRestAction extends BaseRestHandler{ @Inject public TSslRestAction(Settings settings, Client client, RestController controller) { super(settings, controller, client); controller.registerHandler(Method.GET, "/_tssl/state", this); controller.registerHandler(Method.POST, "/_tssl/state", this); } @Override protected void handleRequest(RestRequest request, RestChannel channel, Client client) throws Exception { XContentBuilder builder = JsonXContent.contentBuilder(); builder.startObject(); builder.field("enabled_protocols", SecurityUtil.ENABLED_SSL_PROTOCOLS); builder.field("enabled_chipers", SecurityUtil.ENABLED_SSL_CIPHERS); builder.endObject(); channel.sendResponse(new BytesRestResponse(RestStatus.OK, builder)); } } Register a handler for an endpoint Register a handler for an endpoint Send a JSON response Send a JSON response

Slide 35

Slide 35 text

plugin=de.saly.es.example.tssl.plugin.TSslPlugin version=${project.version} Add es-plugin.properties Add es-plugin.properties

Slide 36

Slide 36 text

Call the new REST endpoint Call the new REST endpoint

Slide 37

Slide 37 text

Testing Testing

Slide 38

Slide 38 text

Unittests Unittests Randomized testing by Randomized testing by extending ElasticsearchIntegrationTest extending ElasticsearchIntegrationTest Extend Extend Start nodes directly in the test method Start nodes directly in the test method Test in real elasticsearch node/cluster Test in real elasticsearch node/cluster https://github.com/tlrx/elasticsearch-test https://github.com/tlrx/elasticsearch-test MultiJvmUnitTest MultiJvmUnitTest DevOps magic does help here DevOps magic does help here Always test with multiple nodes (cluster scenario)

Slide 39

Slide 39 text

If you want use the randomized testing suite, If you want use the randomized testing suite, include test dependencies in pom include test dependencies in pom org.apache.lucene lucene-test-framework test org.elasticsearch elasticsearch test-jar test

Slide 40

Slide 40 text

Distribution Distribution

Slide 41

Slide 41 text

Use Maven for building plugins Use Maven for building plugins because ES is using Maven because ES is using Maven Plugins are best published in Plugins are best published in Or distributed as .zip file Or distributed as .zip file maven central maven central # Install from maven central bin/plugin --install de.saly/elasticsearch-sample-plugin-tssl/1.1 # Install from a .zip file bin/plugin --url file:///Users/.../elasticsearch-sample-plugin-tssl-1.1.zip \ --install elasticsearch-sample-plugin-tssl

Slide 42

Slide 42 text

src/main/assemblies/plugin.xml src/main/assemblies/plugin.xml plugin zip false / true true org.elasticsearch:elasticsearch

Slide 43

Slide 43 text

Use maven-assembly-plugin Use maven-assembly-plugin maven-assembly-plugin false ${project.build.directory}/releases/ ${basedir}/src/main/assemblies/plugin.xml package single !! !!

Slide 44

Slide 44 text

Plugin related configuration options in Plugin related configuration options in elasticsearch.yml elasticsearch.yml # Load this plugin from the classpath # (does only makes sense if plugins.load_classpath_plugins is false) # Order is respected plugin.types: org.company.MyPlugin,com.guhgle.FantasticPlugin # If a plugin listed here is not installed for current node, the node will not start. plugin.mandatory: mapper-attachments,lang-groovy # If its true (which is the default) load all plugins which are in the classpath # No order guarantee plugins.load_classpath_plugins: true

Slide 45

Slide 45 text

Limitations Limitations

Slide 46

Slide 46 text

No trust model (yet) No trust model (yet) ​ Plugins are allowed to do anything ​ Plugins are allowed to do anything With ES 2.0 maybe Java SecurityManager With ES 2.0 maybe Java SecurityManager will be will be utilised utilised No isolation No isolation Same classloader for ES and all plugins Same classloader for ES and all plugins Plugins can interfere with others Plugins can interfere with others your dependencies your dependencies Install plugins on as few nodes as possible Install plugins on as few nodes as possible Consider using dedicated plugin nodes Consider using dedicated plugin nodes Shade Shade

Slide 47

Slide 47 text

Not covered Not covered today

Slide 48

Slide 48 text

Creating a custom transport request/response Creating a custom transport request/response Manipulating the HTTP layer Manipulating the HTTP layer Plugin scopes Plugin scopes (Rest)ActionFilter (Rest)ActionFilter Custom analyzers Custom analyzers Custom mapping types Custom mapping types Site plugins Site plugins ... maybe there will be a Part II of this talk ... maybe there will be a Part II of this talk

Slide 49

Slide 49 text

@hendrikdev22 @hendrikdev22 by geek & poke System.exit(0);