Searching in Neos with Elasticsearch - InspiringCon 2015 in Kolbermoor

Searching of content is a core feature of almost every website, especially for bigger ones. This talk will explain the general approach of proactively indexing content, also highlighting the supported indexing systems like ElasticSearch. It will especially focus on indexing custom data, and show how this can be integrated with data from external systems like Magento. Furthermore, we’ll highlight some features which can be built upon a flexible indexing system, such as tagging or categorization of content. You’ll see that search combined with custom content types solves lots of use cases where custom home-grewn solutions had to be implemented beforehand.

Transcript

1. F I R S T N A M E L A S T N A M E @ S K U R F U E R S T S E A R C H I N G I N N E O S S E B A S T I A N K U R F Ü R S T
2. Sebastian Kurfürst @skurfuerst
4. exply.io Enterprise Search meets Business Intelligence
7. 
8. features (Page) main (ContentCollection) … (Headline) … (Text) roadmap (Page) neostypo3org (Page) Tree of Nodes de en de en de en en en unsere-codesprints (Page) de
9. TYPO3CR is great for Tree Traversal
11. Find all articles written by Sebastian. Display the first three locations tagged with ConferenceLocation. What are the newest pages in a certain category?
12. features (Page) main (ContentCollection) … (Headline) … (Text) roadmap (Page) neostypo3org (Page) unsere-codesprints (Page)
13. features (Page) main (ContentCollection) … (Headline) … (Text) roadmap (Page) neostypo3org (Page) unsere-codesprints (Page)
14. All Documents & Content Currently Relevant Content
15. Currently, TYPO3CR does not yet effectively provide this set-based view on nodes.
16. to the rescue!
17. Getting Started
18. 1. Set up ElasticSearch # ElasticSearch 1.4.4 - config/elasticsearch.yml script.disable_dynamic: sandbox script.groovy.sandbox.class_whitelist: java.util.LinkedHashMap script.groovy.sandbox.receiver_whitelist: java.util.Iterator, 
 java.lang.Object, java.util.Map, java.util.Map$Entry script.groovy.sandbox.enabled: true cluster.name: [PUT_YOUR_CUSTOM_NAME_HERE] network.host: 127.0.0.1 index.number_of_shards: 1 index.number_of_replicas: 0
19. 2. Start ElasticSearch bin/elasticsearch
20. composer require --prefer-source typo3/typo3cr-search @dev composer require --prefer-source flowpack/elasticsearch- contentrepositoryadaptor @dev 2. Require the CR Adaptor TODO: no @dev anymore!
21. /flow nodeindex:build 3. Indexing
22. 4. Debugging Tools http://localhost:9200/_plugin/head/ http://localhost:9200/_plugin/sense/
26. composer require --prefer-source flowpack/searchplugin @dev
28. This is fulltext search.
30. Node Querying
31. Article Category Tag contains tagged with
32. 1. Node References # build up relation in NodeTypes.yaml 'Sandstorm.News:Article': superTypes: ['TYPO3.Neos:Document'] ... properties: tags: type: references ui: label: 'Tags' inspector: editorOptions: # allow only references to tags nodeTypes: ['Sandstorm.News:Tag']
33. 2. Query in TypoScript # replace main content area by a custom TypoScript object prototype(PrimaryContent).newsTag { condition = ${q(node).is('[instanceof Sandstorm.News:Tag]')} type = 'Sandstorm.News:Tag' }
34. 2. Query in TypoScript # inherits from Template by default prototype(Sandstorm.News:Tag) { latestArticlesTaggedWithTag = ${...} }
35. 2. Query in TypoScript latestArticlesTaggedWithTag = ${Search.query(site) # search underneath this site
 .nodeType('Sandstorm.News:Article') # filter by node type .exactMatch('tags', node) # where tag == current tag .limit(3) # first 3 results .sortDesc('publishDate') # and sort by publishing date desc .execute()}
36. 3. Use in Template <f:for each="{latestArticlesTaggedWithTag}" as="singleArticle"> <neos:link.node node="{singleArticle}"> <!-- Render single article as you would like --> </neos:link.node> </f:for>
37. ${Search.query(site)
 .fulltext('Alice') .execute()}
38. Node References together with Search Queries
40. ElasticSearch Core Concepts
41. A T2 T1 T3 Normalized Data in a relational DB A T1 A T2 A T3 Denormalized Data in an index
42. GET Index/Type/Document-ID
43. GET Index/Type/_mapping
44. GET Index/Type/_mapping _all
45. typo3cr-1426882860 typo3cr-1426885219 typo3cr Index Aliases allow index rebuilds!
46. Hey, InspiringCon 2015 Hey, InspiringCon 2015 Tokenization Token Filtering Hey InspiringCon 2015 hey inspiringcon 2015 Indexing Pipeline InspiringCon2015 Search Pipeline InspiringCon 2015 inspiringcon 2015
48. ${Search.query(site)
 .fulltext('Alice') .execute()} .log() 15-03-23 07:20:50 1820 DEBUG Query Log (): {"query":{"filtered":{"query":{"bool":{"must":[{"match_all":[]},{"query_string": {"query":"Alice"}}]}},"filter":{"bool":{"must":[{"term":{"__parentPath":"\/sites\/neosdemotypo3org"}},{"terms":{"__workspace":["live"]}}],"should":[],"must_not":[{"term": {"_hidden":true}},{"range":{"_hiddenBeforeDateTime":{"gt":"now"}}},{"range":{"_hiddenAfterDateTime":{"lt":"now"}}}]}}}},"fields":["__path"],"highlight":{"fields": {"__fulltext*":{"fragment_size":150,"no_match_size":150,"number_of_fragments":2}}}} -- execution time: 10.998010635376 ms -- Total Results: 28 Data/Logs/ElasticSearch.log
51. Aggregations calculate statistical information about the current result. TODO Kibana Screenshot
52. Aggregations calculate statistical information about the current result.
53. Fine-Tuning ElasticSearch+Neos
54. 1. ElasticSearch Schema
55. 1. ElasticSearch Schema TYPO3:
 TYPO3CR:
 Search:
 defaultConfigurationPerType:
 string:
 elasticSearchMapping:
 type: string
 include_in_all: false
 boolean:
 elasticSearchMapping:
 type: boolean
 date:
 elasticSearchMapping:
 type: date
 format: 'date_time_no_millis'
 include_in_all: false
 Settings.yaml NodeTypes.yaml 
 'TYPO3.Neos:Node': &node
 properties:
 '__identifier':
 search:
 elasticSearchMapping:
 type: string
 index: not_analyzed
 include_in_all: false
 
 defaults overrides indexing: '${node.identifier}'
56. 2. ElasticSearch Indexing indexing: '${Indexing.buildAllPathPrefixes(node.parentPath)}' indexing: '${node.identifier}'
57. 3. Fulltext Searching We at InspiringCon (Article) main (ContentCollection) … (Headline) … (Text) collect all content Fulltext Root
58. 3. Fulltext Searching We at InspiringCon (Article) main (ContentCollection) … (Headline) … (Text) h1 h2 ... text
59. 3. Fulltext Searching # predefined in Neos 'TYPO3.Neos:Document':
 search:
 fulltext:
 isRoot: true 'TYPO3.Neos.NodeTypes:Text':
 properties:
 'text':
 search:
 fulltextExtractor: '${Indexing.extractHtmlTags(value)}' 'Sandstorm.News:Article':
 properties:
 'title':
 search:
 fulltextExtractor: '${Indexing.extractInto("h1", value)}'
60. Indexing Additional Data
61. typo3cr-1426882860 search Index Aliases allow to link multiple indices. products-2976886808
62. ElasticSearch Rivers can poll data from other systems.
64. ElasticSearch too big for your project? Use SimpleSearch! composer require --prefer-source flowpack/simplesearch- contentrepositoryadaptor @dev
66. Resources http://www.elasticsearch.org/guide/ README of Flowpack.ElasticSearch.ContentRepositoryAdaptor
67. Thank You!