Spring eXchange - Spring XD Deep Dive

This is a Presentation about the internals of Spring XD, a one stop shop tool to ease the development of Big Data applications.

The recorded talk can be seen at https://skillsmatter.com/skillscasts/5952-a-guided-tour-to-spring-xd-internals

Transcript

1. A Guided Tour to Spring XD Internals Eric BOTTARD, Pivotal The Spring eXchange, London, Nov. 6, 2014
4. Spring XD concepts
9. Process boundary Feature boundary XD Admin Container Container ZK Distributed
10. Process boundary Feature boundary XD Admin Container Container ZK XD Admin Container ZK Distributed Single-Node
11. Process boundary Feature boundary XD Admin Container Container ZK XD Admin Container ZK Distributed Single-Node module module module module module module module module module
12. Stream processing
14. these guys
15. Message Bus
16. Message Bus ✦ Binds module inputs & outputs to a transport
17. Message Bus ✦ Binds module inputs & outputs to a transport ✦ Performs Serialization (Kryo)
18. Message Bus ✦ Binds module inputs & outputs to a transport ✦ Performs Serialization (Kryo) ✦ Rabbit, Redis, Kafka, (local)
19. Partitioning
21. Horizontal Scalability e.g. stateful computation
22. Horizontal Scalability e.g. stateful computation Data Locality e.g. reference data
23. Horizontal Scalability e.g. stateful computation Data Locality e.g. reference data Concurrent Access e.g. files, hdfs
24. Message<T>
25. Message<T> partitionKey[Expression|ExtractorClass]
26. Message<T> Key partitionKey[Expression|ExtractorClass]
27. Message<T> Key partitionKey[Expression|ExtractorClass] partitionSelector[Expression|Class]
28. Message<T> Key partitionKey[Expression|ExtractorClass] partitionSelector[Expression|Class] defaults to hashCode()
29. Message<T> Key selection partitionKey[Expression|ExtractorClass] partitionSelector[Expression|Class] defaults to hashCode()
30. Message<T> Key selection partitionKey[Expression|ExtractorClass] partitionSelector[Expression|Class] % numPartitions defaults to hashCode()
31. Message<T> Key selection partition # partitionKey[Expression|ExtractorClass] partitionSelector[Expression|Class] % numPartitions defaults to hashCode()
32. Demo partitioning
33. Direct binding
34. Bypass the Message Bus
35. Bypass the Message Bus ✦Automatic when producer & consumer are guaranteed to be co-located currently: module.*.count = 0
36. Bypass the Message Bus ✦Automatic when producer & consumer are guaranteed to be co-located currently: module.*.count = 0 ✦Can be disabled module.*.producer.directBindingAllowed=false
37. A closer look at deployment
38. 1AST = parse(definition)
39. 2Validate Options
40. 2Validate Options Rich options metamodel ↪Validation, completion
41. 3Match Containers
42. 4Write Deployment Requests
43. 5Containers pick up from there
44. Demo container matching
45. Modules
46. Modules ✦ Own ApplicationContext ✦ no bean name collision ✦ module options = value placeholders
47. Modules ✦ Own ApplicationContext ✦ no bean name collision ✦ module options = value placeholders ✦ Own ClassLoader (Parent Last) ✦ ⾠ classes on the wire
48. Modules ✦ Own ApplicationContext ✦ no bean name collision ✦ module options = value placeholders ✦ Own ClassLoader (Parent Last) ✦ ⾠ classes on the wire ✦ Simple Module, Composed Module
49. Fault tolerance
50. Apache ZooKeeper Fault Tolerant ~ hierarchical FileSystem Clients can be notified of ✦ node changes ✦ other clients departure
52. ! XD Shell
53. ! XD Shell HTTP POST /streams/myStream “http | file”
54. ! XD Shell HTTP POST /streams/myStream “http | file”
55. XD  Admin ! XD Shell HTTP POST /streams/myStream “http | file”
56. XD  Admin ! XD Shell HTTP POST /streams/myStream “http | file” ZooKeeper Container State
57. XD  Container XD  Container XD  Admin ! XD Shell Assigns Modules to Containers HTTP POST /streams/myStream “http | file” ZooKeeper Container State
58. XD  Container XD  Container XD  Admin ! XD Shell HTTP Module Spring App Context Assigns Modules to Containers HTTP POST /streams/myStream “http | file” ZooKeeper Container State
59. XD  Container XD  Container XD  Admin ! XD Shell File   Module HTTP Module Spring App Context Assigns Modules to Containers HTTP POST /streams/myStream “http | file” ZooKeeper Container State
60. XD  Container XD  Container XD  Admin ! XD Shell File   Module HTTP Module Spring App Context Assigns Modules to Containers Message  Bus HTTP POST /streams/myStream “http | file” ZooKeeper Container State
61. XD  Container XD  Container XD  Admin ! XD Shell Message  Bus File   Module HTTP Module Spring App Context Assigns Modules to Containers ZooKeeper Container State
62. XD  Container XD  Container XD  Admin ! XD Shell Message  Bus File   Module HTTP Module Spring App Context Assigns Modules to Containers HTTP POST /streams/aStream “M1 | M2” ZooKeeper Container State
63. XD  Container XD  Container XD  Admin ! XD Shell Message  Bus File   Module HTTP Module Spring App Context M1 Assigns Modules to Containers HTTP POST /streams/aStream “M1 | M2” ZooKeeper Container State
64. XD  Container XD  Container XD  Admin ! XD Shell Message  Bus File   Module HTTP Module Spring App Context M1 M2 Assigns Modules to Containers HTTP POST /streams/aStream “M1 | M2” ZooKeeper Container State
65. XD  Container XD  Container XD  Admin (leader) ! XD Shell HTTP POST /streams/aStream “M1 | M2” Message  Bus File   Module HTTP Module Spring App Context M1 M2 ZooKeeper Container State XD  Admin XD  Admin
66. XD  Container XD  Admin (leader) ! XD Shell HTTP POST /streams/aStream “M1 | M2” Message  Bus File   Module M2 ZooKeeper Container State XD  Admin XD  Admin
67. XD  Container XD  Admin (leader) ! XD Shell HTTP POST /streams/aStream “M1 | M2” Message  Bus File   Module M2 ZooKeeper Container State XD  Admin XD  Admin HTTP Module M1
68. Demo fault tolerance
70. Coming next
71. Thank You Eric Bottard Pivotal t@ebottard Äericbottard http://projects.spring.io/spring-xd/ ‛
72. Thank You Eric Bottard Pivotal t@ebottard Äericbottard http://projects.spring.io/spring-xd/ ‛
73. Creek designed by Dan Hetteix from the thenounproject.com School designed by Ricardo Augusto Cherem from the thenounproject.com Crystal Ball designed by Johanna Vargas Sanchez from the thenounproject.com Funnel designed by alberto galindo from the thenounproject.com Fingers Crossed designed by caba kosmotesto from the thenounproject.com Search designed by Gianni - Dolce Merda from the thenounproject.com Streamline designed by dw from the thenounproject.com Network designed by Dmitry Baranovskiy from the thenounproject.com Network designed by Edward Boatman from the thenounproject.com Frog designed by Ealancheliyan s from the thenounproject.com