Data Services: {Big,Fast,Smart} Data Processing mit Microservices

Data Processing und Microservices sind ein perfektes Gespann. In dieser Kombination können Microservices dazu verwendet werden, ein flexibles, Event-getriebenes und skalierbares System von lose gekoppelten Datenverarbeitungsaufgaben aufzubauen. Diesen Ansatz nennen wir Dataservices.

In diesem Vortrag stellen wir zunächst die wesentlichen Konzepte und einige Schlüsseltechnologien vor, um Dataservice-Architekturen zu realisieren. Anschließend werden wir die einzelnen Bestandteile einer exemplarischen Datenverarbeitungs-Pipeline schrittweise komponieren und die Showcase-Pipeline in der Cloud zur Ausführung bringen und skalieren.

Transcript

15. I1 Pn P1 P1 Pn O1
24. apiVersion: extensions/v1beta1 kind: Deployment metadata: name: nginx spec: replicas: 3 strategy: type: RollingUpdate template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 livenessProbe: ... readinessProbe: ... --- apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx namespace: default spec: type: NodePort ports: - port: 80 selector: app: nginx --- apiVersion: extensions/v1beta1 kind: Ingress metadata: name: nginx labels: app: nginx namespace: default spec: rules: - host: nginx.cloudkoffer http: paths: - path: / backend: serviceName: nginx servicePort: 80
30. 33 P1 C1 Q1 P1 C1 Q1 Cn P1 C1 Q1 Cn Qn P1 C1 Q1 Q2
42. CDI Extensions Web Fragments Bean Validation 2.0 CDI 2.0 Managed Beans 1.0 JCA 1.7 JPA 2.2 JMS 2.0 JSP 2.3 EL 3.0 EJB 3.2 Batch 1.0 JSF 2.3 Interceptors 1.2 Mail 1.6 Common Annotations 1.3 JTA 1.2 JAX-WS 1.4 JAX-RS 2.1 Concurrency 1.0 JSON-P 1.1 JSON-B 1.0 WebSocket 1.1 JAPSIC 1.1 JACC 1.5 Security 1.0 Servlet 4.0 JCache 1.0
45. @MessageDriven(activationConfig = { @ActivationConfigProperty(propertyName = "serverURIs", propertyValue = "tcp://eclipse-mosquitto:1883"), @ActivationConfigProperty(propertyName = "cleanSession", propertyValue = "false"), @ActivationConfigProperty(propertyName = "automaticReconnect", propertyValue = "true"), @ActivationConfigProperty(propertyName = "filePersistence", propertyValue = "false"), @ActivationConfigProperty(propertyName = "connectionTimeout", propertyValue = "30"), @ActivationConfigProperty(propertyName = "maxInflight", propertyValue = "3"), @ActivationConfigProperty(propertyName = "keepAliveInterval", propertyValue = "5"), @ActivationConfigProperty(propertyName = "topicFilter", propertyValue = "de/qaware/oss/cloud/mqtt"), @ActivationConfigProperty(propertyName = "qos", propertyValue = "1") }) public class MqttSourceMDB implements MQTTListener { @OnMQTTMessage @TransactionAttribute(value = TransactionAttributeType.REQUIRED) @Transactional(Transactional.TxType.REQUIRED) public void onMQTTMessage(String topic, MqttMessage message) { JsonReader reader = Json.createReader(new ByteArrayInputStream(message.getPayload())); JsonObject jsonObject = reader.readObject(); // TODO do stuff with the JSON payload } }
46. JsonObject currentWeather = Json.createObjectBuilder() .add("city", “London") .add("weather", “Drizzle") .build(); StringWriter payload = new StringWriter(); JsonWriter jsonWriter = Json.createWriter(payload); jsonWriter.writeObject(currentWeather); TextMessage msg = session.createTextMessage(payload.toString()); msg.setJMSType("CurrentWeather"); msg.setStringProperty("contentType", "application/vnd.weather.v1+json"); @ActivationConfigProperty(propertyName = "messageSelector", propertyValue = "(JMSType = 'CurrentWeather') AND (contentType = 'application/vnd.weather.v1+json‘)“) JsonReader reader = Json.createReader(new StringReader(body)); JsonObject jsonObject = reader.readObject();