Kogito + Quarkus: A Marriage made on a Cloud

Full recording https://youtu.be/hY_eERFX_DQ

A hands-on walkthrough of the Kogito codebase with a focus on the interaction with Quarkus.

About this Event

KieLive#23: Kogito + Quarkus: the Marriage Made on a Cloud
Quarkus is the secret sauce that makes Kogito cloud-native. But like every special ingredient, you need to learn how to use it! In this hands-on walkthrough we will get an overview of the design principles of our framework, and we will learn some of the nuances of writing a native application or even a Quarkus extension. This presentation is intended for a technical audience, but it is open to everyone: maybe you are the next Kogito contributor?

Link to the live streaming: http://red.ht/KieLive23​

About the invited speaker:

During my PhD, I researched language design and implementation at University of Milan. After three years at UniCredit Bank's R&D department, I have joined Red Hat: I work on the Drools rule engine, the jBPM platform and the Kogito project.

Twitter: https://twitter.com/evacchi​

About the KIE Lives:
The KIE Live Series is composed of live streamings that bring technical information and updates about business automation delivered by the projects under the KIE umbrella: Drools, jBPM, OptaPlanner, and Kogito.

Problems like process automation, decision automation, resource planning solution are the main topics, and of course, we always have in mind recent technology concepts like cloud-native application target for any type of cloud (private/public/hybrid/edge). You can expect to hear from business automation experts who code or/and deliver business automation within big enterprises across the world.

Join us to share, learn, and grow together.

Knowledge is everything

Transcript

1. The Marriage Made on a Cloud
   

   View Slide

2. • Edoardo Vacchi @evacchi
   • Research @ UniMi / MaTe
   • Research @ UniCredit R&D
   • Kogito / Drools / jBPM @ Red Hat
   

   View Slide

3. • You are a team mate ( hey! )
   • You are a wannabe contributor ( wow! )
   • You are just curious ( welcome! )
   

   View Slide

4. Step 1
   • Head over to
   • github.com/kiegroup/kogito-runtimes
   OK, now what?
   

   View Slide

5. • Kogito
   • Compile-Time vs Run-Time
   • Quarkus Extensions
   • Bytecode vs Source-Code Generation
   • GraalVM Native image
   • Scaffolding and Troubleshooting
   

   View Slide

6. Cloud-Native Business Automation for building
   intelligent applications, backed by battle-tested
   capabilities
   

   View Slide

7. View Slide

8. • There is always a pre-processing phase where you prepare your
   program for execution
   • Then, there's actual process execution phase
   

   View Slide

9. • Will that configuration change across runs?
   • Do you have to repackage the application to bundle the new
   configuration?
   

   View Slide

10. • You are building an Immutable Dockerized Microservice in the Cloud
    • You are not going to load new classes at run-time
    • You don’t need Runtime Dependency Injection
    

    View Slide

11. Build-time
    • Pre-process configuration
    • Generate code with configuration options pre-loaded
    • Resolve dependencies
    • Generate “wiring” code -- e.g. factories, startup methods etc.
    

    View Slide

12. Dagger 2
    

    View Slide

13. Dagger 2
    

    View Slide

14. View Slide

15. View Slide

16. Assets Wiring
    Pre-Built
    Assets
    REST/Event
    Handlers
    ∙ “Drools Canonical
    Model”
    ∙ BPMN Fluent API
    ∙ ...
    ∙ DRL
    ∙ BPMN
    ∙ DMN
    ∙ ...
    ∙ JAX-RS Endpoints
    ∙ Cloud Event
    Reactive
    Consumers
    ∙ ...
    ∙ CDI Producers
    ∙ Bean Factories
    ∙ ...
    

    View Slide

17. REST Resources
    class MyModelProcessResource { @Inject Process p; }
    class MyRuleUnitResource { @Inject RuleUnit r; }
    Wiring Code (CDI etc.)
    Application.java
    ApplicationConfig.java
    Processes.java
    Decisions.java
    ...
    (Inferred) Data Model
    (process variables, DMN ItemDefinitions, DMN context, Rule Unit descriptors...)
    e.g. MyModel, MyUnitData
    Typed Knowledge Asset API (and related wiring) exposes the engine
    e.g. Process, RuleUnit, DMNDecision
    

    View Slide

18. REST Resources
    class MyModelProcessResource { @Inject Process p; }
    class MyRuleUnitResource { @Inject RuleUnit r; }
    Wiring Code (CDI etc.)
    Application.java
    ApplicationConfig.java
    Processes.java
    Decisions.java
    ...
    (Inferred) Data Model
    (process variables, DMN ItemDefinitions, DMN context, Rule Unit descriptors...)
    e.g. MyModel, MyUnitData
    Typed Knowledge Asset API (and related wiring) exposes the engine
    e.g. Process, RuleUnit, DMNDecision
    Implementation Detail !
    Heavy Rework in Progress !
    Do Not Rely on The Details !
    

    View Slide

19. • Maven Plug-In
    • Quarkus Extension
    

    View Slide

20. View Slide

21. • Codegen-based framework
    • Plug into the codegen mechanism
    • Produce more code
    

    View Slide

22. … show path here ...
    

    View Slide

23. View Slide

24. • deployment
    • runtime
    

    View Slide

25. View Slide

26. • @BuildStep
    • annotates a method
    • method parameters: required items
    • return type: provided result
    • You can think of it as of
    • a CDI @Produces method or
    • a Spring @Factory
    • With a twist:
    • You can return more than one element at once (with a “trick”)
    @BuildStep
    public SomeBuildItem myBuildStep(
    SomeOtherBuildItem items ) {
    ...
    }
    

    View Slide

27. all: foo.o bar.o
    cc -o out foo.o bar.o
    foo.o: foo.c
    $(CC) -c foo.c -o foo.o
    bar.o: bar.c
    $(CC) -c bar.c -o bar.o
    

    View Slide

28. foo.h
    foo.o
    a.out
    bar.o
    foo.c
    bar.h
    bar.c
    

    View Slide

29. • Think of them as CDI/Spring beans
    • A special interface that represents things that can be
    • produced (provided)
    • consumed (required)
    @BuildStep
    public SomeBuildItem generate(
    SomeOtherBuildItem items) { ... }
    

    View Slide

30. • Think of them as CDI/Spring beans
    • A special interface that represents things that can be
    • produced (provided)
    • consumed (required)
    @BuildStep
    public SomeBuildItem generate(
    SomeOtherBuildItem items) { ... }
    @BuildStep
    public SomeBuildItem generate(
    Collection items) { ... }
    You can
    depend on a collection
    !
    

    View Slide

31. @BuildStep
    public void generate(
    SomeBuildItem items,
    BuildProducer bp) {
    ...
    bp.produce(t);
    }
    BuildProducer generatedBeans,
    BuildProducer resource,
    BuildProducer reflectiveClass,
    BuildProducer genResBI
    

    View Slide

32. @BuildStep
    public void generate(
    SomeBuildItem items,
    BuildProducer bp) {
    ...
    bp.produce(t);
    }
    BuildProducer generatedBeans,
    BuildProducer resource,
    BuildProducer reflectiveClass,
    BuildProducer genResBI
    You can have
    as many
    as you want !
    Simulates
    multiple
    return values !
    

    View Slide

33. In the body of the methods
    // real work occurs here: invoke the code-generation procedure
    Collection generatedFiles = appGen.generate();
    // Write files to disk
    dumpFilesToDisk(appPaths, generatedFiles);
    

    View Slide

34. private void registerResources(Collection generatedFiles,
    BuildProducer resource,
    BuildProducer genResBI) {
    for (GeneratedFile f : generatedFiles) {
    if (f.getType() == GeneratedFile.Type.GENERATED_CP_RESOURCE) {
    genResBI.produce(new GeneratedResourceBuildItem(f.relativePath(), f.contents()));
    resource.produce(new NativeImageResourceBuildItem(f.relativePath()));
    }
    }
    }
    

    View Slide

35. result.add(new ReflectiveHierarchyIgnoreWarningBuildItem(
    DotName.createSimple("org.kie.dmn.api.core.DMNContext")));
    ...
    toReturn.add(new ReflectiveClassBuildItem(true, true, PMML4Result.class));
    ...
    

    View Slide

36. • Avoid scanning class-path
    • Instead use a serialized index
    • Faster lookup: no scanning!
    IndexView index = combinedIndexBuildItem.getIndex();
    DotName classDotName = DotName.createSimple(className);
    return !index.getAnnotations(classDotName).isEmpty() ||
    index.getClassByName(classDotName) != null;
    

    View Slide

37. META-INF/services/io.quarkus.deployment.dev.CompilationProvider:
    org.kie.kogito.quarkus.deployment.ProcessCompilationProvider
    org.kie.kogito.quarkus.deployment.RulesCompilationProvider
    org.kie.kogito.quarkus.deployment.DecisionTablesCompilationProvider
    org.kie.kogito.quarkus.deployment.DMNCompilationProvider
    

    View Slide

38. if (liveReload.isLiveReload()) {
    return;
    }
    

    View Slide

39. View Slide

40. Developer Mode / Hot Reload
    e.g io.quarkus.it.kogito.drools.HotReloadTest
    

    View Slide

41. • integration-tests
    • integration-tests-kogito-plugin
    • integration-tests-quarkus
    • integration-tests-quarkus-decisions
    • integration-tests-quarkus-predictions
    • integration-tests-quarkus-processes
    • integration-tests-quarkus-rules
    • integration-tests-springboot
    

    View Slide

42. View Slide

43. • Quarkus general approach is to generate byte-code
    • In Kogito we generate Java source code
    • In some cases we compile it on-the-fly
    

    View Slide

44. • Templated Approach
    • Useful for Scaffolding
    • Learn more about this in a Future Meeting
    

    View Slide

45. View Slide

46. Here Be
    Dragons
    Charmander Vector by Pokinee (DeviantArt)
    deviantart.com/pokinee/art/Charmander-Vector-427462295
    

    View Slide

47. $ mvn kogito:scaffold
    ...
    [info] Generating...
    

    View Slide

48. • fiddling with generated code
    • target/generated-code/kogito/
    

    View Slide

49. View Slide

50. • Native binary compilation
    • Restriction: “closed-world assumption”
    • Limitations on reflection
    • No dynamic code loading: forbidden ClassLoader#defineClass(...byte[]...)
    • Allows more aggressive optimization (pruning)
    • Static initializers are not lazy* !
    • Evaluated at build time !
    * in Quarkus
    

    View Slide

51. • Native does not prevent dynamic code loading!
    • e.g. dlopen() for dynamic .so/.dll/.dylib loading
    • JVM mode does not prevent pruning!
    • e.g. ProGuard
    • These are just choices of the native image compiler
    

    View Slide

52. META-INF/
    └── native-image
    └── groupID
    └── artifactID
    └── native-image.properties
    └── reflection-config.json
    └── resource-config.json
    └── jni-config.json
    └── proxy-config.json
    ReflectiveClassBuildItem
    

    View Slide

53. @TargetClass(ASMAccessorOptimizer.class)
    final class ASMAccessorOptimizer_Target {
    @Substitute
    public Accessor optimizeAccessor(
    ParserContext pCtx, char[] property, int start, int offset, Object ctx, Object thisRef,
    VariableResolverFactory factory, boolean rootThisRef, Class ingressType) {
    throw new UnsupportedOperationException();
    }
    ...
    }
    

    View Slide

54. @Recorder
    class HelloRecorder {
    public void sayHello(String name) {
    System.out.println("Hello" + name);
    }
    }
    @Record(RUNTIME_INIT)
    @BuildStep
    public void helloBuildStep(HelloRecorder recorder) {
    recorder.sayHello("World");
    }
    

    View Slide

55. public class Example {
    static {
    System.out.println("hello");
    }
    public static void main(String... args) {
    System.out.println("world");
    }
    }
    

    View Slide

56. $ java Example
    hello
    world
    

    View Slide

57. $ native-image --initialize-at-build-time Example
    [example:23074] classlist: 1,032.11 ms, 1.18 GB
    [example:23074] (cap): 2,301.26 ms, 1.18 GB
    [example:23074] setup: 3,609.57 ms, 1.69 GB
    hello
    [example:23074] (clinit): 82.45 ms, 1.73 GB
    [example:23074] (typeflow): 3,032.00 ms, 1.73 GB
    [example:23074] (objects): 2,923.76 ms, 1.73 GB
    [example:23074] (features): 129.59 ms, 1.73 GB
    [example:23074] analysis: 6,307.81 ms, 1.73 GB
    [example:23074] universe: 277.17 ms, 1.73 GB
    [example:23074] (parse): 525.88 ms, 1.73 GB
    [example:23074] (inline): 877.57 ms, 1.78 GB
    [example:23074] (compile): 3,842.94 ms, 1.87 GB
    [example:23074] compile: 5,504.45 ms, 1.87 GB
    [example:23074] image: 463.22 ms, 1.87 GB
    [example:23074] write: 176.80 ms, 1.87 GB
    [example:23074] [total]: 17,528.27 ms, 1.87 GB
    

    View Slide

58. $ native-image --initialize-at-build-time Example
    [example:23074] classlist: 1,032.11 ms, 1.18 GB
    [example:23074] (cap): 2,301.26 ms, 1.18 GB
    [example:23074] setup: 3,609.57 ms, 1.69 GB
    hello
    [example:23074] (clinit): 82.45 ms, 1.73 GB
    [example:23074] (typeflow): 3,032.00 ms, 1.73 GB
    [example:23074] (objects): 2,923.76 ms, 1.73 GB
    [example:23074] (features): 129.59 ms, 1.73 GB
    [example:23074] analysis: 6,307.81 ms, 1.73 GB
    [example:23074] universe: 277.17 ms, 1.73 GB
    [example:23074] (parse): 525.88 ms, 1.73 GB
    [example:23074] (inline): 877.57 ms, 1.78 GB
    [example:23074] (compile): 3,842.94 ms, 1.87 GB
    [example:23074] compile: 5,504.45 ms, 1.87 GB
    [example:23074] image: 463.22 ms, 1.87 GB
    [example:23074] write: 176.80 ms, 1.87 GB
    [example:23074] [total]: 17,528.27 ms, 1.87 GB
    

    View Slide

59. $ ./example
    world
    

    View Slide

60. the check on a static method may not be equivalent from the perspective
    of the native image compiler, which does aggressive code pruning at build
    time.
    If the field is static, then the entire non-true branch is basically treated
    as unreachable; otherwise the native image compiler will try to visit the
    other branch too, which in some cases may mark as reachable code
    things that would not really be reached at runtime.
    this may not be an issue here, but it may be if that code eventually leads
    to some "banned" method call (e.g. ClassLoader#defineClass()), causing
    the native image build to fail.
    

    View Slide

61. private static final Collection readers = List.of(
    readResource("some/path..."), ...
    );
    private static InputStreamReader readResource(InputStream stream) {
    return new java.io.InputStreamReader(stream);
    }
    

    View Slide

62. private final static boolean IS_NATIVE_IMAGE =
    System.getProperty("org.graalvm.nativeimage.imagecode") != null;
    private static final Collection readers = List.of(...);
    private static InputStreamReader readResource(java.io.InputStream stream) {
    if (!IS_NATIVE_IMAGE) {
    return new java.io.InputStreamReader(stream);
    }
    try {
    byte[] bytes = stream.readAllBytes();
    ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(bytes);
    return new InputStreamReader(byteArrayInputStream);
    } catch (java.io.IOException e) {
    throw new java.io.UncheckedIOException(e);
    }
    }
    

    View Slide

63. private final static boolean IS_NATIVE_IMAGE =
    System.getProperty("org.graalvm.nativeimage.imagecode") != null;
    private static final Collection readers = List.of(...);
    private static InputStreamReader readResource(java.io.InputStream stream) {
    if (!IS_NATIVE_IMAGE) {
    return new java.io.InputStreamReader(stream);
    }
    try {
    byte[] bytes = stream.readAllBytes();
    ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(bytes);
    return new InputStreamReader(byteArrayInputStream);
    } catch (java.io.IOException e) {
    throw new java.io.UncheckedIOException(e);
    }
    }
    

    View Slide

64. private final static boolean IS_NATIVE_IMAGE =
    System.getProperty("org.graalvm.nativeimage.imagecode") != null;
    private static final Collection readers = List.of(...);
    private static InputStreamReader readResource(java.io.InputStream stream) {
    if (!IS_NATIVE_IMAGE) {
    return new java.io.InputStreamReader(stream);
    }
    try {
    byte[] bytes = stream.readAllBytes();
    ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(bytes);
    return new InputStreamReader(byteArrayInputStream);
    } catch (java.io.IOException e) {
    throw new java.io.UncheckedIOException(e);
    }
    }
    

    View Slide

65. private static final Collection readers = List.of(...);
    private static InputStreamReader readResource(java.io.InputStream stream) {
    try {
    byte[] bytes = stream.readAllBytes();
    ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(bytes);
    return new InputStreamReader(byteArrayInputStream);
    } catch (java.io.IOException e) {
    throw new java.io.UncheckedIOException(e);
    }
    }
    

    View Slide

66. private boolean isNativeImage() {
    return System.getProperty("org.graalvm.nativeimage.imagecode") != null;
    }
    private static final Collection readers = List.of(...);
    private static InputStreamReader readResource(java.io.InputStream stream) {
    if (!isNativeImage()) {
    return new java.io.InputStreamReader(stream);
    }
    try {
    byte[] bytes = stream.readAllBytes();
    ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(bytes);
    return new InputStreamReader(byteArrayInputStream);
    } catch (java.io.IOException e) {
    throw new java.io.UncheckedIOException(e);
    }
    }
    Not
    Equivalent!
    

    View Slide

67. View Slide

68. I don’t want to contribute to the extension. What else can I do?
    - contribute documentation
    - contribute examples
    - hang out on the Zulip channel / mailing list and help other users out!
    

    View Slide

69. View Slide