Groovy Roadmap

Transcript

1. objectcomputing.com © 2018, Object Computing, Inc. (OCI). All rights reserved.

   No part of these notes may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior, written permission of Object Computing, Inc. (OCI) Groovy 2.5, 3, 4 Roadmap Presented by Dr Paul King

2. Dr Paul King OCI Groovy Lead V.P. and PMC Chair

   Apache Groovy Author: https://www.manning.com/books/groovy-in-action-second-edition Slides: https://speakerdeck.com/paulk/groovy-roadmap Examples repo: https://github.com/paulk-asert/upcoming-groovy @paulk_asert

3. objectcomputing.com © 2018, Object Computing, Inc. (OCI). All rights reserved.

   3 REIMAGINE TOGETHER Grails

4. Groovy Open Collective

5. How did Groovy start? ... As a dynamic complement to

   Java James Strachan 2003: Perhaps its time the JVM had its own dynamic language ... we're starting simple with the nice tuples, sequences, maps from Python & closures from Ruby and being concise & dynamically typed with a Java look-and-feel ... where we end up is anyone's guess right now.

6. What is Groovy? It’s like a super version of Java:

   Supports most Java syntax but allows simpler syntax for many constructs Supports all Java libraries but provides many extensions and its own productivity libraries Has both a static and dynamic nature Extensible language and tooling

7. Java-like or concise shortcuts import java.util.List; import java.util.ArrayList; class Main

   { private List keepShorterThan(List strings, int length) { List result = new ArrayList(); for (int i = 0; i < strings.size(); i++) { String s = (String) strings.get(i); if (s.length() < length) { result.add(s); } } return result; } public static void main(String[] args) { List names = new ArrayList(); names.add("Ted"); names.add("Fred"); names.add("Jed"); names.add("Ned"); System.out.println(names); Main m = new Main(); List shortNames = m.keepShorterThan(names, 4); System.out.println(shortNames.size()); for (int i = 0; i < shortNames.size(); i++) { String s = (String) shortNames.get(i); System.out.println(s); } } } names = ["Ted", "Fred", "Jed", "Ned"] println names shortNames = names.findAll{ it.size() < 4 } println shortNames.size() shortNames.each{ println it }

8. Java-like or concise shortcuts import java.util.List; import java.util.ArrayList; class Main

   { private List keepShorterThan(List strings, int length) { List result = new ArrayList(); for (int i = 0; i < strings.size(); i++) { String s = (String) strings.get(i); if (s.length() < length) { result.add(s); } } return result; } public static void main(String[] args) { List names = new ArrayList(); names.add("Ted"); names.add("Fred"); names.add("Jed"); names.add("Ned"); System.out.println(names); Main m = new Main(); List shortNames = m.keepShorterThan(names, 4); System.out.println(shortNames.size()); for (int i = 0; i < shortNames.size(); i++) { String s = (String) shortNames.get(i); System.out.println(s); } } } names = ["Ted", "Fred", "Jed", "Ned"] println names shortNames = names.findAll{ it.size() < 4 } println shortNames.size() shortNames.each{ println it } given the names "Ted", "Fred", "Jed" and "Ned" display all the names display the number of names having size less than 4 display the names having size less than 4

9. Python code to solve cryptarithmetic def solutions(): # letters =

   ('s', 'e', 'n', 'd', 'm', 'o', 'r', 'y') all_solutions = list() for s in range(1, 10): for e in range(0, 10): for n in range(0, 10): for d in range(0, 10): for m in range(1, 10): for o in range(0, 10): for r in range(0, 10): for y in range(0, 10): if len({s, e, n, d, m, o, r, y}) == 8: send = 1000 * s + 100 * e + 10 * n + d more = 1000 * m + 100 * o + 10 * r + e money = 10000 * m + 1000 * o + 100 * n + 10 * e + y if send + more == money: all_solutions.append((send, more, money)) return all_solutions print(solutions()) S E N D + M O R E = M O N E Y [(9567, 1085, 10652)] Script-like Groovy code can often look like script-like Python code

10. Groovy code to solve cryptarithmetic def solutions() { // letters

    = ['s', 'e', 'n', 'd', 'm', 'o', 'r', 'y'] def all_solutions = [] for (s in 1..<10) for (e in 0..9) for (n in 0..9) for (d in 0..9) for (m in 1..9) for (o in 0..9) for (r in 0..9) for (y in 0..9) if ([s, e, n, d, m, o, r, y].toSet().size() == 8) { def send = 1000 * s + 100 * e + 10 * n + d def more = 1000 * m + 100 * o + 10 * r + e def money = 10000 * m + 1000 * o + 100 * n + 10 * e + y if (send + more == money) all_solutions.add([send, more, money]) } return all_solutions } print(solutions()) S E N D + M O R E = M O N E Y [[9567, 1085, 10652]]

11. Runtime Metaprogramming Wouldn’t it be great if Java’s String class

    had an isBlank() method.

12. Runtime Metaprogramming Wouldn’t it be great if Java’s String class

    had an isBlank() method. Apache Commons Lang StringUtils to the rescue! Or Groovy lets you add it to Java’s String class: String.metaClass.isBlank = { delegate.trim().size() == 0 } And there are 1750+ extensions like this already created for you in the Groovy Development Kit (GDK)

13. Runtime Metaprogramming Wouldn’t it be great if you could override

    object behavior like you can in Ruby.

14. Runtime Metaprogramming Wouldn’t it be great if you could override

    object behavior like you can in Ruby. You can with Groovy: You can participate in an object’s lifecycle There are numerous hooks to tweak behavior class Cave { String xyzzy() { 'Nothing happens' } def methodMissing(String name, args) { metaClass.invokeMethod(this, name.toLowerCase(), args) } } def cave = new Cave() assert cave.xyzzy() == 'Nothing happens' assert cave.XYZZY() == 'Nothing happens'

15. Compile-time Metaprogramming Wouldn’t it be great if you could extend

    the compiler to also know about: immutable objects pattern matching the singleton pattern the delegate pattern lazy object creation <insert your favorite feature here>

16. Compile-time Metaprogramming Wouldn’t it be great if you could extend

    the compiler to also know about: immutable objects pattern matching the singleton pattern the delegate pattern lazy object creation <insert your favorite feature here> Groovy provides an extension mechanism where you can define your own plus we give you 60+ pre-baked ones

17. @Immutable(copyWith = true) @Sortable(excludes = 'authors') @AutoExternalize class Book {

    String title List<String> authors Date published } Compile-time Metaprogramming Or 400+ lines of Java code

18. Matrix manipulation: Java vs Groovy • Same example • Same

    library Array2DRowRealMatrix{{15.1379501385,40.488531856},{21.4354570637,59.5951246537}} import org.apache.commons.math3.linear.*; public class MatrixMain { public static void main(String[] args) { double[][] matrixData = { {1d,2d,3d}, {2d,5d,3d}}; RealMatrix m = MatrixUtils.createRealMatrix(matrixData); double[][] matrixData2 = { {1d,2d}, {2d,5d}, {1d, 7d}}; RealMatrix n = new Array2DRowRealMatrix(matrixData2); RealMatrix o = m.multiply(n); // Invert p, using LU decomposition RealMatrix oInverse = new LUDecomposition(o).getSolver().getInverse(); RealMatrix p = oInverse.scalarAdd(1d).scalarMultiply(2d); RealMatrix q = o.add(p.power(2)); System.out.println(q); } }

19. Matrix manipulation: Java vs Groovy • Same example • Same

    library Array2DRowRealMatrix{{15.1379501385,40.488531856},{21.4354570637,59.5951246537}} Thanks to operator overloading and extensible tooling import org.apache.commons.math3.linear.*; public class MatrixMain { public static void main(String[] args) { double[][] matrixData = { {1d,2d,3d}, {2d,5d,3d}}; RealMatrix m = MatrixUtils.createRealMatrix(matrixData); double[][] matrixData2 = { {1d,2d}, {2d,5d}, {1d, 7d}}; RealMatrix n = new Array2DRowRealMatrix(matrixData2); RealMatrix o = m.multiply(n); // Invert p, using LU decomposition RealMatrix oInverse = new LUDecomposition(o).getSolver().getInverse(); RealMatrix p = oInverse.scalarAdd(1d).scalarMultiply(2d); RealMatrix q = o.add(p.power(2)); System.out.println(q); } }

20. Type systems - a binary toggle? Scala C# Java Haskell

    Ruby JavaScript Python Dynamic Static Groovy

21. Type systems - actually a spectrum Java Dynamic Static Groovy

    Hybrid

22. Groovy’s static nature Java Dynamic Static Groovy Java Groovy Hybrid

    Extensibility added to cater for hybrid cases but can actually be used to check more strictly than possible in Java

23. Groovy’s static nature class Cave { String xyzzy() { 'Nothing

    happens' } def methodMissing(String name, args) { metaClass.invokeMethod(this, name.toLowerCase(), args) } } @TypeChecked(extensions = 'LowerChecker.groovy') def method() { def cave = new Cave() assert cave.xyzzy() == 'Nothing happens' assert cave.XYZZY() == 'Nothing happens' } Hybrid code (partially dynamic) can sometimes be type checked If needed, we can augment type checker to handle such cases

24. class Cave { String xyzzy() { 'Nothing happens' } def

    methodMissing(String name, args) { metaClass.invokeMethod(this, name.toLowerCase(), args) } } @TypeChecked(extensions = 'LowerChecker.groovy') def method() { def cave = new Cave() assert cave.xyzzy() == 'Nothing happens' assert cave.XYZZY() == 'Nothing happens' } Groovy’s static nature methodNotFound { receiver, name, argumentList, argTypes, call -> def result = null withTypeChecker { def candidates = findMethod(receiver, name.toLowerCase(), argTypes) if (candidates && candidates.size() == 1) { result = candidates[0] } } result } LowerChecker.groovy uses extensibility DSL to augment type checker Here we give the type checker a chance to find a lowercase method

25. Groovy’s static nature - extensible type checker java.util.IllegalFormatConversionException: d !=

    java.util.Date System.out.printf("The month is %d", new Date()); [Static type checking] - For placeholder 1 [%d] expected 'int' but was 'java.util.Date' [Static type checking] - For placeholder 1 [%tB] expected 'java.util.Date' but was 'int' printf "The month is %d", new Date() printf "The month is %tB", 42 printf "The month is %tB", new Date() Runtime Compilation

26. Paul’s international travel jet lag cure tip • Let’s create

    a DSL for national park rangers to track animal eating observations • Do some outdoor physical activity upon arrival

27. Groovy’s static nature - extensible type checker species SandhillCrane eats

    ThistleGrasshopper

28. Groovy’s static nature - extensible type checker species SandhillCrane eats

    ThistleGrasshopper Terms in our DSL aren’t hard-coded classes or pre-defined constants but come from an online database of known species via type checker extension (also known as a type provider in F#)

29. Groovy’s static nature - extensible type checker species SandhillCrane eats

    ThistleGrasshopper species SandhillHawk eats DesertPocketMouse [Static type checking] - Unknown species: SandhillHawk. Did you mean: GrayHawk, FerruginousHawk, FiveSpottedHawkMoth, RedTailedHawk? @ line 16, column 13. species SandhillHawk eats DesertPocketMouse ^ 1 error

30. Groovy’s static nature: smart type inferencing def someCondition = new

    Random().nextBoolean() def component = someCondition ? new ArrayDeque() : new Stack() component.clear() // LUB: AbstractCollection or Serializable or Cloneable if (component instanceof ArrayDeque) { component.addFirst(1) // only in ArrayDeque } else if (component instanceof Stack) { component.addElement(2) // only in Stack } if (component instanceof ArrayDeque || component instanceof Stack) { println component.peek() // checked duck typing } Smart type inferencing allows dynamic looking code to be type checked: No casts required Common ancestor or Least Upper Bound (LUB) automatically discovered Union/intersection types used in specific places

31. Groovy by the numbers: Downloads v Popular & growing 2016:

    23M 2017: 50M 2018: 103M currently: approx. 16M+ per month

32. Groovy by the numbers: TIOBE Sept 2019 Groovy ranked 11th.

    Not perfect index but still great to be: Closing in on 10% rating of Java Fastest growing of top ranking languages Almost double the rating of all the other alternative JVM languages combined

33. Groovy by the numbers: Commits v Steady activity across its

    lifespan

34. Groovy by the numbers: Contributors VM ware Spring- Source G

    2O ne Pivotal O CI Hosting/governance: Sponsorship: Contributors:

35. Groovy by the numbers: Issues resolved v Reasonably responsive most

    of the time

36. Groovy by the numbers: Releases v Healthy cadence for most

    of its lifetime JSR tax & leadership transition Apache transition

37. Groovy Roadmap v Groovy 2.5 § 2.5.8 released § Macros,

    AST transformation improvements, various misc. features § JDK 7 minimum, runs on JDK 9/10/11 with warnings v Groovy 3.0 § Beta-3 out now, RC-1 in the next week or two, GA later in 2019 § Parrot parser, various misc. features § JDK 8 minimum, address most JDK 9/10/11 issues v Groovy 4.0 § Alphas out soon § Further module support, split packaging § Indy-only jar

38. Language, Libraries & Tools

39. groovy groovy-all fat jar -> fat pom groovy-ant groovy-bsf optional

    groovy-cli-commons optional groovy-cli-picocli groovy-console groovy-datetime groovy-dateutil groovy -> optional groovy-docgenerator groovy-groovydoc groovy-groovysh groovy-jaxb optional groovy-jmx Modules – Groovy 2.4, Groovy 2.5, Groovy 3.0 groovy-json groovy-json-direct optional removed groovy-jsr223 groovy-macro groovy-nio groovy-servlet groovy-sql groovy-swing groovy-templates groovy-test groovy-test-junit5 groovy-testng groovy-xml groovy-yaml optional

40. 2.5: Tooling improvements GroovyConsole: Horizontal layout, Log to file,Context menus,

    JUnit 5 Groovysh repl: :grab support CliBuilder: Annotation support, commons cli and picocli implementations, improved type safety options, improved converters

41. 2.5: JUnit 5 support via groovy and groovyConsole class MyTest

    { @Test void streamSum() { assert Stream.of(1, 2, 3).mapToInt{ i -> i }.sum() > 5 } @RepeatedTest(value=2, name = "{displayName} {currentRepetition}/{totalRepetitions}") void streamSumRepeated() { assert Stream.of(1, 2, 3).mapToInt{i -> i}.sum() == 6 } private boolean isPalindrome(s) { s == s.reverse() } @ParameterizedTest // requires org.junit.jupiter:junit-jupiter-params @ValueSource(strings = [ "racecar", "radar", "able was I ere I saw elba" ]) void palindromes(String candidate) { assert isPalindrome(candidate) } @TestFactory def dynamicTestCollection() {[ dynamicTest("Add test") { -> assert 1 + 1 == 2 }, dynamicTest("Multiply Test") { -> assert 2 * 3 == 6 } ]} } JUnit5 launcher: passed=8, failed=0, skipped=0, time=246ms

42. 2.5: Includes JUnit 5 native annotations and JUnit frameworks @Grab('net.jqwik:jqwik:1.1.3')

    import net.jqwik.api.* class StringConcatenationTests { @Property void 'size of concatenated string is sum of size of each'( @ForAll String s1, @ForAll String s2 ) { String conc = s1 + s2 assert conc.size() == s1.size() + s2.size() } } import spock.lang.* class Spock2Spec extends Specification { def "my test"() { expect: true } } 2.5.8 and later 3.0.0-beta-2 and later

43. 3.0: New operators: identity import groovy.transform.EqualsAndHashCode @EqualsAndHashCode class Creature {

    String type } def cat = new Creature(type: 'cat') def copyCat = cat def lion = new Creature(type: 'cat') assert cat.equals(lion) // Java logical equality assert cat == lion // Groovy shorthand operator assert cat.is(copyCat) // Groovy identity assert cat === copyCat // operator shorthand assert cat !== lion // negated operator shorthand

44. 3.0: New operators: negated variants assert 45 !instanceof Date assert

    4 !in [1, 3, 5, 7]

45. 3.0: New operators: Elvis assignment import groovy.transform.ToString @ToString class Element

    { String name int atomicNumber } def he = new Element(name: 'Helium') he.with { // name = name != null ? name : 'Hydrogen' // Java name = name ?: 'Hydrogen' // existing Elvis operator atomicNumber ?= 2 // new Elvis assignment shorthand } assert he.toString() == 'Element(Helium, 2)'

46. 3.0: Safe indexing String[] array = ['a', 'b'] assert 'b'

    == array?[1] // get using normal array index array?[1] = 'c' // set using normal array index assert 'c' == array?[1] array = null assert null == array?[1] // return null for all index values array?[1] = 'c' // quietly ignore attempt to set value assert array == null

47. 3.0: GroovyDoc comments as metadata import org.codehaus.groovy.control.* import static groovy.lang.groovydoc.GroovydocHolder.DOC_COMMENT

    def ast = new CompilationUnit().tap { addSource 'myScript.groovy', ''' /** class doco */ class MyClass { /** method doco */ def myMethod() {} } ''' compile Phases.SEMANTIC_ANALYSIS }.ast def classDoc = ast.classes[0].groovydoc assert classDoc.content.contains('class doco') def methodDoc = ast.classes[0].methods[0].groovydoc assert methodDoc.content.contains('method doco') Requires: -Dgroovy.attach.groovydoc=true

48. 3.0: Groovydoc comments: runtime embedding class Foo { /**@ fo

    fum */ def bar() { } } Foo.methods.find{ it.name == 'bar' }.groovydoc.content.contains('fo fum') Requires: -Dgroovy.attach.runtime.groovydoc=true

49. GDK Enhancements

50. Java Enhancements bundled with Groovy v ~300 added in 2.5,

    50+ added in 3.0

51. 2.4: GDK Enhancements byte, int, boolean, float, short, double, char,

    long javax.swing JTabbedPane, ButtonGroup, ListModel, JPopupMenu, JComboBox, JToolBar, MutableComboBoxModel, AbstractButton, JMenuBar, DefaultComboBoxModel, DefaultListModel, JMenu javax.swing.table DefaultTableModel, TableModel, TableColumnModel javax.swing.tree TreePath, MutableTreeNode, DefaultMutableTreeNode, TreeNode javax.script ScriptEngineManager, ScriptEngine groovy.sql GroovyResultSet groovy.lang GroovyObject, ListWithDefault, GString, Closure java.nio.file Path java.io PrintWriter, File, Writer, InputStream, BufferedWriter, Closeable, DataInputStream, OutputStream, Reader, PrintStream, BufferedReader, ObjectOutputStream, ObjectInputStream java.awt Container java.math BigInteger, BigDecimal java.sql ResultSet, Date, Timestamp, ResultSetMetaData java.util AbstractMap, Collection, SortedSet, Calendar, Timer, BitSet, Enumeration, List, Date, ResourceBundle, Map, Iterator, Set, SortedMap, AbstractCollection java.util.regex Pattern, Matcher java.util.concurrent BlockingQueue java.lang StringBuffer, Enum, ClassLoader, Class, Comparable, Iterable, StringBuilder, Character, Long, Appendable, System, Integer, Double, String, Number, Float, Thread, Object, Process, CharSequence, Boolean, Byte, Object[] java.net ServerSocket, Socket, URL org.w3c.dom Element, NodeList byte, int, boolean, float, short, double, char, long

52. 2.4: GDK Enhancements: Object[] Object min() Object min(Closure closure) Object

    min(Comparator comparator) Object[] minus(Iterable removeMe) Object[] minus(Object removeMe) Object[] minus(Object[] removeMe) Object[] plus(Iterable right) Object[] plus(Object right) Object[] plus(Object[] right) Object[] plus(Collection right) Object[] reverse() Object[] reverse(boolean mutate) Object[] reverseEach(Closure closure) Object[] sort() Object[] sort(boolean mutate) Object[] sort(boolean mutate, Closure closure) Object[] sort(boolean mutate, Comparator comparator) Object[] sort(Closure closure) Object[] sort(Comparator comparator) Object[] swap(int i, int j) Object[] tail() Object[] take(int num) Object[] takeRight(int num) Object[] takeWhile(Closure condition) List toList() Object[] toSorted() Object[] toSorted(Closure condition) Object[] toSorted(Comparator comparator) Object[] toUnique() Object[] toUnique(Closure condition) Object[] toUnique(Comparator comparator) Map collectEntries() Map collectEntries(Closure transform) Map collectEntries(Map collector) Map collectEntries(Map collector, Closure transform) List collectMany(Closure projection) Number count(Closure closure) Map countBy(Closure closure) Object[] drop(int num) Object[] dropRight(int num) Object[] dropWhile(Closure condition) Object find(Closure condition) Collection findAll() Collection findAll(Closure condition) Object first() List getAt(EmptyRange range) List getAt(IntRange range) List getAt(ObjectRange range) List getAt(Range range) List getAt(Collection indices) IntRange getIndices() Collection grep() Collection grep(Object filter) Map groupBy(Closure closure) Object head() Object[] init() Object inject(Closure closure) Object inject(Object initialValue, Closure closure) Iterator iterator() Object last() Object max() Object max(Closure closure) Object max(Comparator comparator)

53. 2.5 Nearly 300 GDK Additions java.io File void write(String text,

    boolean writeBom) void write(String text, String charset, boolean writeBom) void append(Object text, boolean writeBom) void append(Writer writer, boolean writeBom) void append(Object text, String charset, boolean writeBom) void append(Writer writer, String charset, boolean writeBom) void append(Reader reader, boolean writeBom) void append(Reader reader, String charset, boolean writeBom) BufferedWriter newWriter(String charset, boolean append, boolean writeBom) java.lang AutoCloseable withCloseable Byte 2 CharSequence isBlank, sha256, md5, digest Class getLocation Iterable 17 Object 2 Object[] 21 String append, append, decodeBase64Url Throwable asString java.math BigDecimal 4 java.nio.file Path 11 java.time 16 145 java.time.chrono ChronoPeriod 3 java.time.temporal 3 7 java.util 10 55 byte 5

54. 2.5 ~300 GDK Additions assert ('a'..'h').chop(2, 4) == [['a', 'b'],

    ['c', 'd', 'e', 'f']] assert 'xyzzy'.replace(xy:'he', z:'l', y: 'o') == 'hello' def map = [ant:1, bee:2, cat:3] map.removeAll { k,v -> k == 'bee' } assert map == [ant:1, cat:3] @TupleConstructor(excludes = 'fullname') class Artist { String first, last, fullname } def elvis = new Artist('Elvis', 'Presley').tap{ fullname = "$first $last" } assert 'abcd'.startsWithAny('ab', 'AB') assert 'abc123'.md5() == 'e99a18c428cb38d5f260853678922e03'

55. 3.0: 50+ GDK Additions groovy.lang Closure 2 GString 8 java.lang

    CharSequence 11 Iterable 2 Object 1 Object[] 5 String 8 Thread 1 java.util Iterator Object average() Object average(Closure closure) List void shuffle() void shuffle(Random rnd) <T> List<T> shuffled() <T> List<T> shuffled(Random rnd) byte 1 double 1 float 1 int 1 long 1 short 1

56. House pricing revisited – Apache Beam … static buildPipeline(Pipeline p)

    { def features = [ 'price', 'bedrooms', 'bathrooms', 'sqft_living', 'sqft_living15', 'lat', 'sqft_above', 'grade', 'view', 'waterfront', 'floors' ] def readCsvChunks = new DoFn<String, double[][]>() { @ProcessElement void processElement(@Element String path, OutputReceiver<double[][]> receiver) throws IOException { def chunkSize = 6000 def table = Table.read().csv(path) table = table.dropWhere(table.column("bedrooms").isGreaterThan(30)) def idxs = 0..<table.rowCount() for (nextChunkIdxs in idxs.shuffled().collate(chunkSize)) { def chunk = table.rows(*nextChunkIdxs) receiver.output(chunk.as().doubleMatrix(*features)) } sleep 2000 } } def fitModel = new DoFn<double[][], double[]>() { @ProcessElement void processElement(@Element double[][] rows, OutputReceiver<double[]> receiver) throws IOException { double[] model = new OLS(rows.collect{ it[1..-1] } as double[][], rows.collect{ it[0] } as double[]).with{ [it.intercept(), *it.coefficients()] } receiver.output(model) } } def evalModel = { double[][] chunk, double[] model -> double intercept = model[0] double[] coefficients = model[1..-1] def predicted = chunk.collect { row -> intercept + Math.dot(row[1..-1] as double[], coefficients) } def residuals = chunk.toList().indexed().collect { idx, row -> predicted[idx] - row[0] } def rmse = sqrt(StatUtils.sumSq(residuals as double[]) / chunk.size()) [rmse, residuals.average(), chunk.size()] as double[] } … Map filename to chunks Map chunk to model Map chunk to stats

57. Compile-time Metaprogramming

58. Compile-time metaprogramming v AST transforms: 11 added in 2.5 with

    major reworking, 2 in 3.0 v Macros, Matchers, Macro methods

59. @ASTTest @AutoClone @AutoExternalize @BaseScript @Bindable @Builder @Canonical @Category @CompileDynamic @CompileStatic

    @ConditionalInterrupt @Delegate @EqualsAndHashCode @ExternalizeMethods @ExternalizeVerifier @Field @Newify @NotYetImplemented @PackageScope @Singleton @Sortable @SourceURI @Synchronized @TailRecursive @ThreadInterrupt @TimedInterrupt @ToString @Trait * @TupleConstructor @TypeChecked @Vetoable @WithReadLock @WithWriteLock AST Transformations – Groovy 2.4, Groovy 2.5, Groovy 3.0 @AutoFinal @AutoImplement @ImmutableBase @ImmutableOptions @MapConstructor @NamedDelegate @NamedParam @NamedParams @NamedVariant @PropertyOptions @VisibilityOptions @GroovyDoc * @NullCheck * Not normally used directly @Grab • @GrabConfig • @GrabResolver • @GrabExclude @Grapes @Immutable @IndexedProperty @InheritConstructors @Lazy Logging: • @Commons • @Log • @Log4j • @Log4j2 • @Slf4j @ListenerList @Mixin

60. @ASTTest @AutoClone @AutoExternalize @BaseScript @Bindable @Builder @Canonical @Category @CompileDynamic @CompileStatic

    @ConditionalInterrupt @Delegate @EqualsAndHashCode @ExternalizeMethods @ExternalizeVerifier @Field @Newify @NotYetImplemented @PackageScope @Singleton @Sortable @SourceURI @Synchronized @TailRecursive @ThreadInterrupt @TimedInterrupt @ToString @Trait @TupleConstructor @TypeChecked @Vetoable @WithReadLock @WithWriteLock AST Transformations – Groovy 2.4, Groovy 2.5, Groovy 3.0 @AutoFinal @AutoImplement @ImmutableBase @ImmutableOptions @MapConstructor @NamedDelegate @NamedParam @NamedParams @NamedVariant @PropertyOptions @VisibilityOptions @GroovyDoc @NullCheck * Improved in 2.5 @Grab • @GrabConfig • @GrabResolver • @GrabExclude @Grapes @Immutable @IndexedProperty @InheritConstructors @Lazy Logging: • @Commons • @Log • @Log4j • @Log4j2 • @Slf4j @ListenerList @Mixin

61. AST Transformations – Groovy 2.4, Groovy 2.5 Numerous annotations have

    additional annotation attributes, e.g @TupleConstructor String[] excludes() default {} String[] includes() default {Undefined.STRING} boolean includeProperties() default true boolean includeFields() default false boolean includeSuperProperties() default false boolean includeSuperFields() default false boolean callSuper() default false boolean force() default false boolean defaults() default true boolean useSetters() default false boolean allNames() default false boolean allProperties() default false String visibilityId() default Undefined.STRING Class pre() default Undefined.CLASS Class post() default Undefined.CLASS

62. AST Transformations AST transforms are easy to use but a

    little bit involved to write Groovy 2.1: wouldn’t it be great if you could define annotations in terms of other annotations

63. AST Transformations AST transforms are easy to use but a

    little bit involved to write Groovy 2.1: wouldn’t it be great if you could define annotations in terms of other annotations Groovy added annotation collectors, also known as meta-annotations But they were only used sparingly until ...

64. AST Transformations – Groovy 2.5 Some existing annotations totally reworked:

    @Canonical and @Immutable are now meta-annotations (annotation collectors)

65. AST Transforms: @Canonical becomes meta-annotation @Canonical => @ToString, @TupleConstructor, @EqualsAndHashCode

66. AST Transforms: @Canonical becomes meta-annotation @Canonical => @ToString, @TupleConstructor, @EqualsAndHashCode

    @AnnotationCollector( value=[ToString, TupleConstructor, EqualsAndHashCode], mode=AnnotationCollectorMode.PREFER_EXPLICIT_MERGED) public @interface Canonical { }

67. @Canonical @Canonical(cache = true, useSetters = true, includeNames = true)

    class Point { int x, y }

68. @Canonical @Canonical(cache = true, useSetters = true, includeNames = true)

    class Point { int x, y } @ToString(cache = true, includeNames = true) @TupleConstructor(useSetters = true) @EqualsAndHashCode(cache = true) class Point { int x, y }

69. AST Transforms: @Immutable becomes meta-annotation @Immutable class Point { int

    x, y }

70. AST Transforms: @Immutable becomes meta-annotation @Immutable class Point { int

    x, y } @ToString(includeSuperProperties = true, cache = true) @EqualsAndHashCode(cache = true) @ImmutableBase @ImmutableOptions @PropertyOptions(propertyHandler = ImmutablePropertyHandler) @TupleConstructor(defaults = false) @MapConstructor(noArg = true, includeSuperProperties = true, includeFields = true) @KnownImmutable class Point { int x, y }

71. AST Transforms: @Immutable enhancements An immutable class with one constructor

    making it dependency injection friendly import groovy.transform.* import groovy.transform.options.* @ImmutableBase @PropertyOptions(propertyHandler = ImmutablePropertyHandler) @Canonical(defaults = false) class Shopper { String first, last Date born List items } println new Shopper('John', 'Smith', new Date(), [])

72. AST Transforms: @TupleConstructor pre/post import groovy.transform.ToString import groovy.transform.TupleConstructor import static

    groovy.test.GroovyAssert.shouldFail @ToString @TupleConstructor( pre = { first = first?.toLowerCase(); assert last }, post = { this.last = first?.toUpperCase() } ) class Actor { String first, last } assert new Actor('Johnny', 'Depp').toString() == 'Actor(johnny, JOHNNY)' shouldFail(AssertionError) { println new Actor('Johnny') }

73. AST Transforms: @TupleConstructor enhancements Visibility can be specified, also works

    with MapConstructor and NamedVariant import groovy.transform.* import static groovy.transform.options.Visibility.PRIVATE @TupleConstructor @VisibilityOptions(PRIVATE) class Person { String name static makePerson(String first, String last) { new Person("$first $last") } } assert 'Jane Eyre' == Person.makePerson('Jane', 'Eyre').name def publicCons = Person.constructors assert publicCons.size() == 0

74. @NamedVariant @CompileStatic/Java friendly named args import ... def chooseBox(Color c,

    Integer size) { boolean special = RED.equals(c) boolean large = size > 12 String desc = "${(large ? 'large ' : '')}" + "${special ? 'special' : 'standard'}" println "You chose a $desc box" } chooseBox(RED, 20) //You chose a large special box chooseBox(BLUE, 10) //You chose a standard box

75. @NamedVariant @CompileStatic/Java friendly named args import ... def chooseBox(Map map)

    { boolean special = RED.equals(map.color) boolean large = (Integer) map.size > 12 String desc = "${(large ? 'large ' : '')}" + "${special ? 'special' : 'standard'}" println "You chose a $desc box" } chooseBox(color: RED, size: 20) chooseBox(size: 10, color: BLUE)

76. @NamedVariant import ... @TypeChecked def chooseBox(Map map) { boolean special

    = RED.equals(map.colour) // Silent fail! boolean large = (Integer) map.size > 12 String desc = "${(large ? 'large ' : '')}" + "${special ? 'special' : 'standard'}" println "You chose a $desc box" } @TypeChecked def method() { chooseBox(color: "red", size: 20) chooseBox(colour: BLUE, size: '10') // GroovyCastException at runtime! } method() If we have parameters of different types the best we can use is: Map<String, Object> which doesn’t give much information to the type checker. Impacts method writer and user.

77. @NamedVariant import ... @TypeChecked @NamedVariant def chooseBox(Color color, Integer size)

    { boolean special = RED.equals(color) boolean large = size > 12 String desc = "${(large ? 'large ' : '')}" + "${special ? 'special' : 'standard'}" println "You chose a $desc box" } @TypeChecked def method() { chooseBox(color: "red", size: 20) chooseBox(colour: BLUE, size: '10') } method()

78. import ... @TypeChecked @NamedVariant def chooseBox(Color color, Integer size) {

    boolean special = RED.equals(color) boolean large = size > 12 String desc = "${(large ? 'large ' : '')}" + "${special ? 'special' : 'standard'}" println "You chose a $desc box" } @TypeChecked def method() { chooseBox(color: "red", size: 20) chooseBox(colour: BLUE, size: '10') } method() @NamedVariant def chooseBox(@NamedParams([ @NamedParam(value = 'color', type = Color), @NamedParam(value = 'size', type = Integer)]) Map __namedArgs) { chooseBox(__namedArgs.color, __namedArgs.size) }

79. import ... @TypeChecked @NamedVariant def chooseBox(Color color, Integer size) {

    boolean special = RED.equals(color) boolean large = size > 12 String desc = "${(large ? 'large ' : '')}" + "${special ? 'special' : 'standard'}" println "You chose a $desc box" } @TypeChecked def method() { chooseBox(color: "red", size: 20) chooseBox(colour: BLUE, size: '10') } method() @NamedVariant def chooseBox(@NamedParams([ @NamedParam(value = 'color', type = Color), @NamedParam(value = 'size', type = Integer)]) Map __namedArgs) { chooseBox(__namedArgs.color, __namedArgs.size) } [Static type checking] - parameter for named arg 'size' has type 'java.lang.String' but expected 'java.lang.Integer' [Static type checking] - parameter for named arg 'color' has type 'java.lang.String' but expected 'java.awt.Color'

80. Other frameworks using AST transforms: Spock, Micronaut, Grails, ... @IgnoreIf({

    !System.getenv('TWITTER_OAUTH_ACCESS_TOKEN') || !System.getenv('TWITTER_OAUTH_ACCESS_TOKEN_SECRET') || !System.getenv('TWITTER_OAUTH_CONSUMER_KEY') || !System.getenv('TWITTER_OAUTH_CONSUMER_SECRET') }) class UpdateStatusSpec extends Specification { @Shared @AutoCleanup EmbeddedServer embeddedServer = ApplicationContext.run(EmbeddedServer) @Shared RxHttpClient httpClient = embeddedServer.applicationContext.createBean(RxHttpClient, embeddedServer.getURL()) void "test post tweet"() { when: UpdateResult result = httpClient.toBlocking() .retrieve(HttpRequest.POST('/update-status', new Message(text:"My New Tweet")), UpdateResult) then: result.url != null } }

81. Macros motivation AST transforms are still a little bit involved

    to write Wouldn’t it be great if we could simplify writing them further Rather than having to know about compiler internal details and data structures, what if it was as simple as writing Groovy itself!

82. Macros motivation AST transforms are still a little bit involved

    to write Wouldn’t it be great if we could simplify writing them further Rather than having to know about compiler internal details and data structures, what if it was as simple as writing Groovy itself! Transforms often need to find some part of the AST to replace and then replace it with a new chunk of AST Matchers help with the finding task -- think regex for AST Macros help with specifying the replacement

83. Without Macros import org.codehaus.groovy.ast.* import org.codehaus.groovy.ast.stmt.* import org.codehaus.groovy.ast.expr.* def ast

    = new ReturnStatement( new ConstructorCallExpression( ClassHelper.make(Date), ArgumentListExpression.EMPTY_ARGUMENTS ) ) def ast = macro { return new Date() }

84. With Macros import org.codehaus.groovy.ast.* import org.codehaus.groovy.ast.stmt.* import org.codehaus.groovy.ast.expr.* def ast

    = new ReturnStatement( new ConstructorCallExpression( ClassHelper.make(Date), ArgumentListExpression.EMPTY_ARGUMENTS ) ) def ast = macro { return new Date() }

85. Macros v Variations: • Expressions, Statements, Classes • Supports variable

    substitution, specifying compilation phase def varX = new VariableExpression('x') def varY = new VariableExpression('y') def pythagoras = macro { return Math.sqrt($v{varX} ** 2 + $v{varY} ** 2).intValue() }

86. Macros v Variations: • Expressions, Statements, Classes • Supports variable

    substitution, specifying compilation phase @Statistics class Person { Integer age String name } def p = new Person(age: 12, name: 'john') assert p.methodCount == 0 assert p.fieldCount == 2 ClassNode buildTemplateClass(ClassNode reference) { def methodCount = constX(reference.methods.size()) def fieldCount = constX(reference.fields.size()) return new MacroClass() { class Statistics { java.lang.Integer getMethodCount() { return $v { methodCount } } java.lang.Integer getFieldCount() { return $v { fieldCount } } } } }

87. AST Matching v AST Matching: • Selective transformations, filtering, testing

    • Supports placeholders Expression transform(Expression exp) { Expression ref = macro { 1 + 1 } if (ASTMatcher.matches(ref, exp)) { return macro { 2 } } return super.transform(exp) }

88. Macro methods motivation Local AST transforms are triggered by an

    annotation Annotations can only be placed in certain places, e.g. classes, methods, fields, etc. What if we opened up the mechanism that causes expansion of “macro” to work for user defined macro methods? This would allow very flexible expansion: macro1(macro2 { 2 } + macro3 { 5 })

89. Macro method examples: match def fact(num) { return match(num) {

    when String then fact(num.toInteger()) when(0 | 1) then 1 when 2 then 2 orElse num * fact(num - 1) } } assert fact("5") == 120 See: https://github.com/touchez-du-bois/akatsuki

90. Macro method examples: doWithData Spock inspired @Grab('org.spockframework:spock-core:1.0-groovy-2.4') import spock.lang.Specification class

    MathSpec extends Specification { def "maximum of two numbers"(int a, int b, int c) { expect: Math.max(a, b) == c where: a | b | c 1 | 3 | 3 7 | 4 | 7 0 | 0 | 0 } }

91. Macro method examples: doWithData See: https://github.com/touchez-du-bois/akatsuki doWithData { dowith: assert

    a + b == c where: a | b || c 1 | 2 || 3 4 | 5 || 9 7 | 8 || 15 }

92. Java Compatibility

93. Groovy has a low learning curve, particularly for Java developers

94. Groovy is Java's friend Seamless integration • IDEs provide cross-language

    compile, navigation, and refactoring • Arbitrarily mix source language • Drop-in replace any class • Overloaded methods • Syntax alignment • Shared data types Java Groovy

95. Groovy 2.5 Java compatibility enhancements v Repeated annotations (JEP 120

    JDK8) v Method parameter names (JEP 118 JDK8) v GDK methods for java.time package (JSR 310 JDK8) v Annotations in more places (JSR 308) v Runs on JDK 9/10/11 (currently with warnings) v Repackaging towards JPMS

96. Parrot looping // classic Java-style do..while loop def count =

    5 def fact = 1 do { fact *= count-- } while(count > 1) assert fact == 120

97. Parrot looping // classic for loop but now with extra

    commas def facts = [] def count = 5 for (int fact = 1, i = 1; i <= count; i++, fact *= i) { facts << fact } assert facts == [1, 2, 6, 24, 120]

98. Parrot looping // multi-assignment def (String x, int y) =

    ['foo', 42] assert "$x $y" == 'foo 42' // multi-assignment goes loopy def baNums = [] for (def (String u, int v) = ['bar', 42]; v < 45; u++, v++) { baNums << "$u $v" } assert baNums == ['bar 42', 'bas 43', 'bat 44']

99. Java-style array initialization def primes = new int[] {2, 3,

    5, 7, 11} assert primes.size() == 5 && primes.sum() == 28 assert primes.class.name == '[I' def pets = new String[] {'cat', 'dog'} assert pets.size() == 2 && pets.sum() == 'catdog' assert pets.class.name == '[Ljava.lang.String;' // traditional Groovy alternative still supported String[] groovyBooks = ['Groovy in Action', 'Making Java Groovy'] assert groovyBooks.every{ it.contains('Groovy') }

100. Better Java syntax support: try with resources class FromResource extends

     ByteArrayInputStream { @Override void close() throws IOException { super.close() println "FromResource closing" } FromResource(String input) { super(input.toLowerCase().bytes) } } class ToResource extends ByteArrayOutputStream { @Override void close() throws IOException { super.close() println "ToResource closing" } }

101. Better Java syntax support: try with resources def wrestle(s) {

     try ( FromResource from = new FromResource(s) ToResource to = new ToResource() ) { to << from return to.toString() } } assert wrestle("ARM was here!").contains('arm') ToResource closing FromResource closing

102. Better Java syntax support: try with resources // some Groovy

     friendliness without explicit types def wrestle(s) { try ( from = new FromResource(s) to = new ToResource() ) { to << from return to.toString() } } assert wrestle("ARM was here!").contains('arm') ToResource closing FromResource closing

103. Better Java syntax support: try with resources // some Groovy

     friendliness without explicit types def wrestle(s) { try ( from = new FromResource(s) to = new ToResource() ) { to << from return to.toString() } } assert wrestle("ARM was here!").contains('arm') ToResource closing FromResource closing But remember Groovy’s IO/Resource extension methods may be better

104. Better Java syntax support: try with resources def wrestle(s) {

     new FromResource(s).withCloseable { from -> new ToResource().withCloseable { to -> to << from to.toString() } } } ToResource closing FromResource closing But remember Groovy’s IO/Resource extension methods may be better

105. Better Java syntax support: try with resources Java 9 def

     a = 1 def resource1 = new Resource(1) try (resource1) { a = 2 } assert Resource.closedResourceIds == [1] assert 2 == a

106. Better Java syntax support: nested blocks { def a =

     1 a++ assert 2 == a } try { a++ // not defined at this point } catch(MissingPropertyException ex) { println ex.message } { { // inner nesting is another scope def a = 'banana' assert a.size() == 6 } def a = 1 assert a == 1 }

107. Better Java syntax support: var (JDK10/11) v Local variables (JDK10)

     v Lambda params (JDK11)

108. Lambdas import static java.util.stream.Collectors.toList (1..10).forEach(e -> { println e })

     assert (1..10).stream() .filter(e -> e % 2 == 0) .map(e -> e * 2) .collect(toList()) == [4, 8, 12, 16, 20]

109. Lambdas – all the shapes // general form def add

     = (int x, int y) -> { def z = y; return x + z } assert add(3, 4) == 7 // curly braces are optional for a single expression def sub = (int x, int y) -> x - y assert sub(4, 3) == 1 // parameter types and // explicit return are optional def mult = (x, y) -> { x * y } assert mult(3, 4) == 12 // no parentheses required for a single parameter with no type def isEven = n -> n % 2 == 0 assert isEven(6) assert !isEven(7) // no arguments case def theAnswer = () -> 42 assert theAnswer() == 42 // any statement requires braces def checkMath = () -> { assert 1 + 1 == 2 } checkMath() // example showing default parameter values (no Java equivalent) def addWithDefault = (int x, int y = 100) -> x + y assert addWithDefault(1, 200) == 201 assert addWithDefault(1) == 101

110. Method pointers (AKA method closures) up to 2.5 Instance method

     Static method Class target Not meaningful: MissingMethodException def asHex = Integer.&toHexString assert asHex(10) == 'a' Instance target def tenPlus = 10G.&add assert tenPlus(1G) == 11G assert tenPlus(2G) == 12G // generally discouraged def asOctal = 42.&toOctalString assert asOctal(10) == '12' • Mechanism to treat an existing method as a Closure • Has the full power of Closures available

111. Method pointers (AKA method closures) 3.0+ • Adds Class/instance method

     support • an extra first ‘instance’ parameter is added to the method • Allows ‘new’ to be used to reference constructor Instance method Static method Class target def file = File.&new def isHidden = File.&isHidden assert isHidden(file('.git')) def asHex = Integer.&toHexString assert asHex(10) == 'a' Instance target def tenPlus = 10G.&add assert tenPlus(1G) == 11G assert tenPlus(2G) == 12G // generally discouraged def asOctal = 42.&toOctalString assert asOctal(10) == '12'

112. Method references (syntax from Java 8+) Groovy 3.0+ • In

     one sense an equivalent concept • But see implementation details later Instance method Static method Class target def file = File::new def isHidden = File::isHidden assert isHidden(file('.git')) def asHex = Integer::toHexString assert asHex(10) == 'a' Instance target def tenPlus = 10G::add assert tenPlus(1G) == 11G assert tenPlus(2G) == 12G // generally discouraged def asOctal = 42::toOctalString assert asOctal(10) == '12'

113. Method references: implementation details • Implemented as method closures for

     dynamic Groovy def upper = String::toUpperCase println 'upper = ' + upper // upper = org.codehaus.groovy.runtime.MethodClosure@7aa9e414 def plus = BigInteger::add def fivePlus = plus.curry(5G).memoizeAtLeast(10) assert fivePlus(3G) == 8G assert fivePlus << 4G == 9G

114. Method references: implementation details • Implemented as method closures for

     dynamic Groovy • Implemented similar to Java for static Groovy def upper = String::toUpperCase println 'upper = ' + upper // upper = org.codehaus.groovy.runtime.MethodClosure@7aa9e414 @groovy.transform.CompileStatic def method() { Function<String, String> lower = String::toLowerCase println 'lower = ' + lower } // lower = MethodRefG$$Lambda$135/0x0000000801376440@5411dd90

115. Method references: implementation details • Implemented as method closures for

     dynamic Groovy • Often slower and higher overheads • But more concise as less type information needed • Full power of Closures: • memoize*, n/r/curry, asWriteable, de/rehydrate, left/rightShift • Implemented similar to Java for static Groovy • Can require more type information when type can’t be inferred • Often faster • A few edge cases currently fall back to method closures but will be fixed over time • If you need @CompileStatic for other reasons but want dynamic behavior, just use method pointers dynamic @CompileStatic .& operator Method closure Method closure :: operator Method closure Native method reference

116. Default methods in interfaces interface Greetable { String target() default

     String salutation() { 'Greetings' } default String greet() { "${salutation()}, ${target()}" } } class Greetee implements Greetable { String name @Override String target() { name } } def daniel = new Greetee(name: 'Daniel') assert 'Greetings, Daniel' == "${daniel.salutation()}, ${daniel.target()}" assert 'Greetings, Daniel' == daniel.greet() Currently implemented using traits

117. Illegal Access Warnings

118. JDK 9+ improvements: Split packages (from beta-2) groovy: groovy.xml.QName groovy.namespace.QName

     groovy-ant: groovy.util (includes AntBuilder) groovy.ant groovy-console: groovy.ui.ConsoleApplet groovy.inspect groovy.console groovy.inspect.swingui groovy.console.ui groovy.ui groovy.console.ui groovy-groovysh: org.codehaus.groovy.tools.shell org.codehaus.groovy.groovysh.tools Deprecated Copied and original deprecated

119. JDK 9+ improvements (split package changes) groovy-jmx: groovy.util.GroovyMBean groovy.jmx groovy-nio:

     org.codehaus.groovy.runtime.WritablePath org.apache.groovy.nio.runtime org.codehaus.groovy.runtime.NioGroovyMethods org.apache.groovy.nio.extensions.NioExtensions groovy-swing (SwingBuilder only produces new classes): org.codehaus.groovy.binding org.apache.groovy.swing.binding org.codehaus.groovy.runtime org.apache.groovy.swing.extensions groovy.model groovy.swing.model groovy.inspect.swingui org.apache.groovy.swing.table

120. JDK 9+ improvements (split package changes) groovy-test: org.codehaus.groovy.runtime.ScriptTestAdapter org.apache.groovy.test groovy.transform.NotYetImplemented

     groovy.test groovy.util (includes GroovyTestCase) groovy.test groovy.lang groovy.test groovy-xml: groovy.util (includes XmlParser & XmlSlurper) groovy.xml org.codehaus.groovy.tools.xml.DomToGroovy org.apache.groovy.xml.tools • Migrate over lifetime of Groovy 3 with some caveats • Don’t mix and match old and new versions of classes • Deprecated classes removed in 4.0 to make jars module friendly

121. Did you know?

122. Groovy works with your favorite big data/ML framework

123. Groovy works with your favorite big data/ML framework Source: https://mxnet.apache.org/versions/master/faq/why_mxnet.html

124. Classification Overview Clustering: • Predicting class of some data Algorithms:

     • Logistic Regression • Naïve Bayes • Stochastic Gradient Descent • K-Nearest Neighbours • Decision Tree • Random Forest • Support Vector Machine Aspects: • Over/underfitting • Ensemble • Confusion Applications: • Image recognition • Speech recognition • Medical daignosis

125. Apache MXNET Object detection … static detectObjects(String modelPath, String imagePath,

     inputShape) { def context = [Context.cpu()] def inputDescriptor = new DataDesc("data", inputShape, DType.Float32(), "NCHW") def objDet = new ObjectDetector(modelPath, [inputDescriptor], context, 0) return objDet.imageObjectDetect(ObjectDetector.loadImageFromFile(imagePath), 3) } def (imagePath, modelPath) = downloadModelImage() def image = ImageIO.read(imagePath as File) def (w, h) = image.with{ [it.width, it.height] } def count = 1 // batch size def channels = 3 // for RGB def inputShape = new Shape(count, channels, w * 0.8 as int, h) def results = detectObjects(modelPath, imagePath, inputShape).sum() def boxes = results.collect {[ xmin: w * it.XMin as int, ymin: h * it.YMin as int, xmax: w * it.XMax as int, ymax: h * it.YMax as int ]} def names = results.collect{ it.className + sprintf(' %.3f', it.probability) } (0..<names.size()).each{ println "${names[it]} ${boxes[it]}" } Image.drawBoundingBox(image, boxes, names) new SwingBuilder().edt { frame(title: "${results.size()} detected objects", size: [w, h], show: true, defaultCloseOperation: DISPOSE_ON_CLOSE) { label(icon: imageIcon(image: image)) } } car 0.996 [xmin:439, ymin:62, xmax:715, ymax:173] bicycle 0.986 [xmin:144, ymin:128, xmax:547, ymax:453] dog 0.919 [xmin:102, ymin:184, xmax:344, ymax:541]

126. Regression with Least Squares Find line of “best fit”: •

     Find intercept and slope of line which minimizes the sum of the square of the residual errors Residual errors

127. House price predictions – scaling regression import … def rows

     = Table.read().csv('kc_house_data.csv') rows = rows.dropWhere(rows.column("bedrooms").isGreaterThan(30)) String[] features = [ 'price', 'bedrooms', 'bathrooms', 'sqft_living', 'sqft_living15', 'lat', 'sqft_above', 'grade', 'view', 'waterfront', 'floors' ] def data = rows.as().doubleMatrix(features) // configure to all run on local machine but could be a cluster (can be hidden in XML) def cfg = new IgniteConfiguration( peerClassLoadingEnabled: true, discoverySpi: new TcpDiscoverySpi( ipFinder: new TcpDiscoveryMulticastIpFinder( addresses: ['127.0.0.1:47500..47509'] ) ) ) static pretty(mdl, features) { def sign = { val -> val < 0 ? '- ' : '+ ' } def valIdx = { idx, val -> sprintf '%.2f*%s', val, features[idx+1] } def valIdxSign = { idx, val -> sign(val) + valIdx(idx, Math.abs(val)) } def valSign = { val -> sign(val) + sprintf('%.2f', Math.abs(val)) } def (w, i) = [mdl.weights, mdl.intercept] [valIdx(0, w.get(0)), *(1..<w.size()).collect{ valIdxSign(it, w.get(it)) }, valSign(i)].join(' ') } Read in data and select features of interest Adjust config for known local mode setup Utility method to pretty print model

128. House prices – scaling regression import … def spark =

     builder().config('spark.master', 'local[8]').appName('HousePrices').orCreate def file = '/path/to/kc_house_data.csv' int k = 5 Dataset<Row> ds = spark.read().format('csv') .options('header': 'true', 'inferSchema': 'true').load(file) double[] splits = [80, 20] def (training, test) = ds.randomSplit(splits) String[] colNames = ds.columns().toList().minus(['id', 'date', 'price']) def assembler = new VectorAssembler(inputCols: colNames, outputCol: 'features') Dataset<Row> dataset = assembler.transform(training) def lr = new LinearRegression(labelCol: 'price', maxIter: 10) def model = lr.fit(dataset) println 'Coefficients:' println model.coefficients().values()[1..-1].collect{ sprintf '%.2f', it }.join(', ') def testSummary = model.evaluate(assembler.transform(test)) printf 'RMSE: %.2f%n', testSummary.rootMeanSquaredError printf 'r2: %.2f%n', testSummary.r2 spark.stop() Coefficients: 41979.78, 80853.89, 0.15, 5412.83, 564343.22, 53834.10, 24817.09, 93195.29, -80662.68, -80694.28, -2713.58, 19.02, -628.67, 594468.23, -228397.19, 21.23, -0.42 RMSE: 187242.12 r2: 0.70

129. House pricing revisited – Apache Beam … static buildPipeline(Pipeline p)

     { def features = [ 'price', 'bedrooms', 'bathrooms', 'sqft_living', 'sqft_living15', 'lat', 'sqft_above', 'grade', 'view', 'waterfront', 'floors' ] def readCsvChunks = new DoFn<String, double[][]>() { @ProcessElement void processElement(@Element String path, OutputReceiver<double[][]> receiver) throws IOException { def chunkSize = 6000 def table = Table.read().csv(path) table = table.dropWhere(table.column("bedrooms").isGreaterThan(30)) def idxs = 0..<table.rowCount() for (nextChunkIdxs in idxs.shuffled().collate(chunkSize)) { def chunk = table.rows(*nextChunkIdxs) receiver.output(chunk.as().doubleMatrix(*features)) } sleep 2000 } } def fitModel = new DoFn<double[][], double[]>() { @ProcessElement void processElement(@Element double[][] rows, OutputReceiver<double[]> receiver) throws IOException { double[] model = new OLS(rows.collect{ it[1..-1] } as double[][], rows.collect{ it[0] } as double[]).with{ [it.intercept(), *it.coefficients()] } receiver.output(model) } } def evalModel = { double[][] chunk, double[] model -> double intercept = model[0] double[] coefficients = model[1..-1] def predicted = chunk.collect { row -> intercept + Math.dot(row[1..-1] as double[], coefficients) } def residuals = chunk.toList().indexed().collect { idx, row -> predicted[idx] - row[0] } def rmse = sqrt(StatUtils.sumSq(residuals as double[]) / chunk.size()) [rmse, residuals.average(), chunk.size()] as double[] } … Map filename to chunks Map chunk to model Map chunk to stats

130. House pricing revisited – Apache Beam … def model2out =

     new DoFn<double[], String>() { @ProcessElement void processElement(@Element double[] ds, OutputReceiver<String> out) { out.output("intercept: ${ds[0]}, coeffs: ${ds[1..-1].join(', ')}".toString()) } } def stats2out = new DoFn<double[], String>() { @ProcessElement void processElement(@Element double[] ds, OutputReceiver<String> out) { out.output("rmse: ${ds[0]}, mean: ${ds[1]}, count: ${ds[2]}".toString()) } } var csvChunks = p .apply(Create.of('/path/to/kc_house_data.csv')) .apply('Create chunks', ParDo.of(readCsvChunks)) var model = csvChunks .apply('Fit chunks', ParDo.of(fitModel)) .apply(Combine.globally(new MeanDoubleArrayCols())) var modelView = model .apply(View.<double[]>asSingleton()) csvChunks .apply(ParDo.of(new EvaluateModel(modelView, evalModel)).withSideInputs(modelView)) .apply(Combine.globally(new AggregateModelStats())) .apply('Log stats', ParDo.of(stats2out)).apply(Log.ofElements()) model .apply('Log model', ParDo.of(model2out)).apply(Log.ofElements()) } def pipeline = Pipeline.create() buildPipeline(pipeline) pipeline.run().waitUntilFinish() INFO: intercept: -3.2504584735458568E7, coeffs: -27862.329468295735, -2269.8423856962077, 198.1365232723505, 2.9357512250793327, 675276.3494316386, -2.323389992226333, 83241.42291377622, 66606.19682072607, 607326.5819472861, -30354.844181689274 INFO: rmse: 214702.83588071115, mean: -1191.2433423574066, count: 21612.0 Map model to output Map stats to output Create and run pipeline Build pipeline

131. House pricing revisited – Apache Beam … var csvChunks =

     p | Create.of('/path/to/kc_house_data.csv') | 'Create chunks' >> ParDo.of(readCsvChunks) var model = csvChunks | 'Fit chunks' >> ParDo.of(fitModel) | Combine.globally(new MeanDoubleArrayCols()) var modelView = model | View.<double[]>asSingleton() csvChunks | ParDo.of(new EvaluateModel(modelView, evalModel)).withSideInputs(modelView) | Combine.globally(new AggregateModelStats()) | 'Log stats' >> ParDo.of(stats2out) | Log.ofElements() model | 'Log model' >> ParDo.of(model2out) | Log.ofElements() } PCollection.metaClass.or = { List arg -> delegate.apply(*arg) } PCollection.metaClass.or = { PTransform arg -> delegate.apply(arg) } String.metaClass.rightShift = { PTransform arg -> [delegate, arg] } Pipeline.metaClass.or = { PTransform arg -> delegate.apply(arg) } def pipeline = Pipeline.create() buildPipeline(pipeline) pipeline.run().waitUntilFinish() INFO: intercept: -3.2504584735458568E7, coeffs: -27862.329468295735, -2269.8423856962077, 198.1365232723505, 2.9357512250793327, 675276.3494316386, -2.323389992226333, 83241.42291377622, 66606.19682072607, 607326.5819472861, -30354.844181689274 INFO: rmse: 214702.83588071115, mean: -1191.2433423574066, count: 21612.0 Data Chunk1 Model1 Stats readCsvChunks csvChunks … Chunk2 Model2 ChunkN ModelN fitModel Model … … ChunkN Stats1 Stats2 StatsN … Chunk2 Chunk1 AggregateModelStats MeanDoubleArrayCols csvChunks model modelView sideInput evalModel

132. Clustering Overview Clustering: • Grouping similar items Algorithm families: •

     Hierarchical • Partitioning k-means, x-means • Density-based • Graph-based Aspects: • Disjoint vs overlapping • Preset cluster number • Dimensionality reduction PCA • Nominal feature support Applications: • Market segmentation • Recommendation engines • Search result grouping • Social network analysis • Medical imaging

133. Whiskey flavors – scaling clustering import … def rows =

     Table.read().csv('whiskey.csv') String[] features = [ 'Distillery', 'Body', 'Sweetness', 'Smoky', 'Medicinal', 'Tobacco', 'Honey', 'Spicy', 'Winey', 'Nutty', 'Malty', 'Fruity', 'Floral' ] def data = rows.as().doubleMatrix(features) def cfg = new IgniteConfiguration(…) Ignition.start(cfg).withCloseable { ignite -> println ">>> Ignite grid started for data: ${data.size()} rows X ${data[0].size()} cols" def trainer = new KMeansTrainer().withDistance(new EuclideanDistance()).withAmountOfClusters(5) def dataCache = new SandboxMLCache(ignite).fillCacheWith(data) def vectorizer = new DoubleArrayVectorizer().labeled(Vectorizer.LabelCoordinate.FIRST) def mdl = trainer.fit(ignite, dataCache, vectorizer) println ">>> KMeans centroids" println features[1..-1].join(', ‘) mdl.centers.each { c -> println c.all().collect{ sprintf '%.4f', it.get() }.join(', ') } dataCache.destroy() } [17:18:07] Ignite node started OK (id=ea92fc20) >>> Ignite grid started for data: 86 rows X 13 cols >>> KMeans centroids Body, Sweetness, Smoky, Medicinal, Tobacco, Honey, Spicy, Winey, Nutty, Malty, Fruity, Floral 2.9091, 1.5455, 2.9091, 2.7273, 0.4545, 0.4545, 1.4545, 0.5455, 1.5455, 1.4545, 1.1818, 0.5455 1.8095, 2.4762, 1.5714, 0.4286, 0.1429, 1.5238, 1.7143, 0.8571, 0.7143, 1.6667, 1.6190, 2.0476 2.5217, 2.3913, 1.4783, 0.0435, 0.0435, 1.6087, 1.4783, 1.9565, 2.0435, 2.0435, 1.9565, 1.3913 1.3704, 2.3704, 1.0000, 0.2593, 0.0370, 0.7778, 0.9259, 0.4074, 1.5185, 1.7407, 2.0000, 2.1111 3.2500, 2.2500, 1.5000, 0.0000, 0.0000, 3.0000, 2.0000, 1.0000, 1.5000, 2.5000, 2.2500, 2.0000 [17:18:07] Ignite node stopped OK [uptime=00:00:00.368]

134. Whiskey flavors – scaling clustering import … def spark =

     builder().config('spark.master', 'local[8]').appName('Whiskey').orCreate def file = '/path/to/whiskey.csv' int k = 5 Dataset<Row> rows = spark.read().format('com.databricks.spark.csv') .options('header': 'true', 'inferSchema': 'true').load(file) String[] colNames = rows.columns().toList().minus(['RowID', 'Distillery']) def assembler = new VectorAssembler(inputCols: colNames, outputCol: 'features') Dataset<Row> dataset = assembler.transform(rows) def clusterer = new KMeans(k: k, seed: 1L) def model = clusterer.fit(dataset) println 'Cluster centers:' model.clusterCenters().each{ println it.values().collect{ sprintf '%.2f', it }.join(', ') } spark.stop() Cluster centers: 1.73, 2.35, 1.58, 0.81, 0.19, 1.15, 1.42, 0.81, 1.23, 1.77, 1.23, 1.31 2.00, 1.00, 3.00, 0.00, 0.00, 0.00, 3.00, 1.00, 0.00, 2.00, 2.00, 2.00 2.86, 2.38, 1.52, 0.05, 0.00, 1.95, 1.76, 2.05, 1.81, 2.05, 2.19, 1.71 1.53, 2.38, 1.06, 0.16, 0.03, 1.09, 1.00, 0.50, 1.53, 1.75, 2.13, 2.28 3.67, 1.50, 3.67, 3.33, 0.67, 0.17, 1.67, 0.50, 1.17, 1.33, 1.17, 0.17

135. 4.0

136. Groovy 4.0 Themes v Consolidation: v Remove split package duplicates

     and other old deprecated classes v Indy-only bytecode v Feature interaction between xforms, traits and joint compiler v Further JDK 9/10 module support v Stream based XmlSlurper, GINQ… v Type checker improvements/TLC v Improved built-in extensions @NonNull? v Pattern matching, improvements for Graal VM, jdk12 switch expressions

137. GINQ • Experimental status from p of persons leftjoin c

     of cities on p.city.name == c.name select p.name, c.name from p of persons groupby p.gender having p.gender == 'Male' select p.gender, max(p.age) from p of persons orderby p.age desc thenby p.name asc select p.name // C# LINQ method calls IQueryable<Product> source = database.Products; var results = source.Where(product => product.ReorderLevel > 20) .Select(product => new { ProductName = string.Concat("@", product.ProductName), UnitPrice = product.UnitPrice });

138. Join us: groovy.apache.org

139. objectcomputing.com © 2018, Object Computing, Inc. (OCI). All rights reserved.

     181 CONNECT WITH US 1+ (314) 579-0066 @objectcomputing objectcomputing.com THANK YOU Find me on twitter @paulk_asert