Implementing inline-*

In this talk, I give an overview of the problems that arise when trying to make Haskell inter-operate with other languages in the same address space.

One popular approach is embedding snippets of foreign languages in Haskell modules, which gave a range of libraries like inline-c, inline-r, clr-inline, inline-rust and inline-js. His pivot throughout the presentation will be inline-java, the library for embedding Java code in Haskell programs, and he will discuss how other inline-* implementations solve these problems in comparison.

If the talk is successful, attendees should leave with some useful insights to tackle their next project which requires bridging Haskell with a new language.

Transcript

1. Implementing inline-* Embedding languages in Haskell Facundo Dom´ ınguez October 12, 2018 Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 1 / 26
2. Take aways 1 Relation of embedding foreign code to other approaches 2 Overview of the implementation challenges Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 2 / 26
3. Multi-language systems Desktop applications Web applications Computing frameworks and libraries Haskell compilers! There many problems already solved. Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 3 / 26
4. Language interoperability Split in processes communicating with some IPC. e.g. client programs and RESTful backends Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 4 / 26
5. Language interoperability Split in processes communicating with some IPC. e.g. client programs and RESTful backends Compile the languages for the same runtime. e.g. .Net Eta Scala Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 4 / 26
6. Language interoperability Split in processes communicating with some IPC. e.g. client programs and RESTful backends Compile the languages for the same runtime. e.g. .Net Eta Scala Have the runtimes of the various languages cooperating on a single process. e.g. FFI clr-inline, inline-java, inline-c, inline-r, inline-rust, language-c-inline Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 4 / 26
7. Why embedded code? Unaddressed costs Converting values from one runtime to the other (a.k.a. marshaling) Managing references to objects in a foreign heap. Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 5 / 26
8. Why embedded code? String showInputDialog(Object message) Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 6 / 26
9. Why embedded code? String showInputDialog(Object message) showInputDialog :: JObject -> IO String Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 6 / 26
10. Why embedded code? String showInputDialog(Object message) showInputDialog :: JObject -> IO String Cost: Maintaining the whole interfaces aligned Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 6 / 26
11. Why embedded code? static bool challengeDialog(int x, int y) { String question = "How much is " + x + " + " + y + "?"; String response = javax.swing.JOptionPane.showInputDialog(question); try { return Integer.parseInt(response) == x + y; } catch (NumberFormatException e) { return false; } } Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 7 / 26
12. Why embedded code? static bool challengeDialog(int x, int y) { String question = "How much is " + x + " + " + y + "?"; String response = javax.swing.JOptionPane.showInputDialog(question); try { return Integer.parseInt(response) == x + y; } catch (NumberFormatException e) { return false; } } challengeDialog :: Int32 -> Int32 -> IO Bool Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 7 / 26
13. Why embedded code? static bool challengeDialog(int x, int y) { String question = "How much is " + x + " + " + y + "?"; String response = javax.swing.JOptionPane.showInputDialog(question); try { return Integer.parseInt(response) == x + y; } catch (NumberFormatException e) { return false; } } challengeDialog :: Int32 -> Int32 -> IO Bool Cost: Managing files with auxiliary foreign definitions Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 7 / 26
14. Why embedded code? challengeDialog :: Int32 -> Int32 -> IO Bool challengeDialog x y = [java| { String question = "How much is " + $x + " + " + $y + "?"; String response = javax.swing.JOptionPane.showInputDialog(question); try { return Integer.parseInt(response) == $x + $y; } catch (NumberFormatException e) { return false; } } |] Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 8 / 26
15. Implementation overview 1 Produce code for a foreign compiler 2 Invoke foreign compiler and link/embed output 3 Generate Haskell code to invoke foreign code Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 9 / 26
16. Producing foreign code module MyHaskellModule { f = ... [java| <text_of_qq1> |] ... g = ... [java| <text_of_qq2> |] ... Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 10 / 26
17. Producing foreign code module MyHaskellModule { f = ... [java| <text_of_qq1> |] ... g = ... [java| <text_of_qq2> |] ... class inline_java_MyPackage_MyHaskellModule123 { static ... method_qq1(...) { <text_of_qq1> } static ... method_qq2(...) { <text_of_qq2> } } Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 10 / 26
18. Finding antiquotations [java| { final Oid krb5Mechanism = new Oid( "1.2.840.113554.1.2.2"); final Oid krb5PrincipalNameType = new Oid("1.2.840.113554.1.2.2.1"); final GSSName serverName = $gssManager.createName ( $subject.getPrincipals().iterator().next().getName() , krb5PrincipalNameType); return Subject.doAs( $subject, new PrivilegedAction<GSSCredential>() { public GSSCredential run() { try { return $gssManager.createCredential(serverName, GSSCredential.INDEFINITE_LIFETIME, krb5Mechanism, GSSCredential.ACCEPT_ONLY); } catch (Exception e) { throw new RuntimeException(e); } } }); } |] -- find $, lex in Haskell, or lex in foreign language Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 11 / 26
19. Producing foreign code class inline_java_MyPackage_MyHaskellModule123 { static ... method_qq1(...) { <text_of_qq1> } static ... method_qq2(... $gssManager, ... $subject) { <text_of_qq2> } } Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 12 / 26
20. Finding the types subject :: J (Class "javax.security.auth.Subject") gssManager :: J (Class "org.ietf.jgss.GSSManager") [java| ... |] :: IO (J (Class "org.ietf.jgss.GSSCredentials")) Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 13 / 26
21. Finding the types Language.Haskell.TH.reify :: Name -> Q Info f = ... where x = ... y = $(do z <- newName "z" z_decl <- [d| $(return (VarP z)) = 2 |] infox <- reify 'x infoy <- reify 'y infoz <- reify z ... ) Parsing Typechecking Optimizations and Core plugins Executable code generation TH Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 14 / 26
22. Finding the types data Expr = Var Id Lit Literal App Expr Arg Lam Id Expr ... idType :: Id -> Type inlineJavaPlugin :: [(Var, Expr)] -> [(Var, Expr)] Parsing Typechecking Optimizations and Core plugins Executable code generation TH Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 15 / 26
23. Finding types [C.block| JavaVM * { JavaVM *jvm; JavaVMInitArgs vm_args; JavaVMOption *options = malloc(sizeof(JavaVMOption) * $(int n)); for(int i = 0; i < $(int n); i++) options[i].optionString = $(char **coptions)[i]; vm_args.version = JNI_VERSION_1_6; vm_args.nOptions = $(int n); vm_args.options = options; vm_args.ignoreUnrecognized = 0; JNI_CreateJavaVM(&jvm, (void**)&jniEnv, &vm_args); free(options); return jvm; } |] Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 16 / 26
24. Producing foreign code class inline_java_MyPackage_MyHaskellModule123 { static ... method_qq1(...) { <text_of_qq1> } static GSSCredentials method_qq2 (GSSManager $gssManager, Subject $subject) { <text_of_qq2> } } Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 17 / 26
25. Implementation overview 1 Produce code for a foreign compiler 2 Invoke foreign compiler and link/embed output 3 Generate Haskell code to invoke foreign code Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 18 / 26
26. Foreign compiler output After compiling the generated foreign code, store it somewhere. inline-c, inline-rust: link it together with the Haskell module. Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 19 / 26
27. Foreign compiler output After compiling the generated foreign code, store it somewhere. inline-c, inline-rust: link it together with the Haskell module. inline-java: store the bytecode in a global table Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 19 / 26
28. Bytecode in inline-java Haskell runtime JVM Bytecode table One entry per module Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 20 / 26
29. Bytecode in inline-java Haskell runtime JVM Bytecode table One entry per module static void hs_inline_java_init(void) __attribute__((constructor)); static void hs_inline_java_init(void) { ... } Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 20 / 26
30. Implementation overview 1 Produce code for a foreign compiler 2 Invoke foreign compiler and link/embed output 3 Generate Haskell code to invoke foreign code Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 21 / 26
31. Producing foreign calls inline-java: JNI defineClass :: ClassName -> ByteString -> IO JClass findClass :: ClassName -> IO JClass getStaticMethodID :: JClass -> MethodName -> MethodTypeSignature -> IO JMethodID callStaticMethod :: JClass -> JMethodID -> [...] -> IO ... Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 22 / 26
32. Producing foreign calls inline-java: JNI defineClass :: ClassName -> ByteString -> IO JClass findClass :: ClassName -> IO JClass getStaticMethodID :: JClass -> MethodName -> MethodTypeSignature -> IO JMethodID callStaticMethod :: JClass -> JMethodID -> [...] -> IO ... inline-c, inline-rust: FFI foreign import cap function_name :: ... -> IO ... Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 22 / 26
33. Producing foreign calls inline-java: JNI defineClass :: ClassName -> ByteString -> IO JClass findClass :: ClassName -> IO JClass getStaticMethodID :: JClass -> MethodName -> MethodTypeSignature -> IO JMethodID callStaticMethod :: JClass -> JMethodID -> [...] -> IO ... inline-c, inline-rust: FFI foreign import cap function_name :: ... -> IO ... inline-js, inline-r: eval :: String -> [...] -> IO ... Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 22 / 26
34. Implementation overview 1 Produce code for a foreign compiler 2 Invoke foreign compiler and link/embed output 3 Generate Haskell code to invoke foreign code Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 23 / 26
35. GHC tweaking inline-c addForeignSource :: ForeignSrcLang -> String -> Q () inline-rust addForeignFilePath :: ForeignSrcLang -> FilePath -> Q () inline-java addCorePlugin :: String -> Q () Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 24 / 26
36. Language interoperability Split in processes communicating with some IPC. e.g. client programs and RESTful backends Compile the languages for the same runtime. e.g. .Net Eta Scala Have the runtimes of the various languages cooperating on a single process. e.g. FFI clr-inline, inline-java, inline-c, inline-r, inline-rust, language-c-inline Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 25 / 26
37. Final impressions Embedding code avoids large binding libraries, has the compiler automate (foreign) build tasks, and admits relatively simple implementations. Facundo Dom´ ınguez (tweag.io) Implementing inline-* October 12, 2018 26 / 26