Foreign Inline Code in Haskell (YOW! Lambda Jam 2014)

VIDEO: https://www.youtube.com/watch?v=52yvHv_Ahvg

Talk and workshop at YOW! Lambda Jam, Brisbane, 2014.

Template Haskell is a meta programming framework for Haskell implemented by the Glasgow Haskell Compiler (GHC), which is widely used as a template meta-programming system for Haskell, to define macros, code generators, or even code transformation engines. Subsequent support for the quasiquoting of arbitrary programming languages greatly simplified writing code generators in Haskell that produce complex C, CUDA, OpenCL, or Objective-C code by writing code templates in the syntax of the generated language.

Additionally, quasiquoting of C-like languages enables a purely library-based system for inline C code in Haskell. This dramatically simplifies language interoperability, and especially, the use of frameworks and libraries written in C-like languages from Haskell. It is, for example, helpful in applications based on native GUI libraries and projects integrating code written in multiple languages.

In this talk, I will explain the concepts of template meta-programming and quasiquoting and how they are used in Template Haskell. I will demonstrate quasiquoting by way of a few simple and intuitive examples. Finally, I will demonstrate the use of inline C code in Haskell and compare it to other forms of language interoperability as provided by Haskell and other functional languages.

This talk will explore the use, but not the implementation of quasiquoting and inline C & Objective-C code. Hence, the material should be accessible to anybody with an intermediate-level working knowledge of Haskell and C.

This slide set also contains the slides for the associated hands on workshop. If you are interested in going through the exercises, you can get the workshop source code templates and solutions from https://github.com/mchakravarty/ylj14-workshop and you can find details on the prerequisite software setup (which requires OS X) at https://gist.github.com/mchakravarty/fb8ecc1be86df1a23b33

Transcript

1. Manuel M T Chakravarty University of New South Wales Foreign Inline Code in Haskell mchakravarty α TacticalGrace TacticalGrace justtesting.org 1 30 minute time slot: 25min talking + 5min [15min The Problem; 5min TH+Quasiquoting; 5min Inline Objective-C]
2. λ Shiny new functional language 2 » Imagine, you have got a shiny, new functional language...
3. 3 » ...and you want to use it to write a great new app...
4. λ 4 » ...then you will need to use many existing frameworks and libraries. » Luckily, any serious language will have a foreign function interface! » Haskell standard includes a simple, but versatile FFI
5. “Problem solved?” 5 » Does that solve the problem of language interoperability? » Let me explain that at an example...
6. “Problem solved?” No! 5 » Does that solve the problem of language interoperability? » Let me explain that at an example...
7. dumpURL :: String -> IO () dumpURL urlString = do urlData <- stringWithContentsOfUrl urlString putStr urlData What we want to write 6
8. dumpURL :: String -> IO () dumpURL urlString = do urlData <- stringWithContentsOfUrl urlString putStr urlData What we want to write char *stringWithContentsOfUrlCstub(char *urlString) { NSURL *url = [NSURL URLWithString:urlString]; [NSString stringWithContentsOfURL:url encoding:NSUTF8StringEncoding error:NULL]; } What we want to call (& need to put into an extra file) 6
9. What we have to write as well… foreign import stringWithContentsOfURLCstub :: CString -> IO CString stringWithContentsOfURL :: String -> IO String stringWithContentsOfURL url = withCString url $ \urlC -> do resultC <- stringWithContentsOfURLCstub urlC result <- peek resultC free resultC return result 7 » FFI code: If you find it confusing, that’s fine, as I want to argue that you shouldn’t write it in the first place. * Anything extensively relying on foreign frameworks will be a pain [Hybrid languages (eg, F# & Scala) have a different set of trade offs, but don’t solve it either.]
10. Bridging Libraries 1-to-1 transliteration of types & functions 8 * Bridging or binding libraries transliterate types & functions * Need to be maintained and documented; track multiple versions of base library * Lack of type safety * Tools and dynamic transliteration (by reflection) help
11. C➙Haskell c2hs A case study GTK+ 9 * c2hs: automation for bridging C libraries to Haskell <https://hackage.haskell.org/package/ c2hs> * Implements large parts of a C compiler front-end * Bridge for the cross-platform Gnome GUI library GTK+ <http://projects.haskell.org/ gtk2hs/>
12. 10 * AFAIK, currently the only fully featured and properly maintained Haskell GUI library * Haskell GTK+ library is used for realistic applications
13. “Does this approach scale?” 11 » Read question * It requires significant resources & constant effort to track the original library * Application frameworks: enormous and growing footprint
14. February 2011 Introduction of GTK+ 3 12 * Base GTK+ development isn’t even particularly fast * Bridging libraries lag behind their base libraries * Another example: Haskell OpenGL library
15. February 2011 Introduction of GTK+ 3 December 2013 GTK+ 3 in Haskell Bridge 12 * Base GTK+ development isn’t even particularly fast * Bridging libraries lag behind their base libraries * Another example: Haskell OpenGL library
16. 13 * Compared to modern application frameworks, GTK+ is small!
17. Bridging libraries don’t scale Too much weight! 14
18. “Interoperability is an old problem — maybe an old solution can help?” 15
19. 16 * Modula-2 with inline 68000 assembly * Assembly code can symbolically refer to Modula-2 entities (e.g., variables)
20. 16 * Modula-2 with inline 68000 assembly * Assembly code can symbolically refer to Modula-2 entities (e.g., variables)
21. 16 * Modula-2 with inline 68000 assembly * Assembly code can symbolically refer to Modula-2 entities (e.g., variables)
22. “Inline C, C++, Objective-C, … in Haskell?” 17 * We don’t want to build front ends for a few more languages into GHC * How do we share entities between the languages?
23. Meta-programming Template Haskell 18 » Generic infrastructure for program manipulation NOTE: I'll run through this quickly; I'll explain the details at the workshop.
24. HASKELL WORKSHOP 2002 19 * Template Haskell: Haskell extension implemented by GHC * Useful for: defining macros, code generators, code transformations… * Other languages have their own variants; eg., MetaOCaml » Let’s look at an example…
25. HASKELL WORKSHOP 2002 #define macros 19 * Template Haskell: Haskell extension implemented by GHC * Useful for: defining macros, code generators, code transformations… * Other languages have their own variants; eg., MetaOCaml » Let’s look at an example…
26. HASKELL WORKSHOP 2002 #define macros [| … |] code generators 19 * Template Haskell: Haskell extension implemented by GHC * Useful for: defining macros, code generators, code transformations… * Other languages have their own variants; eg., MetaOCaml » Let’s look at an example…
27. HASKELL WORKSHOP 2002 #define macros [| … |] code generators trafo (ConE name) = … code transformations 19 * Template Haskell: Haskell extension implemented by GHC * Useful for: defining macros, code generators, code transformations… * Other languages have their own variants; eg., MetaOCaml » Let’s look at an example…
28. $(sel 1 3) (a, b, c) = a 20 * Meta function executed at splice point, generating spliced code
29. $(sel 1 3) (a, b, c) = a meta-programming function 20 * Meta function executed at splice point, generating spliced code
30. $(sel 1 3) (a, b, c) = a meta-programming function splice 20 * Meta function executed at splice point, generating spliced code
31. $(sel 1 3) (a, b, c) = a meta-programming function splice \tup -> case tup of {(x, y, z) -> x} :: (a, b, c) -> a 20 * Meta function executed at splice point, generating spliced code
32. $(sel 1 3) (a, b, c) = a meta-programming function splice \tup -> case tup of {(x, y, z) -> x} :: (a, b, c) -> a $(sel 5 5) (a, b, c, d, e) = e 20 * Meta function executed at splice point, generating spliced code
33. $(sel 1 3) (a, b, c) = a meta-programming function splice \tup -> case tup of {(x, y, z) -> x} :: (a, b, c) -> a $(sel 5 5) (a, b, c, d, e) = e \tup -> case tup of {(x, y, z, v, w) -> w} :: (a, b, c, d, e) -> e 20 * Meta function executed at splice point, generating spliced code
34. sel :: Int -> Int -> ExpQ sel i n = [| \tup -> case tup of {$pat -> $res} |] where pat = tupP (map varP names) res = varE (names !! (i - 1)) names = [mkName $ "v" ++ show i | i <- [1..n]] 21 * Quasiquotations in [|..|] brackets * Explain TH in more detail in the workshop
35. sel :: Int -> Int -> ExpQ sel i n = [| \tup -> case tup of {$pat -> $res} |] where pat = tupP (map varP names) res = varE (names !! (i - 1)) names = [mkName $ "v" ++ show i | i <- [1..n]] type of Haskell expressions 21 * Quasiquotations in [|..|] brackets * Explain TH in more detail in the workshop
36. sel :: Int -> Int -> ExpQ sel i n = [| \tup -> case tup of {$pat -> $res} |] where pat = tupP (map varP names) res = varE (names !! (i - 1)) names = [mkName $ "v" ++ show i | i <- [1..n]] type of Haskell expressions quasi-quotation 21 * Quasiquotations in [|..|] brackets * Explain TH in more detail in the workshop
37. sel :: Int -> Int -> ExpQ sel i n = [| \tup -> case tup of {$pat -> $res} |] where pat = tupP (map varP names) res = varE (names !! (i - 1)) names = [mkName $ "v" ++ show i | i <- [1..n]] type of Haskell expressions quasi-quotation spliced expression (anti quote) 21 * Quasiquotations in [|..|] brackets * Explain TH in more detail in the workshop
38. Processing other languages Generic quasi-quotation 22 * Quoting arbitrary languages
39. HASKELL WORKSHOP 2007 Quasiquoting for any language you can provide a parser for 23
40. Language.C.Quote add n = [cfun| int addConstant(int x) { return x + $int:n; } |] 24 * QQ for C including some GNU extensions, parts of CUDA & OpenCL, and all of Objective-C
41. Language.C.Quote add n = [cfun| int addConstant(int x) { return x + $int:n; } |] quasi-quotation identifier 24 * QQ for C including some GNU extensions, parts of CUDA & OpenCL, and all of Objective-C
42. Language.C.Quote add n = [cfun| int addConstant(int x) { return x + $int:n; } |] quasi-quotation identifier splice identifier 24 * QQ for C including some GNU extensions, parts of CUDA & OpenCL, and all of Objective-C
43. mkMap dev aenv fun arr = return $ CUTranslSkel "map" [cunit| $esc:("#include <accelerate_cuda.h>") extern "C" __global__ void map ($params:argIn, $params:argOut) { const int shapeSize = size(shOut); const int gridSize = $exp:(gridSize dev); int ix; for ( ix = $exp:(threadIdx dev) ; ix < shapeSize ; ix += gridSize ) { $items:(dce x .=. get ix) $items:(setOut "ix" .=. f x) } } |] where ... 25 * Accelerate: embedded high-performance array language for GPUs * Combinators as code skeletons (code templates with holes) * Yellow splices/anti-quotes are the holes (parameters) of the template * Doing this with strings or explicit AST construction would be much worse
44. mkMap dev aenv fun arr = return $ CUTranslSkel "map" [cunit| $esc:("#include <accelerate_cuda.h>") extern "C" __global__ void map ($params:argIn, $params:argOut) { const int shapeSize = size(shOut); const int gridSize = $exp:(gridSize dev); int ix; for ( ix = $exp:(threadIdx dev) ; ix < shapeSize ; ix += gridSize ) { $items:(dce x .=. get ix) $items:(setOut "ix" .=. f x) } } |] where ... 25 * Accelerate: embedded high-performance array language for GPUs * Combinators as code skeletons (code templates with holes) * Yellow splices/anti-quotes are the holes (parameters) of the template * Doing this with strings or explicit AST construction would be much worse
45. mkMap dev aenv fun arr = return $ CUTranslSkel "map" [cunit| $esc:("#include <accelerate_cuda.h>") extern "C" __global__ void map ($params:argIn, $params:argOut) { const int shapeSize = size(shOut); const int gridSize = $exp:(gridSize dev); int ix; for ( ix = $exp:(threadIdx dev) ; ix < shapeSize ; ix += gridSize ) { $items:(dce x .=. get ix) $items:(setOut "ix" .=. f x) } } |] where ... 25 * Accelerate: embedded high-performance array language for GPUs * Combinators as code skeletons (code templates with holes) * Yellow splices/anti-quotes are the holes (parameters) of the template * Doing this with strings or explicit AST construction would be much worse
46. Inlining foreign languages Inline C Code 26
47. Language.C.Inline dumpURL :: String -> IO () dumpURL urlString = do urlData <- putStr urlData stringWithContentsOfUrl urlString 27 * Inline Objective-C * All the FFI code is generated automatically * Again, details in the workshop
48. Language.C.Inline dumpURL :: String -> IO () dumpURL urlString = do urlData <- putStr urlData $(objc ['urlString] ''String [cexp| [NSString stringWithContentsOfURL: [NSURL URLWithString:urlString] encoding:NSUTF8StringEncoding error:NULL] |]) inline Objective-C splice 27 * Inline Objective-C * All the FFI code is generated automatically * Again, details in the workshop
49. Inline C …is simple 28 * Lack of bridge saves a lot of work; instant access to new versions * Bridges require familiarity with native libraries already; inline code leverages that directly * Haskell types are automatically mapped to types of the inline language
50. Inline C …is simple No bridging library needs to be maintained 28 * Lack of bridge saves a lot of work; instant access to new versions * Bridges require familiarity with native libraries already; inline code leverages that directly * Haskell types are automatically mapped to types of the inline language
51. Inline C …is simple No extra documentation No bridging library needs to be maintained 28 * Lack of bridge saves a lot of work; instant access to new versions * Bridges require familiarity with native libraries already; inline code leverages that directly * Haskell types are automatically mapped to types of the inline language
52. Inline C …is simple No extra documentation Some type safety No bridging library needs to be maintained 28 * Lack of bridge saves a lot of work; instant access to new versions * Bridges require familiarity with native libraries already; inline code leverages that directly * Haskell types are automatically mapped to types of the inline language
53. Language integration by language inlining types >< state languages 29 Outline workshop * Get familiar with TH and Language.C.Quote * Run through three exercises using inline Objective-C including a graphical REPL
54. Workshop Part 1 Template Haskell 30 * http://www.haskell.org/haskellwiki/Template_Haskell * http://hackage.haskell.org/package/template-haskell
55. Clone https://github.com/mchakravarty/ylj14-workshop 31
60. printf :: String -> ExpQ printf fmt = gen (parse fmt) 33
61. printf :: String -> ExpQ printf fmt = gen (parse fmt) data Format = D | S | L String parse :: String -> [Format] 33
62. printf :: String -> ExpQ printf fmt = gen (parse fmt) data Format = D | S | L String parse :: String -> [Format] gen :: [Format] -> ExpQ gen [D] = [| \n -> show n |] gen [S] = [| \s -> s |] gen [L s] = [| s |] 33
63. Task ❶ A recursive generator 34
64. printf/ 35
65. printf :: String -> ExpQ printf fmt = gen (parse fmt) [| "" |] printf/ 35
66. printf :: String -> ExpQ printf fmt = gen (parse fmt) [| "" |] gen :: [Format] -> ExpQ -> ExpQ printf/ 35
67. printf :: String -> ExpQ printf fmt = gen (parse fmt) [| "" |] gen :: [Format] -> ExpQ -> ExpQ gen [] prefix = prefix printf/ 35
68. printf :: String -> ExpQ printf fmt = gen (parse fmt) [| "" |] gen :: [Format] -> ExpQ -> ExpQ gen [] prefix = prefix gen (D : fmt) prefix printf/ 35
69. printf :: String -> ExpQ printf fmt = gen (parse fmt) [| "" |] gen :: [Format] -> ExpQ -> ExpQ gen [] prefix = prefix gen (D : fmt) prefix = [| \n -> $(gen fmt [| $prefix ++ show n |]) |] printf/ 35
70. printf :: String -> ExpQ printf fmt = gen (parse fmt) [| "" |] gen :: [Format] -> ExpQ -> ExpQ gen [] prefix = prefix gen (D : fmt) prefix = [| \n -> $(gen fmt [| $prefix ++ show n |]) |] gen (S : fmt) prefix printf/ 35
71. printf :: String -> ExpQ printf fmt = gen (parse fmt) [| "" |] gen :: [Format] -> ExpQ -> ExpQ gen [] prefix = prefix gen (D : fmt) prefix = [| \n -> $(gen fmt [| $prefix ++ show n |]) |] gen (S : fmt) prefix = [| \s -> $(gen fmt [| $prefix ++ s |]) |] printf/ 35
72. printf :: String -> ExpQ printf fmt = gen (parse fmt) [| "" |] gen :: [Format] -> ExpQ -> ExpQ gen [] prefix = prefix gen (D : fmt) prefix = [| \n -> $(gen fmt [| $prefix ++ show n |]) |] gen (S : fmt) prefix = [| \s -> $(gen fmt [| $prefix ++ s |]) |] gen (L s : fmt) prefix printf/ 35
73. printf :: String -> ExpQ printf fmt = gen (parse fmt) [| "" |] gen :: [Format] -> ExpQ -> ExpQ gen [] prefix = prefix gen (D : fmt) prefix = [| \n -> $(gen fmt [| $prefix ++ show n |]) |] gen (S : fmt) prefix = [| \s -> $(gen fmt [| $prefix ++ s |]) |] gen (L s : fmt) prefix = gen fmt [| $prefix ++ s |] printf/ 35
74. Workshop Part 2 Quasi-quotation of C variants 36 * http://hackage.haskell.org/package/language-c-quote
75. gensum/ 37
76. genSum :: Name -> Int -> BlockItem genSum arr n = [citem| { int sum = 0; for (int i = 0; i++; i < $n) sum += $id:(show arr)[i]; } |] gensum/ 37
77. genSum :: Name -> Int -> BlockItem genSum arr n = [citem| { int sum = 0; for (int i = 0; i++; i < $n) sum += $id:(show arr)[i]; } |] custom parser for C block items gensum/ 37
78. genSum :: Name -> Int -> BlockItem genSum arr n = [citem| { int sum = 0; for (int i = 0; i++; i < $n) sum += $id:(show arr)[i]; } |] custom parser for C block items splicing of a name as an identifier gensum/ 37
79. genSum :: Name -> Int -> BlockItem genSum arr n = [citem| { int sum = 0; for (int i = 0; i++; i < $n) sum += $id:(show arr)[i]; } |] custom parser for C block items splicing of a name as an identifier default is to splice as an expression gensum/ 37
80. Task ❷ Loop unrolling 38
81. Replace the for loop by a statement sequence [cstms| … |] quote a statement sequence $stms:… splice a statement sequence 39
82. genSum arr n = [citem| { int sum = 0; $stms:(additions 0) } |] where additions i | i == n = [] | otherwise = [cstms| sum += $id:(show arr)[ $int:i ]; $stms:(additions (i + 1)) |] Replace the for loop by a statement sequence [cstms| … |] quote a statement sequence $stms:… splice a statement sequence 39
83. Workshop Part 3 Inline Objective-C Code 40 * http://hackage.haskell.org/package/language-c-inline
84. {-# LANGUAGE TemplateHaskell, QuasiQuotes #-} import Language.C.Quote.ObjC import Language.C.Inline.ObjC objc_import ["<Foundation/Foundation.h>"] nslog :: String -> IO () nslog msg = $(objc ['msg] ''() [cexp| NSLog(@"Here is a message from Haskell: %@”, ! ! ! ! ! ! msg) |]) objc_emit minimal/ 41 * Complete the skeleton in the exercise pack
85. {-# LANGUAGE TemplateHaskell, QuasiQuotes #-} import Language.C.Quote.ObjC import Language.C.Inline.ObjC objc_import ["<Foundation/Foundation.h>"] nslog :: String -> IO () nslog msg = $(objc ['msg] ''() [cexp| NSLog(@"Here is a message from Haskell: %@”, ! ! ! ! ! ! msg) |]) objc_emit imports for the generated Objective-C code minimal/ 41 * Complete the skeleton in the exercise pack
86. {-# LANGUAGE TemplateHaskell, QuasiQuotes #-} import Language.C.Quote.ObjC import Language.C.Inline.ObjC objc_import ["<Foundation/Foundation.h>"] nslog :: String -> IO () nslog msg = $(objc ['msg] ''() [cexp| NSLog(@"Here is a message from Haskell: %@”, ! ! ! ! ! ! msg) |]) objc_emit free variables imports for the generated Objective-C code minimal/ 41 * Complete the skeleton in the exercise pack
87. {-# LANGUAGE TemplateHaskell, QuasiQuotes #-} import Language.C.Quote.ObjC import Language.C.Inline.ObjC objc_import ["<Foundation/Foundation.h>"] nslog :: String -> IO () nslog msg = $(objc ['msg] ''() [cexp| NSLog(@"Here is a message from Haskell: %@”, ! ! ! ! ! ! msg) |]) objc_emit free variables imports for the generated Objective-C code finalise inline code minimal/ 41 * Complete the skeleton in the exercise pack
88. Task ❸ Bridging classes 42
89. particle/ Particle.hs Represent Haskell record as ObjC class Main.hs Allocate instance and use it 43
90. particle/ Particle.hs Represent Haskell record as ObjC class Main.hs Allocate instance and use it Complete the ObjC class implementation 43
91. Task ❹ A simple application 44
92. app/ Interpreter.hs Haskell interface to package ghc AppDelegate.hs Haskell-Objective-C bridge 45
93. app/ Interpreter.hs Haskell interface to package ghc AppDelegate.hs Haskell-Objective-C bridge Complete AppDelegate.hs Run with open -a HSApp.app 45
94. Thank you! 46
95. Images from http://wikimedia.org http://openclipart.org GitHub repo https://github.com/mchakravarty/language-c-inline Megamax Modula-2 Screenshot by Dirk Steins