case study: cross compiling dhall-json

a quick walk through how to cross compile dhall-json into an executable for raspberry pi

Transcript

1. Case Study cross compiling dhall-json Moritz Angermann, Haskell.sg March Meetup, Singapore
2. Who • Moritz Angermann / zw3rk blog: https:/ /medium.com/@zw3rk twitter: @angerman_io, @zw3rktech, @mobilehaskell Moritz Angermann, Haskell.sg March Meetup, Singapore
3. dhall-json Dhall is a programmable configuration language that is not Turing- complete You can think of Dhall as: JSON + functions + types + imports The dhall-json package provides a Dhall to JSON compiler and a Dhall to YAML compiler. Moritz Angermann, Haskell.sg March Meetup, Singapore
4. Prerequisites • raspberry pi sdk & toolchain-wrappers • ghc @ 5ef6e568 • for the build machine • for arm-linux-gnueabihf • cabal (HEAD) Moritz Angermann, Haskell.sg March Meetup, Singapore
5. I'm feeling lucky $ cabal unpack dhall-json $ cd dhall-json-1.0.11 dhall-json-1.0.11 $ arm-linux-gnueabihf-cabal new-build \ --configure-option=--host=arm-linux-gnueabihf [...] After searching the rest of the dependency tree exhaustively, these were the goals I've had most trouble fulfilling: base, scientific, hashable, unordered-containers, yaml, dhall-json Moritz Angermann, Haskell.sg March Meetup, Singapore
6. I'm feeling a little less lucky dhall-json-1.0.11 $ arm-linux-gnueabihf-cabal new-build \ --configure-option=--host=arm-linux-gnueabihf \ --allow-newer [...] next goal: integer-gmp (dependency of text-format-0.3.1.1) rejecting: integer-gmp-1.0.1.0, integer-gmp-1.0.0.1, integer-gmp-1.0.0.0, integer-gmp-0.5.1.0 (only already installed instances can be used) After searching the rest of the dependency tree exhaustively, these were the goals I've had most trouble fulfilling: integer-gmp, text-format, dhall-json, dhall Moritz Angermann, Haskell.sg March Meetup, Singapore
7. Looks like we need a cabal.project As we are going to have to patch a few packages, we'll create a cabal.project file: packages: . Moritz Angermann, Haskell.sg March Meetup, Singapore
8. Adding text-format-0.3.1.1 Let's unpack text-format-0.3.1.1 $ cabal unpack text-format and add it to the cabal.project file: packages: . text-format-0.3.1.1 Moritz Angermann, Haskell.sg March Meetup, Singapore
9. Drop integer-gmp from text-format And remove the build-depends from text-format.cabal - if impl(ghc >= 6.11) - build-depends: integer-gmp >= 0.2 Moritz Angermann, Haskell.sg March Meetup, Singapore
10. Trying to build again [...] trying: cryptonite-0.25:+integer-gmp next goal: integer-gmp (dependency of cryptonite-0.25:+integer-gmp) rejecting: integer-gmp-1.0.1.0, integer-gmp-1.0.0.1, integer-gmp-1.0.0.0, integer-gmp-0.5.1.0 (only already installed instances can be used) Moritz Angermann, Haskell.sg March Meetup, Singapore
11. Disabling integer-gmp in cryptonite We can simply disable integer-gmp on cryptonite by setting the - integer-gmp flag. Let's add that to the cabal.project file: packages: . text-format-0.3.1.1 package cryptonite flags: -integer-gmp Moritz Angermann, Haskell.sg March Meetup, Singapore
12. Trying to build once again [...] Blaze/ByteString/Builder/Internal/Write.hs:122:10: error: • No instance for (Semigroup Poke) arising from the superclasses of an instance declaration • In the instance declaration for ‘Monoid Poke’ | 122 | instance Monoid Poke where | ^^^^^^^^^^^ [...] Failed to build blaze-builder-0.4.0.2 (which is required by exe:dhall-to-yaml from dhall-json-1.0.11 and exe:dhall-to-json from dhall-json-1.0.11). See the build log above for details. Moritz Angermann, Haskell.sg March Meetup, Singapore
13. Adding head.hackage overlay Adding +repository head.hackage + url: http://head.hackage.haskell.org/ + secure: True + root-keys: [...] + key-threshold: 3 to the cabal.project file and running cabal new-update should make the overlay available. Moritz Angermann, Haskell.sg March Meetup, Singapore
14. Trying to build again Data/Text/Format/Functions.hs:16:7: error: Ambiguous occurrence ‘<>’ It could refer to either ‘GHC.Base.<>’, imported from ‘GHC.Base’ at Data/Text/Format/Functions.hs:22:1-15 or ‘Data.Text.Format.Functions.<>’, defined at Data/Text/Format/Functions.hs:32:1 | 16 | (<>) | ^^^^ [...] Failed to build text-format-0.3.1.1 (which is required by exe:dhall-to-yaml from dhall-json-1.0.11 and exe:dhall-to-json from dhall-json-1.0.11). Moritz Angermann, Haskell.sg March Meetup, Singapore
15. Patching text-format (1) -import GHC.Base +import GHC.Base hiding ((<>)) +import Prelude hiding ((<>)) in text-format-0.3.1.1/Data/Text/Format/Functions.hs Moritz Angermann, Haskell.sg March Meetup, Singapore
16. Trying to build again Data/Text/Format/Int.hs:32:1: error: Could not find module ‘GHC.Integer.GMP.Internals’ Perhaps you meant GHC.Integer.Simple.Internals (needs flag -package-key integer-simple-0.1.1.1) Use -v to see a list of the files searched for. | 32 | import GHC.Integer.GMP.Internals | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [...] Failed to build text-format-0.3.1.1 (which is required by exe:dhall-to-yaml from dhall-json-1.0.11 and exe:dhall-to-json from dhall-json-1.0.11). Moritz Angermann, Haskell.sg March Meetup, Singapore
17. Patching text-format (2) -import GHC.Integer.GMP.Internals [...] -integer 10 (S# i#) = decimal (I# i#) -integer 16 (S# i#) = hexadecimal (I# i#) in text-format-0.3.1.1/Data/Text/Format/Int.hs Moritz Angermann, Haskell.sg March Meetup, Singapore
18. Trying to build once again Data/Text/Format/Int.hs:55:25: error: Ambiguous occurrence ‘<>’ It could refer to either ‘Prelude.<>’, imported from ‘Prelude’ at Data/Text/Format/Int.hs:12:8-27 (and originally defined in ‘GHC.Base’) or ‘Data.Text.Format.Functions.<>’, imported from ‘Data.Text.Format.Functions’ at Data/Text/Format/Int.hs:21:36-39 | 55 | | i < 0 = minus <> go (-i) | ^^ [...] Failed to build text-format-0.3.1.1 (which is required by exe:dhall-to-yaml from dhall-json-1.0.11 and exe:dhall-to-json from dhall-json-1.0.11). Moritz Angermann, Haskell.sg March Meetup, Singapore
19. Patching text-format (3) +import Prelude hiding ((<>)) in text-format-0.3.1.1/Data/Text/Format/Int.hs Moritz Angermann, Haskell.sg March Meetup, Singapore
20. Trying to build yet again Data/Text/Format/Types/Internal.hs:49:10: error: • No instance for (Semigroup Format) arising from the superclasses of an instance declaration • In the instance declaration for ‘Monoid Format’ | 49 | instance Monoid Format where | ^^^^^^^^^^^^^ [...] Failed to build text-format-0.3.1.1 (which is required by exe:dhall-to-yaml from dhall-json-1.0.11 and exe:dhall-to-json from dhall-json-1.0.11). Moritz Angermann, Haskell.sg March Meetup, Singapore
21. Patching text-format (4) + instance Semigroup Format where + Format a <> Format b = Format (a <> b) + instance Monoid Format where - Format a `mappend` Format b = Format (a `mappend` b) + mempty = Format mempty in text-format-0.3.1.1/Data/Text/Format/Types/ Internal.hs Moritz Angermann, Haskell.sg March Meetup, Singapore
22. Building ... Data/Text/Buildable.hs:109:35: error: Ambiguous occurrence ‘<>’ It could refer to either ‘Prelude.<>’, imported from ‘Prelude’ at Data/Text/Buildable.hs:14:8-26 (and originally defined in ‘GHC.Base’) or ‘Data.Text.Format.Functions.<>’, imported from ‘Data.Text.Format.Functions’ at Data/Text/Buildable.hs:22:36-39 | 109 | build a = build (numerator a) <> singleton '/' <> build (denominator a) | ^^ [...] Failed to build text-format-0.3.1.1 (which is required by exe:dhall-to-yaml from dhall-json-1.0.11 and exe:dhall-to-json from dhall-json-1.0.11). Moritz Angermann, Haskell.sg March Meetup, Singapore
23. Patching text-format (4) +import Prelude hiding ((<>)) in text-format-0.3.1.1/Data/Text/Buildable.hs Moritz Angermann, Haskell.sg March Meetup, Singapore
24. And again ... Data/Text/Format.hs:68:30: error: Ambiguous occurrence ‘<>’ It could refer to either ‘Data.Text.Format.Functions.<>’, imported from ‘Data.Text.Format.Functions’ at Data/Text/Format.hs:39:36-39 or ‘Prelude.<>’, imported from ‘Prelude’ at Data/Text/Format.hs:44:1-34 (and originally defined in ‘GHC.Base’) | 68 | where go (f:fs) (y:ys) = f <> y <> go fs ys | ^^ [...] Failed to build text-format-0.3.1.1 (which is required by exe:dhall-to-yaml from dhall-json-1.0.11 and exe:dhall-to-json from dhall-json-1.0.11). Moritz Angermann, Haskell.sg March Meetup, Singapore
25. Patching text-format (5) -import Prelude hiding (exp, print) +import Prelude hiding (exp, print, (<>)) in text-format-0.3.1.1/Data/Text/Format.hs Moritz Angermann, Haskell.sg March Meetup, Singapore
26. Building once more [...] Codec/Compression/Zlib/Stream.hsc:1023 directive const_str cannot be handled in cross-compilation mode [...] Failed to build zlib-0.6.1.2 (which is required by exe:dhall-to-yaml from dhall-json-1.0.11 and exe:dhall-to-json from dhall-json-1.0.11). This means, hsc2hs can't handle const_str directives... Moritz Angermann, Haskell.sg March Meetup, Singapore
27. Fetching zlib from git This specific issue is has been fixed for a while, but no new release of zlib has been cut yet... $ git clone git@github.com:haskell/zlib.git and patch cabal.project -packages: . text-format-0.3.1.1 +packages: . text-format-0.3.1.1 zlib Moritz Angermann, Haskell.sg March Meetup, Singapore
28. Let's build again [...] Failed to build system-filepath-0.4.14 (which is required by exe:dhall-to-yaml from dhall-json-1.0.11 and exe:dhall-to-json from dhall-json-1.0.11). The failure occurred during the configure step. The build process terminated with exit code 126 This usually happens when the package uses build-type: Custom. Moritz Angermann, Haskell.sg March Meetup, Singapore
29. Patching system-filepath After cabal unpack sysmte-filepath and patching cabal.project -packages: . text-format-0.3.1.1 zlib +packages: . text-format-0.3.1.1 zlib system-filepath-0.4.14 we patch system-filepath-0.4.14/system-filepath.cabal -build-type: Custom +build-type: Simple Moritz Angermann, Haskell.sg March Meetup, Singapore
30. And again... [...] cabal: Failed to build distributive-0.5.3 (which is required by exe:dhall-to-yaml from dhall-json-1.0.11 and exe:dhall-to-json from dhall-json-1.0.11). The failure occurred during the configure step. The build process terminated with exit code 126 Again, cabal unpack, adding it to the cabal.project and changing the build-type. The same for the comonad, lens, trifecta and semigroupoid packages as well. Moritz Angermann, Haskell.sg March Meetup, Singapore
31. Building... src/Data/Profunctor/Unsafe.hs:49:1: error: Ambiguous module name ‘Data.Functor.Contravariant’: it was found in multiple packages: base-4.11.0.0 contravariant-1.4.1 | 49 | import Data.Functor.Contravariant (Contravariant(..)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [...] cabal: Failed to build profunctors-5.2.2 (which is required by exe:dhall-to-yaml from dhall-json-1.0.11 and exe:dhall-to-json from dhall-json-1.0.11). Moritz Angermann, Haskell.sg March Meetup, Singapore
32. Patching contravariant exposed-modules: - Data.Functor.Contravariant Data.Functor.Contravariant.Compose Data.Functor.Contravariant.Divisible + if impl(ghc < 8.5) + exposed-modules: + Data.Functor.Contravariant Moritz Angermann, Haskell.sg March Meetup, Singapore
33. Are we done yet? [...] ghc: this operation requires -fexternal-interpreter cabal: Failed to build trifecta-1.7.1.1 (which is required by exe:dhall-to-yaml from dhall-json-1.0.11 and exe:dhall-to-json from dhall-json-1.0.11). Text.Trifecta.Rendering contains: {-# LANGUAGE TemplateHaskell #-} [...] makeClassy ''Fixit Moritz Angermann, Haskell.sg March Meetup, Singapore
34. -fexternal-interpreter Run interpreted code (for GHCi, Template Haskell, Quasi-quoting, or Annotations) in a separate process. • Build == Host ghc <-> iserv • Build /= Host ghc <-> iserv-proxy <~ network ~> remote-iserv Moritz Angermann, Haskell.sg March Meetup, Singapore
35. Building iserv-proxy The provided cabal.project file in $ghc/utils/iserv-proxy: packages: . ../../libraries/libiserv package libiserv flags: +network Building iserv-proxy with cabal new-build: $ cabal new-build [...] Linking /some/rather/long/path/iserv-proxy Moritz Angermann, Haskell.sg March Meetup, Singapore
36. Building remote-iserv The provided cabal.project file in $ghc/utils/remote-iserv: packages: . ../../libraries/libiserv package libiserv flags: +network Building remote-iserv for the Raspberry Pi: $ arm-linux-gnueabihf-cabal new-build --configure-option=--host=arm-linux-gnueabihf [...] Linking /some/rather/long/path/remote-iserv Moritz Angermann, Haskell.sg March Meetup, Singapore
37. Running remote-iserv We'll copy the remote-iserv binary over, and start it on port 5001 with a temporary tmp in the current working directory. $ scp /path/to/remote-iserv pi@raspberrypi $ ssh pi@raspberrypi './remote-iserv ./tmp 5001' Moritz Angermann, Haskell.sg March Meetup, Singapore
38. Using iserv-proxy Adding the relevant ghc-options to the cabal.project file: +package trifecta + ghc-options: -fexternal-interpreter + -pgmi arm-linux-gnueabihf-iserv-proxy + -opti 10.0.1.16 -opti 5001 Moritz Angermann, Haskell.sg March Meetup, Singapore
39. And building ... src/Dhall/Core.hs:475:10: error: • No instance for (Semigroup (Chunks s a)) arising from the superclasses of an instance declaration • In the instance declaration for ‘Monoid (Chunks s a)’ | 475 | instance Monoid (Chunks s a) where | ^^^^^^^^^^^^^^^^^^^ [...] cabal: Failed to build dhall-1.10.0 (which is required by exe:dhall-to-yaml from dhall-json-1.0.11 and exe:dhall-to-json from dhall-json-1.0.11). Moritz Angermann, Haskell.sg March Meetup, Singapore
40. Patching dhall (1) +instance Semigroup (Chunks s a) where + (Chunks xysL zL) <> (Chunks [] zR) = + Chunks xysL (zL <> zR) + (Chunks xysL zL) <> (Chunks ((x, y):xysR) zR) = + Chunks (xysL ++ (zL <> x, y):xysR) zR instance Monoid (Chunks s a) where mempty = Chunks [] mempty - mappend (Chunks xysL zL) (Chunks [] zR) = - Chunks xysL (zL <> zR) - mappend (Chunks xysL zL) (Chunks ((x, y):xysR) zR) = - Chunks (xysL ++ (zL <> x, y):xysR) zR in Dhall.Core Moritz Angermann, Haskell.sg March Meetup, Singapore
41. Patching dhall (2) -import Prelude hiding (const, pi) +import Prelude hiding (const, pi, (<>)) [...] +instance Semigroup a => Semigroup (Parser a) where + (<>) = liftA2 (<>) instance Monoid a => Monoid (Parser a) where mempty = pure mempty - mappend = liftA2 mappend in Dhall.Parser Moritz Angermann, Haskell.sg March Meetup, Singapore
42. And building one more time... [...] Linking /.../dhall-to-yaml ... Linking /.../dhall-to-json ... But do they work? Moritz Angermann, Haskell.sg March Meetup, Singapore
43. Testing (very basic testing ...) $ scp /path/to/dhall-to-yaml pi@raspberrypi: $ ssh pi@raspberrypi './dhall-to-yaml <<< "{ foo = [1, 2, 3] : List Integer, bar = { baz = True } }"' bar: baz: true foo: - 1 - 2 - 3 Moritz Angermann, Haskell.sg March Meetup, Singapore
44. Testing (very basic testing ...) $ scp /path/to/dhall-to-json pi@raspberrypi: $ ssh pi@raspberrypi './dhall-to-json <<< "{ foo = [1, 2, 3] : List Integer, bar = { baz = True } }"' {"bar":{"baz":true},"foo":[1,2,3]} Seems to work ! Moritz Angermann, Haskell.sg March Meetup, Singapore
45. Final cabal.project file packages: . text-format-0.3.1.1 zlib system-filepath-0.4.14 distributive-0.5.3 comonad-5.0.3 semigroupoids-5.2.2 contravariant-1.4.1 lens-4.16 trifecta-1.7.1.1 dhall-1.10.0 package cryptonite flags: -integer-gmp package trifecta ghc-options: -fexternal-interpreter -pgmi iserv-proxy -opti 10.0.1.16 -opti 5001 repository head.hackage url: http://head.hackage.haskell.org/ secure: True root-keys: 07c59cb65787dedfaef5bd5f987ceb5f7e5ebf88b904bbd4c5cbdeb2ff71b740 2e8555dde16ebd8df076f1a8ef13b8f14c66bad8eafefd7d9e37d0ed711821fb 8f79fd2389ab2967354407ec852cbe73f2e8635793ac446d09461ffb99527f6e key-threshold: 3 Moritz Angermann, Haskell.sg March Meetup, Singapore
46. Thank you! Questions? Moritz Angermann, Haskell.sg March Meetup, Singapore