Shake it

Shake it so you don't have to make it.

Talk for the Frankfurt Haskell User Group

Transcript

1. Introduction Using Shake Even Deeper Case Studies The End Shake It - So You Don’t Have To Make It Markus Hauck codecentric Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 1 / 54
2. Introduction Using Shake Even Deeper Case Studies The End Introduction Introduction Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 2 / 54
3. Introduction Using Shake Even Deeper Case Studies The End Motivation Common situation: • lots of files • target: create a result (compiled code, image, . . . ) • complex relationship between files • this talk: use Shake to automate work Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 3 / 54
4. Introduction Using Shake Even Deeper Case Studies The End Make Besides building programs, Make can be used to manage any project where some files must be updated automati- cally from others whenever the others change. • often used to build C(++) programs/libraries • other build systems: cabal, stack, sbt, maven, gradle • most are focused on building programs Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 4 / 54
5. Introduction Using Shake Even Deeper Case Studies The End Maintaining A Make Build Definitely not fun Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 5 / 54
6. Introduction Using Shake Even Deeper Case Studies The End We Wrote Our Own Even worse: custom build Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 6 / 54
7. Introduction Using Shake Even Deeper Case Studies The End Shake Shake is a library for writing build systems. • written in Haskell (of course) • no assumptions about build result • you can build anything! Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 7 / 54
8. Introduction Using Shake Even Deeper Case Studies The End Shake vs Make • what’s the deal about shake? • shake is monadic • make is only applicative • (let’s forget about the unprincipled rest for now) • and Monad is far more powerful than Applicative Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 8 / 54
9. Introduction Using Shake Even Deeper Case Studies The End Dressing Up Dress up for winter Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 9 / 54
10. Introduction Using Shake Even Deeper Case Studies The End Dressing Up Run independent things in parallel Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 10 / 54
11. Introduction Using Shake Even Deeper Case Studies The End Dressing Up • rebuild only files that need to be built > ./Build.hs clean && ./Build.hs > rm coat && ./Build.hs > rm right sock && ./Build.hs Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 11 / 54
12. Introduction Using Shake Even Deeper Case Studies The End Using Shake Using Shake Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 12 / 54
13. Introduction Using Shake Even Deeper Case Studies The End Using Shake Haskell developer explaining the power of Shake Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 13 / 54
14. Introduction Using Shake Even Deeper Case Studies The End It’s a Library • Shake is meant to be used as a library • use Haskell to describe your rules • use rules to build your output main :: IO () main = shakeArgs shakeOptions $ do want ["now.time"] "*.time" %> \out -> do Stdout currentTime <- cmd "date" writeFileChanged out currentTime Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 14 / 54
15. Introduction Using Shake Even Deeper Case Studies The End It’s a Library Awesome because: • Turtle for shell scripts in Haskell • Dhall to handle configs • Wreq for arbitrary http calls • Pandoc for conversions of documents • lens, pipes, conduit, async, . . . Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 15 / 54
16. Introduction Using Shake Even Deeper Case Studies The End Rules you define rules inside the Rules (monad) (%>) :: FilePattern -> (FilePath -> Action ()) -> Rules () -- usage: "filepattern" %> \outPath -> doSomethingWith outPath rules specify an Action to build the outPath Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 16 / 54
17. Introduction Using Shake Even Deeper Case Studies The End And Action The file building action is where you have to do your work generateTime :: FilePath -> Action ByteString generateTime outPath = do putNormal "Asking the gods for the current time" Stdout stdout <- cmd "date" return stdout dateRule :: Rules () dateRule = "*.time*" %> generateTime Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 17 / 54
18. Introduction Using Shake Even Deeper Case Studies The End And Action • Action has a MonadIO instance -> liftIO • use predefined functions in Shake • running external commands • perform tracked IO operations • depend on inputs Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 18 / 54
19. Introduction Using Shake Even Deeper Case Studies The End Writing Rules -- glob patterns "pattern" %> action -- multiple glob (OR) ["pattern1", "pattern2"] %> action -- arbitrary predicates isPrefixOf "some-prefix" ?> action -- build multiple files (AND) ["*.o", "*.hi"] &%> action Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 19 / 54
20. Introduction Using Shake Even Deeper Case Studies The End Writing Actions "pattern" %> \outPath -> do need ["some-input.txt"] somethingSomething outPath Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 20 / 54
21. Introduction Using Shake Even Deeper Case Studies The End Writing Actions Use need to depend on input files need :: [FilePath] -> Action () need ["file1", "file2"] need ["file1"] need ["file2"] • all arguments in the list are built in parallel Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 21 / 54
22. Introduction Using Shake Even Deeper Case Studies The End Running External Commands External commands can be run via cmd: cmd "git commit -m test" cmd "git" ["commit", "-m", "test"] cmd "git" ["commit", "-m", "this is a test"] Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 22 / 54
23. Introduction Using Shake Even Deeper Case Studies The End Running External Commands also supports special arguments: Cwd <path> AddEnv "NAME" "VALUE" Shell Timeout 4.2 WithStdout True EchoStdout True FileStdout <file> and more, see CmdOption on hackage Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 23 / 54
24. Introduction Using Shake Even Deeper Case Studies The End Running External Commands Example: unzip a file cmd [Cwd "/tmp/test/", EchoStderr True] "unzip" ["-o", "test.zip"] Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 24 / 54
25. Introduction Using Shake Even Deeper Case Studies The End Running External Commands Another: run latexmk cmd [Cwd cwd ,WithStdout True ,EchoStdout False ,EchoStderr False ,Stdin "" ] bin ["-g", "-shell-escape", "-pdf", inp] Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 25 / 54
26. Introduction Using Shake Even Deeper Case Studies The End Running External Commands The output is also flexible: examples = do Stdout stdout <- cmd "date" (Exit code, Stderr stderr) <- cmd "date" CmdTime t <- cmd "sleep 1" Process handle <- cmd "wget haskell.org" return () See CmdResult Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 26 / 54
27. Introduction Using Shake Even Deeper Case Studies The End Working With Files Shake provides many helpful functions: copyFile old new copyFileChanged old new readFile file writeFileChanged file content removeFiles dir [pattern1, pattern2] removeFilesAfter withTempFile withTempDir -- ... many more Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 27 / 54
28. Introduction Using Shake Even Deeper Case Studies The End Working With Files • when possible always prefer Shake versions • automatic tracking of input • -Changed functions are handy to avoid unnecessary rebuilds Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 28 / 54
29. Introduction Using Shake Even Deeper Case Studies The End Reports • shake can produce reports in html and other formats • the html version is interactively explorable • let’s look at a report! Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 29 / 54
30. Introduction Using Shake Even Deeper Case Studies The End Even Deeper Even Deeper Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 30 / 54
31. Introduction Using Shake Even Deeper Case Studies The End Even Deeper Haskell developers realizing the full power of Shake Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 31 / 54
32. Introduction Using Shake Even Deeper Case Studies The End Even Deeper • up to this point: how to use shake for most things • recall: it’s a library • extension points to customize it to your needs Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 32 / 54
33. Introduction Using Shake Even Deeper Case Studies The End Track arbitrary IO actions • recall: Action has a MonadIO instance • we can therefore use arbitrary IO actions • instead of liftIO you shoudl use traced: traced :: String -> IO a -> Action a download :: String -> FilePath -> Action () download uri outPath = traced "named" $ do r <- Wreq.get uri BL.writeFile target (r ^. Wreq.responseBody) Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 33 / 54
34. Introduction Using Shake Even Deeper Case Studies The End Non-file dependencies Shake also supports tracking of other things example :: Action () example = do home <- getEnv "HOME" contents <- getDirectoryContents "." doSomething home contents Even more powerful: we can define our own using “Oracle rules” Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 34 / 54
35. Introduction Using Shake Even Deeper Case Studies The End Oracle rules newtype GitHash = GitHash () deriving (Show,Typeable,Eq,Hashable,Binary,NFData) shakeArgs shakeOptions $ do addOracle $ \(GitHash ()) -> fromStdout <$> cmd "git" ["rev-parse", "--short", "HEAD"] "some-file" %> \out -> do hash <- askOracle (GitHash ()) doSomething hash Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 35 / 54
36. Introduction Using Shake Even Deeper Case Studies The End Oracle rules • with oracles, you can depend on anything you want • gotcha: will always be run in a build if required • though they only invalidate others if sth. changed Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 36 / 54
37. Introduction Using Shake Even Deeper Case Studies The End Using Resources • mosts tasks won’t be cpu-bound • and resources are not infinite • if we need a list of 1000 images.. • some form of limit would be good • shake: resources and throttles Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 37 / 54
38. Introduction Using Shake Even Deeper Case Studies The End Using Resources • finite resources with limited number of slots: newResource • throttle how many actions are run in time period: newThrottle main :: IO () main = shakeArgs shakeOptions $ do -- max 10 disk usages disk <- newResource "Disk" 10 -- max 5 api calls per 60s api <- newThrottle "API" 5 60 "*.txt" %> \out -> withResource disk 1 $ withResource api 1 $ someAction out Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 38 / 54
39. Introduction Using Shake Even Deeper Case Studies The End Show Me The Monads Already The two important monadic datatypes in Shake: • Monad to generate rules: Rules :: * -> * newtype Rules a = Rules (WriterT SRules (ReaderT ShakeOptions IO) a) • Monad to describe build actions: Action :: * -> * newtype Action a = Action (ReaderT (S Global Local) (ContT () IO) a) • Action has an MonadIO instance Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 39 / 54
40. Introduction Using Shake Even Deeper Case Studies The End Show Me The Monads Already Monads Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 40 / 54
41. Introduction Using Shake Even Deeper Case Studies The End Case Studies Case Studies Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 41 / 54
42. Introduction Using Shake Even Deeper Case Studies The End Case Studies Haskell developer eager to apply Shake Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 42 / 54
43. Introduction Using Shake Even Deeper Case Studies The End Case Studies • presentations using reveal.js or LaTeX • images are created somehow (graphviz, download) • haskell code is checked via hlint • write in markdown, convert via pandoc • build both beamer and reveal.js presentation • developing RAWs for photography Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 43 / 54
44. Introduction Using Shake Even Deeper Case Studies The End Presentation: Pictures - Manually • google for picture • download picture • resize picture • include in presentation • where to store it? git? Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 44 / 54
45. Introduction Using Shake Even Deeper Case Studies The End Presentation: Pictures - Automatically • reference image in presentation • define how to download and how to resize in image/*.src files { url = "http://cool.image.de/cool.jpg", transformations = ["-resize 800x600", "-caption Cool-Image"] } • define how to convert .dot to .png (graphviz) Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 45 / 54
46. Introduction Using Shake Even Deeper Case Studies The End How to do it Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 46 / 54
47. Introduction Using Shake Even Deeper Case Studies The End Presentation: Source Code • including code quickly leads to a mess • write code in slide works • modifying is a nightmare • let’s shake it Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 47 / 54
48. Introduction Using Shake Even Deeper Case Studies The End Presentation: Source Code the plan: • find all snippets in the presentation • extract them into files • check them with hlint Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 48 / 54
49. Introduction Using Shake Even Deeper Case Studies The End Presentation: Source Code • lucky: I’m using pandoc (amazing!) • pandoc allows us to parse and modify the AST extractCodeBlocks :: Pandoc -> [String] extractCodeBlocks = query codeBlocks where codeBlocks (CodeBlock (_,classes,_) content) | "haskell" elem classes = [content] | otherwise = [] codeBlocks _ = [] Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 49 / 54
50. Introduction Using Shake Even Deeper Case Studies The End Presentation: Build It This whole presentation is built with Shake • all figures are compiled from sources • images are downloaded • latex is compiled • reveal.js downloaded Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 50 / 54
51. Introduction Using Shake Even Deeper Case Studies The End Case Study: Developing RAWs • camera produces RAW files with the sensor data • nondestructive editors (Adobe Lightroom, RawTherapee, . . . ) create “sidecar” files • develop a .jpg from the RAW using this sidecar • batch export your images using the editor • problems: • this process is very expensive • we want to avoid it if nothing changed • we don’t want to remember which ones have changed • and more would be nice too (resize, rename, delete, . . . ) Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 51 / 54
52. Introduction Using Shake Even Deeper Case Studies The End Case Study: Developing RAWs • shake was perfect for this • we can need dynamically all RAW files • run editor to develop the .jpg based on sidecar • produce a smaller resized copy for sharing • label the copy with watermark and exif information • use resources to limit parallel developing of pictures Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 52 / 54
53. Introduction Using Shake Even Deeper Case Studies The End The End The End Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 53 / 54
54. Introduction Using Shake Even Deeper Case Studies The End The End Devs show gratitude for the new build Markus Hauck (codecentric) Shake It - So You Don’t Have To Make It 54 / 54