Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

From Zero to Haskell: Lessons Learned

From Zero to Haskell: Lessons Learned

Susan Potter

June 15, 2019
Tweet

More Decks by Susan Potter

Other Decks in Programming

Transcript

  1. finger $(whoami) Name: Susan Potter Logged in since Sun Jan

    18 18:30 1996 (GMT) on tty1 - 23 years writing software in industry - Server-side/backend/infrastructure engineering, mostly - Former SRE; really care about operational ergonomics Today: - Building new backend services, test & automation in Haskell - Infrastructure and CI/CD with Nix/NixOS/Hydra - Still babysit a bloated Rails webapp Previously: trading systems and SaaS products Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 1 / 44
  2. Overview Clickstreams (collecting & reporting API) Figure 1: High write

    throughput and unreliable runtime dependencies killed original Ruby service Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 3 / 44
  3. Overview Streaming data replication with scrubbing Figure 2: Soft real-time

    replication needs to ensure PII data is scrubbed Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 4 / 44
  4. Overview Developer, testing & deploy tools Figure 3: Developer tools

    scattered across copy-pasta Bash/Ruby scripts across multiple repos Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 5 / 44
  5. Overview Agenda Illustrated examples of team feedback: • Security/Privacy Awareness

    • Operational Debuggability Human factors review • Teaching/Learning • Managing Up • Setting Expectations Take-Aways Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 6 / 44
  6. Security/Privacy Awareness Security/Privacy Awareness: Motivation Need to scrub sensitive PII:

    • between production and staging • before sharing with third-parties Security gates Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 7 / 44
  7. Security/Privacy Awareness Security/Privacy Awareness: The Big Idea ”Susan, you keep

    telling us about the powers of the Haskell type system, when do we actually use it for great good?” –Team Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 8 / 44
  8. Security/Privacy Awareness Security/Privacy Awareness: The Big Idea How do we

    get type errors when we want them? • Reification turning terms into types Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 10 / 44
  9. Security/Privacy Awareness Security/Privacy Awareness: The Big Idea How do we

    get type errors when we want them? • Reification turning terms into types Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 10 / 44
  10. Security/Privacy Awareness Security/Privacy Awareness: Language Extensions {-# LANGUAGE DataKinds #-}

    {-# LANGUAGE KindSignatures #-} {-# LANGUAGE GADTs #-} -- OPTIONAL {-# LANGUAGE NoImplicitPrelude #-} -- OPTIONAL {-# LANGUAGE OverloadedStrings #-} -- OPTIONAL {-# LANGUAGE TypeApplications #-} -- OPTIONAL Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 11 / 44
  11. Security/Privacy Awareness Security/Privacy Awareness: Imports import Control.Applicative import Control.Monad import

    Data.Eq import Data.Function import Data.Maybe import Data.Semigroup import Data.Text import GHC.Err (error) import GHC.Show import GHC.Types Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 12 / 44
  12. Security/Privacy Awareness Security/Privacy Awareness: Data Types EmailState data EmailState =

    Unscrubbed | Scrubbed deriving (Eq, Show) Email data Email (s :: EmailState) = MkEmail { emailAddress :: Text } deriving (Eq, Show) Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 13 / 44
  13. Security/Privacy Awareness Security/Privacy Awareness: Data Types EmailState data EmailState =

    Unscrubbed | Scrubbed deriving (Eq, Show) Email data Email (s :: EmailState) = MkEmail { emailAddress :: Text } deriving (Eq, Show) Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 13 / 44
  14. Security/Privacy Awareness Security/Privacy Awareness: -XTypeApplications -- Before enabling TypeApplications --

    >>> MkEmail "[email protected]" :: Email 'Scrubbed -- MkEmail {emailAddress = "[email protected]"} -- it :: Email 'Scrubbed -- After: -- >>> MkEmail @'Unscrubbed "[email protected]" -- MkEmail {emailAddress = "[email protected]"} -- it :: Email 'Unscrubbed Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 14 / 44
  15. Security/Privacy Awareness Security/Privacy Awareness: -XDataKinds -- >>> import Scrubbed --

    >>> :set -XDataKinds -- >>> :kind! 'Scrubbed -- 'Scrubbed :: EmailState -- = 'Scrubbed Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 15 / 44
  16. Security/Privacy Awareness Security/Privacy Awareness: -XKindSignatures Remember this? data Email (s

    :: EmailState) -- this is the kind signature! = MkEmail { emailAddress :: Text } deriving (Eq, Show) -- >>> :kind! Email Int -- <interactive>:1:7: error: -- • Expected kind ‘EmailState’, but ‘Int’ has kind ‘*’ -- • In the first argument of ‘Email’, namely ‘Int’ -- In the type ‘Email Int’ Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 16 / 44
  17. Security/Privacy Awareness Security/Privacy Awareness: -XKindSignatures data Email (s :: EmailState)

    -- this is the kind signature! = MkEmail { emailAddress :: Text } deriving (Eq, Show) -- >>> :kind! Email 'Unscrubbed -- Email 'Unscrubbed :: * -- = Email 'Unscrubbed Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 17 / 44
  18. Security/Privacy Awareness Security/Privacy Awareness: Back to the team … ”Nice

    tricks but what do they buy us, Susan?” – Team Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 18 / 44
  19. Security/Privacy Awareness Security/Privacy Awareness: Introducer -- >>> Just unscrubbedEmail =

    readEmail "[email protected]" → -- unscrubbedEmail :: Email 'Unscrubbed readEmail :: Text -> Maybe (Email 'Unscrubbed) readEmail = pure . MkEmail @'Unscrubbed -- silly version for slides → Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 19 / 44
  20. Security/Privacy Awareness Security/Privacy Awareness: State Transitions -- >>> scrubEmail (MkUsername

    "username0") <$> readEmail "[email protected]" → -- Just (MkEmail {emailAddress = "[email protected]"}) → -- it :: Maybe (Email 'Scrubbed) scrubEmail :: Username -> Email 'Unscrubbed -> Email 'Scrubbed → scrubEmail username (MkEmail unscrubbedEmail) = MkEmail @'Scrubbed (mkEmailAddress $ getUsername username) → Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 20 / 44
  21. Security/Privacy Awareness Security/Privacy Awareness: State Transitions -- >>> import Text.StringRandom

    -- >>> stringRandomIO "[0-9a-zA-Z][0-9a-zA-Z.+]{2,8}" >>= pure . randomizeEmail → -- MkEmail {emailAddress = "[email protected]"} -- it :: Email 'Scrubbed randomizeEmail :: Text -> Email 'Scrubbed randomizeEmail randomizedPrefix = MkEmail @'Scrubbed (mkEmailAddress randomizedPrefix) -- ^^^^^^^^^^ using -XTypeApplications Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 21 / 44
  22. Security/Privacy Awareness Security/Privacy Awareness: Relies on two things… • Hiding

    MkEmail data constructor • Only providing functions for valid state transitions Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 22 / 44
  23. Security/Privacy Awareness Security/Privacy Awareness: Relies on two things… • Hiding

    MkEmail data constructor • Only providing functions for valid state transitions Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 22 / 44
  24. Security/Privacy Awareness Security/Privacy Awareness: Relies on two things… • Hiding

    MkEmail data constructor • Only providing functions for valid state transitions Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 22 / 44
  25. Security/Privacy Awareness Security/Privacy Awareness: -XGADTs Now I started going wild

    (reflection, going back to terms): -- OPTIONAL: -- GADT can act as a witness of EmailState -- if we need to reflect at runtime data SEmailState :: EmailState -> Type where SUnscrubbed :: SEmailState 'Unscrubbed SScrubbed :: SEmailState 'Scrubbed Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 23 / 44
  26. Security/Privacy Awareness Security/Privacy Awareness: Team Verdict ”Wow, stop! Too much

    type acrobatics, Susan.” – Team Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 24 / 44
  27. Security/Privacy Awareness Security/Privacy Awareness: Backtrack to just enough • Didn’t

    really need to reflect using a witness in this case. • Keep is simple, Susan (KISS) • Removed GADT witness and reflection functions and then… ”Cool, we like this just enough solution.” – Team Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 26 / 44
  28. Operational Debuggability Operational Debuggability: Motivation TREND Our systems are larger

    today: • more external integrations • more ”micro”-services • more storage & compute needs (for new categories of problems) • more runtime dependencies (monitoring, metrics, orchestration) • more deployment strategies And … Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 27 / 44
  29. Operational Debuggability Operational Debuggability: Semi-Structured Logging Figure 4: Getting even

    semi-strutured log records in a Rails app is like herding cats Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 28 / 44
  30. Operational Debuggability Operational Debuggability: Semi-Structured Logging 1 data LogMessage a

    2 = MkLogMessage 3 { logMessageService :: !Service -- compile 4 , logMessageVersion :: !Version -- compile 5 , logMessageEnv :: !Env -- on start 6 , logMessageRuntimeId :: !RuntimeId -- on start 7 , logMessageTime :: !UTCTime -- runtime 8 , logMessageThreadId :: !ThreadId -- runtime 9 , logMessagePayload :: a -- runtime } deriving (Generic) Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 29 / 44
  31. Operational Debuggability Operational Debuggability: Semi-Structured Logging mkLog :: (Loggable a,

    MonadIO m) => a -> m (LogMessage a) class Loggable a where formatLog :: a -> LogOutputType instance Loggable a => Loggable (LogMessage a) where formatLog (MkLogMessage t tid rid v s c e p) = ... Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 30 / 44
  32. Operational Debuggability Operational Debuggability: Semi-Structured Logging -- Request/Response specific types

    type Username = Text data PasswordReset = PasswordReset Username instance Loggable PasswordReset where formatLog (PasswordReset username) = ... Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 31 / 44
  33. Operational Debuggability Operational Debuggability: Semi-Structured Logging Use formatted protocol and

    ingest structure. We us CEE-enhanced syslog @cee: {"time":"20190613T041811654", "tid":167, "pid":33451, "ver":"f73cfffe", "svc":"dk", "cmp":"tel", "env":"s1", "p":{"username":"someuser", "action":"password-reset"}} → → Now we can query structured logs not search for arbitrary text @severity:"error" svc:"dk" env:"prod" payload_status:[500 TO 599] → Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 32 / 44
  34. Operational Debuggability Operational Debuggability: Semi-Structured Logging Use formatted protocol and

    ingest structure. We us CEE-enhanced syslog @cee: {"time":"20190613T041811654", "tid":167, "pid":33451, "ver":"f73cfffe", "svc":"dk", "cmp":"tel", "env":"s1", "p":{"username":"someuser", "action":"password-reset"}} → → Now we can query structured logs not search for arbitrary text @severity:"error" svc:"dk" env:"prod" payload_status:[500 TO 599] → Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 32 / 44
  35. Operational Debuggability Operational Debuggability: Semi-Structured Logging • BAD: Team found

    logging setup more involved (compared to Rails) • GOOD: First service work easily extracted in to a shared library for the next • GOOD: Simple type features used so far but some boilerplate • GOOD: Structured log record consistency (unlike first pass in Rails) • TODO: Trees That Grow to extend LogMessage in new ways • TODO: Type families to change output type • TODO: instance Generic a => Loggable a where ... Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 33 / 44
  36. Operational Debuggability Operational Debuggability: Semi-Structured Logging • BAD: Team found

    logging setup more involved (compared to Rails) • GOOD: First service work easily extracted in to a shared library for the next • GOOD: Simple type features used so far but some boilerplate • GOOD: Structured log record consistency (unlike first pass in Rails) • TODO: Trees That Grow to extend LogMessage in new ways • TODO: Type families to change output type • TODO: instance Generic a => Loggable a where ... Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 33 / 44
  37. Operational Debuggability Operational Debuggability: Semi-Structured Logging • BAD: Team found

    logging setup more involved (compared to Rails) • GOOD: First service work easily extracted in to a shared library for the next • GOOD: Simple type features used so far but some boilerplate • GOOD: Structured log record consistency (unlike first pass in Rails) • TODO: Trees That Grow to extend LogMessage in new ways • TODO: Type families to change output type • TODO: instance Generic a => Loggable a where ... Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 33 / 44
  38. Operational Debuggability Operational Debuggability: WIP Kill Switches Kill switches are

    like feature flags EXCEPT: • they prevent access to a run-time resources like a database or cache or flaky API getKillSwitch :: ??? f => f (Either a b) gracefullyFail :: ??? f => f (a -> c) normalOps :: ??? f => f (b -> c) branch getKillSwitch gracefullyFail normalOps Look familiar? ;) Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 34 / 44
  39. Operational Debuggability Operational Debuggability: WIP Kill Switches Selectives allow: •

    static over- and under-estimation of computation tree • under-estimation can be used to find computations that always run • over-estimation can be used to find computations that sometimes run • generate an on-call endpoint mapping to help tired brain debug at 3AM • Still a work-in-progress Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 35 / 44
  40. Operational Debuggability Operational Debuggability: WIP Kill Switches Selectives allow: •

    static over- and under-estimation of computation tree • under-estimation can be used to find computations that always run • over-estimation can be used to find computations that sometimes run • generate an on-call endpoint mapping to help tired brain debug at 3AM • Still a work-in-progress Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 35 / 44
  41. Operational Debuggability Operational Debuggability: WIP Kill Switches Selectives allow: •

    static over- and under-estimation of computation tree • under-estimation can be used to find computations that always run • over-estimation can be used to find computations that sometimes run • generate an on-call endpoint mapping to help tired brain debug at 3AM • Still a work-in-progress Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 35 / 44
  42. Operational Debuggability Operational Debuggability: WIP Kill Switches Selectives allow: •

    static over- and under-estimation of computation tree • under-estimation can be used to find computations that always run • over-estimation can be used to find computations that sometimes run • generate an on-call endpoint mapping to help tired brain debug at 3AM • Still a work-in-progress Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 35 / 44
  43. Operational Debuggability Operational Debuggability: WIP Kill Switches Selectives allow: •

    static over- and under-estimation of computation tree • under-estimation can be used to find computations that always run • over-estimation can be used to find computations that sometimes run • generate an on-call endpoint mapping to help tired brain debug at 3AM • Still a work-in-progress Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 35 / 44
  44. Operational Debuggability Operational Debuggability: WIP Kill Switches Selectives allow: •

    static over- and under-estimation of computation tree • under-estimation can be used to find computations that always run • over-estimation can be used to find computations that sometimes run • generate an on-call endpoint mapping to help tired brain debug at 3AM • Still a work-in-progress Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 35 / 44
  45. Human Factors Teaching/Learning in industry • Industry technical leaders untrained

    at teaching • ”Success” criteria often flawed or unmeasurable • Bottom-up foundational learning with top-down practical practice • Setup simple working dev envs; introduce the tools of trade • Team-level learning, not just individual Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 37 / 44
  46. Human Factors Teaching/Learning in industry • Industry technical leaders untrained

    at teaching • ”Success” criteria often flawed or unmeasurable • Bottom-up foundational learning with top-down practical practice • Setup simple working dev envs; introduce the tools of trade • Team-level learning, not just individual Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 37 / 44
  47. Human Factors Teaching/Learning in industry • Industry technical leaders untrained

    at teaching • ”Success” criteria often flawed or unmeasurable • Bottom-up foundational learning with top-down practical practice • Setup simple working dev envs; introduce the tools of trade • Team-level learning, not just individual Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 37 / 44
  48. Human Factors Teaching/Learning in industry • Industry technical leaders untrained

    at teaching • ”Success” criteria often flawed or unmeasurable • Bottom-up foundational learning with top-down practical practice • Setup simple working dev envs; introduce the tools of trade • Team-level learning, not just individual Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 37 / 44
  49. Human Factors Teaching/Learning in industry • Industry technical leaders untrained

    at teaching • ”Success” criteria often flawed or unmeasurable • Bottom-up foundational learning with top-down practical practice • Setup simple working dev envs; introduce the tools of trade • Team-level learning, not just individual Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 37 / 44
  50. Human Factors Team-level learning (safe-to-fail experiments) • Define hypothesis •

    Design experiment • Document results • Share recommendations back to team • Team Discusses • Ticket cleanup for failures and successes Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 38 / 44
  51. Human Factors Managing Up • Past project failures (postmortems) •

    Propose solution to core problems • Offer PoC evidence that it satisfies a core requirement • Frame results to your audience (technical vs non-technical) • Incremental rollout, show results early, increase trust Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 39 / 44
  52. Human Factors Managing Up • Past project failures (postmortems) •

    Propose solution to core problems • Offer PoC evidence that it satisfies a core requirement • Frame results to your audience (technical vs non-technical) • Incremental rollout, show results early, increase trust Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 39 / 44
  53. Human Factors Managing Up • Past project failures (postmortems) •

    Propose solution to core problems • Offer PoC evidence that it satisfies a core requirement • Frame results to your audience (technical vs non-technical) • Incremental rollout, show results early, increase trust Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 39 / 44
  54. Human Factors Managing Up • Past project failures (postmortems) •

    Propose solution to core problems • Offer PoC evidence that it satisfies a core requirement • Frame results to your audience (technical vs non-technical) • Incremental rollout, show results early, increase trust Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 39 / 44
  55. Human Factors Managing Up • Past project failures (postmortems) •

    Propose solution to core problems • Offer PoC evidence that it satisfies a core requirement • Frame results to your audience (technical vs non-technical) • Incremental rollout, show results early, increase trust Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 39 / 44
  56. Human Factors Set Expectations With: • Management (above) • Your

    technical peers • Your business peers • Your team By: • Be realistic • Iterative delivery • It will never be rainbows and unicorns, sorry! Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 40 / 44
  57. Human Factors Set Expectations With: • Management (above) • Your

    technical peers • Your business peers • Your team By: • Be realistic • Iterative delivery • It will never be rainbows and unicorns, sorry! Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 40 / 44
  58. In Closing Recap: Takeaways from our experience so far •

    Stay within a novelty budget • Give people new to Haskell working dev envs from day one! (Frustration budget) • Teach thinking tools over syntax • Promote team experiments and learning over time Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 41 / 44
  59. In Closing Recap: Takeaways from our experience so far •

    Stay within a novelty budget • Give people new to Haskell working dev envs from day one! (Frustration budget) • Teach thinking tools over syntax • Promote team experiments and learning over time Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 41 / 44
  60. In Closing Recap: Takeaways from our experience so far •

    Stay within a novelty budget • Give people new to Haskell working dev envs from day one! (Frustration budget) • Teach thinking tools over syntax • Promote team experiments and learning over time Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 41 / 44
  61. In Closing Recap: Takeaways from our experience so far •

    Stay within a novelty budget • Give people new to Haskell working dev envs from day one! (Frustration budget) • Teach thinking tools over syntax • Promote team experiments and learning over time Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 41 / 44
  62. Fertig/Fin/The End Questions? Accounts GitHub @mbbx6spp Keybase @mbbx6spp LinkedIn /in/susanpotter

    Twitter @SusanPotter I <3 silly hats. Thank you! Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 42 / 44
  63. Appendix: Answers/links to answers in Q&A (post-talk) Materials on thinking

    tools vs syntax • Parametricity • Well Typed blog post from 2015 about ’Parametricity’ • Slide from talk ’Parametricity: Types Are Documentation’ • Theorems for free! • The algebra of [algebraic] data types • Algebra and Calculus of Algebraic Data Types • Abstractions of typed functional programming (aka typeclassopedia) • Typeclassopedia (Haskell wiki) • Typeclasses • Functors, Applicatives, and Monads in Pictures Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 43 / 44
  64. Appendix: Answers/links to answers in Q&A (post-talk) Examples of Haskell/Nix

    development environments • Gist of an example Nix shell with GHC + hlint + hoogle + stylish-haskell devenv Susan Potter From Zero to Haskell: Lessons Learned 2019-06-15 44 / 44