Using F# in Production at ClearTax

Using F# in Production at ClearTax

E5bd30aad31d00afe67bff9175db96b1?s=128

Ankit Solanki

October 15, 2016
Tweet

Transcript

  1. Using F# in Production Functional Conf, 2016 Ankit Solanki, cleartax.in

    @_anks, ankit@cleartax.in
  2. About me • Co-founder at ClearTax • Lifetime functional programming

    novice • Still learning!
  3. Outline • History: the product and why we picked F#

    • Wins & Losses • Scaling-up with the language • Learnings
  4. The Product • cleartds.com • SAAS product for preparing TDS

    Returns • Quarterly deadlines for submission of TDS Returns • Launched in early 2014 (Pre-YC)
  5. Product Requirements • Accuracy • Cannot afford to make any

    mistakes • 1,000s of people in every single TDS return • Constraints: • Arcane file format used by the department • Lack of documentation
  6. Product Requirements • Flexibility • Rules change every quarter •

    File format changes every quarter • Constraints: • Changes announced with no warning • Updated formats applicable from the day of release
  7. Product Requirements • Simplicity / Expressiveness / Speed • Need

    ability to launch fast • Needed to be able to run the product on auto-pilot for large chunks of time (after launch) • Ability to quickly dip-in every quarter and make changes • Constraints: • Lack of engineering resources • Lack of time
  8. File Format • Flat file format, like CSV files •

    Different types of lines • Mode-based, hierarchical format • Large number of fields
 and line types • Definition changes over time • Field means X in 2015 but Y in 2016 • Different variants based on use case (submission from department, download from department) and time (quarter, year, form type) • Delta-submissions in case of filing a revised return. Multiple modes of correction
  9. • File format main blocker • Experimented with a few

    approaches • Procedural code, DSLs, Mapping tools • Found F# Type Providers
  10. Type Providers • Compile-time ability to deal with structured data

    formats • Generate types & metadata based on sample data, at compile time • Parser to read input data of that type
  11. Type Providers Demo

  12. Using type providers drastically simplified our actual parsing,
 and made

    our code super-readable deduction.Tds <- record.``TDS / TCS -Income Tax for the period`` deduction.Surcharge <- record.``TDS / TCS -Surcharge for the period`` deduction.EducationCess <- record.``TDS / TCS -Cess``
  13. • We tested the file parser / generator • Simple

    strategy: • Parse sample file into your data model • Generate it • Do line-by-line, field-by-field comparison • Automated a large part of this with F#
  14. • We ended up writing the whole product in F#

    • But we did not focus on making it ‘functional’ • Using F# it as a better C# or Java (at first) • Core components (business logic) written in mostly functional style • Glue (controllers) written more in an imperative style
  15. Our philosophy with FP • Be pragmatic • Grow with

    the language • Experiment, bring in parts that you feel are valuable
  16. Product launch • We wanted to launch in 6-8 weeks

    • Very focused execution • We learned parts of the language and limited ourselves to only those parts • Started out with mostly the basics: pattern matching, currying, pipelines
  17. Initial period

  18. Pipelines // Pipeline operator itself is very simple
 
 let

    (|>) x fn = fn x
 
 // Usage is natural
 
 let square x = x * x
 let double x = 2 * x
 
 [1 ; 2; 3 ] |> List.map square |> List.map double
 // [2 ; 8; 18 ] // A little more natural than composition, at least for beginners // Can be arbitrarily long
 users
 |> List.map validateUsers
 |> List.filter isValid
 |> List.map getUserId
  19. Pattern Matching • Since you can pattern match on a

    tuple with n elements, this results in the ability to flatten nested logic into a simple decision table • Huge win for readability • We might have gone over-board with this
  20. Partial Application • Ability to encapsulate in a functional manner

    • Big aha moment when I finally understood this // Validations would be context dependent let validateDeduction year quarter returnType deduction = … // But at a higher level, you want a simpler signature type ValidateFn<'A> = 'A -> ValidationResult // So you can just freeze the context sensitive parameters // to get the specific validator you want let currentValidator = validateDeduction 2016 Q4 Original
  21. Code as Data // Type definitions type IsColumnVisible = (TdsReturn

    -> bool) list type IsColumnEditable<'T> = ('T -> bool) list type ColumnSpecification<'T> = (Quotations.Expr * IsColumnVisible * IsColumnEditable<'T>) list // Column visibility specifiers let showAlways = ... let showForRevised = ... let showForGovernment = ... // Editing specifiers let editAlways = fun (d : Deduction) -> true let editWhenNil = fun (d : Deduction) -> d.Amount = 0 let editWhenDateIsPresent = fun (d : Deduction) -> d.Date |> Option.isSome // UI Specification let (deductionColumns : ColumnSpecification<Deduction>) = [ <@ fun (c : Deduction) -> c.Date @> , [ showAlways ] , [ editAlways ] <@ fun (c : Deduction) -> c.SectionCode @> , [ showForRevised ] , [ editWhenNil ] <@ fun (c : Deduction) -> c.Amount @> , [ showForRevised ; showForGovernment ] , [ editWhenNil ; editWhenDateIsPresent ] ]
  22. ORM • ORM – we picked a C#-specific ORM [ServiceStack.OrmLite]

    early on • ORM was ideal for the product use case (bulk inserts, updates, simple conceptual model) • It actually worked pretty well with F# (with a minimal wrapper) • Had some issues, will go into detail later let loadDeductionsByName name = // Where clause let condition = <% fun (d : Deduction) -> d.Name = name %> // Order-by clause let ordering = <% fun (d : Deduction) -> d.CreateTimestamp %> // Pagination let currentPage = { page = 1 ; pageSize = 10 } // This executes: // select * from deduction where name = ? order by create_timestamp limit 10 DbHelper.LoadPageWhen currentPage condition ordering
  23. Mistakes were made though.

  24. Design Issue: Not leveraging types • One of the main

    mistakes we made • ORM layer was unable to deal with F# specific types (records, tuples, discriminated unions) • This resulted in nullable values introduced in the data model • Polluted the whole codebase • Right solution would have been to isolate this in a data layer
  25. More type problems • Same base types (example: Tax Deduction)

    used throughout the product • In some ways, made things simpler • But also led to unnecessary complexity • UI may not need to know about some fields, but it still gets them • Too much capability stuffed into a single entity • We should have defined more granular types • If I started over, would have spent more time getting the types right, building a layered architecture
  26. Performance: Lazy Evaluation F# Sequences and their transformations are lazy

    let squares = [1 ; 2 ] |> Seq.map (fun i -> printfn "%d" i i * i ) printfn "%A" squares // 1 // 2 // seq [1; 4] printfn "%A" squares // prints again // 1 // 2 // seq [1; 4] Usually, laziness is what we want. But responsibility lies with caller. Sometimes, this can lead to very expensive operations being repeated.
  27. Performance: Expression Trees • Or “Code Quotations” – language level

    feature of F# • Expression trees that you can work with programatically • Example: 
 let ordering = <% fun (d : Deduction) -> d.CreateTimestamp %> • Possible to inspect this expression and do code generation, evaluate it, get the property name it refers to, etc
  28. Performance: Expression Trees (continued) • These are actually fairly expensive

    to build • Relatively slow, even when compared to operations like creating a new function • First version of our application used quotations while generating the final TDS return • One quotation per field, per line • Profiler said that 99.99% time was spent building the trees or traversing these trees • Refactored the code to get a 100x speed improvement
  29. Data Structure Selection • This was a relatively minor issue,

    but still painful • F# has its own parallel data structures (List, Map, etc) • Different from standard C# structures • Immutable in design • Picking the right data structure was tricky • Libraries would work with C# lists, not F# lists, we had to convert • Also, default list in F# is a linked-list, not a array list
  30. Some tooling issues

  31. Tooling Issues: IDE • Visual Studio tooling for F# was

    not great in the beginning • We had crashes, slowdowns, etc • Example: • F# does not allow cyclic dependencies • Order of files in the proejct matters • Visual Studio (2013) actually did not have options to insert file at a particular location or re-order files • We hand-edited project files for a long time
  32. IDE (continued) • F# support in Visual Studio is better

    now • Still not ideal, not on-par with C# • F# compiler much slower than C# compiler, for example • Will take some time to catch up • F# story outside Windows is also good, now
  33. Tooling Issues: Language / Compiler Versions • Hit by this

    several times • Subtle differences in complier versions or language level support caused compilation fail during deployment • "Works on my machine", though • Resolving this was very painful
  34. Maturing with the language

  35. Computation Expressions • Syntactic sugar for monads • (Let's not

    talk about monads) • Will let you design 'workflows', flatten your logic even further • Simplified our business logic
  36. // Sugared syntax using the maybe computation expression // 'maybe'

    is not a built-in let lateFine = maybe { // CreditDate, TdsDate are option types, can be None if not entered let! creditDate = deduction.CreditDate let! deductionDate = deduction.TdsDate let diffInMonths = getDifferenceInMonth creditDate deductionDate return (calculateFine deduction.taxDeducted diffInMonths }
  37. // De-sugared let lateFine = match creditDate with | None

    -> None | Some c -> match deductionDate with | None -> None | Some d -> let diffInMonths = getDifferenceInMonth c d calculateFine deduction.taxDeducted diffInMonths
  38. Big win with 'glue' code • The ‘glue’ logic in

    controllers (written in a mostly imperative style) ended up being more and more complex • Needed to handle different use cases, UI states • Things became difficult to reason about • We started to use computation expressions to simplify it (where suitable)
  39. Type annotations are a win • Initially, we omitted type

    annotations in most places • “Complier is smart enough to infer types, why do I need to mention them?” • Code was difficult to read, and error messages were a little cryptic at times • Type inference would pick up wrong type for the function if you make a mistake • We started adding annotations to functions • Also started defining type aliases for common combinations
 
 let d : ColumnSpecification<Deduction> = …
 is more readable than
 let d : ((Quotations.Expr * IsColumnVisible * IsColumnEditable<'T>) list)
  40. Union types • If doing a do-over, would especially focus

    on algebraic types / union types • Data model would be richer if we had made proper use • Mostly ended up using union types at leaf level (individual fields), did not use at higher levels (entity level, composition of entities)
  41. Final thoughts

  42. On-boarding engineers • We thought F# would be hard for

    people to learn • Current team: One person with prior Haskell experience, one JavaScript programmer and one fresher • Everyone was able to ramp-up and start using F# very quickly • In-depth understanding takes time though • Basics are really easy to pick up
  43. Design matters • Language will not solve problems for you

    • No substitute for good design • The areas where we took shortcuts came back to haunt us • Functional languages like F# do help, if you are willing to listen • Not a silver bullet
  44. F# on the CLR • When we started, fewer F#

    specific libraries • Ability to use any C# library, but they often didn't feel natural • We made compromises here. Used some sub-optimal tools • Situation is better now. F# ecosystem is vibrant and growing!
  45. F# at ClearTax right now • TDS Product fully built

    & running on F# • Some features of ClearTax built in F# • Majority of ClearTax is still C# • Mostly because of tooling issues: not possible to mix and match languages within a singe 'project' • Browser based testing – Canopy, a F# DSL over Selenium
  46. Q&A

  47. Thank you! You can contact me on Twitter: _anks or,

    ankit@cleartax.in
  48. Resources for F# • Online • F# For Fun And

    Profit • F# Programming WikiBook • Books • F# Deep Dives • F# Applied