Upgrade to Pro — share decks privately, control downloads, hide ads and more …

On Notation

Xuanyi
September 29, 2022

On Notation

My slides for my DataEngConfAU 2022 talk, complete with additional material (FAQs and other supplementary material)

Xuanyi

September 29, 2022
Tweet

More Decks by Xuanyi

Other Decks in Technology

Transcript

  1. Follow @chewxy on Twitter
    बह#$पते परथमं वाचो अ1ं यत
    परैरत नामधेय6दधानाः
    यदेषां शरे;ठं यद=र>मासीA>ेणा
    तदेषां CनDहतं गुहाGवः
    On Notation
    (or YAML MUST DIE)

    View full-size slide

  2. Follow @chewxy on Twitter
    On Problem Solving

    View full-size slide

  3. Follow @chewxy on Twitter
    How Do We Solve Problems?
    1. Simplify the problem
    2. Solve the simplified problem(s) recursively
    3. Solve the initial problem

    View full-size slide

  4. Follow @chewxy on Twitter
    How Do We Solve Problems?
    1. Simplify the problem
    a. Breaking down the problem into component problems
    b. Model the problem with less resolution
    2. Solve the simplified problem(s) recursively
    3. Solve the initial problem

    View full-size slide

  5. Follow @chewxy on Twitter
    How Do We Solve Problems?
    1. Simplify the problem
    a. Breaking down the problem into component problems
    b. Model the problem with less resolution
    2. Solve the simplified problem(s) recursively
    3. Solve the initial problem

    View full-size slide

  6. Follow @chewxy on Twitter
    How Do We Solve Problems?
    1. Simplify the problem
    a. Breaking down the problem into component problems
    b. Model the problem with less resolution
    2. Solve the simplified problem(s) recursively
    3. Solve the initial problem
    a. Combine solutions
    b. Increase resolution for the solution

    View full-size slide

  7. Follow @chewxy on Twitter
    How Do We Represent Problems?
    github.com/owulveryck/wardleytogo

    View full-size slide

  8. Follow @chewxy on Twitter
    「白马非马」,可乎?
    曰:可。
    曰:何哉?
    曰:马者,所以命形也;白者
    所以命色也。命色者非命形也
    。故曰:「 白马非马」。
    Language:
    A Unifying Theme

    View full-size slide

  9. Follow @chewxy on Twitter
    What I’ve Been Thinking About
    Deep Learning in Go
    https://gorgonia.org

    View full-size slide

  10. Follow @chewxy on Twitter
    Neural Network Notations
    y = σ(Wx + b)
    yi
    l = f(Σj
    wi,j
    l xi
    l-1 + bi
    l)
    y = f(wi
    xi + b)
    y = f ⟨w|x⟩ + |b⟩

    View full-size slide

  11. Follow @chewxy on Twitter
    Neural Network Notations
    y = σ(Wx + b)
    yi
    l = f(Σj
    wi,j
    l xi
    l-1 + bi
    l)
    y = f(wi
    xi + b)
    y = f ⟨w|x⟩ + |b⟩

    View full-size slide

  12. Follow @chewxy on Twitter
    Abstraction, Abstraction, Abstraction*
    λ
    As an ML (of the ISWIM kind) - 2017
    * just don’t look under the lambda
    λ>> let x = Matrix(2,2) of float32 WithInit(GlorotN)
    λ>> let y = Matrix(2,2) of float32 WithInit(Uniform)
    λ>> let z = mul x y
    λ>> :∂ z (* manually trigger differentiation of z wrt its inputs *)
    λ>> (:deriv x) = y (* check that the partial deriv of x is equal to y
    *)
    True
    λ>> let id = fun x -> x in id x
    ...

    View full-size slide

  13. Follow @chewxy on Twitter
    As An Algol Descendent (Go-like) - 2018
    Commands, Commands Commands
    λ>> x = Matrix((2,2), Float32, WithInit(GlorotN))
    λ>> y = Matrix((2,2), Float32, WithInit(Uniform))
    λ>> z = MatMul(x, y)
    λ>> x.∂ == y // check that the partial deriv of x is equal to y
    true
    λ>> func id(x [a]) [a] { return x }
    id(x): ∀[a]: a → a
    ...

    View full-size slide

  14. Follow @chewxy on Twitter
    As An APL - v1 2018 - 2022, v2 2022
    life ← {⊃1 ⍵ ∨.∧ 3 4 = +/ +⌿ ¯1 0 1 ∘.⊖ ¯1 0 1 ⌽¨ ⊂⍵}

    View full-size slide

  15. Follow @chewxy on Twitter
    As An APL - v1 2018 - 2022, v2 2022
    x ← ⎕?GN 2 2
    y ← ⎕?U 2 2
    z ← x +.× y
    ∂ x = y ⍝ check that partial deriv of x = y
    1

    View full-size slide

  16. Follow @chewxy on Twitter
    INiGo

    View full-size slide

  17. Follow @chewxy on Twitter
    INiGo
    INiGo - Iverson Notation in Go

    View full-size slide

  18. Follow @chewxy on Twitter
    INiGo
    INiGo - Iverson Notation in Go
    MoNTOYA - Materializer of Names That Otherwise Yield Aught

    View full-size slide

  19. Follow @chewxy on Twitter
    INiGo - History
    ● v1 - MNG (2018 - 2022)
    ● v2 - INiGo (current)

    View full-size slide

  20. Follow @chewxy on Twitter
    INiGo
    ● Slightly different notation from traditional APLs (GNU APL or Dyalog APL)
    ● Tree Calculus-like semantics and a Jay-style shapely type system
    ● More symbols:
    ○ ⚠ - Mark function as unsafe
    ○ ∇ - Auto differentiation (𝖄 is used for recursion instead)
    ○ ∂ - Partial derivative of a value
    ○ 🀱 - Horizontal stack
    ○ 🁣 - Vertical stack
    ○ ⌘ - Command/Escape hatch
    ○ ⍼ - Plot
    ○ ⎄ - Unpivot

    View full-size slide

  21. Follow @chewxy on Twitter
    Change The Way You Think

    View full-size slide

  22. Follow @chewxy on Twitter
    Think different.

    View full-size slide

  23. Follow @chewxy on Twitter
    +.×

    View full-size slide

  24. Follow @chewxy on Twitter
    +.×
    Inner Product

    View full-size slide

  25. Follow @chewxy on Twitter
    +.×

    View full-size slide

  26. Follow @chewxy on Twitter
    +.×

    View full-size slide

  27. Follow @chewxy on Twitter
    +.×

    View full-size slide

  28. Follow @chewxy on Twitter
    +.×

    View full-size slide

  29. Follow @chewxy on Twitter
    On + and ×
    1 + 1
    2

    View full-size slide

  30. Follow @chewxy on Twitter
    On + and ×
    1 + 1
    2
    1 2 3 4 × 1 10 100 1000
    1 20 300 4000

    View full-size slide

  31. Follow @chewxy on Twitter
    Side Note: Reduction
    +/ 1 2 3 4
    10

    View full-size slide

  32. Follow @chewxy on Twitter
    +.×

    View full-size slide

  33. Follow @chewxy on Twitter
    +.×
    a b c
    d
    e
    f

    View full-size slide

  34. Follow @chewxy on Twitter
    +.×
    a b c
    d
    e
    f

    View full-size slide

  35. Follow @chewxy on Twitter
    +.×
    a b c
    d
    e
    f
    a×d

    View full-size slide

  36. Follow @chewxy on Twitter
    +.×
    a b c
    d
    e
    f
    a×d b×e

    View full-size slide

  37. Follow @chewxy on Twitter
    +.×
    a b c
    d
    e
    f
    a×d b×e c×f

    View full-size slide

  38. Follow @chewxy on Twitter
    +.×
    a b c
    d
    e
    f
    a×d b×e c×f

    View full-size slide

  39. Follow @chewxy on Twitter
    +.×
    a b c
    d
    e
    f
    a×d + b×e + c×f

    View full-size slide

  40. Follow @chewxy on Twitter
    +.×

    View full-size slide

  41. Follow @chewxy on Twitter
    +.×

    View full-size slide

  42. Follow @chewxy on Twitter
    +.×

    View full-size slide

  43. Follow @chewxy on Twitter
    +.×

    View full-size slide

  44. Follow @chewxy on Twitter
    +.×

    View full-size slide

  45. Follow @chewxy on Twitter
    ⋀.⋀

    View full-size slide

  46. Follow @chewxy on Twitter
    1 2 3 4 +.× 5 6 7 8
    70

    View full-size slide

  47. Follow @chewxy on Twitter
    1 2 3 4 +.× 5 6 7 8
    70
    1 2 3 4 ,.× 5 6 7 8
    5 12 21 32

    View full-size slide

  48. Follow @chewxy on Twitter
    1 2 3 4 +.× 5 6 7 8
    70
    1 2 3 4 ,.× 5 6 7 8
    5 12 21 32
    1 2 3 4 ,.{⍺'×'⍵','} 5 6 7 8
    1 × 5 , 2 × 6 , 3 × 7 , 4 × 8 ,

    View full-size slide

  49. Follow @chewxy on Twitter
    Beautiful APL
    c (×÷+.×) p

    View full-size slide

  50. Follow @chewxy on Twitter
    cp ← {({(⍳∘(⌈/⍵))⍵}|+\(⍵-(+/÷≢)⍵))⌷⍺}
    Real Life APL

    View full-size slide

  51. Follow @chewxy on Twitter
    Things I Discovered By Virtue of New Notation
    ● Rediscovery of differentiation by means of skolemization of dual numbers.
    ● S(·) → + → × → ✳
    ● Various isomorphisms that may be useful for developing neural networks.

    View full-size slide

  52. Follow @chewxy on Twitter
    On Problems (Redux)

    View full-size slide

  53. Follow @chewxy on Twitter
    github.com/chewxy/annotation-mode.el

    View full-size slide

  54. Follow @chewxy on Twitter
    Programming in the large
    vs
    Programming in the small

    View full-size slide

  55. Follow @chewxy on Twitter
    “Software engineering is programming, integrated over
    time”
    - Titus Winters

    View full-size slide

  56. Follow @chewxy on Twitter

    View full-size slide

  57. Follow @chewxy on Twitter
    On Good Notation(s)

    View full-size slide

  58. Follow @chewxy on Twitter

    View full-size slide

  59. Follow @chewxy on Twitter
    The Properties of a Good Notation
    ● Ease of Expressing Constructs in Problem Solving
    ● Suggestivity
    ● Subordination of Detail
    ● Economy
    ● Amenable to Formal Proofs

    View full-size slide

  60. Follow @chewxy on Twitter
    The Properties of a Good Notation
    ● Ease of Expressing Constructs in Problem Solving
    ● Suggestivity
    ● Subordination of Detail
    ● Economy
    ● Amenable to Formal Proofs

    View full-size slide

  61. Follow @chewxy on Twitter
    The Properties of a Good Notation
    ● Ease of Expressing Constructs in Problem Solving
    ● Suggestivity
    ● Subordination of Detail
    ● Economy
    ● Amenable to Formal Proofs

    View full-size slide

  62. Follow @chewxy on Twitter
    The Properties of a Good Notation
    ● Ease of Expressing Constructs in Problem Solving
    ● Suggestivity
    ● Subordination of Detail
    ● Economy
    ● Amenable to Formal Proofs

    View full-size slide

  63. Follow @chewxy on Twitter
    Multiple Notation In Use
    ● Specification and implementation
    ○ Type language vs term language
    ○ Googlers Protobuf transformers

    View full-size slide

  64. Follow @chewxy on Twitter
    Multiple Notation In Use
    ● Specification and implementation
    ○ Type language vs term language
    ○ Googlers Protobuf transformers
    ● Mixed computational models
    ○ Regular expressions
    ○ SQL

    View full-size slide

  65. Follow @chewxy on Twitter
    Multiple Notation In Use
    ● Specification and implementation
    ○ Type language vs term language
    ○ Googlers Protobuf transformers
    ● Mixed computational models
    ○ Regular expressions
    ○ SQL
    ● Hierarchical use
    ○ Functional core, imperative shell
    ○ Imperative core, functional BAU (e.g. NaughtyDog’s Game Oriented Assembly Lisp )

    View full-size slide

  66. Follow @chewxy on Twitter
    Multiple Notation In Use
    ● Specification and implementation
    ○ Type language vs term language
    ○ Googlers Protobuf transformers
    ● Mixed computational models
    ○ Regular expressions
    ○ SQL
    ● Hierarchical use
    ○ Functional core, imperative shell
    ○ Imperative core, functional BAU (e.g. NaughtyDog’s GOAL - OpenGOAL.dev)
    ● DSLs and Mixins
    ● Intensional and extensional views w/ periodic unification
    ● Multi-surface languages (e.g. Racket, JetBrains MPS, etc)

    View full-size slide

  67. Follow @chewxy on Twitter
    Poly(glot) Pleasure
    “A change in perspective is worth 80 IQ points”
    - Alan Kay

    View full-size slide

  68. Follow @chewxy on Twitter
    Properties of Good Notations
    ● Good interoperation/integration
    ● Fit for purpose
    ● Distinctive

    View full-size slide

  69. Follow @chewxy on Twitter
    Good Interop
    Gold standard:
    1 + 1

    View full-size slide

  70. Follow @chewxy on Twitter
    Good Interop
    Gold standard:
    1 + 1
    💩 standard:
    λm.λn.λf.λx.mf(nfx) λf.λx.fx λf.λx.fx

    View full-size slide

  71. Follow @chewxy on Twitter
    Good Interop
    Gold standard:
    1 + 1
    Whelmed standard:
    import arith ..
    arith.add(arith.one, arith.one)
    💩 standard:
    λm.λn.λf.λx.mf(nfx) λf.λx.fx λf.λx.fx

    View full-size slide

  72. Follow @chewxy on Twitter
    Properties of Good Notations
    ● Good interoperation/integration
    ● Fit for purpose
    ● Distinctive

    View full-size slide

  73. Follow @chewxy on Twitter
    “It can scarcely be denied that the supreme goal of all theory is to make the irreducible
    basic elements as simple and as few as possible without having to surrender the
    adequate representation of a single datum of experience.”
    - Albert Einstein

    View full-size slide

  74. Follow @chewxy on Twitter
    Properties of Good Notations
    ● Good interoperation/integration
    ● Fit for purpose
    ● Distinctive

    View full-size slide

  75. Follow @chewxy on Twitter
    Reality Knocks On My Door…

    View full-size slide

  76. Follow @chewxy on Twitter
    Recommendations
    ● Invent specific languages to solve your specific problem.
    ● Invent multiple languages to solve one problem.
    ● Take a hundred metre view:
    ○ Survey similarities
    ○ Examine exceptions
    ○ Understand unifiers

    View full-size slide

  77. Follow @chewxy on Twitter
    Alternatives
    Cue/Dhall instead of YAML
    Protobuf instead of json
    dbt for “dependent” type checking of
    databases
    Malloy for “modular” SQL
    Nix for reproducible builds

    View full-size slide

  78. Follow @chewxy on Twitter
    Alternatives
    Cue/Dhall instead of YAML
    Protobuf instead of json
    dbt for “dependent” type checking of
    databases
    Malloy for “modular” SQL
    Nix for reproducible builds
    Deterministic Name Algebra
    }

    View full-size slide

  79. Follow @chewxy on Twitter
    Takeaways

    View full-size slide

  80. Follow @chewxy on Twitter
    Takeaways
    ● Language := rules + elements that the rules work on.
    ● Solving problems := establishing rules/constraints on a bunch of things.
    ● Notation := how language is written.
    ● Changing the way we write changes the way we solve problems.
    ● Use more different notations to help us think through and solve problems.
    ● Don’t add more cruft to the already fragile ecosystem.

    View full-size slide

  81. Follow @chewxy on Twitter
    @chewxy on Twitter
    Thank You
    Mei Lee
    Gary Miller
    Sarah Catanzaro
    Dane Laban
    Gajan Suthokumar
    Adam Burgess
    Daniel Owji
    Darrell Chua
    Gareth Seneque

    View full-size slide

  82. Follow @chewxy on Twitter
    Appendix

    View full-size slide

  83. Follow @chewxy on Twitter
    FAQs - On INiGo’s Creation
    ● Why not lisp/Have you heard of Racket?
    ○ (((o(°▽°)o)))
    ● Why not Rust?
    ○ People keep comparing Rust to Go. They should really compare Rust with chess. Both are tactical
    games. Go is more of a strategy game.
    ● Why not Haskell/Ocaml?
    ○ Both are pure functional programming languages. Ocaml is pure, and Haskell is extra virgin. I
    can’t afford the purity because I have a day job.
    ● Why don’t you use Dyalog APL?
    ○ I don’t understand their business model, which appears to be stuck 20 years in the past. Plus, I
    control my own language.

    View full-size slide

  84. Follow @chewxy on Twitter
    FAQs - On Use of INiGo and APLs
    ● How do you type INiGo or APL code?
    ○ With a keyboard. Specifically, all the keyboards I use either have QMK or ZMK as a firmware. The
    QMK keyboards have a special APL layer, which allows direct input.
    ● Are the symbols necessary?
    ○ Yes and no. They may be replaced by words or any other combination of symbols on a keyboard
    (like J-lang does). But I feel the symbols are especially well thought out and are suggestive of
    what they do.
    ○ e.g ⍉ is transpose and ⌽ is reverse/rotate along the Y axis. These give visual cues as to what
    you’re doing.

    View full-size slide

  85. Follow @chewxy on Twitter
    FAQs
    ● What is the Sanskrit text background in the first slide?
    ○ Book 10 of the Rig Veda, on knowledge
    ○ The verse reads:
    “Bṛhaspati, when the wise established name-giving - the first principles of Vāc (Language) - all that
    was excellent and pure was brought forth to light”
    ● What is the Chinese text in the background of the Language slide?
    ○ The White Horse Dialogue by Gongsun Long (320-250 B.C)
    ○ The words observes a linguistic paradox - that a white horse is not a horse
    ○ The crux of the paradox is summed up by Bill Clinton:
    “that depends on what your definition of ‘is’ is”.

    View full-size slide

  86. Follow @chewxy on Twitter
    FAQs - on Representing AI-ish Things
    ● Have you heard of Dyna?
    Yes. I have also used it. It’s a declarative language for maths that I feel is better
    than traditional mathematical language.
    ● The mathematical notations on neural networks aren’t all equally showing the
    same information!
    First, that’s not a question. And yes, I am aware of that. Surely I get to bias
    things a little to make my point.

    View full-size slide

  87. Follow @chewxy on Twitter
    FAQs
    ● What is with the scary head picture on the ALGOL slide?
    ○ Algol is known as the demon star in the constellation of Perseus.
    ○ Perseus killed Medusa, a Gorgon. The demon star refers to Medusa.
    ○ Gorgonia was named after the Gorgons.

    View full-size slide

  88. Follow @chewxy on Twitter
    FAQs on Multiple Notation
    ● Didn’t we already go through this with the mess of UML and the like in the 90s?
    You want us to go back to writing specs like that?
    Yes. We lacked wisdom then. We are wiser now. Knowing when to use things
    like UML for problem solving and when to use it for specification is wisdom.

    View full-size slide

  89. Follow @chewxy on Twitter
    Supplementary Material

    View full-size slide

  90. Follow @chewxy on Twitter
    The Hierarchy of Operations in the Arithmetic Universe is about to Change

    *
    *

    View full-size slide

  91. Follow @chewxy on Twitter
    The Hierarchy of Operations in the Arithmetic Universe is about to Change
    Traditionally:
    ● Repeated S ⇒ +
    ● Repeated + ⇒ ×
    ● Repeated × ⇒ *

    View full-size slide

  92. Follow @chewxy on Twitter
    The Hierarchy of Operations in the Arithmetic Universe is about to Change
    Traditionally:
    ● Repeated S ⇒ +
    ● Repeated + ⇒ ×
    ● Repeated × ⇒ *
    But:
    ab ≠ ba
    Compiler writers love the commutativity trick!

    View full-size slide

  93. Follow @chewxy on Twitter
    The Hierarchy of Operations in the Arithmetic Universe is about to Change
    Is there an operation ✳ that is a “next level”, and still maintains commutativity,
    associativity and distributivity?

    View full-size slide

  94. Follow @chewxy on Twitter
    The Hierarchy of Operations in the Arithmetic Universe is about to Change
    Is there an operation ✳ that is a “next level”, and still maintains commutativity,
    associativity and distributivity?
    a ✳ b = aln b

    View full-size slide

  95. Follow @chewxy on Twitter
    The Hierarchy of Operations in the Arithmetic Universe is about to Change
    Is there an operation ✳ that is a “next level”, and still maintains commutativity,
    associativity and distributivity?
    a ✳ b = aln b
    ✓ (a ✳ b) ✳ c = a ✳ (b ✳ c)
    ✓ a ✳ b = b ✳ a
    ✓ a ✳ (b × c) = a ✳ b × a ✳ c
    Identity: e
    Inverse Identity: e1/ln(a)

    View full-size slide

  96. Follow @chewxy on Twitter
    The Hierarchy of Operations in the Arithmetic Universe is about to Change
    Is there an operation ✳ that is a “next level”, and still maintains commutativity,
    associativity and distributivity?
    a ✳ b = aln b
    ✓ (a ✳ b) ✳ c = a ✳ (b ✳ c)
    ✓ a ✳ b = b ✳ a
    ✓ a ✳ (b × c) = a ✳ b × a ✳ c
    Identity: e
    Inverse Identity: e1/ln(a)
    a, b, c > 0

    View full-size slide