$30 off During Our Annual Pro Sale. View Details »

Hanging on in Quiet Desperation: Time & Programming

Hanging on in Quiet Desperation: Time & Programming

Time has a profound impact on the complexity of the systems we build.

A significant amount of this software complexity comes from either
an inability to recall previous states or the inability to
understand how a state was arrived at.

From the foundations of AI, LISP and functional programming [1],
to causality in distributed systems [2], to the more grungy practices
of immutable infrastructure, or the unreasonable effectiveness
of fact-based approaches to large scale data systems; the ability
to adequately cope with time, and the change and conflict it
inevitable creates, is a common thread to being able to build and
reason about these systems.

This talk looks at the impact of time on system design. We will
walk through examples of large-scale systems and their battles
with complexity. At the end of the talk, the audience should start to
see the common spectre of time and have an appreciation of
how understanding time is fundamental to maintaining clarity,
correctness and reliability in systems.

[1] Situations, Actions, and Causal Laws - John
McCarthy - http://www.dtic.mil/dtic/tr/fulltext/u2/785031.pdf

[2] Times, Clocks and the Ordering of Events in a Distributed
System - Leslie Lamport - https://amturing.acm.org/p558-lamport.pdf

Mark Hibberd

May 23, 2018
Tweet

More Decks by Mark Hibberd

Other Decks in Technology

Transcript

  1. Hanging on in Quiet Desperation:
    @markhibberd
    Time & Programming

    View Slide

  2. Ticking away the moments that make up a dull day.
    You fritter and waste the hours in an offhand way.

    And then one day you find ten years have got behind you.
    No one told you where to run, you missed the starting gun.”
    - Roger Waters

    View Slide

  3. Complexity.

    View Slide

  4. 1986.

    View Slide

  5. View Slide

  6. Changeability.
    Conformity.
    Complexity.
    Invisibility.

    View Slide

  7. 2006.

    View Slide

  8. View Slide

  9. 2011.

    View Slide

  10. View Slide

  11. Present Day.

    View Slide

  12. 1948.

    View Slide

  13. View Slide

  14. 1958-1963.

    View Slide

  15. View Slide

  16. The big idea.

    View Slide

  17. If we can reason
    about all the states of
    a system.

    View Slide

  18. And we can
    understand how each
    state is arrived at.

    View Slide

  19. We can make better decisions, and
    start to unwind the
    interdependencies and complexity
    caused by time and ordering.

    View Slide

  20. View Slide

  21. View Slide

  22. Present Day.

    View Slide

  23. 2004-2006.

    View Slide

  24. Disconnected networks.

    View Slide

  25. View Slide

  26. View Slide

  27. View Slide

  28. View Slide

  29. View Slide

  30. View Slide

  31. {dock:0
    ,red:0
    ,purple:0}
    {dock:0
    ,red:0
    ,purple:0}
    {dock:0
    ,red:0
    ,purple:0}
    owner|id|item|count
    red |0 |x |99
    red |1 |y |1
    dock |2 |z |33

    View Slide

  32. {dock:3
    ,red:0
    ,purple:0}
    {dock:0
    ,red:0
    ,purple:1}
    {dock:0
    ,red:7
    ,purple:0}

    View Slide

  33. {dock:5
    ,red:0
    ,purple:0}
    {dock:0
    ,red:7
    ,purple:1}
    {dock:0
    ,red:7
    ,purple:1}

    View Slide

  34. {dock:5
    ,red:0
    ,purple:0}
    {dock:3
    ,red:7
    ,purple:1}
    {dock:0
    ,red:7
    ,purple:1}

    View Slide

  35. {dock:7
    ,red:7
    ,purple:3}
    {dock:7
    ,red:7
    ,purple:3}
    {dock:0
    ,red:8
    ,purple:1}

    View Slide

  36. {dock:7
    ,red:8
    ,purple:3}
    {dock:7
    ,red:8
    ,purple:3}
    {dock:7
    ,red:8
    ,purple:3}

    View Slide

  37. View Slide

  38. 2009-2010.

    View Slide

  39. The most complex system I
    ever worked on.

    View Slide

  40. A web API with 2 resources.

    View Slide

  41. POST /activate
    POST /check
    Call to activate a product, validate and
    register license details
    Check license and validate installation
    details.

    View Slide

  42. What could possibly go
    wrong?

    View Slide

  43. View Slide

  44. POST /activate POST /check

    View Slide

  45. POST /activate POST /check
    POST /check2007

    View Slide

  46. POST /activate?magik=true POST /check
    POST /check2007

    View Slide

  47. POST /activate?magik=true POST /check
    POST /check2007
    POST /activatev3

    View Slide

  48. POST /activate?magik=true POST /check
    POST /check2007
    POST /activatev3
    POST /check9

    View Slide

  49. POST /activate?magik=true POST /check
    POST /check2007
    POST /activatev3
    POST /activate-new POST /check9

    View Slide

  50. POST /activate?magik=true POST /check
    POST /check2007
    POST /activatev3
    POST /activate-new?legacy=1 POST /check9

    View Slide

  51. POST /activate?magik=true POST /check
    POST /check2007
    POST /activatev3
    POST /activate-new?legacy=1 POST /check9

    View Slide

  52. POST /activate?magik=true POST /check
    POST /check2007
    POST /activatev3
    POST /activate-new?legacy=1 POST /check9

    View Slide

  53. POST /activate?magik=true POST /check
    POST /check2007
    POST /activatev3
    POST /activate-new?legacy=1 POST /check9

    View Slide

  54. POST /activate?magik=true POST /check
    POST /check2007
    POST /activatev3
    POST /activate-new?legacy=1 POST /check9

    View Slide

  55. From: https://stripe.com/blog/api-versioning

    View Slide

  56. 2013-2017.

    View Slide

  57. Full circle.
    A machine learning problem.

    View Slide

  58. We need to describe the
    world, now and in the past.

    View Slide

  59. cause(π)(α)

    View Slide

  60. at(I)(desk)

    View Slide

  61. at(I)(desk)
    at(desk)(home)

    View Slide

  62. at(I)(desk)
    at(desk)(home)
    ∴ at(I)(home)

    View Slide

  63. TX CUSTOMER ITEM AMOUNT TIMESTAMP
    1 X0001 Pen $5 2016-08-01T12:24:13
    2 X0001 Notepad $20 2017-01-01T02:19:01
    3 X0002 Calculator $55 2015-01-01T09:55:21
    4 X0003 Book $30 2017-07-01T15:34:53

    View Slide

  64. CUSTOMER LOCATION
    X0001 NY
    X0002 SINGAPORE
    X0003 HONG KONG
    X0004 SYDNEY
    X0005 SYDNEY

    View Slide

  65. CUSTOMER LOCATION TIMESTAMP
    X0001 NY 2016-08-01T00:00:00
    X0002 SINGAPORE 2017-08-04T00:00:00
    X0003 HONG KONG 2016-01-21T00:00:00
    X0004 SYDNEY 2016-08-01T00:00:00
    X0005 SYDNEY 2014-12-24T00:00:00

    View Slide

  66. CUSTOMER LOCATION TIMESTAMP
    X0001 NY 2016-08-01T00:00:00
    X0002 SINGAPORE 2017-08-04T00:00:00
    X0003 HONG KONG 2016-01-21T00:00:00
    X0004 SYDNEY 2016-08-01T00:00:00
    X0005 SYDNEY 2014-12-24T00:00:00
    X0001 LONDON 2017-09-07T00:00:00

    View Slide

  67. What is in a time?

    View Slide

  68. World Time System Time

    View Slide

  69. When is this
    fact valid?
    When did we
    learn this fact?

    View Slide

  70. Intervals.
    X0001 NY 2016-08-01T00:00:00
    X0001 LONDON 2017-09-07T00:00:00

    View Slide

  71. Instants.
    1 X0001 Pen $5 2016-08-01T12:24:13

    View Slide

  72. Horizons.
    1 X0001 Pen $5 2016-08-01T12:24:13

    View Slide

  73. Horizons.
    1 X0001 Pen $5 1234

    View Slide

  74. 2013-2017.

    View Slide

  75. Operations as a system.

    View Slide

  76. This is complexity.

    View Slide

  77. Techniques!

    View Slide

  78. How does time impact the system?

    View Slide

  79. View Slide

  80. View Slide

  81. 2014.

    View Slide

  82. towards
    annɛx

    View Slide

  83. Semantic versioning is not the
    whole story.

    View Slide

  84. Open world.

    View Slide

  85. Time matters.

    View Slide

  86. commit

    View Slide

  87. ci

    View Slide

  88. publish

    View Slide

  89. platform test

    View Slide

  90. production

    View Slide

  91. performance

    View Slide

  92. cve

    View Slide

  93. eminence
    boxer
    napoleon
    snowball

    View Slide

  94. boxer 1.2.1
    commit: bd2f074…02
    api-signature: […]
    feature: it-works
    artifact:
    tag: bin/boxer
    flags: […]
    address: e2f1…bc74

    View Slide

  95. :boxer
    :has :feature “multi-part-put”
    :has :commit “abcd-1345”
    :has :branch “master”
    :snowball
    :is :compatable-with atom/…
    :napoleon
    :semver >= 1.3 < 1.4
    :no-cve +annex.example.com@HEAD

    View Slide

  96. View Slide

  97. View Slide

  98. View Slide

  99. View Slide

  100. View Slide

  101. View Slide

  102. View Slide

  103. Present Day.

    View Slide

  104. Artifacts over constructs.

    View Slide

  105. Time and ordering are out to get us.

    View Slide

  106. Techniques abound.

    View Slide

  107. If we can reason
    about all the states of
    a system.

    View Slide

  108. And we can
    understand how each
    state is arrived at.

    View Slide

  109. We can make better decisions, and
    start to unwind the
    interdependencies and complexity
    caused by time and ordering.

    View Slide

  110. Hanging on in Quiet Desperation:
    @markhibberd
    Time & Programming

    View Slide