Hanging on in Quiet Desperation: Time & Programming

Time has a profound impact on the complexity of the systems we build.

A significant amount of this software complexity comes from either
an inability to recall previous states or the inability to
understand how a state was arrived at.

From the foundations of AI, LISP and functional programming [1],
to causality in distributed systems [2], to the more grungy practices
of immutable infrastructure, or the unreasonable effectiveness
of fact-based approaches to large scale data systems; the ability
to adequately cope with time, and the change and conflict it
inevitable creates, is a common thread to being able to build and
reason about these systems.

This talk looks at the impact of time on system design. We will
walk through examples of large-scale systems and their battles
with complexity. At the end of the talk, the audience should start to
see the common spectre of time and have an appreciation of
how understanding time is fundamental to maintaining clarity,
correctness and reliability in systems.

[1] Situations, Actions, and Causal Laws - John
McCarthy - http://www.dtic.mil/dtic/tr/fulltext/u2/785031.pdf

[2] Times, Clocks and the Ordering of Events in a Distributed
System - Leslie Lamport - https://amturing.acm.org/p558-lamport.pdf

Transcript

1. Hanging on in Quiet Desperation: @markhibberd Time & Programming
2. Ticking away the moments that make up a dull day. You fritter and waste the hours in an offhand way. … And then one day you find ten years have got behind you. No one told you where to run, you missed the starting gun.” - Roger Waters “
3. Complexity.
4. 1986.
6. Changeability. Conformity. Complexity. Invisibility.
7. 2006.
9. 2011.
11. Present Day.
12. 1948.
14. 1958-1963.
16. The big idea.
17. If we can reason about all the states of a system.
18. And we can understand how each state is arrived at.
19. We can make better decisions, and start to unwind the interdependencies and complexity caused by time and ordering.
22. Present Day.
23. 2004-2006.
24. Disconnected networks.
31. {dock:0 ,red:0 ,purple:0} {dock:0 ,red:0 ,purple:0} {dock:0 ,red:0 ,purple:0} owner|id|item|count red |0 |x |99 red |1 |y |1 dock |2 |z |33
32. {dock:3 ,red:0 ,purple:0} {dock:0 ,red:0 ,purple:1} {dock:0 ,red:7 ,purple:0}
33. {dock:5 ,red:0 ,purple:0} {dock:0 ,red:7 ,purple:1} {dock:0 ,red:7 ,purple:1}
34. {dock:5 ,red:0 ,purple:0} {dock:3 ,red:7 ,purple:1} {dock:0 ,red:7 ,purple:1}
35. {dock:7 ,red:7 ,purple:3} {dock:7 ,red:7 ,purple:3} {dock:0 ,red:8 ,purple:1}
36. {dock:7 ,red:8 ,purple:3} {dock:7 ,red:8 ,purple:3} {dock:7 ,red:8 ,purple:3}
38. 2009-2010.
39. The most complex system I ever worked on.
40. A web API with 2 resources.
41. POST /activate POST /check Call to activate a product, validate and register license details Check license and validate installation details.
42. What could possibly go wrong?
44. POST /activate POST /check
45. POST /activate POST /check POST /check2007
46. POST /activate?magik=true POST /check POST /check2007
47. POST /activate?magik=true POST /check POST /check2007 POST /activatev3
48. POST /activate?magik=true POST /check POST /check2007 POST /activatev3 POST /check9
49. POST /activate?magik=true POST /check POST /check2007 POST /activatev3 POST /activate-new POST /check9
50. POST /activate?magik=true POST /check POST /check2007 POST /activatev3 POST /activate-new?legacy=1 POST /check9
51. POST /activate?magik=true POST /check POST /check2007 POST /activatev3 POST /activate-new?legacy=1 POST /check9
52. POST /activate?magik=true POST /check POST /check2007 POST /activatev3 POST /activate-new?legacy=1 POST /check9
53. POST /activate?magik=true POST /check POST /check2007 POST /activatev3 POST /activate-new?legacy=1 POST /check9
54. POST /activate?magik=true POST /check POST /check2007 POST /activatev3 POST /activate-new?legacy=1 POST /check9
55. From: https://stripe.com/blog/api-versioning
56. 2013-2017.
57. Full circle. A machine learning problem.
58. We need to describe the world, now and in the past.
59. cause(π)(α)
60. at(I)(desk)
61. at(I)(desk) at(desk)(home)
62. at(I)(desk) at(desk)(home) ∴ at(I)(home)
63. TX CUSTOMER ITEM AMOUNT TIMESTAMP 1 X0001 Pen $5 2016-08-01T12:24:13 2 X0001 Notepad $20 2017-01-01T02:19:01 3 X0002 Calculator $55 2015-01-01T09:55:21 4 X0003 Book $30 2017-07-01T15:34:53
64. CUSTOMER LOCATION X0001 NY X0002 SINGAPORE X0003 HONG KONG X0004 SYDNEY X0005 SYDNEY
65. CUSTOMER LOCATION TIMESTAMP X0001 NY 2016-08-01T00:00:00 X0002 SINGAPORE 2017-08-04T00:00:00 X0003 HONG KONG 2016-01-21T00:00:00 X0004 SYDNEY 2016-08-01T00:00:00 X0005 SYDNEY 2014-12-24T00:00:00
66. CUSTOMER LOCATION TIMESTAMP X0001 NY 2016-08-01T00:00:00 X0002 SINGAPORE 2017-08-04T00:00:00 X0003 HONG KONG 2016-01-21T00:00:00 X0004 SYDNEY 2016-08-01T00:00:00 X0005 SYDNEY 2014-12-24T00:00:00 X0001 LONDON 2017-09-07T00:00:00
67. What is in a time?
68. World Time System Time
69. When is this fact valid? When did we learn this fact?
70. Intervals. X0001 NY 2016-08-01T00:00:00 X0001 LONDON 2017-09-07T00:00:00
71. Instants. 1 X0001 Pen $5 2016-08-01T12:24:13
72. Horizons. 1 X0001 Pen $5 2016-08-01T12:24:13
73. Horizons. 1 X0001 Pen $5 1234
74. 2013-2017.
75. Operations as a system.
76. This is complexity.
77. Techniques!
78. How does time impact the system?
81. 2014.
82. towards annɛx
83. Semantic versioning is not the whole story.
84. Open world.
85. Time matters.
86. commit
87. ci
88. publish
89. platform test
90. production
91. performance
92. cve
93. eminence boxer napoleon snowball
94. boxer 1.2.1 commit: bd2f074…02 api-signature: […] feature: it-works artifact: tag: bin/boxer flags: […] address: e2f1…bc74
95. :boxer :has :feature “multi-part-put” :has :commit “abcd-1345” :has :branch “master” :snowball :is :compatable-with atom/… :napoleon :semver >= 1.3 < 1.4 :no-cve +annex.example.com@HEAD
103. Present Day.
104. Artifacts over constructs.
105. Time and ordering are out to get us.
106. Techniques abound.
107. If we can reason about all the states of a system.
108. And we can understand how each state is arrived at.
109. We can make better decisions, and start to unwind the interdependencies and complexity caused by time and ordering.
110. Hanging on in Quiet Desperation: @markhibberd Time & Programming