Programming in the Large: Architecture and Experimentation

42d9867a0fee0fa6de6534e9df0f1e9b?s=47 Mark Hibberd
December 11, 2014

Programming in the Large: Architecture and Experimentation

Building robust, quality systems is hard. We trade off organizational issues against technical decisions; the ability to deliver quickly against our ability to change; and the ability to build systems easily against the ability to run those systems in production. However, good architectural decisions can free us to choose the right tools and techniques, allowing us to manage these challenges and concentrate on solving real problems rather than our made up ones.

In this talk, we will run through some stereotypical projects, come to terms with legacy systems, and look at the properties of robust architectures. In particular we are interested in how architectures lend themselves to experimentation and change in terms of both function and technology.

We will attempt to ground the discussion with examples from my past projects. Looking at where things have worked well and probably of more interest, where they really have not.

This was presented at the YOW! conference in Australia, Melbourne 04/12/2014, Brisbane 08/12/2014, Sydney 11/12/2014 (http://2014.yowconference.com.au/).

42d9867a0fee0fa6de6534e9df0f1e9b?s=128

Mark Hibberd

December 11, 2014
Tweet

Transcript

  1. 17.
  2. 34.
  3. 43.

    /call server /check we spent a lot of time “fire

    fighting” /check2 /check2z /v3check
  4. 44.

    /call server /check we spent a lot of time “fire

    fighting” /check2 /check2z /v3check
  5. 52.
  6. 61.

    ?

  7. 73.
  8. 90.
  9. 100.
  10. 101.
  11. 102.
  12. 103.
  13. 104.
  14. 105.
  15. 108.
  16. 109.
  17. 110.
  18. 111.
  19. 112.
  20. 113.
  21. 114.
  22. 115.
  23. 116.

    x x

  24. 120.

    it was really only feasible to change if one team

    was working on both “systems”
  25. 128.

    “... with proper design, the features come cheaply. This approach

    is arduous, but continues to succeed.” Dennis Ritchie
  26. 133.
  27. 137.
  28. 140.
  29. 150.
  30. 166.
  31. 167.
  32. 183.

    R make sure you can run this straight away external

    indexer support mega-code-search-tool
  33. 184.

    R make sure you can run this straight away external

    indexer support mega-code-search-tool
  34. 200.

    @ambiata we deal with ingesting and processing lots of data

    100s TB / per day / per customer scientific experiment and measurement is key experiments affect users directly researchers / non-specialist engineers produce code
  35. 240.

    package publish the machine wall-time: 13411s cpu-time: 429130s records: 19

    million histogram: a: 13million b: 2million c: 4million
  36. 241.

    package publish the machine wall-time: 13411s cpu-time: 429130s records: 19

    million histogram: a: 13million b: 2million c: 4million aggregate over time
  37. 243.

    package publish the machine cross check everything wall-time: 13411s cpu-time:

    429130s records: 19 million histogram: a: 13million b: 2million c: 4million
  38. 244.

    package publish the machine cross check everything wall-time: 13411s cpu-time:

    429130s records: 19 million histogram: a: 13million b: 2million c: 4million