Upgrade to Pro — share decks privately, control downloads, hide ads and more …


01dad3d7bf0ae06c552a9e8c07ab6bfa?s=47 ShaD
October 15, 2018


Delivering security products without shooting in a foot

Talk about improving the infrastructure for developing, testing and delivering security tools. Our experience of smoothing the difference between security idealism and engineering friendliness.



October 15, 2018

More Decks by ShaD

Other Decks in Programming


  1. Delivering security products without shooting in a foot Dmytro Shapovalov

    Infrastructure Engineer @ Cossack Labs
  2. We will talking about: — Approaches, not instruments — CI/CD

    — Infrastructure — Workflow — Usability
  3. Why is the delivery a bit different in security?

  4. An attractive target for attackers

  5. Highly qualified and very demanding community

  6. Significant risks and bigger cost of each mistake Public responsibility

  7. What is the problem? Security teams create complex security products,

    developers want to have an easy way to use them
  8. Why developing is so hard? — ingenious ideas — chaotic

    strategy changing — do not care about compatibility — testing in hothouse conditions — hard to deploy — hard to use properly
  9. Developers expect stability from libraries, but tend to make breaking

    changes in own products
  10. Avoid creating useless products!

  11. Take care of the delivery!

  12. Let's look for a solution!

  13. Our company specific: — we create crypthographic products — compact

    size of team — mostly engineering staff — horizontal structure — open-source products — multiple languages / platforms
  14. What is critical for us? — automation — clear, transparent

    processes — responsibility — openness — safety — use own products — write docs
  15. Let's do something!

  16. Let's divide solution into the parts: — architecture — workflow

    standards and rules — infrastructure — CI/CD
  17. Are you involved into the planning process?

  18. What questions should we answer during planning?

  19. What are customers' values?

  20. How will it be arranged?

  21. — on what platforms should our application work? — how

    many and what are the types of the components? — usability bounds of the different components? — what are the possible topology variants?
  22. — how will they communicate? — how do we plan

    to scale? — what external dependencies will the product have? — what are possible vulnerabilities and risks in the planned architecture?
  23. Use existing standards wherever possible!

  24. How will it be delivered?

  25. — source code — package management systems — docker —

    pre-built images
  26. Give the choice for a lazy user!

  27. What are the external systems and how do we plan

    to integrate with? — basic functional systems — an application in which we integrate — databases — libraries — auxiliary systems — monitoring — SIEM
  28. How will it be monitored?

  29. How will application handle errors? — standard messages — standard

    codes — common library
  30. How do you plan to do logging, tracing and metrics

  31. Logging — events — human-readable — standard output methods Formats

    and outputs File STDERR Syslog Plain text ✔ ✔ - JSON ✔ ✔ ✔ CEF ✔ ✔ ✔ syslog - - ✔
  32. Metrics — availability — components' health — performance — statistics

    Output — prometheus standard
  33. Tracing — detail analyze of successive data processing — intrusion

    detection — performance tuning — deep debugging Output — OpenTracing — OpenCensus — JSON
  34. How will it be supported? — fast deploy scripts —

    troubleshooting information gathering tools
  35. Workflow standards and rules — synchronize all team members —

    unify approaches — make the development results more expected and clear for our customers — simplify automation
  36. The main idea is to simplify internal processes!

  37. The rules must: — be as simple and clear as

    possible — be kept actual — have a minimal impact on productivity — only correct, not break existing workflow
  38. Our internal standards and rules — versioning — naming —

    git flow — CI/CD
  39. Versioning Semantic Versioning https://semver.org Format: a.b.c, where: a — major

    version, increases with significant changes of product b — minor version, increases with current releases, backward compatibility may not be guaranteed c — build version, increases with bug hotfixes, minor features' improvements
  40. Naming Element Rule Example application CamelCase AcraServer binary debian-style acra-connector

    package debian-style libthemis- dev0.10.0+stretch amd64.deb
  41. Git flow and CI/CD Name Usage CI/CD automation stable Releases

    all tests, packaging, delivering master Current state all tests, packaging, delivering username/ GH999_issue_des c Dev, fixes manual
  42. The earlier you implement the internal standards, the cheaper the

    support will be later!
  43. Infrastructure Part Requirements core safety, stable developing speed up, easy

    cross- communications CI/CD full automation, tests, packaging, delivery
  44. CI/CD

  45. None
  46. CircleCI — quick check of basic functionality

  47. Buildbot — unit tests — multiple platforms — crossplatform tests

    — integration tests — pre-release checks — packages building
  48. Buildbot architecture — core — CI/CD engine — scenarios

  49. CI/CD rules — emulate real customer environment — use system

    libraries — strict configurations — scenarios according to docs — easily maintained and extended — developers can add scenarios
  50. Exact methodology do not change rules on the fly

  51. CI/CD flow

  52. Docker

  53. Docker image sizes — large image size is typical —

    most developers do not pay attention to the size — there are ways to significantly reduce the image — we reduced the size from ≈950MB to ≈15MB
  54. What technologies did we use? — multi-stage builds — build

    from scratch You can find all dockerfiles at: https://github.com/cossacklabs/acra/tree/master/docker
  55. Take care of the consumer!

  56. Image tags Static tags — version tag — git commit

    tag Sliding tags — stable, latest — master, current
  57. Image labels — label-schema standard — Cossack Labs internal standard

  58. How do we use docker compose? As a demonstration stand!

  59. What was uncomfortable? — generate and manually place keys —

    configure parameters inside docker-compose file manually — one very simple example of configuration — topology was unclear — security perimeter was not visible
  60. What now? — multiple demonstation stands — all stands can

    be launched in one action — confiurations can be easily extended and secured * — self-documented — topology described
  61. Summary — complex security products require careful and accurate handling

    — iplemented standards — built large testing platform — release cycle: from once a year to once a couple weeks — from burst developing to planned releases — increased flexibility of our team — from concept researching to implementing usable features
  62. Our products became closer to consumers!

  63. Links — Cossack Labs official site https://cossacklabs.com — GitHub https://github.com/cossacklabs

    — Docker Hub https://hub.docker.com/r/cossacklabs/
  64. Dmytro Shapovalov shad@cossacklabs.com shadinua shad.in.ua shad.in.ua