delivering_security_products.pdf

Delivering security products without shooting in a foot

Talk about improving the infrastructure for developing, testing and delivering security tools. Our experience of smoothing the difference between security idealism and engineering friendliness.

Transcript

1. Delivering security products without shooting in a foot Dmytro Shapovalov Infrastructure Engineer @ Cossack Labs
2. We will talking about: — Approaches, not instruments — CI/CD — Infrastructure — Workflow — Usability
3. Why is the delivery a bit different in security?
4. An attractive target for attackers
5. Highly qualified and very demanding community
6. Significant risks and bigger cost of each mistake Public responsibility
7. What is the problem? Security teams create complex security products, developers want to have an easy way to use them
8. Why developing is so hard? — ingenious ideas — chaotic strategy changing — do not care about compatibility — testing in hothouse conditions — hard to deploy — hard to use properly
9. Developers expect stability from libraries, but tend to make breaking changes in own products
10. Avoid creating useless products!
11. Take care of the delivery!
12. Let's look for a solution!
13. Our company specific: — we create crypthographic products — compact size of team — mostly engineering staff — horizontal structure — open-source products — multiple languages / platforms
14. What is critical for us? — automation — clear, transparent processes — responsibility — openness — safety — use own products — write docs
15. Let's do something!
16. Let's divide solution into the parts: — architecture — workflow standards and rules — infrastructure — CI/CD
17. Are you involved into the planning process?
18. What questions should we answer during planning?
19. What are customers' values?
20. How will it be arranged?
21. — on what platforms should our application work? — how many and what are the types of the components? — usability bounds of the different components? — what are the possible topology variants?
22. — how will they communicate? — how do we plan to scale? — what external dependencies will the product have? — what are possible vulnerabilities and risks in the planned architecture?
23. Use existing standards wherever possible!
24. How will it be delivered?
25. — source code — package management systems — docker — pre-built images
26. Give the choice for a lazy user!
27. What are the external systems and how do we plan to integrate with? — basic functional systems — an application in which we integrate — databases — libraries — auxiliary systems — monitoring — SIEM
28. How will it be monitored?
29. How will application handle errors? — standard messages — standard codes — common library
30. How do you plan to do logging, tracing and metrics gathering?
31. Logging — events — human-readable — standard output methods Formats and outputs File STDERR Syslog Plain text ✔ ✔ - JSON ✔ ✔ ✔ CEF ✔ ✔ ✔ syslog - - ✔
32. Metrics — availability — components' health — performance — statistics Output — prometheus standard
33. Tracing — detail analyze of successive data processing — intrusion detection — performance tuning — deep debugging Output — OpenTracing — OpenCensus — JSON
34. How will it be supported? — fast deploy scripts — troubleshooting information gathering tools
35. Workflow standards and rules — synchronize all team members — unify approaches — make the development results more expected and clear for our customers — simplify automation
36. The main idea is to simplify internal processes!
37. The rules must: — be as simple and clear as possible — be kept actual — have a minimal impact on productivity — only correct, not break existing workflow
38. Our internal standards and rules — versioning — naming — git flow — CI/CD
39. Versioning Semantic Versioning https://semver.org Format: a.b.c, where: a — major version, increases with significant changes of product b — minor version, increases with current releases, backward compatibility may not be guaranteed c — build version, increases with bug hotfixes, minor features' improvements
40. Naming Element Rule Example application CamelCase AcraServer binary debian-style acra-connector package debian-style libthemis- dev0.10.0+stretch amd64.deb
41. Git flow and CI/CD Name Usage CI/CD automation stable Releases all tests, packaging, delivering master Current state all tests, packaging, delivering username/ GH999_issue_des c Dev, fixes manual
42. The earlier you implement the internal standards, the cheaper the support will be later!
43. Infrastructure Part Requirements core safety, stable developing speed up, easy cross- communications CI/CD full automation, tests, packaging, delivery
44. CI/CD
46. CircleCI — quick check of basic functionality
47. Buildbot — unit tests — multiple platforms — crossplatform tests — integration tests — pre-release checks — packages building
48. Buildbot architecture — core — CI/CD engine — scenarios
49. CI/CD rules — emulate real customer environment — use system libraries — strict configurations — scenarios according to docs — easily maintained and extended — developers can add scenarios
50. Exact methodology do not change rules on the fly
51. CI/CD flow
52. Docker
53. Docker image sizes — large image size is typical — most developers do not pay attention to the size — there are ways to significantly reduce the image — we reduced the size from ≈950MB to ≈15MB
54. What technologies did we use? — multi-stage builds — build from scratch You can find all dockerfiles at: https://github.com/cossacklabs/acra/tree/master/docker
55. Take care of the consumer!
56. Image tags Static tags — version tag — git commit tag Sliding tags — stable, latest — master, current
57. Image labels — label-schema standard — Cossack Labs internal standard
58. How do we use docker compose? As a demonstration stand!
59. What was uncomfortable? — generate and manually place keys — configure parameters inside docker-compose file manually — one very simple example of configuration — topology was unclear — security perimeter was not visible
60. What now? — multiple demonstation stands — all stands can be launched in one action — confiurations can be easily extended and secured * — self-documented — topology described
61. Summary — complex security products require careful and accurate handling — iplemented standards — built large testing platform — release cycle: from once a year to once a couple weeks — from burst developing to planned releases — increased flexibility of our team — from concept researching to implementing usable features
62. Our products became closer to consumers!
63. Links — Cossack Labs official site https://cossacklabs.com — GitHub https://github.com/cossacklabs — Docker Hub https://hub.docker.com/r/cossacklabs/
64. Dmytro Shapovalov shad@cossacklabs.com shadinua shad.in.ua shad.in.ua