delivering_security_products.pdf

Slide 1

Slide 1 text

Delivering security products without shooting in a foot Dmytro Shapovalov Infrastructure Engineer @ Cossack Labs

Slide 2

Slide 2 text

We will talking about: — Approaches, not instruments — CI/CD — Infrastructure — Workflow — Usability

Slide 3

Slide 3 text

Why is the delivery a bit different in security?

Slide 4

Slide 4 text

An attractive target for attackers

Slide 5

Slide 5 text

Highly qualified and very demanding community

Slide 6

Slide 6 text

Significant risks and bigger cost of each mistake Public responsibility

Slide 7

Slide 7 text

What is the problem? Security teams create complex security products, developers want to have an easy way to use them

Slide 8

Slide 8 text

Why developing is so hard? — ingenious ideas — chaotic strategy changing — do not care about compatibility — testing in hothouse conditions — hard to deploy — hard to use properly

Slide 9

Slide 9 text

Developers expect stability from libraries, but tend to make breaking changes in own products

Slide 10

Slide 10 text

Avoid creating useless products!

Slide 11

Slide 11 text

Take care of the delivery!

Slide 12

Slide 12 text

Let's look for a solution!

Slide 13

Slide 13 text

Our company specific: — we create crypthographic products — compact size of team — mostly engineering staff — horizontal structure — open-source products — multiple languages / platforms

Slide 14

Slide 14 text

What is critical for us? — automation — clear, transparent processes — responsibility — openness — safety — use own products — write docs

Slide 15

Slide 15 text

Let's do something!

Slide 16

Slide 16 text

Let's divide solution into the parts: — architecture — workflow standards and rules — infrastructure — CI/CD

Slide 17

Slide 17 text

Are you involved into the planning process?

Slide 18

Slide 18 text

What questions should we answer during planning?

Slide 19

Slide 19 text

What are customers' values?

Slide 20

Slide 20 text

How will it be arranged?

Slide 21

Slide 21 text

— on what platforms should our application work? — how many and what are the types of the components? — usability bounds of the different components? — what are the possible topology variants?

Slide 22

Slide 22 text

— how will they communicate? — how do we plan to scale? — what external dependencies will the product have? — what are possible vulnerabilities and risks in the planned architecture?

Slide 23

Slide 23 text

Use existing standards wherever possible!

Slide 24

Slide 24 text

How will it be delivered?

Slide 25

Slide 25 text

— source code — package management systems — docker — pre-built images

Slide 26

Slide 26 text

Give the choice for a lazy user!

Slide 27

Slide 27 text

What are the external systems and how do we plan to integrate with? — basic functional systems — an application in which we integrate — databases — libraries — auxiliary systems — monitoring — SIEM

Slide 28

Slide 28 text

How will it be monitored?

Slide 29

Slide 29 text

How will application handle errors? — standard messages — standard codes — common library

Slide 30

Slide 30 text

How do you plan to do logging, tracing and metrics gathering?

Slide 31

Slide 31 text

Logging — events — human-readable — standard output methods Formats and outputs File STDERR Syslog Plain text ✔ ✔ - JSON ✔ ✔ ✔ CEF ✔ ✔ ✔ syslog - - ✔

Slide 32

Slide 32 text

Metrics — availability — components' health — performance — statistics Output — prometheus standard

Slide 33

Slide 33 text

Tracing — detail analyze of successive data processing — intrusion detection — performance tuning — deep debugging Output — OpenTracing — OpenCensus — JSON

Slide 34

Slide 34 text

How will it be supported? — fast deploy scripts — troubleshooting information gathering tools

Slide 35

Slide 35 text

Workflow standards and rules — synchronize all team members — unify approaches — make the development results more expected and clear for our customers — simplify automation

Slide 36

Slide 36 text

The main idea is to simplify internal processes!

Slide 37

Slide 37 text

The rules must: — be as simple and clear as possible — be kept actual — have a minimal impact on productivity — only correct, not break existing workflow

Slide 38

Slide 38 text

Our internal standards and rules — versioning — naming — git flow — CI/CD

Slide 39

Slide 39 text

Versioning Semantic Versioning https://semver.org Format: a.b.c, where: a — major version, increases with significant changes of product b — minor version, increases with current releases, backward compatibility may not be guaranteed c — build version, increases with bug hotfixes, minor features' improvements

Slide 40

Slide 40 text

Naming Element Rule Example application CamelCase AcraServer binary debian-style acra-connector package debian-style libthemis- dev0.10.0+stretch amd64.deb

Slide 41

Slide 41 text

Git flow and CI/CD Name Usage CI/CD automation stable Releases all tests, packaging, delivering master Current state all tests, packaging, delivering username/ GH999_issue_des c Dev, fixes manual

Slide 42

Slide 42 text

The earlier you implement the internal standards, the cheaper the support will be later!

Slide 43

Slide 43 text

Infrastructure Part Requirements core safety, stable developing speed up, easy cross- communications CI/CD full automation, tests, packaging, delivery

Slide 44

Slide 44 text

CI/CD

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

CircleCI — quick check of basic functionality

Slide 47

Slide 47 text

Buildbot — unit tests — multiple platforms — crossplatform tests — integration tests — pre-release checks — packages building

Slide 48

Slide 48 text

Buildbot architecture — core — CI/CD engine — scenarios

Slide 49

Slide 49 text

CI/CD rules — emulate real customer environment — use system libraries — strict configurations — scenarios according to docs — easily maintained and extended — developers can add scenarios

Slide 50

Slide 50 text

Exact methodology do not change rules on the fly

Slide 51

Slide 51 text

CI/CD flow

Slide 52

Slide 52 text

Docker

Slide 53

Slide 53 text

Docker image sizes — large image size is typical — most developers do not pay attention to the size — there are ways to significantly reduce the image — we reduced the size from ≈950MB to ≈15MB

Slide 54

Slide 54 text

What technologies did we use? — multi-stage builds — build from scratch You can find all dockerfiles at: https://github.com/cossacklabs/acra/tree/master/docker

Slide 55

Slide 55 text

Take care of the consumer!

Slide 56

Slide 56 text

Image tags Static tags — version tag — git commit tag Sliding tags — stable, latest — master, current

Slide 57

Slide 57 text

Image labels — label-schema standard — Cossack Labs internal standard

Slide 58

Slide 58 text

How do we use docker compose? As a demonstration stand!

Slide 59

Slide 59 text

What was uncomfortable? — generate and manually place keys — configure parameters inside docker-compose file manually — one very simple example of configuration — topology was unclear — security perimeter was not visible

Slide 60

Slide 60 text

What now? — multiple demonstation stands — all stands can be launched in one action — confiurations can be easily extended and secured * — self-documented — topology described

Slide 61

Slide 61 text

Summary — complex security products require careful and accurate handling — iplemented standards — built large testing platform — release cycle: from once a year to once a couple weeks — from burst developing to planned releases — increased flexibility of our team — from concept researching to implementing usable features

Slide 62

Slide 62 text

Our products became closer to consumers!

Slide 63

Slide 63 text

Links — Cossack Labs official site https://cossacklabs.com — GitHub https://github.com/cossacklabs — Docker Hub https://hub.docker.com/r/cossacklabs/

Slide 64

Slide 64 text

Dmytro Shapovalov shad@cossacklabs.com shadinua shad.in.ua shad.in.ua