RailsWorld_2023_-_Everything_We_Learned_the_Hard_Way_Implementing_ActiveRecord_Encryption.pdf

Slide 1

Slide 1 text

Everything We Learned the Hard Way Implementing ActiveRecord::Encryption Rails World 2023 - Kylie Stradley

Slide 2

Slide 2 text

Kylie Stradley Senior Product Security Engineer at GitHub Matt Langlois Senior Product Security Engineer at GitHub You may recognize me from “ActiveRecord::Encryption, Stop Hackers from Reading your Data”

Slide 3

Slide 3 text

What is ActiveRecord::Encryption? Automatic encryption of database records on save and decryption on access

Slide 4

Slide 4 text

We upgraded previously encrypted columns and plaintext columns to ActiveRecord::Encryption

Slide 5

Slide 5 text

Key generation bottleneck Easy to use API Before: bespoke internal strategy Maintaining divergent column encryption code

Slide 6

Slide 6 text

Easy to use API After: ActiveRecord:: Encryption Less bespoke code Keys derived at time of writing code Easily upgrade columns to encrypted Centralized key rotation

Slide 7

Slide 7 text

We wanted to show how straightforward and easy column encryption can be

Slide 8

Slide 8 text

But… deploying a new column encryption strategy wasn’t always straightforward and easy

Slide 9

Slide 9 text

This talk is for

Slide 10

Slide 10 text

This talk is for (this will be some of you) Converting from an existing column encryption strategy

Slide 11

Slide 11 text

This talk is for (this will be some of you) Those converting from an existing column encryption strategy (likely some more of you) Those working in distributed systems

Slide 12

Slide 12 text

This talk is for (this will be some of you) Those converting from an existing column encryption strategy (this should be all of you) Those who will rotate their encryption keys (likely some more of you) Those working in distributed systems

Slide 13

Slide 13 text

We learned a lot about deploying and maintaining resilient security software

Slide 14

Slide 14 text

We did it with no downtime

Slide 15

Slide 15 text

We learned the hard way

Slide 16

Slide 16 text

What We Thought We Knew About Encryption ● The hardest part of encryption is key management ● One byte in, one byte out ● The price of “it just works” is performance

Slide 17

Slide 17 text

What we thought we knew “Key Management is the hardest part of encryption”

Slide 18

Slide 18 text

What we thought “Key management is the hardest part of encryption” meant ● FIPS Compliance

Slide 19

Slide 19 text

What we thought “Key management is the hardest part of encryption” meant ● FIPS Compliance - meeting with AES GCM 256 encryption

Slide 20

Slide 20 text

What we thought “Key management is the hardest part of encryption” meant ● FIPS Compliance - meeting with AES GCM 256 encryption ● Nonce Reuse

Slide 21

Slide 21 text

What we thought “Key management is the hardest part of encryption” meant ● FIPS Compliance - meeting with AES GCM 256 encryption ● Nonce Reuse - add current year as anti nonce exhaustion data, effectively deriving new key each year per column

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

What we learned

Slide 25

Slide 25 text

Key Deployment is actually an extremely difficult part of encryption

Slide 26

Slide 26 text

Let’s say you have one Rails server Rails Server Database Encrypts record with key Encrypted Record

Slide 27

Slide 27 text

Let’s say you have one Rails server…and you add a new key Key Update Mechanism Rails Server Database Append new key Encrypts record with new key Encrypted Record Encrypted Record

Slide 28

Slide 28 text

Let’s say you have one Rails server…and you add a new key Key Update Mechanism Rails Server Database Append new key Encrypted Record Encrypted Record Decrypts with latest key

Slide 29

Slide 29 text

Let’s say you have one Rails server…and you add a new key Key Update Mechanism Rails Server Database Append new key Encrypted Record Encrypted Record Decrypts with next key

Slide 30

Slide 30 text

Let’s say your app is a bit bigger than just one Rails server… ● A fleet of app servers ● Serving 1000s of requests per second ● 400,000 encryptions per day ● 75,000,000 decryptions per day

Slide 31

Slide 31 text

Let’s say your app is a bit bigger than just one Rails server… Rails Server Database Encrypted Record Rails Server Encrypted Record Encrypted Record Encrypts record with key Encrypts record with key

Slide 32

Slide 32 text

Let’s say your app is a bit bigger than just one Rails server… How will you update the key for all of your app servers? Rails Server Database Encrypted Record Rails Server Encrypted Record Encrypted Record Encrypts record with new key Encrypts record with key Key Update Mechanism Encrypted Record Encrypted Record Append new key

Slide 33

Slide 33 text

Let’s say your app is a bit bigger than just one Rails server… How will you update the key for all of your app servers? Rails Server Database Encrypted Record Rails Server Encrypted Record Encrypted Record Encrypts record with new key Key Update Mechanism Encrypted Record Encrypted Record Append new key Decrypts with latest key

Slide 34

Slide 34 text

Just appending a new key didn’t work in our distributed system

Slide 35

Slide 35 text

What we learned “Key management is the hardest part of encryption” means ● In a distributed system, to rotate keys, we also had to distribute keys

Slide 36

Slide 36 text

What we learned “Key management is the hardest part of encryption” means ● In a distributed system, to rotate keys, we also had to distribute keys ● We needed a solution that ensured encryption would happen with new keys only once they were propagated to ALL servers

Slide 37

Slide 37 text

What we learned -A two key strategy allows us to deliver seamless decryption and encryption during key rotation Rails Server Database Encrypted Record Rails Server Encrypted Record Encrypted Record Key Update Mechanism Encrypted Record Append new decryption key Decrypts with latest key

Slide 38

Slide 38 text

Slide 39

Slide 39 text

Slide 40

Slide 40 text

Splitting our key into two lets us avoid plaintext mode while re-encrytping records

Slide 41

Slide 41 text

What you can learn from our experience

Slide 42

Slide 42 text

What you can learn from our experience - Key Management ● We learned this during a planned test of our key rotation strategy

Slide 43

Slide 43 text

What you can learn from our experience - Key Management ● We learned this during a planned test of our key rotation strategy - a better strategy is knowledge of your system

Slide 44

Slide 44 text

What you can learn from our experience - Key Management ● We learned this during a planned test of our key rotation strategy - a better strategy is knowledge of your system ● Understand your capabilities for updating keys for your production servers

Slide 45

Slide 45 text

Understand how you will update keys for production servers Can you update all keys at once? How can you roll out keys to prevent decryption failure?

Slide 46

Slide 46 text

Slide 47

Slide 47 text

Understand how you will update keys for production servers Do you update keys via push or pull? What triggers a push? If you pull, do you poll for updates? How often? Can you update all keys at once? How can you roll out keys to prevent decryption failure? How and when is data migrated? Is your database sharded? Will data ever move between shards? How will you re-encrypt records for key rotation?

Slide 48

Slide 48 text

Slide 49

Slide 49 text

What we thought we knew “One byte in, one byte out. All of our ciphertexts are the same size”

Slide 50

Slide 50 text

What we thought “All of our ciphertexts are the same size” meant ● AES GCM 256 works like a stream cipher

Slide 51

Slide 51 text

What we thought “All of our ciphertexts are the same size” meant ● AES GCM 256 is a block cipher - ciphertexts will be 128 bits

Slide 52

Slide 52 text

What we thought “All of our ciphertexts are the same size” meant ● AES GCM 256 is a block cipher - ciphertexts will be 128 bits ● Plaintext columns may need to be resized

Slide 53

Slide 53 text

What we thought “All of our ciphertexts are the same size” meant ● AES GCM 256 is a block cipher - ciphertexts will be 128 bits ● Plaintext columns may need to be resized - just need to be able to hold 128 bits

Slide 54

Slide 54 text

What we thought “All of our ciphertexts are the same size” meant ● AES GCM 256 is a block cipher - ciphertexts will always be 128 bits ● Plaintext columns may need to be resized - just need to be able to hold 128 bits ● Our previous encryption scheme stored some metadata

Slide 55

Slide 55 text

What we thought “All of our ciphertexts are the same size” meant ● AES GCM 256 is a block cipher - ciphertexts will always be 256 bits ● Plaintext columns may need to be resized - just need to be able to hold 256 bits ● Our previous encryption scheme stored some metadata - but not quite the same amount Rails does

Slide 56

Slide 56 text

What we learned

Slide 57

Slide 57 text

Ciphertext != Encrypted Record

Slide 58

Slide 58 text

What we learned “All of our ciphertexts are the same size” means ● All of our ciphertext are the same size, however ciphertext is not all that is stored in the database

Slide 59

Slide 59 text

Slide 60

Slide 60 text

Database Record Comparison Previous Strategy "\x02\e\x9FU#\xD3\xA8p\x84l\b\xED\x96\xEA3z,\xDFZ\xC 9-\"\\\xC1\xC7\x86\xA1\xCC\x8A\x8D]\xDB\xC2\x99\xE7`S )#\xCCp\xFD\x81\x9E\x01|$" ActiveRecord::Encryption { "p"⇒"nf6EwvS6rsSEErZyjTIRhgy7Gdkj4Wj4iQOtIRphP2p4xeozIL2xdDcO3sA=", "h"⇒ { "iv"⇒"TFXpap3Wpf4OEvtZ", "at"⇒"mbg4Au2ECg2iumLb1WhTsw==", "e"⇒"QVNDSUktOEJJVA==", "i"⇒"MTEwYQ==", "anti_nonce_exhaustion_data"⇒"MjAyMw==" } } Key ID Encrypted message Encrypted message (payload) Message headers

Slide 61

Slide 61 text

What we learned “All of our ciphertexts are the same size” means ● All of our ciphertext are the same size, however ciphertext is not all that is stored in the database ● “Anti-nonce-exhaustion-data” is quite a bit of text ● Recommendation for all columns to be converted to mysql TEXT before conversion to ActiveRecord::Encryption – we do not allow indexing on encrypted columns

Slide 62

Slide 62 text

What you can learn from our experience

Slide 63

Slide 63 text

What you can learn from our experience - Ciphertext vs Encrypted Record size ● Understand the amount of bits of any metadata/headers you are storing alongside ciphertext

Slide 64

Slide 64 text

Slide 65

Slide 65 text

Database Record ActiveRecord::Encryption { "p"⇒"nf6EwvS6rsSEErZyjTIRhgy7Gdkj4Wj4iQOtIRphP2p4xeozIL2xdDcO3sA=", "h"⇒ { "iv"⇒"TFXpap3Wpf4OEvtZ", "at"⇒"mbg4Au2ECg2iumLb1WhTsw==", "e"⇒"QVNDSUktOEJJVA==", "i"⇒"MTEwYQ==", "anti_nonce_exhaustion_data"⇒"MjAyMw==" } } There is a reason the rest of these tags are single/double characters

Slide 66

Slide 66 text

What you can learn from our experience - Ciphertext vs Encrypted Record size ● Understand the amount of bits of any metadata/headers you are storing alongside ciphertext - there is a reason tags are commonly one character ● Make it easy for your engineers

Slide 67

Slide 67 text

What you can learn from our experience - Ciphertext vs Encrypted Record size ● Understand the amount of bits of any metadata/headers you are storing alongside ciphertext - there is a reason “i” is commonly used as the tag name for anti-nonce exhaustion data! ● Make it easy for your engineers - consider longevity when recommending column length

Slide 68

Slide 68 text

What we thought we knew “The price of ‘it just works!’ is performance”

Slide 69

Slide 69 text

What we thought “it just works!” meant ● When something “just works!” you pay a price, we assumed this would be performance

Slide 70

Slide 70 text

Slide 71

Slide 71 text

What we thought “it just works!” meant ● When something “just works!” you pay a price, we assumed this would be performance – moving from one encryption scheme to another so considered negligible ● Through monitoring we thought we found out “it just works” was a function of idempotency

Slide 72

Slide 72 text

What we learned

Slide 73

Slide 73 text

Some data is just extra special https://github.blog/changelog/2022-08-18-false-alert-flags-will-appear-in-users-security-log-due-to-a-bug-in-2fa-recovery-events/

Slide 74

Slide 74 text

Our upgrade strategy relied on a type to feature flag Plaintext Encrypt? Model.save Database Ciphertext Type: Text Type: Encrypted Attribute

Slide 75

Slide 75 text

Our upgrade strategy relied on a type to feature flag … but we neglected to delegate changed_in_place, resulting in records that looked like they had changed Model.save Database Ciphertext Type: Encrypted Attribute changed_in_place? ciphertext plaintext != true

Slide 76

Slide 76 text

We fixed this by delegating the changed_in_place? method to the EncryptedAttributeType Model.save Database Ciphertext Type: Encrypted Attribute changed_in_place? decrypted ciphertext plaintext != false

Slide 77

Slide 77 text

.encrypt was being called twice...

Slide 78

Slide 78 text

…but luckily .encrypt is idempotent

Slide 79

Slide 79 text

…and the bug has been fixed

Slide 80

Slide 80 text

What we learned “It just works” means ● It did just work for most cases

Slide 81

Slide 81 text

What we learned “It just works” means ● It did just work for most cases - but we had a special case ● Monitoring can help catch special cases

Slide 82

Slide 82 text

What we learned “It just works” means ● It did just work for most cases - but we had a special case ● Monitoring can help catch special cases - in production

Slide 83

Slide 83 text

What we learned “It just works” means ● It did just work for most cases - but we had a special case ● Monitoring can help catch special cases - in production ● Some data is just special and you need to take extra time and care to get it right

Slide 84

Slide 84 text

What you can learn from our experience

Slide 85

Slide 85 text

What you can learn from our experience - “It just works!” ● Encrypted data is being encrypted for a reason - there may be special monitors or side effects associated with these records

Slide 86

Slide 86 text

Despite all of this

Slide 87

Slide 87 text

We delivered a seamless column encryption strategy ● We were no longer maintaining divergent column encryption code ● New, easy process to upgrade plaintext columns to encrypted ● Encryption keys are now derived at time of writing code instead of generated – greatly increasing adoption ● Maintained our SLOs

Slide 88

Slide 88 text

And keeping a few things in mind, you can too ● Build with Key Rotation and Key Management in mind first ● Understand why a column should be encrypted and what side effects there are on those records

Slide 89

Slide 89 text

Deploying seamless column encryption was not seamless but it’s doable and worthwhile

Slide 90

Slide 90 text

Further Reading ● https://gh.io/ar-encryption-blog-post ● https://gh.io/ar-encryption-blog-post-2 ● https://gh.io/ar-encryption-guide ● Rails Conf Talk - “ActiveRecord::Encryption, Stop Hackers from Reading your Data” ● The “Real World Cryptography” book by David Wong ● Google’s “Building Secure and Reliable Systems” book Rails World 2023 - Kylie Stradley

Slide 91

Slide 91 text

Rails World 2023 - Kylie Stradley