RailsWorld_2023_-_Everything_We_Learned_the_Hard_Way_Implementing_ActiveRecord_Encryption.pdf

GitHub encrypts your data at rest, as well as specific sensitive database columns. What you may not know is we recently replaced our in-house db column encryption strategy with ActiveRecord::Encryption, in place. While we were able to complete this transition seamlessly for GitHub’s developers, the process was not quite seamless for our team and some of our customers.

Learn why despite making all these mistakes we feel the migration was worth it to our team, GitHub developers and most importantly, GitHub customers.

https://rubyonrails.org/world/agenda/day-1/4b-kylie-stradley-session

Transcript

1. Everything We Learned the Hard Way Implementing ActiveRecord::Encryption Rails World 2023 - Kylie Stradley
2. Kylie Stradley Senior Product Security Engineer at GitHub Matt Langlois Senior Product Security Engineer at GitHub You may recognize me from “ActiveRecord::Encryption, Stop Hackers from Reading your Data”
3. What is ActiveRecord::Encryption? Automatic encryption of database records on save and decryption on access
4. We upgraded previously encrypted columns and plaintext columns to ActiveRecord::Encryption
5. Key generation bottleneck Easy to use API Before: bespoke internal strategy Maintaining divergent column encryption code
6. Easy to use API After: ActiveRecord:: Encryption Less bespoke code Keys derived at time of writing code Easily upgrade columns to encrypted Centralized key rotation
7. We wanted to show how straightforward and easy column encryption can be
8. But… deploying a new column encryption strategy wasn’t always straightforward and easy
9. This talk is for
10. This talk is for (this will be some of you) Converting from an existing column encryption strategy
11. This talk is for (this will be some of you) Those converting from an existing column encryption strategy (likely some more of you) Those working in distributed systems
12. This talk is for (this will be some of you) Those converting from an existing column encryption strategy (this should be all of you) Those who will rotate their encryption keys (likely some more of you) Those working in distributed systems
13. We learned a lot about deploying and maintaining resilient security software
14. We did it with no downtime
15. We learned the hard way
16. What We Thought We Knew About Encryption • The hardest part of encryption is key management • One byte in, one byte out • The price of “it just works” is performance
17. What we thought we knew “Key Management is the hardest part of encryption”
18. What we thought “Key management is the hardest part of encryption” meant • FIPS Compliance
19. What we thought “Key management is the hardest part of encryption” meant • FIPS Compliance - meeting with AES GCM 256 encryption
20. What we thought “Key management is the hardest part of encryption” meant • FIPS Compliance - meeting with AES GCM 256 encryption • Nonce Reuse
21. What we thought “Key management is the hardest part of encryption” meant • FIPS Compliance - meeting with AES GCM 256 encryption • Nonce Reuse - add current year as anti nonce exhaustion data, effectively deriving new key each year per column
22. What we thought “Key management is the hardest part of encryption” meant • FIPS Compliance - meeting with AES GCM 256 encryption • Nonce Reuse - add current year as anti nonce exhaustion data, effectively deriving new key each year per column • Secure Key Storage
23. What we thought “Key management is the hardest part of encryption” meant • FIPS Compliance - meeting with AES GCM 256 encryption • Nonce Reuse - add current year as anti nonce exhaustion data, effectively deriving new key each year per column • Secure Key Storage - we use Hashicorp Vault
24. What we learned
25. Key Deployment is actually an extremely difficult part of encryption
26. Let’s say you have one Rails server Rails Server Database Encrypts record with key Encrypted Record
27. Let’s say you have one Rails server…and you add a new key Key Update Mechanism Rails Server Database Append new key Encrypts record with new key Encrypted Record Encrypted Record
28. Let’s say you have one Rails server…and you add a new key Key Update Mechanism Rails Server Database Append new key Encrypted Record Encrypted Record Decrypts with latest key
29. Let’s say you have one Rails server…and you add a new key Key Update Mechanism Rails Server Database Append new key Encrypted Record Encrypted Record Decrypts with next key
30. Let’s say your app is a bit bigger than just one Rails server… • A fleet of app servers • Serving 1000s of requests per second • 400,000 encryptions per day • 75,000,000 decryptions per day
31. Let’s say your app is a bit bigger than just one Rails server… Rails Server Database Encrypted Record Rails Server Encrypted Record Encrypted Record Encrypts record with key Encrypts record with key
32. Let’s say your app is a bit bigger than just one Rails server… How will you update the key for all of your app servers? Rails Server Database Encrypted Record Rails Server Encrypted Record Encrypted Record Encrypts record with new key Encrypts record with key Key Update Mechanism Encrypted Record Encrypted Record Append new key
33. Let’s say your app is a bit bigger than just one Rails server… How will you update the key for all of your app servers? Rails Server Database Encrypted Record Rails Server Encrypted Record Encrypted Record Encrypts record with new key Key Update Mechanism Encrypted Record Encrypted Record Append new key Decrypts with latest key
34. Just appending a new key didn’t work in our distributed system
35. What we learned “Key management is the hardest part of encryption” means • In a distributed system, to rotate keys, we also had to distribute keys
36. What we learned “Key management is the hardest part of encryption” means • In a distributed system, to rotate keys, we also had to distribute keys • We needed a solution that ensured encryption would happen with new keys only once they were propagated to ALL servers
37. What we learned -A two key strategy allows us to deliver seamless decryption and encryption during key rotation Rails Server Database Encrypted Record Rails Server Encrypted Record Encrypted Record Key Update Mechanism Encrypted Record Append new decryption key Decrypts with latest key
38. What we learned -A two key strategy allows us to deliver seamless decryption and encryption during key rotation Rails Server Database Encrypted Record Rails Server Encrypted Record Encrypted Record Key Update Mechanism Encrypted Record Append new decryption key Decrypts with next key
39. What we learned -A two key strategy allows us to deliver seamless decryption and encryption during key rotation Rails Server Database Encrypted Record Rails Server Encrypted Record Encrypted Record Key Update Mechanism Encrypted Record Decrypts with latest key Latest decryption key propagated to all servers Update encryption key to use new value Encrypts record with new key Encrypted Record
40. Splitting our key into two lets us avoid plaintext mode while re-encrytping records
41. What you can learn from our experience
42. What you can learn from our experience - Key Management • We learned this during a planned test of our key rotation strategy
43. What you can learn from our experience - Key Management • We learned this during a planned test of our key rotation strategy - a better strategy is knowledge of your system
44. What you can learn from our experience - Key Management • We learned this during a planned test of our key rotation strategy - a better strategy is knowledge of your system • Understand your capabilities for updating keys for your production servers
45. Understand how you will update keys for production servers Can you update all keys at once? How can you roll out keys to prevent decryption failure?
46. Understand how you will update keys for production servers Do you update keys via push or pull? What triggers a push? If you pull, do you poll for updates? How often? Can you update all keys at once? How can you roll out keys to prevent decryption failure?
47. Understand how you will update keys for production servers Do you update keys via push or pull? What triggers a push? If you pull, do you poll for updates? How often? Can you update all keys at once? How can you roll out keys to prevent decryption failure? How and when is data migrated? Is your database sharded? Will data ever move between shards? How will you re-encrypt records for key rotation?
48. What you can learn from our experience - Key Management • We learned this during a planned test of our key rotation strategy - a better strategy is knowledge of your system • Understand your capabilities for updating keys for your production servers - This lets you design rollout with key rotation as top priority
49. What we thought we knew “One byte in, one byte out. All of our ciphertexts are the same size”
50. What we thought “All of our ciphertexts are the same size” meant • AES GCM 256 works like a stream cipher
51. What we thought “All of our ciphertexts are the same size” meant • AES GCM 256 is a block cipher - ciphertexts will be 128 bits
52. What we thought “All of our ciphertexts are the same size” meant • AES GCM 256 is a block cipher - ciphertexts will be 128 bits • Plaintext columns may need to be resized
53. What we thought “All of our ciphertexts are the same size” meant • AES GCM 256 is a block cipher - ciphertexts will be 128 bits • Plaintext columns may need to be resized - just need to be able to hold 128 bits
54. What we thought “All of our ciphertexts are the same size” meant • AES GCM 256 is a block cipher - ciphertexts will always be 128 bits • Plaintext columns may need to be resized - just need to be able to hold 128 bits • Our previous encryption scheme stored some metadata
55. What we thought “All of our ciphertexts are the same size” meant • AES GCM 256 is a block cipher - ciphertexts will always be 256 bits • Plaintext columns may need to be resized - just need to be able to hold 256 bits • Our previous encryption scheme stored some metadata - but not quite the same amount Rails does
56. What we learned
57. Ciphertext != Encrypted Record
58. What we learned “All of our ciphertexts are the same size” means • All of our ciphertext are the same size, however ciphertext is not all that is stored in the database
59. What we learned “All of our ciphertexts are the same size” means • All of our ciphertext are the same size, however ciphertext is not all that is stored in the database • “Anti-nonce-exhaustion-data” is quite a bit of text
60. Database Record Comparison Previous Strategy "\x02\e\x9FU#\xD3\xA8p\x84l\b\xED\x96\xEA3z,\xDFZ\xC 9-\"\\\xC1\xC7\x86\xA1\xCC\x8A\x8D]\xDB\xC2\x99\xE7`S )#\xCCp\xFD\x81\x9E\x01|$" ActiveRecord::Encryption { "p"⇒"nf6EwvS6rsSEErZyjTIRhgy7Gdkj4Wj4iQOtIRphP2p4xeozIL2xdDcO3sA=", "h"⇒ { "iv"⇒"TFXpap3Wpf4OEvtZ", "at"⇒"mbg4Au2ECg2iumLb1WhTsw==", "e"⇒"QVNDSUktOEJJVA==", "i"⇒"MTEwYQ==", "anti_nonce_exhaustion_data"⇒"MjAyMw==" } } Key ID Encrypted message Encrypted message (payload) Message headers
61. What we learned “All of our ciphertexts are the same size” means • All of our ciphertext are the same size, however ciphertext is not all that is stored in the database • “Anti-nonce-exhaustion-data” is quite a bit of text • Recommendation for all columns to be converted to mysql TEXT before conversion to ActiveRecord::Encryption – we do not allow indexing on encrypted columns
62. What you can learn from our experience
63. What you can learn from our experience - Ciphertext vs Encrypted Record size • Understand the amount of bits of any metadata/headers you are storing alongside ciphertext
64. What you can learn from our experience - Ciphertext vs Encrypted Record size • Understand the amount of bits of any metadata/headers you are storing alongside ciphertext - there is a reason tags are commonly one character
65. Database Record ActiveRecord::Encryption { "p"⇒"nf6EwvS6rsSEErZyjTIRhgy7Gdkj4Wj4iQOtIRphP2p4xeozIL2xdDcO3sA=", "h"⇒ { "iv"⇒"TFXpap3Wpf4OEvtZ", "at"⇒"mbg4Au2ECg2iumLb1WhTsw==", "e"⇒"QVNDSUktOEJJVA==", "i"⇒"MTEwYQ==", "anti_nonce_exhaustion_data"⇒"MjAyMw==" } } There is a reason the rest of these tags are single/double characters
66. What you can learn from our experience - Ciphertext vs Encrypted Record size • Understand the amount of bits of any metadata/headers you are storing alongside ciphertext - there is a reason tags are commonly one character • Make it easy for your engineers
67. What you can learn from our experience - Ciphertext vs Encrypted Record size • Understand the amount of bits of any metadata/headers you are storing alongside ciphertext - there is a reason “i” is commonly used as the tag name for anti-nonce exhaustion data! • Make it easy for your engineers - consider longevity when recommending column length
68. What we thought we knew “The price of ‘it just works!’ is performance”
69. What we thought “it just works!” meant • When something “just works!” you pay a price, we assumed this would be performance
70. What we thought “it just works!” meant • When something “just works!” you pay a price, we assumed this would be performance – moving from one encryption scheme to another so considered negligible
71. What we thought “it just works!” meant • When something “just works!” you pay a price, we assumed this would be performance – moving from one encryption scheme to another so considered negligible • Through monitoring we thought we found out “it just works” was a function of idempotency
72. What we learned
73. Some data is just extra special https://github.blog/changelog/2022-08-18-false-alert-flags-will-appear-in-users-security-log-due-to-a-bug-in-2fa-recovery-events/
74. Our upgrade strategy relied on a type to feature flag Plaintext Encrypt? Model.save Database Ciphertext Type: Text Type: Encrypted Attribute
75. Our upgrade strategy relied on a type to feature flag … but we neglected to delegate changed_in_place, resulting in records that looked like they had changed Model.save Database Ciphertext Type: Encrypted Attribute changed_in_place? ciphertext plaintext != true
76. We fixed this by delegating the changed_in_place? method to the EncryptedAttributeType Model.save Database Ciphertext Type: Encrypted Attribute changed_in_place? decrypted ciphertext plaintext != false
77. .encrypt was being called twice...
78. …but luckily .encrypt is idempotent
79. …and the bug has been fixed
80. What we learned “It just works” means • It did just work for most cases
81. What we learned “It just works” means • It did just work for most cases - but we had a special case • Monitoring can help catch special cases
82. What we learned “It just works” means • It did just work for most cases - but we had a special case • Monitoring can help catch special cases - in production
83. What we learned “It just works” means • It did just work for most cases - but we had a special case • Monitoring can help catch special cases - in production • Some data is just special and you need to take extra time and care to get it right
84. What you can learn from our experience
85. What you can learn from our experience - “It just works!” • Encrypted data is being encrypted for a reason - there may be special monitors or side effects associated with these records
86. Despite all of this
87. We delivered a seamless column encryption strategy • We were no longer maintaining divergent column encryption code • New, easy process to upgrade plaintext columns to encrypted • Encryption keys are now derived at time of writing code instead of generated – greatly increasing adoption • Maintained our SLOs
88. And keeping a few things in mind, you can too • Build with Key Rotation and Key Management in mind first • Understand why a column should be encrypted and what side effects there are on those records
89. Deploying seamless column encryption was not seamless but it’s doable and worthwhile
90. Further Reading • https://gh.io/ar-encryption-blog-post • https://gh.io/ar-encryption-blog-post-2 • https://gh.io/ar-encryption-guide • Rails Conf Talk - “ActiveRecord::Encryption, Stop Hackers from Reading your Data” • The “Real World Cryptography” book by David Wong • Google’s “Building Secure and Reliable Systems” book Rails World 2023 - Kylie Stradley
91. Rails World 2023 - Kylie Stradley