Crucible

Transcript

1. Crucible Towards a Safer Bosh Job Interface Christopher Brown James Myers March 24th, 2017 Pivotal
2. Overview Motivation Implementation Results 1
3. Overview Motivation Implementation Results 1
4. Overview Motivation Implementation Results 1
5. Motivation
6. Infinite Power The Bosh job interface lets you run a custom script as root. This is extremely powerful and makes Bosh viable for most systems. For most of the things we run we don’t need this power and end up reimplementing ways to be rid of it in each job. 2
7. bash is complicated find foo | while read A; do touch $A; done 3
8. bash is complicated find foo | while read A; do touch $A; done What if the file name has a backslash at the start? 3
9. bash is complicated find foo | while read -r A; do touch $A; done What if the file name has a backslash at the start? 3
10. bash is complicated find foo | while read -r A; do touch $A; done What if the file name has a space in it? 3
11. bash is complicated find foo | while read -r A; do touch "$A"; done What if the file name has a space in it? 3
12. bash is complicated find foo | while read -r A; do touch "$A"; done What if the file name starts with a space? 3
13. bash is complicated find foo | while IFS="" read -r A; do touch "$A"; done What if the file name starts with a space? 3
14. bash is complicated find foo | while IFS="" read -r A; do touch "$A"; done 3
15. bash is complicated find foo | while IFS="" read -r A; do touch "$A"; done 3
16. bash is complicated find foo | while IFS="" read -r A; do touch "$A"; done What if the file name contains a new line? 3
17. bash is complicated find foo -print0 | while IFS="" read -rd A; do touch "$A"; done What if the file name contains a new line? 3
18. bash is complicated find foo -print0 | while IFS="" read -rd A; do touch "$A"; done 3
19. bash is complicated find foo -print0 | while IFS="" read -rd A; do touch "$A"; done 3
20. bash is complicated find foo -print0 | while IFS="" read -rd A; do touch "$A"; done What if the file name starts with a hyphen? 3
21. bash is complicated find foo -print0 | while IFS="" read -rd A; do touch -- "$A"; done What if the file name starts with a hyphen? 3
22. bash is complicated find foo -print0 | while IFS="" read -rd A; do touch -- "$A"; done 3
23. bash is complicated find foo -print0 | while IFS="" read -rd A; do touch -- "$A"; done *doesn’t work on OS X 3
24. bash is too powerful awk -W interactive '{lineWithDate="echo [`date +%̈Y-%m-%d %H:%M:%S%z`̈] "̈ $0 ""̈; system(lineWithDate) }' 4
25. bash is unforgiving 5
26. bash is unforgiving II rm -rf "$STEAMROOT/"* 6
27. bash is unforgiving II rm -rf "/"* 6
28. Poor Job Isolation Security 7
29. Poor Job Isolation Security • Jobs run as root by default 7
30. Poor Job Isolation Security • Jobs run as root by default • Jobs can read the configuration and credentials from other jobs 7
31. Poor Job Isolation II Performance 8
32. Poor Job Isolation II Performance • Jobs suffer from noisy neighbors 8
33. Poor Job Isolation II Performance • Jobs suffer from noisy neighbors • Generic performance fixes must be copied and pasted across releases 8
34. “Enablement” vcap −→ mysql What if we wanted to make it so that each job ran as its own user? 9
35. Lack of Consistency Logging 10
36. Lack of Consistency Signalling 10
37. Lack of Consistency Operator Experience 10
38. Implementation
39. runC An open standard for containers using cgroups. Closely parallels Bosh job lifecycle. • runc exec • runc kill • runc stop -f Complex configuration 11
40. Configuration I check process bbs with pidfile /var/vcap/sys/run/crucible/bbs.pid start program /var/vcap/packages/crucible/bin/crucible start bbs stop program /var/vcap/packages/crucible/bin/crucible stop bbs group vcap 12
41. Configuration II --- run: path: /var/vcap/packages/bbs/bin/bbs args: [--config, /var/vcap/jobs/bbs/config/bbs.json] env: - GODEBUG=netdns=cgo lifecycle: show_stacks_on_stall: true limits: open_files: 100_000 sysctl: net.ipv4.tcp_fin_timeout: 10 net.ipv4.tcp_tw_reuse: 1 directories: - /var/vcap/data/bbs 13
42. Results
43. Releases Converted • Diego 39 files changed, 179 insertions+, 667 deletions− 14
44. Releases Converted • Diego 39 files changed, 179 insertions+, 667 deletions− • UAA 6 files changed, 41 insertions+, 217 deletions− 14
45. Releases Converted • Diego 39 files changed, 179 insertions+, 667 deletions− • UAA 6 files changed, 41 insertions+, 217 deletions− • CAPI (Cloud Controller job) 5 files changed, 50 insertions+, 130 deletions− 14
46. Releases Converted • Diego 39 files changed, 179 insertions+, 667 deletions− • UAA 6 files changed, 41 insertions+, 217 deletions− • CAPI (Cloud Controller job) 5 files changed, 50 insertions+, 130 deletions− • Routing 8 files changed, 34 insertions+, 118 deletions− 14
47. Releases Converted • Diego 39 files changed, 179 insertions+, 667 deletions− • UAA 6 files changed, 41 insertions+, 217 deletions− • CAPI (Cloud Controller job) 5 files changed, 50 insertions+, 130 deletions− • Routing 8 files changed, 34 insertions+, 118 deletions− We found diminishing returns after this. Many other releases were assessed and didn’t show any new challenges. 14
48. Releases Not Converted • Garden For Garden to have any benefit from being in a container it needs to be runnable as non-root. This is being worked on but is still not stable. 15
49. Releases Not Converted • Garden For Garden to have any benefit from being in a container it needs to be runnable as non-root. This is being worked on but is still not stable. • Container Networking This needs to run as root. There are still potentially lifecycle and simplicity improvements by running in a container but no security benefit. 15
50. Security • Jobs run in their own mount namespace. This means that they cannot read the configuration or credentials of other jobs. 16
51. Security • Jobs run in their own mount namespace. This means that they cannot read the configuration or credentials of other jobs. • Jobs automatically run as the vcap user. 16
52. Security • Jobs run in their own mount namespace. This means that they cannot read the configuration or credentials of other jobs. • Jobs automatically run as the vcap user. • The job is restricted to only writing to its persistent and ephemeral storage directories. Everything else is mounted as read-only. 16
53. Performance • Limits can be placed on the memory and CPU that a job receives. 17
54. Performance • Limits can be placed on the memory and CPU that a job receives. • Setting which can safely be automatically applied in order to improve performance can automatically be applied.[Damato, 2017] 17
55. Rough Edges Mount Interface We didn’t end up with a good abstraction that wasn’t just the mount(8) command in YAML form. 18
56. Rough Edges Mount Interface We didn’t end up with a good abstraction that wasn’t just the mount(8) command in YAML form. Packaging Format How do we distribute this? Required collocated release? Modify the Bosh team’s stemcell? 18
57. Rough Edges Mount Interface We didn’t end up with a good abstraction that wasn’t just the mount(8) command in YAML form. Packaging Format How do we distribute this? Required collocated release? Modify the Bosh team’s stemcell? Lifecycle Hooks We found that some releases needed bash in some form to start e.g. keytool. We can’t predict this. A small auditable hook will probably need to stay. 18
58. Areas for More Exploration User Creation We could easily create a new user for each job and then delete it when the job it stopped. Brings other complexities around re-mapping file permissions and shared files.  19
59. Areas for More Exploration II Logical Path Remapping Re-mapping internal paths to decouple from Bosh system paths. 20
60. Areas for More Exploration II Logical Path Remapping Re-mapping internal paths to decouple from Bosh system paths. System Path Container Path /var/vcap/jobs/bbs/config /config 20
61. Areas for More Exploration II Logical Path Remapping Re-mapping internal paths to decouple from Bosh system paths. System Path Container Path /var/vcap/jobs/bbs/config /config /var/vcap/store/bbs /store 20
62. Areas for More Exploration II Logical Path Remapping Re-mapping internal paths to decouple from Bosh system paths. System Path Container Path /var/vcap/jobs/bbs/config /config /var/vcap/store/bbs /store CredHub FUSE Filesystem /credentials 20
63. Areas for More Exploration II Logical Path Remapping Re-mapping internal paths to decouple from Bosh system paths. System Path Container Path /var/vcap/jobs/bbs/config /config /var/vcap/store/bbs /store CredHub FUSE Filesystem /credentials This gets rid of vcap and lets jobs be collocated and uncollocated after they’ve been deployed. 20
64. Questions? James Myers    jfmyers9  jmyers@pivotal.io Christopher Brown    xoebus  cbrown@pivotal.io  https://github.com/xoebus/talks 20
65. References I Damato, J. (2017). How setting the tz environment variable avoids thousands of system calls. https://blog.packagecloud.io/eng/2017/02/21/set- environment-variable-save-thousands-of-system-calls/. Paul, I. (2015). Scary steam for linux bug erases all the personal files on your pc. http://www.pcworld.com/article/2871653/scary-steam-for- linux-bug-erases-all-the-personal-files-on-your-pc.html. 21
66. References II Safin, A. (2017). The collapse of the unix philosophy. https://kukuruku.co/post/the-collapse-of-the-unix- philosophy/. 22