Free Containers

Codecademy provides millions of users with a development environment powered by Docker. In this deck I go over some of the issues we face allowing users to run arbitrary code in a Docker container.

Transcript

1. Free Containers
2. About Codecademy Learning Platform Free development environment for learning. Supports many languages and frameworks Java, Javascript, Ruby, Python, ... Sqlite, Angular, Ruby on Rails, Shell, Sass, Git, ...
3. Learning Environment
4. Overview of Infrastucture ECS ‑> Container Spawning Service ‑> Containers For Users Single container image for all courses (it's big). Users (ws) ‑> Proxy ‑> Containers in VPC
5. Container Spawning Service JSON { " S e r v i c e s " : [ { " N a m e " : " e i n " , " D e s i r e d C o u n t " : 3 0 0 , " M a x P e n d i n g " : 1 2 , " C o n f i g " : { " I m a g e " : " c o d e c a d e m y / e i n " } , " H o s t C o n f i g " : { " C P U S h a r e s " : 5 1 2 , " N e t w o r k M o d e " : " b r i d g e " , " M e m o r y " : 2 6 8 4 3 5 4 5 6 , " P o r t B i n d i n g s " : { " 4 0 0 0 / t c p " : [ { " H o s t P o r t " : " 0 " } ] } } } ] }
6. Yay! Free Containers For Everyone. Everyone can learn. Time to go home.
8. Let's add some IPTABLES i p t a b l e s - I F O R W A R D 4 - m s t a t e - - s t a t e N E W - j D R O P i p t a b l e s - I F O R W A R D 1 - p u d p - - d p o r t 5 3 - j A C C E P T i p t a b l e s - I D O C K E R - I S O L A T I O N - i d o c k e r 0 \ - p t c p - m t c p - - d p o r t 4 0 0 0 - j A C C E P T i p t a b l e s - I D O C K E R - I S O L A T I O N - i d o c k e r 0 \ - p t c p - m t c p - - d p o r t 4 4 3 - j A C C E P T i p t a b l e s - I D O C K E R - I S O L A T I O N - i d o c k e r 0 \ - p t c p - m t c p - - d p o r t 8 0 - j A C C E P T
9. Some time later.... Based on the data from our abuse team, an instance in your account is being used in a Denial of Service (DoS) attack. It's in one of your VPC's and sending out a lot of traffic on port 53.
10. Proposal: Docker should assist in bandwidth limiting containers
11. The notoriously difficult to learn t c
12. SHAPING When traffic is shaped, its rate of transmission is under control. Shaping may be more than lowering the available bandwidth ‑ it is also used to smooth out bursts in traffic for better network behaviour. Shaping occurs on egress. QDISCS qdisc is short for 'queueing discipline'. Whenever the kernel needs to send a packet to an interface, it is enqueued to the qdisc configured for that interface. Immediately afterwards, the kernel tries to get as many packets as possible from the qdisc, for giving them to the network adaptor driver. CLASSES Some qdiscs can contain classes.....
13. CLASSLES QDISCS s f q Stochastic Fairness Queueing reorders queued traffic so each 'session' gets to send a packet in turn. t b f The Token Bucket Filter is suited for slowing traffic down to a precisely configured rate. Scales well to large bandwidths. A few others, check the man page for tc.
14. Finding veth device for docker container veth.sh $ v e t h f 3 2 d 5 3 c 2 c 7 6 2 v e t h a e 9 a 4 2 f Creating a classless qdisc for veth device # S e t q l e n o f v e t h d e v i c e # o t h e r w i s e a d d i n g a q d i s c d r o p s e v e r y t h i n g i p l i n k s e t v e t h a e 9 a 4 2 f q l e n 1 0 0 0 t c q d i s c a d d d e v v e t h a e 9 a 4 2 f r o o t t b f \ r a t e 5 0 0 k b i t \ b u r s t 3 2 k b i t \ l a t e n c y 5 0 m s
15. Configuring t c at container creation time Listen to docker start system events listen.go Get network device Apply tc settings
16. Containerize Give container access to docker sock when starting container d o c k e r r u n - i t - v / v a r / r u n / d o c k e r . s o c k : / v a r / r u n / d o c k e r . s o c k . . . Let it run commands in host namespace with nsenter d o c k e r r u n - i t - v / p r o c : / m n t / p r o c / 1 / n s / m n t . . . n s e n t e r - - m o u n t = / m n t / p r o c / 1 / n s / m n t - - < c o m m a n d t o r u n i n h o s t > Allowing containers to run commands in host namespace is dangerous, be careful.
17. What do when JAVA runs out of memory and starts thrashing on the disk you create docker containers on.
18. Apply blkio limits with cgroups Find container block device from docker api i n f o , e r r : = c l i . C o n t a i n e r I n s p e c t ( c t x , c o n t a i n e r I D ) d e v i c e N a m e , o k : = i n f o . G r a p h D r i v e r . D a t a [ " D e v i c e N a m e " ] Find block device major:minor device pair for container d m s e t u p i n f o - - c o l u m n s - - n o h e a d i n g s - o m a j o r , m i n o r < d e v i c e N a m e > Set limit on containers access to block device with cgroup e c h o " < m a j o r : m i n o r > < r a t e > " > / s y s / f s / c g r o u p / b l k i o / d o c k e r / $ { c o n t a i n e r I D } / b l k i o . t h r o t t l e . r e a d _ b p s _ d e v i c e n s e n t e r - - m o u n t = / m n t / p r o c / 1 / n s / m n t - - c g s e t / d o c k e r / $ { c o n t a i n e r
19. Bonus Content: What to do when when your containers are being used to mine bitcoins, while your working on your talk about how great it is to let users run arbitrary code on your containers.
20. Reluctantly throttle all containers with cgroups Can also use docker api c g s e t / d o c k e r / $ { c o n t a i n e r I D } - r c p u . c f s _ p e r i o d _ u s = 1 0 0 0 0 0 c g s e t / d o c k e r / $ { c o n t a i n e r I D } - r c p u . c f s _ q u o t a _ u s = 5 0 0 0 0 Find and stop bitcoin mining container cryptono.sh Monitor container cpu usage with cgroup c a t / s y s / f s / c g r o u p / c p u / d o c k e r / $ { c o n t a i n e r I D } / c p u a c c t . u s a g e No code for this yet.....
21. If you're interested in writing the code, or helping to improve our docker infrastructure consider working with us. https://www.codecademy.com/about/jobs