Zekun Li - There and Back Again: Disable and re-enable garbage collector at Instagram

Slide 1

Slide 1 text

THERE AND BACK AGAIN: GARBAGE COLLECTION AT INSTAGRAM Zekun Li 李李泽昆 Instagram Infrastructure zekun@fb.com

Slide 2

Slide 2 text

AGENDA 1 Background 2 Why and How Instagram disable GC 3 Why and How Instagram re-enable GC, mostly

Slide 3

Slide 3 text

BACKGROUND

Slide 4

Slide 4 text

MEMORY MANAGEMENT IN PYTHON 4 Reference Count Garbage Collector for cyclic reference

Slide 5

Slide 5 text

MEMORY MANAGEMENT IN PYTHON 5 foo = [] bar = [] __main__ bar foo del foo del bar

Slide 6

Slide 6 text

MEMORY MANAGEMENT IN PYTHON 6 foo = [] bar = [] foo.append(bar) bar.append(foo) __main__ bar foo del foo del bar

Slide 7

Slide 7 text

Process 2 Memory Page Memory Page COPY-ON-WRITE SEMANTIC 7 Process 1 World Hello

Slide 8

Slide 8 text

INSTAGRAM TECH STACK 8

Slide 9

Slide 9 text

DISABLE GARBAGE COLLECTION

Slide 10

Slide 10 text

WHY DISABLE GC? 10

Slide 11

Slide 11 text

OBSERVATION 11 500 MB 350 MB 150 MB 150 MB 150 MB 150 MB 150 MB 150 MB

Slide 12

Slide 12 text

Slide 13

Slide 13 text

WHERE'S MY SHARED MEMORY? 13

Slide 14

Slide 14 text

PROFILING 14

Slide 15

Slide 15 text

WHY GARBAGE COLLECTION? 15 /* GC information is stored BEFORE the object structure. */ typedef union _gc_head { struct { union _gc_head *gc_next; union _gc_head *gc_prev; Py_ssize_t gc_refs; } gc; double dummy; /* force worst-case alignment */ } PyGC_Head;

Slide 16

Slide 16 text

Memory Page Memory Page 16 PyGC_HEAD PyObject

Slide 17

Slide 17 text

17 CAN WE DISABLE THE GC? YES, WE HAVE REFERENCE COUNT

Slide 18

Slide 18 text

HOW TO DISABLE GC? 18

Slide 19

Slide 19 text

DISABLE GC 19 gc.disable() atexit.register(os._exit, 0) gc.set_threshold(0) # fixed in python 3.6

Slide 20

Slide 20 text

RESULTS Memory util -15% Shared memory +100mb Instruction/cycle +10%

Slide 21

Slide 21 text

RE-ENABLE GARBAGE COLLECTION

Slide 22

Slide 22 text

WHY DISABLE GC IS BAD? 22

Slide 23

Slide 23 text

600M 1200M 0 3000 AFTER ONE YEAR 23

Slide 24

Slide 24 text

24 HOW TO MAKE GC COW FRIENDLY?

Slide 25

Slide 25 text

25 REDESIGN PYGC_HEAD

Slide 26

Slide 26 text

REDESIGN GC HEAD 26 /* GC information is stored BEFORE the object structure. */ typedef union _gc_head { struct { union _gc_head *gc_next; union _gc_head *gc_prev; Py_ssize_t gc_refs; } gc; double dummy; /* force worst-case alignment */ } PyGC_Head;

Slide 27

Slide 27 text

REDESIGN GC HEAD 27 /* GC information is stored BEFORE the object structure. */ typedef union _gc_head { struct { union _gc_head *head; } gc_ptr; double dummy; /* force worst-case alignment */ } PyGC_Head_Ptr;

Slide 28

Slide 28 text

Memory Page Memory Page Memory Page Memory Page Memory Page REDESIGN GC HEAD 28 PyGC_HEAD PyObject PyGC_HEAD_Ptr

Slide 29

Slide 29 text

WORKING PROOF 29 lists = [] strs = [] for i in range(16000): lists.append([]) for j in range(40): strs.append(' ' * 8) ~60MB ~0.9MB

Slide 30

Slide 30 text

RESULTS

Slide 31

Slide 31 text

POINTERS TAKE SPACE 31 Memory Page Memory Page 16 BYTES 1,000,000 OBJECTS 80 PROCESSES ~1 GB

Slide 32

Slide 32 text

32 FREEZE OBJECTS

Slide 33

Slide 33 text

33 FREEZE OBJECTS static PyObject * gc_freeze(PyObject *module) { for (int i = 0; i < NUM_GENERATIONS; ++i) { gc_list_merge(GEN_HEAD(i), &permanent_generation.head); generations[i].count = 0; } Py_RETURN_NONE; } gc.freeze() # upstream to python 3.7!

Slide 34

Slide 34 text

34 NO GC BEFORE FORK?

Slide 35

Slide 35 text

Memory Page 35 MEMORY POOL Memory Page PyGC_HEAD PyObject Collection Memory Page Fork Memory Page CoW!!

Slide 36

Slide 36 text

RESULTS GC Enabled GC Disabled

Slide 37

Slide 37 text

WRAP UP 1 Disable garbage collection to avoid copy-on-write 2 Re-enable garbage collection by freezing objects 3 No gc before fork to avoid copy-on-write by re-allocation

Slide 38

Slide 38 text

https://instagram-engineering.com