Out of memory or plenty to spare? 1 Joshua Thijssen jaytaph The fine details of reading memory consumption

Disclaimer 2 No PHP Pretty advanced stuff

How much memory is our server using? 3

4

5

6 How much memory is our server using?

$ free -m total used free shared buffers cached Mem: 3963 3500 462 0 722 1263 -/+ buffers/cache: 1515 2448 Swap: 400 20 379 7

Active / Total Objects (% used) : 2187767 / 2283870 (95.8%) Active / Total Slabs (% used) : 261417 / 261421 (100.0%) Active / Total Caches (% used) : 114 / 192 (59.4%) Active / Total Size (% used) : 1013948.21K / 1024061.72K (99.0%) Minimum / Average / Maximum Object : 0.02K / 0.45K / 4096.00K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 825808 825378 99% 0.98K 206452 4 825808K ext4_inode_cache 522921 512677 98% 0.19K 24901 21 99604K dentry 394290 364506 92% 0.10K 10110 39 40440K buffer_head 166914 142785 85% 0.04K 1686 99 6744K ext4_extent_status 143756 142975 99% 0.05K 1732 83 6928K jbd2_inode 63700 62887 98% 0.56K 9100 7 36400K radix_tree_node 42273 29552 69% 0.06K 671 63 2684K kmalloc-64 23870 19345 81% 0.12K 770 31 3080K kmalloc-96 16884 14623 86% 0.06K 268 63 1072K anon_vma_chain 12144 12024 99% 0.12K 368 33 1472K kernfs_node_cache 11424 10582 92% 0.19K 544 21 2176K vm_area_struct 10044 7897 78% 0.03K 81 124 324K kmalloc-32 9331 9135 97% 0.56K 1333 7 5332K inode_cache 7434 6552 88% 0.06K 118 63 472K anon_vma 6528 6528 100% 0.62K 1088 6 4352K proc_inode_cache 3312 2330 70% 0.25K 207 16 828K filp 2490 2458 98% 0.05K 30 83 120K ftrace_event_field 2294 2207 96% 0.12K 74 31 296K kmalloc-128 1596 1440 90% 0.19K 76 21 304K kmalloc-192 1568 1476 94% 0.07K 28 56 112K Acpi-Operand 1112 1068 96% 1.00K 278 4 1112K kmalloc-1024 1104 1076 97% 0.09K 24 46 96K ftrace_event_file 882 518 58% 0.19K 42 21 168K cred_jar 880 644 73% 0.25K 55 16 220K skbuff_head_cache 828 802 96% 0.65K 138 6 552K shmem_inode_cache 648 552 85% 0.11K 18 36 72K jbd2_journal_head 608 523 86% 0.25K 38 16 152K kmalloc-256 600 500 83% 0.50K 75 8 300K kmalloc-512 468 294 62% 0.11K 13 36 52K task_delay_info 465 302 64% 0.12K 15 31 60K pid 8 slabtop

9 g 3 z x

10

Processes 11

12 Operating system (kernel) Process 1 Process 2 Process 3 Process 4 Kernel POV

13 Operating system (kernel) Process 1 Process POV

➡ Every process can use up to 4 GB of memory. ➡ EVEN IF YOUR COMPUTER DOES NOT HAVE 4GB OF MEMORY!!!!! 14

15 0x00000000 0x00010000 0xC0000000 0xFFFFFFFF 1 GB 3 GB program + data stack shared

16 Enhance

17 4 kb 4 kb 4 kb 4 kb 4 kb 4 kb 4 kb 0x12340000 0x12341000 0x12342000 0x12343000 0x12344000 0x12345000 0x12346000 0x12347000

C 0x00010000 0x00011000 B A 0x00012000 Virtual Physical B 0x00054000 A 0x00CF2000 C 0x00EF7000

19 C1 0x00010000 0x00011000 B1 A1 0x00012000 C2 0x00010000 0x00011000 B2 A2 0x00012000 Virtual Virtual B1 A1 A2 B2 C2 C1 Physical

B1 20 C1 B1 A1 A1 C2 B2 A2 A2 B2 C2 C3 B3 A3 A3 B3 C3 C4 B4 A4 C1 Physical Virtual Virtual

21 C1 B1 A1 C2 B2 A2 C3 B3 A3 C4 B4 A4 Virtual Virtual Swap B1 A1 B2 C2 A3 B3 C3 A2 C1 Physical C4 A4 B4 A2 C1

22 So how can we use more memory?

Ask the OS brk() / mmap() 23

24 0x00000000 0x00010000 0xC0000000 0xFFFFFFFF 1 GB 3 GB program + data stack Heap shared

25

[root@localhost ~]# pmap -x 1271 1271: php-fpm: master process (/etc/php-fpm.conf) Address Kbytes RSS Dirty Mode Mapping 001ee000 248 8 0 r-x-- libgssapi_krb5.so.2.2 0022c000 4 4 4 r---- libgssapi_krb5.so.2.2 0022d000 4 4 4 rw--- libgssapi_krb5.so.2.2 0022f000 28 4 0 r-x-- libcrypt-2.12.so 00236000 4 4 4 r---- libcrypt-2.12.so 00237000 4 4 4 rw--- libcrypt-2.12.so .... 08048000 3400 204 0 r-x-- php-fpm 0839a000 328 140 20 rw--- php-fpm 083ec000 96 32 32 rw--- [ anon ] 092b4000 1316 1176 1176 rw--- [ anon ] ... af483000 4 4 4 rw-s- zero (deleted) af484000 28 0 0 r--s- gconv-modules.cache af48b000 131072 0 0 rw-s- zero (deleted) b748b000 160 4 4 rw--- [ anon ] b74b3000 2048 8 0 r---- locale-archive b76b3000 1312 124 124 rw--- [ anon ] b77fb000 4 4 4 rw-s- zero (deleted) b77fc000 4 4 4 rw-s- zero (deleted) b77fd000 4 4 4 rw-s- zero (deleted) b77fe000 4 4 4 rw-s- zero (deleted) b77ff000 4 4 4 rw--- [ anon ] bf876000 84 48 48 rw--- [ stack ] -------- ------- ------- ------- ------- total kB 155544 - - - 26

... 00aed000 36 4 0 r-x-- gmp.so 00af6000 8 8 8 rw--- gmp.so 00af8000 264 8 0 r-x-- libgmp.so.3.5.0 00b3a000 4 4 4 rw--- libgmp.so.3.5.0 00b3b000 36 4 0 r-x-- json.so 00b44000 4 4 4 rw--- json.so 00b45000 260 8 0 r-x-- phar.so 00b86000 8 8 8 rw--- phar.so 00b88000 72 4 0 r-x-- sockets.so 00b9a000 8 8 8 rw--- sockets.so 00b9c000 40 4 0 r-x-- xml.so 00ba6000 4 4 4 rw--- xml.so 00bad000 12 4 0 r-x-- libdl-2.12.so 00bb0000 4 4 4 r---- libdl-2.12.so 00bb1000 4 4 4 rw--- libdl-2.12.so 00bb2000 16 4 0 r-x-- libplc4.so 00bb6000 4 4 4 r---- libplc4.so 00bb7000 4 4 4 rw--- libplc4.so 00bb8000 88 8 0 r-x-- zip.so 00bce000 8 8 8 rw--- zip.so 00bd0000 48 20 0 r-x-- libnss_files-2.12.so 00bdc000 4 4 4 r---- libnss_files-2.12.so 00bdd000 4 4 4 rw--- libnss_files-2.12.so 00c24000 8 4 0 r-x-- gettext.so 00c26000 4 4 4 rw--- gettext.so 00c5c000 8 4 0 r-x-- ctype.so 00c5e000 4 4 4 rw--- ctype.so 00d06000 40 8 0 r-x-- simplexml.so 00d10000 4 4 4 rw--- simplexml.so ... 27

28

[root@localhost ~]# cat /proc/1271/status Name: php-fpm State: S (sleeping) Tgid: 1271 Pid: 1271 PPid: 1 TracerPid: 0 Uid: 0 0 0 0 Gid: 0 0 0 0 Utrace: 0 FDSize: 32 Groups: 0 VmPeak: 155548 kB VmSize: 155548 kB VmLck: 0 kB VmHWM: 3484 kB VmRSS: 3484 kB VmData: 3160 kB VmStk: 88 kB VmExe: 3400 kB VmLib: 14668 kB VmPTE: 76 kB VmSwap: 0 kB Threads: 1 SigQ: 0/3855 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000000001000 SigCgt: 0000000180014a06 CapInh: 0000000000000000 CapPrm: ffffffffffffffff CapEff: ffffffffffffffff CapBnd: ffffffffffffffff Cpus_allowed: 1 Cpus_allowed_list: 0 Mems_allowed: 1 Mems_allowed_list: 0 voluntary_ctxt_switches: 5155 nonvoluntary_ctxt_switches: 3 29

30

31 fork()

32 Process 1 Process 2 fork()

33 C1 B1 A1 C1` B1` A1` A1 B1 C1 Physical Virtual Virtual fork() =>

34 C1 B1 A1 C1` B2 A1` A1 B1 C1 Physical Virtual Virtual fork() => B2

Everything is f*rked up 35

36

37 Most memory tools returns your virtual memory usage, which is pointless...

http://farm1.static.flickr.com/73/163450213_18478d3aa6_d.jpg 38

39 Find me on twitter: @jaytaph Find me for development and training: www.noxlogic.nl Find me on email: jthijssen@noxlogic.nl Find me for blogs: www.adayinthelifeof.nl

➡ https://techtalk.intersec.com/2013/07/memory-part-2- understanding-process-memory/ ➡ http://locklessinc.com/articles/memory_usage/ ➡ http://rhaas.blogspot.nl/2012/01/linux-memory- reporting.html ➡ http://people.freebsd.org/~lstewart ➡ http://deathbytape.com/post/110371790629/intro- virtual-memoryarticles/cpumemory.pdf ➡ http://nikic.github.com/2011/12/12/How-big-are-PHP- arrays-really-Hint-BIG.html 40 303 See Other