Tweet
Tweet

Slide 1

Slide 1 text

SYSTEMS PROGRAMMING IS FOR EVERYONE by Julia Evans Stripe twitter: @b0rk blog: jvns.ca Tweet questions to @b0rk

Slide 2

Slide 2 text

ὁὁ bugs ὁὁ

Slide 3

Slide 3 text

perl | go | c++ | fortran php | python | java | smalltalk INTERCAL | BASIC | erlang

Slide 4

Slide 4 text

MOSTLY LINUX ὂ

Slide 5

Slide 5 text

YOUR PROGRAM = BLACK BOX

Slide 6

Slide 6 text

DEBUGGING: look at the source code add print statements know the programming language

Slide 7

Slide 7 text

DEBUGGING: look at the source code add print statements know the programming language ★★★ be a wizard★★★

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

THIS TALK Wizard school (or, an operating systems primer) Chapter 1: The Case of the Mystery Config File Chapter 2: The Case of the French Website Chapter 3: The Case of the Slow Program

Slide 10

Slide 10 text

WIZARD SCHOOL -OR- WHY YOU SHOULD ❤ YOUR OPERATING SYSTEM

Slide 11

Slide 11 text

WHAT IS AN OPERATING SYSTEM FOR?

Slide 12

Slide 12 text

When I go to http://google.com, kernel code runs for: Typing in the address Handling every network packet Writing history files to disk Allocating memory Communicating with the graphics card

Slide 13

Slide 13 text

HOW TO CALL OPERATING SYSTEM CODE

Slide 14

Slide 14 text

★★★ SYSTEM CALLS!!! ★★★

Slide 15

Slide 15 text

SYSTEM CALLS: AN OS'S INTERFACE open a file! (o p e n ) start a program! (e x e c v e ) change a file's permissions! (c h m o d )

Slide 16

Slide 16 text

WHAT WE'VE LEARNED Your OS does tons of stuff Programs tell it what to do using system calls

Slide 17

Slide 17 text

USING SYSTEMS KNOWLEDGE TO DEBUG

Slide 18

Slide 18 text

CHAPTER 1: THE CASE OF THE MYSTERY CONFIG FILE

Slide 19

Slide 19 text

Does bash use . b a s h _ p r o f i l e or . b a s h r c ??!??

Slide 20

Slide 20 text

STRACE = ⚡ WIZARDRY ⚡

Slide 21

Slide 21 text

STRACE = TRACING SYSTEM CALLS

Slide 22

Slide 22 text

HOW TO STRACE $ s t r a c e g o o g l e - c h r o m e e x e c v e ( " / u s r / b i n / g o o g l e - c h r o m e " , [ " g o o g l e - c h r o m e " ] , [ / * 5 1 v a r s * / ] b r k ( 0 ) = 0 x 1 2 4 f 0 0 0 a c c e s s ( " / e t c / l d . s o . n o h w c a p " , F _ O K ) = - 1 E N O E N T ( N o s u c h f i l e o

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

OPEN strace ­e open bash

Slide 25

Slide 25 text

BASHRC WINS!

Slide 26

Slide 26 text

OTHER AWESOME SYSTEM CALLS w r i t e for log files e x e c v e for starting programs r e c v f r o m for receiving data

Slide 27

Slide 27 text

STRACE ZINE

Slide 28

Slide 28 text

OPENSNOOP LINUX ὂ & OS X ἴ ! https://github.com/iovisor/bcc

Slide 29

Slide 29 text

opensnoop(8) NAME opensnoop ­ Trace open() syscalls. Uses Linux eBPF/bcc. ... AUTHOR Brendan Gregg

Slide 30

Slide 30 text

CHAPTER 2: THE CASE OF THE POLISH WEBSITE

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

???

Slide 34

Slide 34 text

NETWORK SPYING TO THE RESCUE

Slide 35

Slide 35 text

TCPDUMP sudo tcpdump port 5000 ­i any ­o polish­website.pcap

Slide 36

Slide 36 text

WIRESHARK wireshark polish­website.pcap

Slide 37

Slide 37 text

Accept­Language: en­US,en;q=0.5\r\n

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

WIRESHARK: ASSIGNER OF BLAME

Slide 45

Slide 45 text

why tcpdump is fast: BPF (berkeley packet filter) src ip 192.168.0.1 or dst ip 192.168.0.1

Slide 46

Slide 46 text

NETWORK SPYING TOOLS ngrep tcpdump wireshark mitmproxy

Slide 47

Slide 47 text

CHAPTER 3: THE CASE OF THE SLOW PROGRAM

Slide 48

Slide 48 text

3 SLOW PROGRAMS 1. CPU time 2. too many writes 3. waiting for a slow server

Slide 49

Slide 49 text

MYSTERY PROGRAM #1

Slide 50

Slide 50 text

$ time python mystery_1.py 0.09user 0.01system 0:02.11elapsed 5%CPU

Slide 51

Slide 51 text

WHAT IS IT WAITING FOR?

Slide 52

Slide 52 text

LET'S LOOK INTO THE KERNEL'S SOUL

Slide 53

Slide 53 text

/PROC/P I D /STACK $ p g r e p - f m y s t e r y _ 1 3 1 7 2 8 $ s u d o c a t / p r o c / 3 1 7 2 8 / s t a c k [ < f f f f f f f f 8 1 7 0 2 4 6 7 > ] s k _ w a i t _ d a t a + 0 x 1 0 7 / 0 x 1 2 0 [ < f f f f f f f f 8 1 7 6 7 1 1 2 > ] t c p _ r e c v m s g + 0 x 2 e 2 / 0 x b 8 0 [ < f f f f f f f f 8 1 7 9 4 d 6 e > ] i n e t _ r e c v m s g + 0 x 7 e / 0 x b 0 [ < f f f f f f f f 8 1 6 f d b 6 b > ] s o c k _ r e c v m s g + 0 x 3 b / 0 x 5 0 [ < f f f f f f f f 8 1 6 f d d c 1 > ] S Y S C _ r e c v f r o m + 0 x e 1 / 0 x 1 6 0 [ < f f f f f f f f 8 1 6 f f 1 c e > ] S y S _ r e c v f r o m + 0 x e / 0 x 1 0 [ < f f f f f f f f 8 1 8 2 4 4 f 2 > ] e n t r y _ S Y S C A L L _ 6 4 _ f a s t p a t h + 0 x 1 6 / 0 x 7 1 [ < f f f f f f f f f f f f f f f f > ] 0 x f f f f f f f f f f f f f f f

Slide 54

Slide 54 text

WE WIN! IT WAS THE NETWORK!

Slide 55

Slide 55 text

OUR SERVER @app.route('/') def slow(): time.sleep(2) return "Hi!" app.run()

Slide 56

Slide 56 text

MYSTERY PROGRAM #2 $ time python mystery_2.py 2.74user 0.00system 2.74 seconds 99%CPU

Slide 57

Slide 57 text

USE A PYTHON PROFILER

Slide 58

Slide 58 text

total = 0 for i in xrange(14000000): total += i

Slide 59

Slide 59 text

MYSTERY PROGRAM #3

Slide 60

Slide 60 text

(REALLY A MYSTERY)

Slide 61

Slide 61 text

$ time python mystery_3.py 0.03user 4.11system 4.40 seconds elapsed 94%CPU

Slide 62

Slide 62 text

DEMO DEMO

Slide 63

Slide 63 text

29,81% [kernel] _aesni_enc1 1,88% [kernel] ecryptfs_write_begin ▒ 1,51% [kernel] raw_spin_lock

Slide 64

Slide 64 text

MYSTERY SOLVED filename = '/home/bork/fake.txt'

Slide 65

Slide 65 text

❤ dstat ❤

Slide 66

Slide 66 text

❤ perf top ❤

Slide 67

Slide 67 text

WORKS GREAT WITH THE JVM & NODE

Slide 68

Slide 68 text

rust_mutex system_call_fastpath ttwu_.. en.. enq.. enqu.. futex_wait main::.. [unknown] _raw_.. [unknown] futex_wake do_futex __ticket_spin_lock __schedule __lll_lock_wait wake_futex acti.. schedule sys_futex futex_wait_que.. dea.. __lll_unlock_wake wake_up_state _raw_spin_lock __tic.. __ticket_spin_lock de.. try_to_wake_up se.. _raw_spin_lock ttwu_.. system_call_fastpath deq.. d.. sys_futex futex_wait_setup do_futex

Slide 69

Slide 69 text

YOUR PROGRAM = BLACK BOX

Slide 70

Slide 70 text

THERE ARE A LOT OF AWESOME TOOLS

Slide 71

Slide 71 text

LEARN YOUR OPERATING SYSTEM (or your browser.)

Slide 72

Slide 72 text

Recurse Center

Slide 73

Slide 73 text

PROGRAMMING PROGRAMMING EXPERIMENTS EXPERIMENTS

Slide 74

Slide 74 text

RULES OF RULES OF PROGRAMMING PROGRAMMING EXPERIMENTS EXPERIMENTS it doesn't have to work 1. you don't have to finish it 2. you have to learn something 3.

Slide 75

Slide 75 text

EXPERIMENT 1: WRITE AN EXPERIMENT 1: WRITE AN OPERATING SYSTEM OPERATING SYSTEM remember it doesn't have to work

Slide 76

Slide 76 text

WHAT I LEARNED WHAT I LEARNED having more than once process is hard filesystems are hard operating systems are SO COMPLICATED

Slide 77

Slide 77 text

EXPERIMENT 2: EXPERIMENT 2: DATABASE SURGERY DATABASE SURGERY

Slide 78

Slide 78 text

HOW DOES SQLITE HOW DOES SQLITE WORK? WORK?

Slide 79

Slide 79 text

FUN.SQLITE FUN.SQLITE id | word 1 | greedy 2 | greediness 3 | greener

Slide 80

Slide 80 text

$ hexdump fun.sqlite |.............{.n| |.a.R.D.4.%......| |................| |...y.n._.N.>.,.$| |................| |..............F.| |..EAcevedo.E...D| |Accra's.D...CAcc| |ra.C..#BAccentur| |e's.B...AAccentu| |re.A..!@Acapulco| |'s.@...?Acapulco|

Slide 81

Slide 81 text

A DATABASE IS A A DATABASE IS A TREE TREE

Slide 82

Slide 82 text

static MemPage *btreePageFromDbPage(DbPage *pDbPage, Pgno pgno, BtSha // actual code printf("Read a btree page, page number %d\n", pgno); // added by me // actual code }

Slide 83

Slide 83 text

sqlite> select * from fun where id = 1; Read a btree page, page number 1 Read a btree page, page number 5 Read a btree page, page number 828 Read a btree page, page number 10 Read a btree page, page number 2 Read a btree page, page number 76 Read a btree page, page number 6 1|A's

Slide 84

Slide 84 text

sqlite> select * from fun where id = 20; Read a btree page, page number 1 Read a btree page, page number 5 Read a btree page, page number 828 Read a btree page, page number 10 Read a btree page, page number 2 Read a btree page, page number 76 Read a btree page, page number 6 20|Aaliyah

Slide 85

Slide 85 text

sqlite> select * from fun where id = 80000; Read a btree page, page number 1 Read a btree page, page number 5 Read a btree page, page number 1198 Read a btree page, page number 992 Read a btree page, page number 2 Read a btree page, page number 1813 Read a btree page, page number 449 80000|scarfs

Slide 86

Slide 86 text

WHAT I LEARNED WHAT I LEARNED databases tables are trees databases are made of pages i can read some of the SQLite source code!

Slide 87

Slide 87 text

EXPERIMENT 3: EXPERIMENT 3: WRITE A TCP STACK WRITE A TCP STACK

Slide 88

Slide 88 text

EXPERIMENT 3: EXPERIMENT 3: WRITE A TCP STACK WRITE A TCP STACK IN PYTHON IN PYTHON

Slide 89

Slide 89 text

No content

Slide 90

Slide 90 text

ip_header = IP(dst=dest_ip, src=src_ip) syn = TCP(dport=80, sport=59333, ack=0, flags="S") # Send the SYN packet to Google response = srp(ip_header + syn)

Slide 91

Slide 91 text

WHAT I LEARNED WHAT I LEARNED how TCP packets are put together! you can write a 10% working TCP from scratch in 2 weeks python can't keep up

Slide 92

Slide 92 text

EXPERIMENT 4: EXPERIMENT 4: CONCURRENCY CONCURRENCY

Slide 93

Slide 93 text

int counter; void *AddThings(void *threadid) { for (int i = 0; i < 10000; i++) counter += 1; pthread_exit(NULL); }

Slide 94

Slide 94 text

WRONG ANSWER WRONG ANSWER

Slide 95

Slide 95 text

MUTEX MUTEX pthread_mutex_lock(&mutex); counter += 1;

Slide 96

Slide 96 text

"ATOM" "ATOM" __sync_add_and_fetch(&counter, 1);

Slide 97

Slide 97 text

WHAT I LEARNED WHAT I LEARNED atoms are faster than mutexes

Slide 98

Slide 98 text

I BLOG MY EXPERIMENTS I BLOG MY EXPERIMENTS

Slide 99

Slide 99 text

"can you discuss the pros and cons of using a lock-free approach for implementing a thread-safe hashmap?"

Slide 100

Slide 100 text

DO ENOUGH DO ENOUGH EXPERIMENTS EXPERIMENTS END UP WITH ACTUAL END UP WITH ACTUAL KNOWLEDGE KNOWLEDGE

Slide 101

Slide 101 text

THANKS! Julia Evans twitter: @b0rk learn more by reading my blog: http://jvns.ca Come ask me questions & get a zine!