Tweet
Tweet

Slide 1

Slide 1 text

PHPΞϓϦέʔγϣϯʹ͓͚Δ eBPFͷ࢖͍ॴ 2024/08/19 Sohei Iwahori (GREE, Inc.)

Slide 2

Slide 2 text

who? » Sohei Iwahori (@egmc) » GREE, Inc. » Πϯϑϥ / Monitoring Unit Leader / Senior Lead Engineer » ओʹήʔϜͷΠϯϑϥͱ؂ࢹγεςϜ » SRE NEXT 2025ͷCo-Chairʢnew!ʣ

Slide 3

Slide 3 text

who?ʢ؆୯ͳྺ࢙ʣ » SIer͔Βελʔτ » PHPͰͷόοΫΤϯυ։ൃΛ݁ߏ௕͘ʢPHP5.xͷ࣌୅ʣ » ελʔτΞοϓͰΫϥ΢υΠϯϑϥ΋ؚΊͯશ෦ » ݱ৬ͰঃʑʹԼͷϨΠϠʔʹ » ʢ8͸શવ௥͍͑ͯͳ͍͕ࠓͰ΋Կ΋ߟ͑ͣʹॻ͚Δͷ͸͓ͦΒ͘PHPʣ

Slide 4

Slide 4 text

ࣗࣾͷࣄۀͱPHP » ήʔϜɾΞχϝࣄۀΛத৺ͱͨ͠ϞόΠϧΠϯλʔωοταʔϏε » Ψϥέʔͷ࣌୅͔Βଘࡏ͢ΔϓϩμΫτʙݱࡏ·ͰجຊతʹPHPΛར༻ » όʔδϣϯ͸શମతʹ͸গ͠ݹΊɺ7.x͕ϝΠϯ » ྺ࢙ͷ௕͍ϓϩμΫτ΄ͲڊେͳίʔυϕʔεΛ͍࣋ͬͯΔ » APMͱͯ͠͸NewRelic->DatadogΛར༻

Slide 5

Slide 5 text

Agenda » PHPerࢹ఺ͰͷPHPͷੈք » PHPͷੈքʹ͓͚ΔՄ؍ଌੑͷΧόʔൣғ » PHPΞϓϦέʔγϣϯʹ͓͚ΔeBPFͷ࢖͍ॴ » ࣮ࡍʹ࢖ͬͯΈͨ࿩ » ࠓ౓࢖ͬͯΈ͍ͨ࿩ʢ͕࣌ؒ͋Ε͹ʣ » ·ͱΊ

Slide 6

Slide 6 text

PHPerࢹ఺ͰͷPHPͷੈք

Slide 7

Slide 7 text

PHPerࢹ఺ͰͷPHPͷੈք » ϓϩηεʢঢ়ଶʣ͸୹໋Ͱ͋Δ » ϓϩηεؒͷঢ়ଶڞ༗͸ݶఆతͰ͋Δ » PHP͸άϧʔͰ͋Δ

Slide 8

Slide 8 text

ϓϩηεʢঢ়ଶʣ͸୹໋Ͱ͋Δ » webͷίϯςΩετ » ϦΫΤετ୯ҐͰجຊతʹঢ়ଶ͕ഁغ͞ΕΔ » ҰఆͷϦΫΤετΛࡹ͍ͨΒϓϩηε͸࠶ੜ੒͞ΕΔ » (Α͘࢖ΘΕΔapache+mod-php΋php-fpmͳͲ) » webҎ֎ͷίϯςΩετ » ༗໊Ͳ͜ΖͷLaravelͷΩϡʔϫʔΧʔ͸ಉ͘͡Ұఆ਺ϦΫΤετΛࡹ͍ͨΓɺϝϞϦ ࢖༻ྔ͕͖͍͠஋Λ௒͑ͨΒࣗ਎Λkillͯ͠࠶ىಈͯ͠΋Β͏Α͏ͳઃܭʹͳ͍ͬͯΔ

Slide 9

Slide 9 text

ϓϩηεؒͷঢ়ଶڞ༗͸ݶఆతͰ͋Δ » mod-php΍php-fpmͰ͸ͦ΋ͦ΋ϓϩηε͕෼͔Ε͍ͯΔͷͰάϩʔόϧͳঢ়ଶڞ༗͕ग़ དྷͳ͍ » apcu֦ுΛར༻ͯ͠ɺmmapͳͲͰϓϩηεؒͰڞ༗ՄೳͳΩʔόϦϡʔετΞΛར༻͢Δ ͔֎෦ͷmemcachedͳͲΛར༻͢Δ » ΞϓϦέʔγϣϯͷ಺෦ঢ়ଶΛࣗ਎ͰϝτϦΫεͰग़ྗ͢Δͷ͕ଞݴޠʹൺ΂ͯ΍΍೉͍͠ » prometheus_client_php Ͱ͸apcu/redisͳͲΛར༻͍ͯ͠Δ1 1 https://github.com/PromPHP/prometheus_client_php

Slide 10

Slide 10 text

PHP͸άϧʔͰ͋Δ(1/2) https://www.php.net/manual/ja/faq.installation.php

Slide 11

Slide 11 text

PHP͸άϧʔͰ͋Δ(2/2) » PHPຊମ͔ΒϦϯΫ » libcryptoͳͲ » γεςϜϥΠϒϥϦʢlibxxxʣʹϦϯΫͯ͠PHPଆʹΠϯλʔϑ ΣʔεΛఏڙ͢Δ » memcached.so + libmemcached » curl.so + libcurl » ... » C֦ுϨΠϠʔࣗମͰͷػೳఏڙ » mysqlnd » apcu » phpredis » ...

Slide 12

Slide 12 text

PHPͷੈքʹ͓͚Δ Մ؍ଌੑͷΧόʔൣғ

Slide 13

Slide 13 text

PHPΞϓϦέʔγϣϯʢϢʔβʔίʔυʣ

Slide 14

Slide 14 text

APMར༻࣌(1/3)

Slide 15

Slide 15 text

APMར༻࣌(2/3)

Slide 16

Slide 16 text

APMར༻࣌(3/3)

Slide 17

Slide 17 text

PHPΞϓϦέʔγϣϯʹ ͓͚ΔeBPFͷ࢖͍ॴ

Slide 18

Slide 18 text

PHPຊମʹ࢖͍ͬͯ͘(1/2) » USDT͕࢖͑Δ2 » ΋ͪΖΜuprobe΋ » ࢖͍ॴ͸͋Γͦ͏͕ͩɺ΍΍৚͕݅໘౗ Ͱ͋ΓɺAPMͰΧόʔ͞ΕΔ෦෼Ͱ΋͋ ΔͷͰ೉͍͠ͱ͜Ζ 2 https://www.php.net/manual/ja/features.dtrace.dtrace.php

Slide 19

Slide 19 text

PHPຊମʹ࢖͍ͬͯ͘(2/2) » USDT͕ར༻ग़དྷΔ৚݅ » enable-dtrace͕༗ޮʹͳ͍ͬͯΔ » USE_ZEND_DTRACE=1؀ڥม਺͕ઃఆ͞Ε͍ͯΔ » ಺෦తʹ͸؀ڥม਺ΛΈͯɺτϨʔεϙΠϯτΛؚΉؔ਺ʹϙΠϯλ(execute_ex->dtrace_execute_ex) Λஔ͖׵͍͑ͯΔ3 » pidΛࢦఆ͠ͳ͚Ε͹probeΛΞλονग़དྷͳ͍ » bpftrace͸--usdt-file-activation͕͋Δ͕ɺPHPͷϓϩηεण໋͸୹͍ͷͰ௥͍͔͚ଓ͚Δ͜ͱ͸ग़དྷͳ͍ 3 https://github.com/php/php-src/blob/97b3b4552d343a830e67a63aa9b93cf4f2b1158f/Zend/zend.c#L941-L953

Slide 20

Slide 20 text

֦ுϨΠϠʹ࢖͍ͬͯ͘ » C֦ு΍ɺ͔ͦ͜ΒϦϯΫ͞Ε͍ͯΔڞ༗ϥΠϒϥϦΛuprobeͰτϨʔε͢ Δ » APMͰΧόʔ͞Ε͍ͯͳ͍෦෼ͰϐϯϙΠϯτʹ৘ใΛಘ͍ͨ৔߹ʹ༗༻ » ࠓճ͓࿩͢Δέʔε΋͜ͷ෦෼

Slide 21

Slide 21 text

֎ͷੈքͰ࢖͍ͬͯ͘ » ωοτϫʔΫ·ΘΓͷঢ়ଶ֬ೝͳͲ » strace/tcpdumpͳͲΑΓϐϯϙΠϯτͰௐ͍ࠪͨ͠έʔεͷΧόʔ » ಛʹPHPಛ༗ͷ࿩Ͱ͸ͳ͍͕૊Έ߹ΘͤͯௐࠪͰ͖Δͱศར

Slide 22

Slide 22 text

࣮ࡍʹ࢖ͬͯΈͨ

Slide 23

Slide 23 text

ෛՙࢼݧʹ͓͚ΔMemcachedͷ ߴτϥϑΟοΫݪҼௐࠪ

Slide 24

Slide 24 text

ෛՙࢼݧʹ͓͚ΔϘτϧωοΫௐࠪͰͷར༻ » ͋Δ৽نҊ݅ͰAWS্ͷ؀ڥͰෛՙࢼݧΛ࣮ࢪ͍ͯͨ͠ͱ͜Ζ͋ΔϙΠϯτ ͰϨΠςϯγ͕ѱ͘ͳͬͨ » 2୆த1୆ͷElasticache(Memcached)΁ͷτϥϑΟοΫ͕ଟ͗ͯ͢ଳҬΛ࢖͍ ੾͍ͬͯͨ » Memcached΁ͷΠϯλʔϑΣʔε͸ϑϨʔϜϫʔΫʢFuelPHPʣʹ͓೚ͤ » ෼ࢄΞϧΰϦζϜΛมߋͨ͠ΒτϥϑΟοΫͷภΔϊʔυ͕มΘͬͨ » ͦΕͧΕͷϊʔυίϚϯυͷ਺͸΄΅ಉҰͰҧ͍͸τϥϑΟοΫͷΈ

Slide 25

Slide 25 text

ෛՙࢼݧʹ͓͚ΔϘτϧωοΫௐࠪͰͷར༻

Slide 26

Slide 26 text

ෛՙࢼݧʹ͓͚ΔϘτϧωοΫௐࠪͰͷར༻ » ൃߦ͞ΕΔίϚϯυ਺͸ಉҰͳͷͰ෼ࢄࣗମ͸ग़དྷ͍ͯΔ » ঢ়گతʹ͸ಛఆͷΩʔͰڊେͳΦϒδΣΫτ͕΍ΓͱΓ͞Ε͍ͯΔΑ͏ʹݟ ͑Δ » ཪͱΓΛߦ͏ͨΊʹbpftraceͰlibmemcachedʹuprobeΛ࢓ֻ͚τϨʔεΛ औͬͯΈͨ

Slide 27

Slide 27 text

ෛՙࢼݧʹ͓͚ΔϘτϧωοΫௐࠪͰͷར༻ » ൃߦ͞ΕΔίϚϯυ਺͸ಉҰͳͷͰ෼ࢄࣗମ͸ग़དྷ͍ͯΔ » ঢ়گతʹ͸ಛఆͷΩʔͰڊେͳΦϒδΣΫτ͕΍ΓͱΓ͞Ε͍ͯΔΑ͏ʹݟ ͑Δ » ཪͱΓΛߦ͏ͨΊʹbpftraceͰlibmemcachedʹuprobeΛ࢓ֻ͚τϨʔεΛ औͬͯΈͨ

Slide 28

Slide 28 text

ෛՙࢼݧʹ͓͚ΔϘτϧωοΫௐࠪͰͷར༻ sudo BPFTRACE_STRLEN=200 bpftrace -e 'uprobe:/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/libmemcached.so.11.0.0:memcached_set { printf("----");time(); printf("key_length: %d\nkey: %s\n", arg2, str(arg1)); printf("val_length: %d\nval: %s\n", arg4, str(arg3) );}'

Slide 29

Slide 29 text

ෛՙࢼݧʹ͓͚ΔϘτϧωοΫௐࠪͰͷར༻ » fuelphpͷmemcachedυϥΠό͕memcachedʹετΞ͢ΔࡍʹΦϦδφϧͷ ΩʔΛੜ੒ͨ͠ΩʔʹϚοϐϯά » Ϛοϐϯά৘ใͷΠϯσοΫε͸୯ҰͷΦϒδΣΫτʹ֨ೲ͞Ε͍ͯͨ » ΩʔͷϚοϐϯά৘ใͳͷͰຖճࢀরɺߋ৽͞Ε͍ͯͨ » ౰֘Ҋ݅Ͱ͸Ωʔ͕૿͑΍͍͢܏޲ʹ͋ͬͨͨΊɺΦϒδΣΫτ͕ංେԽͯ͠ ͍ͨ » ࠷ऴతʹϑϨʔϜϫʔΫͷػߏΛ࢖Θͣͳ͍࣮૷ʹஔ͖׵͑໰୊Λղফ

Slide 30

Slide 30 text

mysqlndʹ͓͚Δ caching_sha2_passwordͷڍಈௐࠪ ʢby @ts4thʣ

Slide 31

Slide 31 text

caching_sha2_passwordͷڍಈௐࠪ » MySQL8.0Ͱcaching_sha2_password͕σϑΥϧτʹͳΓɺ8.1Ͱmysql_native_password͕ deprecatedʹͳͬͨ » ηΩϡΞ͡Όͳ͍ܦ࿏Ͱ͸mysqld͔Βެ։伴͕औಘ͞ΕɺͦͷΩʔʹΑΓύεϫʔυ͕҉߸Խ͞Εૹ৴ ͞ΕΔ » get-server-public-keyΦϓγϣϯͰ໌ࣔతʹϦΫΤετͰ͖Δ » σϑΥϧτͰ͸SSL_MODE=PREFERREDͳͷͰmysqlίϚϯυͳͲlibmysqlΛར༻͢Δ؀ڥ͸࠷ॳʹSSL઀ ଓΛࢼΈΔ » ରͯ͠PHPͷmysqlnd֦ு͸get-server-public-key૬౰ͷΦϓγϣϯͳͲ͕ͳ͘ڍಈʹෆ͕҆͋ͬͨ » ͦ͜ͰbpftraceͳͲΛར༻ͯ͠ڍಈௐࠪΛߦ͏͜ͱʹͨ͠

Slide 32

Slide 32 text

caching_sha2_passwordͷڍಈௐࠪ » mysqlnd.so΁͸ͦͷ··Ͱ͸symbolͷղܾʹࣦഊͯ͠uprobeͷΞλον͕ग़དྷͳ ͍ » σόοάγϯϘϧ php7.4-mysql-dbgsym ύοέʔδΛ௥Ճ͢Δ͜ͱͰҎԼͷؔ࿈ ͨؔ͠਺ʹΞλονͰ͖ΔΑ͏ʹͳΔ » mysqlnd_caching_sha2_handle_server_response » mysqlnd_caching_sha2_get_auth_data » ެ։伴΁ͷΞΫηε͸libcrypto.soͷBIO_new_mem_bufؔ਺ʹΞλον͢Δ͜ͱͰ ެ։伴͕࢖ΘΕ͍ͯΔ͜ͱΛ֬ೝͨ͠

Slide 33

Slide 33 text

caching_sha2_passwordͷڍಈௐࠪ #ίʔυύεͷ֬ೝ sudo bpftrace -e 'uprobe:/usr/lib/php/20190902/mysqlnd.so:mysqlnd_caching_sha2_handle_server_response {printf("here\n");}' #ެ։伴Λ֬ೝ͠ɺmysqldଆ͔Βఏࣔ͞Ε͍ͯΔ΋ͷͱͷҰகΛ֬ೝ $ sudo BPFTRACE_STRLEN=200 bpftrace -e 'uprobe:/lib/x86_64-linux-gnu/libcrypto.so.1.1:BIO_new_mem_buf {printf("%s\n", str(arg0));}' Attaching 1 probe... -----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv6agEXz7+WMj0uYjneXo IsWGu7pjdQB6/C3mShqTVvqnc2OgmjQ+K25qDU8MZc/yFiAVgFbQ4PFIiFIxdr2D faBBIW1fw/BMqXGRTXIa6aqDMWShcC9XNuWdsp1y3V $ sudo cat /var/lib/mysql/public_key.pem -----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv6agEXz7+WMj0uYjneXo IsWGu7pjdQB6/C3mShqTVvqnc2OgmjQ+K25qDU8MZc/yFiAVgFbQ4PFIiFIxdr2D faBBIW1fw/BMqXGRTXIa6aqDMWShcC9XNuWdsp1y3V1XUwWneW5R+TOmoDsx4igX ieH0xa6L8oYWM+KZNNvKD5+ESo2i9sAF+SZAgF3ZSYfET+I68KMHN/zMAR2l0Q5i YVlrKc33cAK6kabz6EeeufOzgRM6EFNrw0kMjZ5RVPsAeLO/r8NmcB4Fn7DTcpuz hkgcmbcPEAzaiI6yR0MiiDWPyBJeZgk0f96pD2iv6cpujgqWndQFc69v0OyAz7gR cwIDAQAB -----END PUBLIC KEY-----

Slide 34

Slide 34 text

caching_sha2_passwordͷڍಈௐࠪ » mysqlndͰ͸σϑΥϧτͷ -ssl-mode=PREFERRED ૬౰ͷڍಈͰ͸ͳ͘ɺ-- ssl-mode=DISABLED --get-server-public-key ͷΑ͏ͳಈ࡞Ͱ͋Δ͜ͱ͕ ֬ೝͰ͖ͨ » ຊ݅͸gdbͱperfͷผղ΋͋ΔͷͰͪ͜Β΋ΑΖ͚͠Ε͹͝ཡ͍ͩ͘͞ » Debugging mysqlnd.so ʙmysql_native_password͕ഇࢭ͞ΕΔະདྷʹඋ͑ͯʙ4 4 https://labs.gree.jp/blog/2023/08/22599/

Slide 35

Slide 35 text

ࠓ౓࢖ͬͯΈ͍ͨ࿩ʢ͕࣌ؒ͋Ε͹ʣ

Slide 36

Slide 36 text

ࠓ౓࢖ͬͯΈ͍ͨ࿩1 USDTͷ׆༻ » USDTͷ׆༻ » ྫ͑͹ར༻͍ͯ͠ͳ͍ίʔυͷ͋ͿΓग़͠ͳͲ » PIDΛಈతʹ௥͍͔͚͍ͯ͘࢓૊Έ͸ඞཁɾɾ » ύεϕʔε͘Β͍Ͱ൑ఆͯ͠BPF_MAPʹೖΕͯɾɾɾ͘Β͍ͷ࡞ΓͳΒ ·͋·͋ܰྔʹग़དྷͦ͏ʁ

Slide 37

Slide 37 text

ࠓ౓࢖ͬͯΈ͍ͨ࿩2 ֎ͷੈ քͷ؍ଌ » TCPϨΠϠʔͷSYN࠶ૹͷՄࢹԽ » ebpf_exporterΛར༻ » libbpfʹΑΓෳ਺όʔδϣϯͷΧʔωϧ Ͱಈ࡞͍ͤͨ͞ɾɾ

Slide 38

Slide 38 text

·ͱΊ

Slide 39

Slide 39 text

» PHPݻ༗ͷΦϒβʔόϏϦςΟʹ͓͚Δ೉͠͞ͱͯ͠άϩʔόϧͳঢ়ଶอ࣋ͷ೉ ͠͞ͱ୹໋ͳϓϩηεͱ͍͏ͷ͕͋Δ » PHPͷτϨʔεϙΠϯτ͸༗༻ͦ͏͕ͩɺ্هͷ໰୊ʹΑΓ΍΍׆༻ʹ೉͕͋Δ » eBPFΛར༻͢Δ͜ͱͰC֦ுɺٴͼϦϯΫઌͷϥΠϒϥϦͷڍಈௐ͕ࠪखܰʹ ϐϯϙΠϯτͰߦ͑ͯPHP։ൃऀతʹ΋خ͍͠৔໘͸͋Δ » uprobeͰͷ֦ு·ΘΓͷௐࠪ͸Ұา໨ͱͯ͠ͱ͖ͬͭ΍͍͢ͷͰPHP։ൃऀ޲ ͚ʹ΋͓͢͢Ί » ʢIDEͰͷରԠΈ͍ͨͳͷ΋͸ͳͦ͞͏ʁͳͷͰͦ͏͍ͬͨ΋ͷ͕͋ΔͱΑΓḿ Δ͔΋͠Εͳ͍Ͱ͢Ͷʣ

Slide 40

Slide 40 text

Thank you for listening

Slide 41

Slide 41 text

appendix » Tracing mysqlnd.so and mysqld (with perf and bpftrace) » https://labs.gree.jp/blog/2023/08/22599/ » Debugging mysqlnd.so ʙmysql_native_password͕ഇࢭ͞ΕΔະདྷʹඋ͑ͯʙ » https://labs.gree.jp/blog/2023/07/22581/ » caching_sha2_password as the Preferred Authentication Plugin » https://dev.mysql.com/doc/refman/8.0/en/upgrading-from-previous- series.html#upgrade-caching-sha2-password