Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

出了问题不要靠猜

Avatar for LI Daobing LI Daobing
September 14, 2013
Programming
4.1k
40

 出了问题不要靠猜

Avatar for LI Daobing

LI Daobing

September 14, 2013

More Decks by LI Daobing

See All by LI Daobing
How to attack TLS in PQC decade, part I
Avatar for LI Daobing lidaobing
0
56
HTTP协议相关的若干安全问题
Avatar for LI Daobing lidaobing
9
1.2k
Debian & Packaging
Avatar for LI Daobing lidaobing
1
600
Java 质量保障
Avatar for LI Daobing lidaobing
3
330
OAuth: How and Why?
Avatar for LI Daobing lidaobing
1
190
从 Struts 迁移到 Spring MVC，以及为什么？
Avatar for LI Daobing lidaobing
2
680
glusterfs 文件系统
Avatar for LI Daobing lidaobing
2
230
如何学习 Shell
Avatar for LI Daobing lidaobing
3
370

Other Decks in Programming

See All in Programming
AHC070解法紹介
Avatar for eijirou eijirou
0
120
AIは賢い。でも実行環境は？ CLIおじさんがAI時代に伝えたいこと ~ CLIおじさんがAI時代に伝えたいこと ~
Avatar for curekoshimizu curekoshimizu
1
250
フロントエンドUIフレームワークのこれまでとこれから
Avatar for TOMIKAWA Sotaro ssssota
5
2.8k
モデルのリファクタリングが難しいと思ったら、そもそも複雑だったのはビジネス仕様だった ? / is-the-business-domain-the-real-complexity
Avatar for hatsu hatsu38
0
350
XP祭りでしか伝わらないフリップネタ #xpjug
Avatar for bayashi murabayashi
0
150
手動確認はもう限界 〜XCUITestでCustom URL Schemeの遷移を起動種別ごとに自動テストする〜 / Testing Custom URL Schemes with XCUITest
Avatar for otouto otouto
0
310
Augmenting AI with the Power of Jakarta EE
Avatar for ivargrimstad ivargrimstad
0
200
Omarchy Tokyo やると聞いて UMPC 買ってセットアップしてきた
Avatar for Fumiaki MATSUSHIMA mtsmfm
0
150
仕様駆動開発による爆速プロダクト開発 / Bakusoku Spec Driven Development
Avatar for Keisuke Kobayashi kobakei
0
120
技術的負債の返済は、AI時代の複利で効く投資 — 経営としての意思決定とその遂行
Avatar for curekoshimizu curekoshimizu
0
1.5k
更なる可用性を求めて、5年間運用したKotlinのアプリケーションをGoでリプレイスする話
Avatar for Kensuke Taguchi ken_tunc
0
300
変化を抱擁するドキュメントの作り方 - ビジネスルール駆動開発がもたらす、コードとの新しい関係
Avatar for ioki ioki
2
220

Featured

See All Featured
Why Mistakes Are the Best Teachers: Turning Failure into a Pathway for Growth
Avatar for Umar Saidu Auna auna
0
290
SEO in 2025: How to Prepare for the Future of Search
Avatar for Michael King ipullrank
3
3.8k
Tell your own story through comics
Avatar for letsgokoyo letsgokoyo
1
1.1k
世界の人気アプリ100個を分析して見えたペイウォール設計の心得
Avatar for Akihiro Kokubo akihiro_kokubo
PRO
74
42k
How GitHub (no longer) Works
Avatar for Zach Holman holman
316
150k
Producing Creativity
Avatar for Steve Smith orderedlist
PRO
348
41k
Taking LLMs out of the black box: A practical guide to human-in-the-loop distillation
Avatar for Ines Montani inesmontani
PRO
3
2.4k
Mobile First: as difficult as doing things right
Avatar for Swwweet swwweet
225
10k
Put a Button on it: Removing Barriers to Going Fast.
Avatar for kastner kastner
60
4.6k
Ten Tips & Tricks for a 🌱 transition
Avatar for Manu Carrasco Molina stuffmc
1
230
Ecommerce SEO: The Keys for Success Now & Beyond - #SERPConf2024
Avatar for Aleyda Solis aleyda
1
2.2k
How Software Deployment tools have changed in the past 20 years
Avatar for Geshan Manandhar geshan
1
34k

Transcript

  1. ग़ྃ问题ෆཁᯪᘃ LI Daobing <[email protected]> ࣣڇӠଘ储 2013-09-14 Saturday, September 14, 13

  2. • ཥಓฌ <[email protected]> • Debian Developer • 2004 ೥䇖࢝઀৮ Debian

    • qterm, ibus, scim ౳软݅แಘ维护ਓ员 • 䇖ݯ爱޷ऀ • தจ维جඦՊલ؅ཧ员 • ࢀ༩维护 iso-codes, translationproject.org • python-lunardate, capistrano-scm-jenkins ࡞ऀ • manpages-zh, douban-ruby, iptux 现೚维护ਓ员 • զ对䇖ݯత؃๏ • ࢀ༩Ұ࿽项໨ɼ贡ݙզత时间ੋҼ为认ಉଞతཧ೦ɼࣕෆੋ૝վ଄ଞతཧ೦ • Github: http://github.com/lidaobing https://speakerdeck.com/lidaobing Saturday, September 14, 13

  3. ໨录 • ဓҰ࿽ Bug 讲ى • զ؃౸తཚ৅ • զਪᣑతղႊํҊ Saturday,

    September 14, 13

  4. Ұ࿽حոత Bug • ᠓盘త্传෰务 • Windows Լ IE, Chrome 浏览ثԼਖ਼ৗ

    • Linux Լ Firefox, Chrome 浏览ثԼਖ਼ৗ • Windows Լత Firefox ෆਖ਼ৗ, ্传ࣦ 败ɻ Saturday, September 14, 13

  5. ೗ԿఆҐBug Saturday, September 14, 13

  6. ೗ԿఆҐBug • टઌ查؃浏览ثత߇੍୆ɼ观࡯ੋ൱༗错误(౎຅༗) Saturday, September 14, 13

  7. ೗ԿఆҐBug • टઌ查؃浏览ثత߇੍୆ɼ观࡯ੋ൱༗错误(౎຅༗) • ༻ Wireshark ፊ获请ٻแɼ发现ࣦ败తҊྫ෰务୺ฦ ճ412(௨ৗ为200) Saturday, September

    14, 13

  8. ೗ԿఆҐBug • टઌ查؃浏览ثత߇੍୆ɼ观࡯ੋ൱༗错误(౎຅༗) • ༻ Wireshark ፊ获请ٻแɼ发现ࣦ败తҊྫ෰务୺ฦ ճ412(௨ৗ为200) • 对ൺ

    HTTP Requestɼ发现ࣦ败Ҋྫత HTTP 头༗ “Content-length”ɼࣕଖଞ৘ႎԼ为 “Content- Length” Saturday, September 14, 13

  9. ೗ԿఆҐBug • टઌ查؃浏览ثత߇੍୆ɼ观࡯ੋ൱༗错误(౎຅༗) • ༻ Wireshark ፊ获请ٻแɼ发现ࣦ败తҊྫ෰务୺ฦ ճ412(௨ৗ为200) • 对ൺ

    HTTP Requestɼ发现ࣦ败Ҋྫత HTTP 头༗ “Content-length”ɼࣕଖଞ৘ႎԼ为 “Content- Length” • ༻ telnet 发ૹෆಉత请ٻདྷ验证结论 Saturday, September 14, 13

  10. ೗ԿఆҐBug Saturday, September 14, 13

  11. ೗ԿఆҐBug • 䉯认ݪҼੋ෰务୺对 Content-Length త େখࣸහײɼ༩标।ෆූɼ଄੒ྃ这࿽ Bug Saturday, September 14,

    13

  12. ೗ԿఆҐBug • 䉯认ݪҼੋ෰务୺对 Content-Length త େখࣸහײɼ༩标।ෆූɼ଄੒ྃ这࿽ Bug • ௨஌෰务୺䇖发ਓ员ਚշम䐾 Saturday,

    September 14, 13

  13. ޷Bug/ᆀBug Saturday, September 14, 13

  14. ޷Bug/ᆀBug • ೳ࠶现త Bug बੋ޷ Bug Saturday, September 14, 13

  15. ޷Bug/ᆀBug • ೳ࠶现త Bug बੋ޷ Bug • ೗Ռෆೳ࠶现ɼೳ፤౸䇗ৗ栈҃ऀ详细 ೔ࢤత໵ੋ޷Bug Saturday,

    September 14, 13

  16. ޷Bug/ᆀBug • ೳ࠶现త Bug बੋ޷ Bug • ೗Ռෆೳ࠶现ɼೳ፤౸䇗ৗ栈҃ऀ详细 ೔ࢤత໵ੋ޷Bug •

    对ဋᆀ Bug, ଞత࠷େ༻处बੋಜଅ㟬补 ॆ೔ࢤ Saturday, September 14, 13

  17. 㠥Ұ࿽حոతBug • ဓެ࢘访问᠓᜾༗12sԆ迟 • ௚઀ ping 㠳应኷շ • ဓଖଞ᠓络དྷݯ访问຅༗问题 •

    监߇຅༗问题, ໵຅༗٬户དྷ๊ԇࠑࣄ Saturday, September 14, 13

  18. ઌᘃᘃ؃ Saturday, September 14, 13

  19. ઌᘃᘃ؃ • ެ࢘࿏༝ग़问题ྃʁ • ॏ启ҰԼ࿏༝试试؃ʁ • ୠ为ॄ㜮୞༗访问զ们ࣗݾ᠓᜾ձग़ࣄʁ Saturday, September 14,

    13

  20. ઌᘃᘃ؃ • ެ࢘࿏༝ग़问题ྃʁ • ॏ启ҰԼ࿏༝试试؃ʁ • ୠ为ॄ㜮୞༗访问զ们ࣗݾ᠓᜾ձग़ࣄʁ • ਓ඼问题ʁ •

    ၏ఔং员还ੋ།෺Ұ఺ൺ较޷ Saturday, September 14, 13

  21. ઌᘃᘃ؃ • ެ࢘࿏༝ग़问题ྃʁ • ॏ启ҰԼ࿏༝试试؃ʁ • ୠ为ॄ㜮୞༗访问զ们ࣗݾ᠓᜾ձग़ࣄʁ • ਓ඼问题ʁ •

    ၏ఔং员还ੋ།෺Ұ఺ൺ较޷ • ෆ؅ɼ൓ਖ਼຅ਓ๊ԇ这ࣄ • 99% త༻户ձ۰౸问题时ෆձ报ࠂ Saturday, September 14, 13

  22. ෼ੳ Saturday, September 14, 13

  23. ෼ੳ • tcpdump፛แ • ٬户୺: 发ྃ6࿽ SYN แɼ౸࠷޳Ұ࿽࠽Ꮕ౸ SYN/ACK •

    ෰务୺: 䉯实Ꮕ౸ྃ6࿽ SYN แ, ࣕ׌䉯实લ໘5 ࿽ SYN แ౎຅༗ճ䐾 • ဓଖଞ᠓络访问ɼ䉯实ࡏट࿽ SYN แ೭޳ฦճ Saturday, September 14, 13

  24. ෼ੳ • tcpdump፛แ • ٬户୺: 发ྃ6࿽ SYN แɼ౸࠷޳Ұ࿽࠽Ꮕ౸ SYN/ACK •

    ෰务୺: 䉯实Ꮕ౸ྃ6࿽ SYN แ, ࣕ׌䉯实લ໘5 ࿽ SYN แ౎຅༗ճ䐾 • ဓଖଞ᠓络访问ɼ䉯实ࡏट࿽ SYN แ೭޳ฦճ • 结论: 问题᪑࿏༝/线࿏Ꮰ䎔ɼ䉯实ੋ෰务୺త问题 Saturday, September 14, 13

  25. ෼ੳ Saturday, September 14, 13

  26. ෼ੳ • ࠶ॏ৽෼ੳ tcpdump త记录ɼ发现લ5࿽ SYN แ带ྃ timestamp, ୈ6࿽຅带 •

    尝试䎔ᎃ٬户୺ TCP timestamp, ॠ间㠳应 • 尝试䎔ᎃ෰务୺ TCP timestamp, ॠ间㠳应 Saturday, September 14, 13

  27. ෼ੳ • ࠶ॏ৽෼ੳ tcpdump త记录ɼ发现લ5࿽ SYN แ带ྃ timestamp, ୈ6࿽຅带 •

    尝试䎔ᎃ٬户୺ TCP timestamp, ॠ间㠳应 • 尝试䎔ᎃ෰务୺ TCP timestamp, ॠ间㠳应 • 结论: Bug ༩ timestamp ૬䎔ɼୠနવᏠ๏ ղ释为ॄ㜮ଖଞ᠓络຅问题 Saturday, September 14, 13

  28. ղႊํҊA Saturday, September 14, 13

  29. ղႊํҊA • 䎔ᎃ෰务୺త TCP timestamp • TCP timestamp ձӨ㠳传输଎౓ɼୠӨ 㠳ෆେ

    Saturday, September 14, 13

  30. ղႊํҊA • 䎔ᎃ෰务୺త TCP timestamp • TCP timestamp ձӨ㠳传输଎౓ɼୠӨ 㠳ෆେ

    • ٙ问နવଘࡏɼ为ॄ㜮෰务ثෆ㠳应๭ ࠣ timestamp แ, ༗时ީຢೳਖ਼ৗ㠳应 Saturday, September 14, 13

  31. ෼ੳ Saturday, September 14, 13

  32. ෼ੳ • ؃಺֩ݯ码 (CTO 亲ࣗૢ౛) • ೖޱصث༗େྔత൓޲୅ཧɼ୺ޱ经ৗෆ䭧༻ɼॴҎ䇖 启ྃTIME_WAIT ୺ޱॏ༻ •

    䇖启ྃ TIME_WAIT ୺ޱॏ༻޳ɼ෰务୺ཁٻಉҰ࿽IPత SYN แ timestamp ඞ须ੋ顺ংత • 办ެ᠓络ੋ಺᠓, ኂ׌对զ们׭᠓访问኷频ൟɼ导கނো 发ੜ Saturday, September 14, 13

  33. ෼ੳ • ؃಺֩ݯ码 (CTO 亲ࣗૢ౛) • ೖޱصث༗େྔత൓޲୅ཧɼ୺ޱ经ৗෆ䭧༻ɼॴҎ䇖 启ྃTIME_WAIT ୺ޱॏ༻ •

    䇖启ྃ TIME_WAIT ୺ޱॏ༻޳ɼ෰务୺ཁٻಉҰ࿽IPత SYN แ timestamp ඞ须ੋ顺ংత • 办ެ᠓络ੋ಺᠓, ኂ׌对զ们׭᠓访问኷频ൟɼ导கނো 发ੜ • ؃དྷፙ౸ਅ实తݪҼྃ, ༗຅༗ߋ޷஍ղႊํҊʁ Saturday, September 14, 13

  34. ղႊํҊB • nginx ߴ൛ຊቮ经ࢧ࣋ keep-alive • ঋ级 nginx, 启༻ keep-alive,

    ߱௿୺ޱ઎༻ • 䎔闭୺ޱॏ༻ɼኂՃ㖘୺ޱ਺త监߇࿨ 报ܯ Saturday, September 14, 13

  35. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 Saturday, September 14, 13

  36. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 • 换࿽浏览ث㠧 Saturday, September 14,

    13

  37. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 • 换࿽浏览ث㠧 • ॏ启, ਗ਼

    cookie, ਗ਼ cache, ॏ৽ొ录 Saturday, September 14, 13

  38. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 • 换࿽浏览ث㠧 • ॏ启, ਗ਼

    cookie, ਗ਼ cache, ॏ৽ొ录 • 现场ඃഁᆀಘׯׯ净净 Saturday, September 14, 13

  39. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 • 换࿽浏览ث㠧 • ॏ启, ਗ਼

    cookie, ਗ਼ cache, ॏ৽ొ录 • 现场ඃഁᆀಘׯׯ净净 • 对ဋಹࠣෆқ࠶现త Bug, ब损ࣦྃҰ࣍म 䐾తصձ Saturday, September 14, 13

  40. զॴ؃౸తཚ৅ • 尝试༻试错๏ղႊॴ༗问题 Saturday, September 14, 13

  41. զॴ؃౸తཚ৅ • 尝试༻试错๏ղႊॴ༗问题 • ঋ级ґ赖แ, ঋ级ᎎ݅, ... Saturday, September 14,

    13

  42. զॴ؃౸తཚ৅ • 尝试༻试错๏ղႊॴ༗问题 • ঋ级ґ赖แ, ঋ级ᎎ݅, ... • ਵศᘃҰ࿽ݪҼɼվ䫆۟୅码ɼવ޳ ॏ৽测试

    Saturday, September 14, 13

  43. զॴ؃౸తཚ৅ • ᠍গ൓ল Saturday, September 14, 13

  44. զॴ؃౸తཚ৅ • ᠍গ൓ল • 寻ፙ౸ workaround ศ认为问题ղႊྃ Saturday, September 14,

    13

  45. զॴ؃౸తཚ৅ • ᠍গ൓ল • 寻ፙ౸ workaround ศ认为问题ղႊྃ • ຅༗༻测试ݻԽ Bug,

    ༰қ产ੜճ归Bug Saturday, September 14, 13

  46. զॴ؃౸తཚ৅ • ᠍গ൓ল • 寻ፙ౸ workaround ศ认为问题ղႊྃ • ຅༗༻测试ݻԽ Bug,

    ༰қ产ੜճ归Bug • Ұ࿽ Bug Մೳࡏଟ处ग़现ɼ຅༗尝试፺ࡧଖଞ༗ Bug త஍ํ Saturday, September 14, 13

  47. զॴ؃౸తཚ৅ • ᠍গ൓ল • 寻ፙ౸ workaround ศ认为问题ղႊྃ • ຅༗༻测试ݻԽ Bug,

    ༰қ产ੜճ归Bug • Ұ࿽ Bug Մೳࡏଟ处ग़现ɼ຅༗尝试፺ࡧଖଞ༗ Bug త஍ํ • ຅༗௨过补ॆ೔ࢤ౳खஈདྷ߱௿೔޳ఆҐBugత难 ౓ Saturday, September 14, 13

  48. զॴ؃౸తཚ৅ • ւԸ๏则: 㑌Ұى严ॏࣄނతഎ޳ɼඞવ ༗29࣍轻ඍࣄނ࿨300ىະ਱ઌஹҎٴ 1000ىࣄނ隐ױ Saturday, September 14, 13

  49. զ们ਪᣑతํ๏ • ग़ྃ问题ෆཁᯪᘃ • 浏览ث • ෰务୺೔ࢤ • ፛แ޻۩ Saturday,

    September 14, 13

  50. 浏览ث؃ॄ㜮ʁ Saturday, September 14, 13

  51. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? Saturday, September 14, 13

  52. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? • ᠓络请ٻੋ൱发ग़ʁ Saturday, September 14,

    13

  53. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? • ᠓络请ٻੋ൱发ग़ʁ • 发ग़త请ٻੋ൱ਖ਼䉯ɿ URL,

    ํ๏, ࢀ਺, Accept, Cookie Saturday, September 14, 13

  54. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? • ᠓络请ٻੋ൱发ग़ʁ • 发ग़త请ٻੋ൱ਖ਼䉯ɿ URL,

    ํ๏, ࢀ਺, Accept, Cookie • ظ๬తฦճ值ੋॄ㜮ʁ Saturday, September 14, 13

  55. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? • ᠓络请ٻੋ൱发ग़ʁ • 发ग़త请ٻੋ൱ਖ਼䉯ɿ URL,

    ํ๏, ࢀ਺, Accept, Cookie • ظ๬తฦճ值ੋॄ㜮ʁ • Request-Id Saturday, September 14, 13

  56. Request-Id • 对㑌࣍请ٻ产ੜҰ࿽།ҰతId Saturday, September 14, 13

  57. Request-Id • 对㑌࣍请ٻ产ੜҰ࿽།ҰతId • 该 Id Ҏ HTTP Response Header

    తํࣜ 发ૹ౸٬户୺ Saturday, September 14, 13

  58. Request-Id • 对㑌࣍请ٻ产ੜҰ࿽།ҰతId • 该 Id Ҏ HTTP Response Header

    తํࣜ 发ૹ౸٬户୺ • ՄҎࡏ nginx 层໘实现҃ऀࡏ业务逻辑层 ໘实现 Saturday, September 14, 13

  59. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ Saturday, September 14, 13

  60. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ • 时间: 䇖࢝时间ɼ总໣时 Saturday, September 14, 13

  61. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ • 时间: 䇖࢝时间ɼ总໣时 • 谁: ༻户Id, Session-Id,

    Request-Id Saturday, September 14, 13

  62. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ • 时间: 䇖࢝时间ɼ总໣时 • 谁: ༻户Id, Session-Id,

    Request-Id • ၏ॄ㜮: URL, ํ๏ɼXHR?, format, ࢀ਺(஫ҙ อ护ີ码) Saturday, September 14, 13

  63. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ • 时间: 䇖࢝时间ɼ总໣时 • 谁: ༻户Id, Session-Id,

    Request-Id • ၏ॄ㜮: URL, ํ๏ɼXHR?, format, ࢀ਺(஫ҙ อ护ີ码) • 结Ռੋॄ㜮ʁ Saturday, September 14, 13

  64. ෰务୺೔ࢤ؃ॄ㜮ʁ • 对ଖଞ෰务త请ٻ • 邮݅ɼ୹৴ɼଖଞ෰务... • 记录请ٻ详৘࿨໣时 Saturday, September 14,

    13

  65. ୠੋ经ৗ຅༗଍䭧త೔ࢤ Saturday, September 14, 13

  66. wireshark tcpdump • ፛แ޻۩ • wireshark: ༗ք໘ • tcpdump: sudo

    tcpdump -n -s 4096 -w 1.log port 80 Saturday, September 14, 13

  67. ଖଞ޻۩ • strace/dtruss: ܥ统调༻᪑᪨޻۩ • lsof: ྻग़จ݅ଧ䇖৘ႎ • valgrind: 查಺ଘᔔ࿐

    • ltrace: 查询库调༻ Saturday, September 14, 13

  68. 总结 • զ们ਪਸ௨过Ұ䝅ܥ统తํ๏དྷ෼ੳ问题ɼ寻ፙ 问题తࠜݯ • զ们൓对୞ᯪ试错๏དྷղႊ问题 • ೳ࠶现త Bug ੋ޷

    Bug, ೗Ռෆೳ࠶现ɼ໵ཁ፤౸ 对应త᠓络请ٻ࿨೔ࢤ • ೗Ռ这࣍ղႊෆྃ Bug, ಹ㜮बվળ㟬త೔ࢤɼ䉯 อԼ࣍ Bug ग़现త时ީೳղႊଞ Saturday, September 14, 13

  69. զ们团队 Saturday, September 14, 13

  70. Q&A Saturday, September 14, 13