Upgrade to Pro — share decks privately, control downloads, hide ads and more …

出了问题不要靠猜

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for LI Daobing LI Daobing
September 14, 2013
Programming
4.1k
40

 出了问题不要靠猜

Avatar for LI Daobing

LI Daobing

September 14, 2013

More Decks by LI Daobing

See All by LI Daobing
How to attack TLS in PQC decade, part I
Avatar for LI Daobing lidaobing
0
47
HTTP协议相关的若干安全问题
Avatar for LI Daobing lidaobing
9
1.2k
Debian & Packaging
Avatar for LI Daobing lidaobing
1
580
Java 质量保障
Avatar for LI Daobing lidaobing
3
300
OAuth: How and Why?
Avatar for LI Daobing lidaobing
1
160
从 Struts 迁移到 Spring MVC，以及为什么？
Avatar for LI Daobing lidaobing
2
670
glusterfs 文件系统
Avatar for LI Daobing lidaobing
2
210
如何学习 Shell
Avatar for LI Daobing lidaobing
3
340

Other Decks in Programming

See All in Programming
Datadog × OpenTelemetry 入門と実践のあいだ
Avatar for Max Kento kn_to_maxpno
1
160
LLM本来の能力を解き放つサンドボックス技術とAI民主化への適用
Avatar for Yuku Kotani yukukotani
3
4.1k
正しくソフトウェアを作る、前提を疑うための認知の視点 / doubt-premise
Avatar for MinoDriven minodriven
21
6.7k
Honoでのサプライチェーン侵害対策 〜 3つのライブラリに学ぶ
Avatar for Yusuke Wada yusukebe
6
1.2k
TypeScript+Orvalで実現する型安全かつ堅牢でスケーラブルなマルチチャネル通知基盤 / TSKaigi Night talks ~after conference~
Avatar for doriven d0riven
0
340
Hunting Vulnerabilities in Symfony with LLMs
Avatar for Vincent Amstoutz vinceamstoutz
0
540
DynamoDBには集計系のクエリがないけどなんとかしたい
Avatar for 村上優稀 musan
1
140
Contextとはなにか
Avatar for chiroruxx chiroruxx
1
330
Oxcを導入して開発体験が向上した話
Avatar for Yuji Yamaguchi yug1224
4
320
LLMによるContent Moderationの本番運用の裏側と品質担保への挑戦
Avatar for suikabar suikabar
3
690
技術記事、AIに書かせるか、自分で書くか？ 〜それでも私が自分の手で書く理由〜 / #QiitaConference
Avatar for Junichi Ito jnchito
2
1.4k
AIとASP.NET Coreで雑Webアプリを作った話
Avatar for Mayuki Sawatari mayuki
0
640

Featured

See All Featured
Mozcon NYC 2025: Stop Losing SEO Traffic
Avatar for Sam Torres samtorres
1
250
Building Adaptive Systems
Avatar for Chris Keathley keathley
44
3.1k
Build The Right Thing And Hit Your Dates
Avatar for Maggie C. maggiecrowley
39
3.2k
Unlocking the hidden potential of vector embeddings in international SEO
Avatar for Frank van Dijk frankvandijk
0
840
svc-hook: hooking system calls on ARM64 by binary rewriting
Avatar for Akira Moroo retrage
2
300
From Legacy to Launchpad: Building Startup-Ready Communities
Avatar for Dug Song dugsong
0
230
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
Avatar for Eileen M. Uchitelle eileencodes
28
3.5k
The Pragmatic Product Professional
Avatar for Laura Van Doore lauravandoore
37
7.3k
CSS Pre-Processors: Stylus, Less & Sass
Avatar for Bermon Painter bermonpainter
360
30k
Put a Button on it: Removing Barriers to Going Fast.
Avatar for kastner kastner
60
4.3k
Documentation Writing (for coders)
Avatar for Carmen Chung carmenintech
77
5.4k
Breaking role norms: Why Content Design is so much more than writing copy - Taylor Woolridge
Avatar for UX Y'all uxyall
0
320

Transcript

  1. ग़ྃ问题ෆཁᯪᘃ LI Daobing <[email protected]> ࣣڇӠଘ储 2013-09-14 Saturday, September 14, 13

  2. • ཥಓฌ <[email protected]> • Debian Developer • 2004 ೥䇖࢝઀৮ Debian

    • qterm, ibus, scim ౳软݅แಘ维护ਓ员 • 䇖ݯ爱޷ऀ • தจ维جඦՊલ؅ཧ员 • ࢀ༩维护 iso-codes, translationproject.org • python-lunardate, capistrano-scm-jenkins ࡞ऀ • manpages-zh, douban-ruby, iptux 现೚维护ਓ员 • զ对䇖ݯత؃๏ • ࢀ༩Ұ࿽项໨ɼ贡ݙզత时间ੋҼ为认ಉଞతཧ೦ɼࣕෆੋ૝վ଄ଞతཧ೦ • Github: http://github.com/lidaobing https://speakerdeck.com/lidaobing Saturday, September 14, 13

  3. ໨录 • ဓҰ࿽ Bug 讲ى • զ؃౸తཚ৅ • զਪᣑతղႊํҊ Saturday,

    September 14, 13

  4. Ұ࿽حոత Bug • ᠓盘త্传෰务 • Windows Լ IE, Chrome 浏览ثԼਖ਼ৗ

    • Linux Լ Firefox, Chrome 浏览ثԼਖ਼ৗ • Windows Լత Firefox ෆਖ਼ৗ, ্传ࣦ 败ɻ Saturday, September 14, 13

  5. ೗ԿఆҐBug Saturday, September 14, 13

  6. ೗ԿఆҐBug • टઌ查؃浏览ثత߇੍୆ɼ观࡯ੋ൱༗错误(౎຅༗) Saturday, September 14, 13

  7. ೗ԿఆҐBug • टઌ查؃浏览ثత߇੍୆ɼ观࡯ੋ൱༗错误(౎຅༗) • ༻ Wireshark ፊ获请ٻแɼ发现ࣦ败తҊྫ෰务୺ฦ ճ412(௨ৗ为200) Saturday, September

    14, 13

  8. ೗ԿఆҐBug • टઌ查؃浏览ثత߇੍୆ɼ观࡯ੋ൱༗错误(౎຅༗) • ༻ Wireshark ፊ获请ٻแɼ发现ࣦ败తҊྫ෰务୺ฦ ճ412(௨ৗ为200) • 对ൺ

    HTTP Requestɼ发现ࣦ败Ҋྫత HTTP 头༗ “Content-length”ɼࣕଖଞ৘ႎԼ为 “Content- Length” Saturday, September 14, 13

  9. ೗ԿఆҐBug • टઌ查؃浏览ثత߇੍୆ɼ观࡯ੋ൱༗错误(౎຅༗) • ༻ Wireshark ፊ获请ٻแɼ发现ࣦ败తҊྫ෰务୺ฦ ճ412(௨ৗ为200) • 对ൺ

    HTTP Requestɼ发现ࣦ败Ҋྫత HTTP 头༗ “Content-length”ɼࣕଖଞ৘ႎԼ为 “Content- Length” • ༻ telnet 发ૹෆಉత请ٻདྷ验证结论 Saturday, September 14, 13

  10. ೗ԿఆҐBug Saturday, September 14, 13

  11. ೗ԿఆҐBug • 䉯认ݪҼੋ෰务୺对 Content-Length త େখࣸහײɼ༩标।ෆූɼ଄੒ྃ这࿽ Bug Saturday, September 14,

    13

  12. ೗ԿఆҐBug • 䉯认ݪҼੋ෰务୺对 Content-Length త େখࣸහײɼ༩标।ෆූɼ଄੒ྃ这࿽ Bug • ௨஌෰务୺䇖发ਓ员ਚշम䐾 Saturday,

    September 14, 13

  13. ޷Bug/ᆀBug Saturday, September 14, 13

  14. ޷Bug/ᆀBug • ೳ࠶现త Bug बੋ޷ Bug Saturday, September 14, 13

  15. ޷Bug/ᆀBug • ೳ࠶现త Bug बੋ޷ Bug • ೗Ռෆೳ࠶现ɼೳ፤౸䇗ৗ栈҃ऀ详细 ೔ࢤత໵ੋ޷Bug Saturday,

    September 14, 13

  16. ޷Bug/ᆀBug • ೳ࠶现త Bug बੋ޷ Bug • ೗Ռෆೳ࠶现ɼೳ፤౸䇗ৗ栈҃ऀ详细 ೔ࢤత໵ੋ޷Bug •

    对ဋᆀ Bug, ଞత࠷େ༻处बੋಜଅ㟬补 ॆ೔ࢤ Saturday, September 14, 13

  17. 㠥Ұ࿽حոతBug • ဓެ࢘访问᠓᜾༗12sԆ迟 • ௚઀ ping 㠳应኷շ • ဓଖଞ᠓络དྷݯ访问຅༗问题 •

    监߇຅༗问题, ໵຅༗٬户དྷ๊ԇࠑࣄ Saturday, September 14, 13

  18. ઌᘃᘃ؃ Saturday, September 14, 13

  19. ઌᘃᘃ؃ • ެ࢘࿏༝ग़问题ྃʁ • ॏ启ҰԼ࿏༝试试؃ʁ • ୠ为ॄ㜮୞༗访问զ们ࣗݾ᠓᜾ձग़ࣄʁ Saturday, September 14,

    13

  20. ઌᘃᘃ؃ • ެ࢘࿏༝ग़问题ྃʁ • ॏ启ҰԼ࿏༝试试؃ʁ • ୠ为ॄ㜮୞༗访问զ们ࣗݾ᠓᜾ձग़ࣄʁ • ਓ඼问题ʁ •

    ၏ఔং员还ੋ།෺Ұ఺ൺ较޷ Saturday, September 14, 13

  21. ઌᘃᘃ؃ • ެ࢘࿏༝ग़问题ྃʁ • ॏ启ҰԼ࿏༝试试؃ʁ • ୠ为ॄ㜮୞༗访问զ们ࣗݾ᠓᜾ձग़ࣄʁ • ਓ඼问题ʁ •

    ၏ఔং员还ੋ།෺Ұ఺ൺ较޷ • ෆ؅ɼ൓ਖ਼຅ਓ๊ԇ这ࣄ • 99% త༻户ձ۰౸问题时ෆձ报ࠂ Saturday, September 14, 13

  22. ෼ੳ Saturday, September 14, 13

  23. ෼ੳ • tcpdump፛แ • ٬户୺: 发ྃ6࿽ SYN แɼ౸࠷޳Ұ࿽࠽Ꮕ౸ SYN/ACK •

    ෰务୺: 䉯实Ꮕ౸ྃ6࿽ SYN แ, ࣕ׌䉯实લ໘5 ࿽ SYN แ౎຅༗ճ䐾 • ဓଖଞ᠓络访问ɼ䉯实ࡏट࿽ SYN แ೭޳ฦճ Saturday, September 14, 13

  24. ෼ੳ • tcpdump፛แ • ٬户୺: 发ྃ6࿽ SYN แɼ౸࠷޳Ұ࿽࠽Ꮕ౸ SYN/ACK •

    ෰务୺: 䉯实Ꮕ౸ྃ6࿽ SYN แ, ࣕ׌䉯实લ໘5 ࿽ SYN แ౎຅༗ճ䐾 • ဓଖଞ᠓络访问ɼ䉯实ࡏट࿽ SYN แ೭޳ฦճ • 结论: 问题᪑࿏༝/线࿏Ꮰ䎔ɼ䉯实ੋ෰务୺త问题 Saturday, September 14, 13

  25. ෼ੳ Saturday, September 14, 13

  26. ෼ੳ • ࠶ॏ৽෼ੳ tcpdump త记录ɼ发现લ5࿽ SYN แ带ྃ timestamp, ୈ6࿽຅带 •

    尝试䎔ᎃ٬户୺ TCP timestamp, ॠ间㠳应 • 尝试䎔ᎃ෰务୺ TCP timestamp, ॠ间㠳应 Saturday, September 14, 13

  27. ෼ੳ • ࠶ॏ৽෼ੳ tcpdump త记录ɼ发现લ5࿽ SYN แ带ྃ timestamp, ୈ6࿽຅带 •

    尝试䎔ᎃ٬户୺ TCP timestamp, ॠ间㠳应 • 尝试䎔ᎃ෰务୺ TCP timestamp, ॠ间㠳应 • 结论: Bug ༩ timestamp ૬䎔ɼୠနવᏠ๏ ղ释为ॄ㜮ଖଞ᠓络຅问题 Saturday, September 14, 13

  28. ղႊํҊA Saturday, September 14, 13

  29. ղႊํҊA • 䎔ᎃ෰务୺త TCP timestamp • TCP timestamp ձӨ㠳传输଎౓ɼୠӨ 㠳ෆେ

    Saturday, September 14, 13

  30. ղႊํҊA • 䎔ᎃ෰务୺త TCP timestamp • TCP timestamp ձӨ㠳传输଎౓ɼୠӨ 㠳ෆେ

    • ٙ问နવଘࡏɼ为ॄ㜮෰务ثෆ㠳应๭ ࠣ timestamp แ, ༗时ީຢೳਖ਼ৗ㠳应 Saturday, September 14, 13

  31. ෼ੳ Saturday, September 14, 13

  32. ෼ੳ • ؃಺֩ݯ码 (CTO 亲ࣗૢ౛) • ೖޱصث༗େྔత൓޲୅ཧɼ୺ޱ经ৗෆ䭧༻ɼॴҎ䇖 启ྃTIME_WAIT ୺ޱॏ༻ •

    䇖启ྃ TIME_WAIT ୺ޱॏ༻޳ɼ෰务୺ཁٻಉҰ࿽IPత SYN แ timestamp ඞ须ੋ顺ংత • 办ެ᠓络ੋ಺᠓, ኂ׌对զ们׭᠓访问኷频ൟɼ导கނো 发ੜ Saturday, September 14, 13

  33. ෼ੳ • ؃಺֩ݯ码 (CTO 亲ࣗૢ౛) • ೖޱصث༗େྔత൓޲୅ཧɼ୺ޱ经ৗෆ䭧༻ɼॴҎ䇖 启ྃTIME_WAIT ୺ޱॏ༻ •

    䇖启ྃ TIME_WAIT ୺ޱॏ༻޳ɼ෰务୺ཁٻಉҰ࿽IPత SYN แ timestamp ඞ须ੋ顺ংత • 办ެ᠓络ੋ಺᠓, ኂ׌对զ们׭᠓访问኷频ൟɼ导கނো 发ੜ • ؃དྷፙ౸ਅ实తݪҼྃ, ༗຅༗ߋ޷஍ղႊํҊʁ Saturday, September 14, 13

  34. ղႊํҊB • nginx ߴ൛ຊቮ经ࢧ࣋ keep-alive • ঋ级 nginx, 启༻ keep-alive,

    ߱௿୺ޱ઎༻ • 䎔闭୺ޱॏ༻ɼኂՃ㖘୺ޱ਺త监߇࿨ 报ܯ Saturday, September 14, 13

  35. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 Saturday, September 14, 13

  36. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 • 换࿽浏览ث㠧 Saturday, September 14,

    13

  37. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 • 换࿽浏览ث㠧 • ॏ启, ਗ਼

    cookie, ਗ਼ cache, ॏ৽ొ录 Saturday, September 14, 13

  38. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 • 换࿽浏览ث㠧 • ॏ启, ਗ਼

    cookie, ਗ਼ cache, ॏ৽ొ录 • 现场ඃഁᆀಘׯׯ净净 Saturday, September 14, 13

  39. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 • 换࿽浏览ث㠧 • ॏ启, ਗ਼

    cookie, ਗ਼ cache, ॏ৽ొ录 • 现场ඃഁᆀಘׯׯ净净 • 对ဋಹࠣෆқ࠶现త Bug, ब损ࣦྃҰ࣍म 䐾తصձ Saturday, September 14, 13

  40. զॴ؃౸తཚ৅ • 尝试༻试错๏ղႊॴ༗问题 Saturday, September 14, 13

  41. զॴ؃౸తཚ৅ • 尝试༻试错๏ղႊॴ༗问题 • ঋ级ґ赖แ, ঋ级ᎎ݅, ... Saturday, September 14,

    13

  42. զॴ؃౸తཚ৅ • 尝试༻试错๏ղႊॴ༗问题 • ঋ级ґ赖แ, ঋ级ᎎ݅, ... • ਵศᘃҰ࿽ݪҼɼվ䫆۟୅码ɼવ޳ ॏ৽测试

    Saturday, September 14, 13

  43. զॴ؃౸తཚ৅ • ᠍গ൓ল Saturday, September 14, 13

  44. զॴ؃౸తཚ৅ • ᠍গ൓ল • 寻ፙ౸ workaround ศ认为问题ղႊྃ Saturday, September 14,

    13

  45. զॴ؃౸తཚ৅ • ᠍গ൓ল • 寻ፙ౸ workaround ศ认为问题ղႊྃ • ຅༗༻测试ݻԽ Bug,

    ༰қ产ੜճ归Bug Saturday, September 14, 13

  46. զॴ؃౸తཚ৅ • ᠍গ൓ল • 寻ፙ౸ workaround ศ认为问题ղႊྃ • ຅༗༻测试ݻԽ Bug,

    ༰қ产ੜճ归Bug • Ұ࿽ Bug Մೳࡏଟ处ग़现ɼ຅༗尝试፺ࡧଖଞ༗ Bug త஍ํ Saturday, September 14, 13

  47. զॴ؃౸తཚ৅ • ᠍গ൓ল • 寻ፙ౸ workaround ศ认为问题ղႊྃ • ຅༗༻测试ݻԽ Bug,

    ༰қ产ੜճ归Bug • Ұ࿽ Bug Մೳࡏଟ处ग़现ɼ຅༗尝试፺ࡧଖଞ༗ Bug త஍ํ • ຅༗௨过补ॆ೔ࢤ౳खஈདྷ߱௿೔޳ఆҐBugత难 ౓ Saturday, September 14, 13

  48. զॴ؃౸తཚ৅ • ւԸ๏则: 㑌Ұى严ॏࣄނతഎ޳ɼඞવ ༗29࣍轻ඍࣄނ࿨300ىະ਱ઌஹҎٴ 1000ىࣄނ隐ױ Saturday, September 14, 13

  49. զ们ਪᣑతํ๏ • ग़ྃ问题ෆཁᯪᘃ • 浏览ث • ෰务୺೔ࢤ • ፛แ޻۩ Saturday,

    September 14, 13

  50. 浏览ث؃ॄ㜮ʁ Saturday, September 14, 13

  51. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? Saturday, September 14, 13

  52. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? • ᠓络请ٻੋ൱发ग़ʁ Saturday, September 14,

    13

  53. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? • ᠓络请ٻੋ൱发ग़ʁ • 发ग़త请ٻੋ൱ਖ਼䉯ɿ URL,

    ํ๏, ࢀ਺, Accept, Cookie Saturday, September 14, 13

  54. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? • ᠓络请ٻੋ൱发ग़ʁ • 发ग़త请ٻੋ൱ਖ਼䉯ɿ URL,

    ํ๏, ࢀ਺, Accept, Cookie • ظ๬తฦճ值ੋॄ㜮ʁ Saturday, September 14, 13

  55. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? • ᠓络请ٻੋ൱发ग़ʁ • 发ग़త请ٻੋ൱ਖ਼䉯ɿ URL,

    ํ๏, ࢀ਺, Accept, Cookie • ظ๬తฦճ值ੋॄ㜮ʁ • Request-Id Saturday, September 14, 13

  56. Request-Id • 对㑌࣍请ٻ产ੜҰ࿽།ҰతId Saturday, September 14, 13

  57. Request-Id • 对㑌࣍请ٻ产ੜҰ࿽།ҰతId • 该 Id Ҏ HTTP Response Header

    తํࣜ 发ૹ౸٬户୺ Saturday, September 14, 13

  58. Request-Id • 对㑌࣍请ٻ产ੜҰ࿽།ҰతId • 该 Id Ҏ HTTP Response Header

    తํࣜ 发ૹ౸٬户୺ • ՄҎࡏ nginx 层໘实现҃ऀࡏ业务逻辑层 ໘实现 Saturday, September 14, 13

  59. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ Saturday, September 14, 13

  60. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ • 时间: 䇖࢝时间ɼ总໣时 Saturday, September 14, 13

  61. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ • 时间: 䇖࢝时间ɼ总໣时 • 谁: ༻户Id, Session-Id,

    Request-Id Saturday, September 14, 13

  62. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ • 时间: 䇖࢝时间ɼ总໣时 • 谁: ༻户Id, Session-Id,

    Request-Id • ၏ॄ㜮: URL, ํ๏ɼXHR?, format, ࢀ਺(஫ҙ อ护ີ码) Saturday, September 14, 13

  63. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ • 时间: 䇖࢝时间ɼ总໣时 • 谁: ༻户Id, Session-Id,

    Request-Id • ၏ॄ㜮: URL, ํ๏ɼXHR?, format, ࢀ਺(஫ҙ อ护ີ码) • 结Ռੋॄ㜮ʁ Saturday, September 14, 13

  64. ෰务୺೔ࢤ؃ॄ㜮ʁ • 对ଖଞ෰务త请ٻ • 邮݅ɼ୹৴ɼଖଞ෰务... • 记录请ٻ详৘࿨໣时 Saturday, September 14,

    13

  65. ୠੋ经ৗ຅༗଍䭧త೔ࢤ Saturday, September 14, 13

  66. wireshark tcpdump • ፛แ޻۩ • wireshark: ༗ք໘ • tcpdump: sudo

    tcpdump -n -s 4096 -w 1.log port 80 Saturday, September 14, 13

  67. ଖଞ޻۩ • strace/dtruss: ܥ统调༻᪑᪨޻۩ • lsof: ྻग़จ݅ଧ䇖৘ႎ • valgrind: 查಺ଘᔔ࿐

    • ltrace: 查询库调༻ Saturday, September 14, 13

  68. 总结 • զ们ਪਸ௨过Ұ䝅ܥ统తํ๏དྷ෼ੳ问题ɼ寻ፙ 问题తࠜݯ • զ们൓对୞ᯪ试错๏དྷղႊ问题 • ೳ࠶现త Bug ੋ޷

    Bug, ೗Ռෆೳ࠶现ɼ໵ཁ፤౸ 对应త᠓络请ٻ࿨೔ࢤ • ೗Ռ这࣍ղႊෆྃ Bug, ಹ㜮बվળ㟬త೔ࢤɼ䉯 อԼ࣍ Bug ग़现త时ީೳղႊଞ Saturday, September 14, 13

  69. զ们团队 Saturday, September 14, 13

  70. Q&A Saturday, September 14, 13