Upgrade to Pro — share decks privately, control downloads, hide ads and more …

出了问题不要靠猜

03dd26c9abda17bc6e3781c77e21ea2e?s=47 LI Daobing
September 14, 2013

 出了问题不要靠猜

03dd26c9abda17bc6e3781c77e21ea2e?s=128

LI Daobing

September 14, 2013
Tweet

Transcript

  1. ग़ྃ问题ෆཁᯪᘃ LI Daobing <lidaobing@gmail.com> ࣣڇӠଘ储 2013-09-14 Saturday, September 14, 13

  2. • ཥಓฌ <lidaobing@gmail.com> • Debian Developer • 2004 ೥䇖࢝઀৮ Debian

    • qterm, ibus, scim ౳软݅แಘ维护ਓ员 • 䇖ݯ爱޷ऀ • தจ维جඦՊલ؅ཧ员 • ࢀ༩维护 iso-codes, translationproject.org • python-lunardate, capistrano-scm-jenkins ࡞ऀ • manpages-zh, douban-ruby, iptux 现೚维护ਓ员 • զ对䇖ݯత؃๏ • ࢀ༩Ұ࿽项໨ɼ贡ݙզత时间ੋҼ为认ಉଞతཧ೦ɼࣕෆੋ૝վ଄ଞతཧ೦ • Github: http://github.com/lidaobing https://speakerdeck.com/lidaobing Saturday, September 14, 13
  3. ໨录 • ဓҰ࿽ Bug 讲ى • զ؃౸తཚ৅ • զਪᣑతղႊํҊ Saturday,

    September 14, 13
  4. Ұ࿽حոత Bug • ᠓盘త্传෰务 • Windows Լ IE, Chrome 浏览ثԼਖ਼ৗ

    • Linux Լ Firefox, Chrome 浏览ثԼਖ਼ৗ • Windows Լత Firefox ෆਖ਼ৗ, ্传ࣦ 败ɻ Saturday, September 14, 13
  5. ೗ԿఆҐBug Saturday, September 14, 13

  6. ೗ԿఆҐBug • टઌ查؃浏览ثత߇੍୆ɼ观࡯ੋ൱༗错误(౎຅༗) Saturday, September 14, 13

  7. ೗ԿఆҐBug • टઌ查؃浏览ثత߇੍୆ɼ观࡯ੋ൱༗错误(౎຅༗) • ༻ Wireshark ፊ获请ٻแɼ发现ࣦ败తҊྫ෰务୺ฦ ճ412(௨ৗ为200) Saturday, September

    14, 13
  8. ೗ԿఆҐBug • टઌ查؃浏览ثత߇੍୆ɼ观࡯ੋ൱༗错误(౎຅༗) • ༻ Wireshark ፊ获请ٻแɼ发现ࣦ败తҊྫ෰务୺ฦ ճ412(௨ৗ为200) • 对ൺ

    HTTP Requestɼ发现ࣦ败Ҋྫత HTTP 头༗ “Content-length”ɼࣕଖଞ৘ႎԼ为 “Content- Length” Saturday, September 14, 13
  9. ೗ԿఆҐBug • टઌ查؃浏览ثత߇੍୆ɼ观࡯ੋ൱༗错误(౎຅༗) • ༻ Wireshark ፊ获请ٻแɼ发现ࣦ败తҊྫ෰务୺ฦ ճ412(௨ৗ为200) • 对ൺ

    HTTP Requestɼ发现ࣦ败Ҋྫత HTTP 头༗ “Content-length”ɼࣕଖଞ৘ႎԼ为 “Content- Length” • ༻ telnet 发ૹෆಉత请ٻདྷ验证结论 Saturday, September 14, 13
  10. ೗ԿఆҐBug Saturday, September 14, 13

  11. ೗ԿఆҐBug • 䉯认ݪҼੋ෰务୺对 Content-Length త େখࣸහײɼ༩标।ෆූɼ଄੒ྃ这࿽ Bug Saturday, September 14,

    13
  12. ೗ԿఆҐBug • 䉯认ݪҼੋ෰务୺对 Content-Length త େখࣸහײɼ༩标।ෆූɼ଄੒ྃ这࿽ Bug • ௨஌෰务୺䇖发ਓ员ਚշम䐾 Saturday,

    September 14, 13
  13. ޷Bug/ᆀBug Saturday, September 14, 13

  14. ޷Bug/ᆀBug • ೳ࠶现త Bug बੋ޷ Bug Saturday, September 14, 13

  15. ޷Bug/ᆀBug • ೳ࠶现త Bug बੋ޷ Bug • ೗Ռෆೳ࠶现ɼೳ፤౸䇗ৗ栈҃ऀ详细 ೔ࢤత໵ੋ޷Bug Saturday,

    September 14, 13
  16. ޷Bug/ᆀBug • ೳ࠶现త Bug बੋ޷ Bug • ೗Ռෆೳ࠶现ɼೳ፤౸䇗ৗ栈҃ऀ详细 ೔ࢤత໵ੋ޷Bug •

    对ဋᆀ Bug, ଞత࠷େ༻处बੋಜଅ㟬补 ॆ೔ࢤ Saturday, September 14, 13
  17. 㠥Ұ࿽حոతBug • ဓެ࢘访问᠓᜾༗12sԆ迟 • ௚઀ ping 㠳应኷շ • ဓଖଞ᠓络དྷݯ访问຅༗问题 •

    监߇຅༗问题, ໵຅༗٬户དྷ๊ԇࠑࣄ Saturday, September 14, 13
  18. ઌᘃᘃ؃ Saturday, September 14, 13

  19. ઌᘃᘃ؃ • ެ࢘࿏༝ग़问题ྃʁ • ॏ启ҰԼ࿏༝试试؃ʁ • ୠ为ॄ㜮୞༗访问զ们ࣗݾ᠓᜾ձग़ࣄʁ Saturday, September 14,

    13
  20. ઌᘃᘃ؃ • ެ࢘࿏༝ग़问题ྃʁ • ॏ启ҰԼ࿏༝试试؃ʁ • ୠ为ॄ㜮୞༗访问զ们ࣗݾ᠓᜾ձग़ࣄʁ • ਓ඼问题ʁ •

    ၏ఔং员还ੋ།෺Ұ఺ൺ较޷ Saturday, September 14, 13
  21. ઌᘃᘃ؃ • ެ࢘࿏༝ग़问题ྃʁ • ॏ启ҰԼ࿏༝试试؃ʁ • ୠ为ॄ㜮୞༗访问զ们ࣗݾ᠓᜾ձग़ࣄʁ • ਓ඼问题ʁ •

    ၏ఔং员还ੋ།෺Ұ఺ൺ较޷ • ෆ؅ɼ൓ਖ਼຅ਓ๊ԇ这ࣄ • 99% త༻户ձ۰౸问题时ෆձ报ࠂ Saturday, September 14, 13
  22. ෼ੳ Saturday, September 14, 13

  23. ෼ੳ • tcpdump፛แ • ٬户୺: 发ྃ6࿽ SYN แɼ౸࠷޳Ұ࿽࠽Ꮕ౸ SYN/ACK •

    ෰务୺: 䉯实Ꮕ౸ྃ6࿽ SYN แ, ࣕ׌䉯实લ໘5 ࿽ SYN แ౎຅༗ճ䐾 • ဓଖଞ᠓络访问ɼ䉯实ࡏट࿽ SYN แ೭޳ฦճ Saturday, September 14, 13
  24. ෼ੳ • tcpdump፛แ • ٬户୺: 发ྃ6࿽ SYN แɼ౸࠷޳Ұ࿽࠽Ꮕ౸ SYN/ACK •

    ෰务୺: 䉯实Ꮕ౸ྃ6࿽ SYN แ, ࣕ׌䉯实લ໘5 ࿽ SYN แ౎຅༗ճ䐾 • ဓଖଞ᠓络访问ɼ䉯实ࡏट࿽ SYN แ೭޳ฦճ • 结论: 问题᪑࿏༝/线࿏Ꮰ䎔ɼ䉯实ੋ෰务୺త问题 Saturday, September 14, 13
  25. ෼ੳ Saturday, September 14, 13

  26. ෼ੳ • ࠶ॏ৽෼ੳ tcpdump త记录ɼ发现લ5࿽ SYN แ带ྃ timestamp, ୈ6࿽຅带 •

    尝试䎔ᎃ٬户୺ TCP timestamp, ॠ间㠳应 • 尝试䎔ᎃ෰务୺ TCP timestamp, ॠ间㠳应 Saturday, September 14, 13
  27. ෼ੳ • ࠶ॏ৽෼ੳ tcpdump త记录ɼ发现લ5࿽ SYN แ带ྃ timestamp, ୈ6࿽຅带 •

    尝试䎔ᎃ٬户୺ TCP timestamp, ॠ间㠳应 • 尝试䎔ᎃ෰务୺ TCP timestamp, ॠ间㠳应 • 结论: Bug ༩ timestamp ૬䎔ɼୠနવᏠ๏ ղ释为ॄ㜮ଖଞ᠓络຅问题 Saturday, September 14, 13
  28. ղႊํҊA Saturday, September 14, 13

  29. ղႊํҊA • 䎔ᎃ෰务୺త TCP timestamp • TCP timestamp ձӨ㠳传输଎౓ɼୠӨ 㠳ෆେ

    Saturday, September 14, 13
  30. ղႊํҊA • 䎔ᎃ෰务୺త TCP timestamp • TCP timestamp ձӨ㠳传输଎౓ɼୠӨ 㠳ෆେ

    • ٙ问နવଘࡏɼ为ॄ㜮෰务ثෆ㠳应๭ ࠣ timestamp แ, ༗时ީຢೳਖ਼ৗ㠳应 Saturday, September 14, 13
  31. ෼ੳ Saturday, September 14, 13

  32. ෼ੳ • ؃಺֩ݯ码 (CTO 亲ࣗૢ౛) • ೖޱصث༗େྔత൓޲୅ཧɼ୺ޱ经ৗෆ䭧༻ɼॴҎ䇖 启ྃTIME_WAIT ୺ޱॏ༻ •

    䇖启ྃ TIME_WAIT ୺ޱॏ༻޳ɼ෰务୺ཁٻಉҰ࿽IPత SYN แ timestamp ඞ须ੋ顺ংత • 办ެ᠓络ੋ಺᠓, ኂ׌对զ们׭᠓访问኷频ൟɼ导கނো 发ੜ Saturday, September 14, 13
  33. ෼ੳ • ؃಺֩ݯ码 (CTO 亲ࣗૢ౛) • ೖޱصث༗େྔత൓޲୅ཧɼ୺ޱ经ৗෆ䭧༻ɼॴҎ䇖 启ྃTIME_WAIT ୺ޱॏ༻ •

    䇖启ྃ TIME_WAIT ୺ޱॏ༻޳ɼ෰务୺ཁٻಉҰ࿽IPత SYN แ timestamp ඞ须ੋ顺ংత • 办ެ᠓络ੋ಺᠓, ኂ׌对զ们׭᠓访问኷频ൟɼ导கނো 发ੜ • ؃དྷፙ౸ਅ实తݪҼྃ, ༗຅༗ߋ޷஍ղႊํҊʁ Saturday, September 14, 13
  34. ղႊํҊB • nginx ߴ൛ຊቮ经ࢧ࣋ keep-alive • ঋ级 nginx, 启༻ keep-alive,

    ߱௿୺ޱ઎༻ • 䎔闭୺ޱॏ༻ɼኂՃ㖘୺ޱ਺త监߇࿨ 报ܯ Saturday, September 14, 13
  35. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 Saturday, September 14, 13

  36. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 • 换࿽浏览ث㠧 Saturday, September 14,

    13
  37. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 • 换࿽浏览ث㠧 • ॏ启, ਗ਼

    cookie, ਗ਼ cache, ॏ৽ొ录 Saturday, September 14, 13
  38. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 • 换࿽浏览ث㠧 • ॏ启, ਗ਼

    cookie, ਗ਼ cache, ॏ৽ొ录 • 现场ඃഁᆀಘׯׯ净净 Saturday, September 14, 13
  39. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 • 换࿽浏览ث㠧 • ॏ启, ਗ਼

    cookie, ਗ਼ cache, ॏ৽ొ录 • 现场ඃഁᆀಘׯׯ净净 • 对ဋಹࠣෆқ࠶现త Bug, ब损ࣦྃҰ࣍म 䐾తصձ Saturday, September 14, 13
  40. զॴ؃౸తཚ৅ • 尝试༻试错๏ղႊॴ༗问题 Saturday, September 14, 13

  41. զॴ؃౸తཚ৅ • 尝试༻试错๏ղႊॴ༗问题 • ঋ级ґ赖แ, ঋ级ᎎ݅, ... Saturday, September 14,

    13
  42. զॴ؃౸తཚ৅ • 尝试༻试错๏ղႊॴ༗问题 • ঋ级ґ赖แ, ঋ级ᎎ݅, ... • ਵศᘃҰ࿽ݪҼɼվ䫆۟୅码ɼવ޳ ॏ৽测试

    Saturday, September 14, 13
  43. զॴ؃౸తཚ৅ • ᠍গ൓ল Saturday, September 14, 13

  44. զॴ؃౸తཚ৅ • ᠍গ൓ল • 寻ፙ౸ workaround ศ认为问题ղႊྃ Saturday, September 14,

    13
  45. զॴ؃౸తཚ৅ • ᠍গ൓ল • 寻ፙ౸ workaround ศ认为问题ղႊྃ • ຅༗༻测试ݻԽ Bug,

    ༰қ产ੜճ归Bug Saturday, September 14, 13
  46. զॴ؃౸తཚ৅ • ᠍গ൓ল • 寻ፙ౸ workaround ศ认为问题ղႊྃ • ຅༗༻测试ݻԽ Bug,

    ༰қ产ੜճ归Bug • Ұ࿽ Bug Մೳࡏଟ处ग़现ɼ຅༗尝试፺ࡧଖଞ༗ Bug త஍ํ Saturday, September 14, 13
  47. զॴ؃౸తཚ৅ • ᠍গ൓ল • 寻ፙ౸ workaround ศ认为问题ղႊྃ • ຅༗༻测试ݻԽ Bug,

    ༰қ产ੜճ归Bug • Ұ࿽ Bug Մೳࡏଟ处ग़现ɼ຅༗尝试፺ࡧଖଞ༗ Bug త஍ํ • ຅༗௨过补ॆ೔ࢤ౳खஈདྷ߱௿೔޳ఆҐBugత难 ౓ Saturday, September 14, 13
  48. զॴ؃౸తཚ৅ • ւԸ๏则: 㑌Ұى严ॏࣄނతഎ޳ɼඞવ ༗29࣍轻ඍࣄނ࿨300ىະ਱ઌஹҎٴ 1000ىࣄނ隐ױ Saturday, September 14, 13

  49. զ们ਪᣑతํ๏ • ग़ྃ问题ෆཁᯪᘃ • 浏览ث • ෰务୺೔ࢤ • ፛แ޻۩ Saturday,

    September 14, 13
  50. 浏览ث؃ॄ㜮ʁ Saturday, September 14, 13

  51. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? Saturday, September 14, 13

  52. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? • ᠓络请ٻੋ൱发ग़ʁ Saturday, September 14,

    13
  53. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? • ᠓络请ٻੋ൱发ग़ʁ • 发ग़త请ٻੋ൱ਖ਼䉯ɿ URL,

    ํ๏, ࢀ਺, Accept, Cookie Saturday, September 14, 13
  54. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? • ᠓络请ٻੋ൱发ग़ʁ • 发ग़త请ٻੋ൱ਖ਼䉯ɿ URL,

    ํ๏, ࢀ਺, Accept, Cookie • ظ๬తฦճ值ੋॄ㜮ʁ Saturday, September 14, 13
  55. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? • ᠓络请ٻੋ൱发ग़ʁ • 发ग़త请ٻੋ൱ਖ਼䉯ɿ URL,

    ํ๏, ࢀ਺, Accept, Cookie • ظ๬తฦճ值ੋॄ㜮ʁ • Request-Id Saturday, September 14, 13
  56. Request-Id • 对㑌࣍请ٻ产ੜҰ࿽།ҰతId Saturday, September 14, 13

  57. Request-Id • 对㑌࣍请ٻ产ੜҰ࿽།ҰతId • 该 Id Ҏ HTTP Response Header

    తํࣜ 发ૹ౸٬户୺ Saturday, September 14, 13
  58. Request-Id • 对㑌࣍请ٻ产ੜҰ࿽།ҰతId • 该 Id Ҏ HTTP Response Header

    తํࣜ 发ૹ౸٬户୺ • ՄҎࡏ nginx 层໘实现҃ऀࡏ业务逻辑层 ໘实现 Saturday, September 14, 13
  59. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ Saturday, September 14, 13

  60. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ • 时间: 䇖࢝时间ɼ总໣时 Saturday, September 14, 13

  61. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ • 时间: 䇖࢝时间ɼ总໣时 • 谁: ༻户Id, Session-Id,

    Request-Id Saturday, September 14, 13
  62. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ • 时间: 䇖࢝时间ɼ总໣时 • 谁: ༻户Id, Session-Id,

    Request-Id • ၏ॄ㜮: URL, ํ๏ɼXHR?, format, ࢀ਺(஫ҙ อ护ີ码) Saturday, September 14, 13
  63. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ • 时间: 䇖࢝时间ɼ总໣时 • 谁: ༻户Id, Session-Id,

    Request-Id • ၏ॄ㜮: URL, ํ๏ɼXHR?, format, ࢀ਺(஫ҙ อ护ີ码) • 结Ռੋॄ㜮ʁ Saturday, September 14, 13
  64. ෰务୺೔ࢤ؃ॄ㜮ʁ • 对ଖଞ෰务త请ٻ • 邮݅ɼ୹৴ɼଖଞ෰务... • 记录请ٻ详৘࿨໣时 Saturday, September 14,

    13
  65. ୠੋ经ৗ຅༗଍䭧త೔ࢤ Saturday, September 14, 13

  66. wireshark tcpdump • ፛แ޻۩ • wireshark: ༗ք໘ • tcpdump: sudo

    tcpdump -n -s 4096 -w 1.log port 80 Saturday, September 14, 13
  67. ଖଞ޻۩ • strace/dtruss: ܥ统调༻᪑᪨޻۩ • lsof: ྻग़จ݅ଧ䇖৘ႎ • valgrind: 查಺ଘᔔ࿐

    • ltrace: 查询库调༻ Saturday, September 14, 13
  68. 总结 • զ们ਪਸ௨过Ұ䝅ܥ统తํ๏དྷ෼ੳ问题ɼ寻ፙ 问题తࠜݯ • զ们൓对୞ᯪ试错๏དྷղႊ问题 • ೳ࠶现త Bug ੋ޷

    Bug, ೗Ռෆೳ࠶现ɼ໵ཁ፤౸ 对应త᠓络请ٻ࿨೔ࢤ • ೗Ռ这࣍ղႊෆྃ Bug, ಹ㜮बվળ㟬త೔ࢤɼ䉯 อԼ࣍ Bug ग़现త时ީೳղႊଞ Saturday, September 14, 13
  69. զ们团队 Saturday, September 14, 13

  70. Q&A Saturday, September 14, 13