Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Parsing case study in Go / Go Conference mini 2023 Winter IN KYOTO

Parsing case study in Go / Go Conference mini 2023 Winter IN KYOTO

Ken’ichiro Oyama

December 01, 2023
Tweet

More Decks by Ken’ichiro Oyama

Other Decks in Technology

Transcript

  1. Parsing case study in Go খࢁ݈Ұ࿠ / GMO PEPABO inc.

    2023.12.2 Go Conference mini 2023 Winter IN KYOTO 1
  2. 6 ຊηογϣϯʹ͓͚Δύʔε ύʔε • ຊηογϣϯʹ͓͚Δύʔεͱ͸ʮจࣈྻ΍όΠτྻ͔Βҙຯͷ͋Δߏ଄ԽσʔλΛநग़͠׆༻ ͢Δ͜ͱʯΛࢦ͢ɻҰൠతͳʮύʔεʯͷఆٛΑΓ΋গ͠޿͍ղऍɻ • จࣈྻ΍όΠτྻ … string,

    []byte, io.Reader • ߏ଄Խσʔλ … struct, map, slice, chan • ຊηογϣϯͰ͸σγϦΞϥΠζ΍σίʔυ΋ؚΉʢͳͷͰɺ͋·Γʮύʔεʯͷఆٛʹͩ͜ΘΒͳ͍Ͱ΋Β͑Δͱ خ͍͠Ͱ͢🙏 ʣɻ
  3. 10 README.mdͷҰ෦͚ͩࠩ͠ସ͑Δ github.com/k1LoW/repin • README.mdͳͲɺෳ਺ߦʹΘͨΔจࣈྻͷҰ෦͚ͩΛࠩ͠ସ͑ΔͨΊͷπʔϧ/ϥΠϒϥϦ • ։࢝ΩʔϫʔυͱऴྃΩʔϫʔυΛࢦఆ͢ΔͱͦͷΩʔϫʔυؒͷจࣈྻΛࠩ͠ସ͑Δ • ྫ: ։࢝Ωʔϫʔυ

    ```console ɺऴྃΩʔϫʔυ ``` ʹͯ͠ίϚϯυͷϔϧϓͷग़ ྗΛ࠷৽ʹࠩ͠ସ͑ • ॲཧͷಛ௃ • ͻͱ͔ͨ·Γͷෳ਺ߦͷจࣈྻΛೖྗͱͯͦ͠ͷ్தͷจࣈྻΛࠩ͠ସ͑Δ • ෳ਺ߦΛ·ͱΊͯॲཧ͢Δඞཁ͕͋Δ • ܾ·ͬͨߏจͱ͍ͬͨ΋ͷ͸ͳ͍
  4. 11 README.mdͷҰ෦͚ͩࠩ͠ସ͑Δ ύʔεͷ࣮૷ • ࣈ۟ղੳΛbufio.ScannerͰ࣮૷ • ಠࣗbufio.SplitFuncΛॻ͘͜ͱͰΩʔϫʔυ֎ͱΩʔϫʔυؒͰ෼ׂ • ࣈ۟ղੳ݁ՌΛ Scan()

    Ͱड͚औ͍ͬͯ͘ʢ []string ʣ • ॱ൪ʹΩʔϫʔυ֎ -> Ωʔϫʔυؒ -> Ωʔϫʔυ֎ -> … • ͋ͱ͸Ωʔϫʔυؒͷจࣈྻ͚ͩࠩ͠ସ͑ͭͭ૊Έཱͯͯग़ྗ͢Δ͚ͩ
  5. 14 ඪ४ग़ྗʹྲྀΕ͍ͯΔϩάΛύΠϓͰड͚औͬͯ৭෇͚͢Δ github.com/k1LoW/colr • ඪ४ೖྗʹྲྀΕΔϩάʹରͯ͠ࢦఆͨ͠จࣈྻʹϚονͨ͠Β͚ͦͩ͜৭෇͚Λ͢Δπʔϧ • ϩά ”Error” ΍ “status:500”

    ͳͲΛڧௐ͢Δ͜ͱͰϩάΛΈ΍͘͢͢ΔͨΊͷ΋ͷ • tail -F /var/log/nginx/access.log | colr POST GET 404 500 search • ॲཧͷಛ௃ • ೖྗ͸ऴ୺͕ෆ໌ͷετϦʔϜɻॱ࣍ॲཧ͍ͯ͘͠ඞཁ͕͋Δ • ୯ʹ৭෇͚ͳͷͰɺϩάͷߏจΛߟྀ͢Δඞཁ͸ͳ͍ɻ1ߦͮͭॲཧͯ͠ྑ͍ɻ
  6. 16 ඪ४ग़ྗʹྲྀΕ͍ͯΔϩάΛύΠϓͰड͚औͬͯ৭෇͚͢Δ ύʔεͷ࣮૷ʢHandle(context.Context, io.Reader) <-chan stringʣ • bufio.NewReader(os.Stdin) -> ReadString(‘\n’)Ͱߦ୯Ґʹ෼ׂ

    • ߦ͝ͱʹ৭෇͚ ( Paint() ) • ಺෦͸࣮࣭strings.ReplaceAll • ৭෇͚݁ՌΛ chan stringʹྲྀ͢ • ʢ࣮͸ʣp.Handle()͕ඪ४ग़ྗΛड͚औͬ ͯ΋ྑ͍ • ʢ࣮͸ʣchan []byte Ͱ΋ྑ͍
  7. 20 VIEWςʔϒϧͰ࢖༻͞Ε͍ͯΔςʔϒϧͷҰཡΛऔಘ͢Δ ςʔϒϧ໊ͷ൑ఆ • “FROM” ͔ “JOIN” ͷ࣍ͷτʔΫϯ͕ςʔϒϧ໊ ͱ͍͏൑ఆͰे෼ •

    ಺ɺ“WITH” Ͱఆٛ͞ΕͨҰ࣌తͳςʔϒ ϧ໊͚ͩআ͘ • VIEWςʔϒϧͷDDLͷಛ௃Λཧղ͓͚ͯ͠͹શ ͯΛτʔΫϯ൑ผ͢Δඞཁ͸ͳ͍
  8. 22 σʔλϕʔεʹྲྀΕΔύέοτΛղੳͯ͠ߏ଄ԽΫΤϦϩάΛ࡞Δ github.com/k1LoW/tcpdp • σʔλϕʔεʹྲྀΕΔύέοτΛղੳͯ͠σʔλϕʔεͷߏ଄ԽΫΤϦϩάΛ࡞੒͢Δπʔϧ • 3ύλʔϯʹରԠ 1. ΫϥΠΞϯτͱσʔλϕʔεͷؒʹTCPϓϩΩγαʔόͱͯ͠ىಈ 2.

    libpcapΛ࢖ͬͯtcpdumpͱಉ͡࢓૊ΈͰύέοτΛΩϟϓνϟ͢Δ 3. tcpdump͕μϯϓͨ͠pcapϑΝΠϧΛղੳ͢Δ • ॲཧͷಛ௃ • ෳ਺ͷΫϥΠΞϯτ͔Βͷ઀ଓ͕͋Δ • 1ͭͷTCPύέοτʹΫΤϦ͕ऩ·Δͱ͸ݶΒͳ͍ • σʔλϕʔεϓϩτίϧҙຯͷ͋Δ௨৴͸ΫϥΠΞϯτ-αʔόؒͰԿճ͔ԟ෮͢Δ • 3ύλʔϯͷύέοτऔಘʹରԠ͢Δඞཁ͕͋Δ͕ύʔεॲཧ͸ڞ௨Խ͍ͨ͠
  9. 27 ͜͜·Ͱͷֶͼ ࣈ۟ղੳ͸έʔεʹΑͬͯ͸ࡶʹͰ͖Δ • ೖྗΛઌ಄͔ΒॱʹಡΈࠐΜͰ͍͘ͷ͕ϕʔγοΫͳख๏ • τʔΫϯʹ෼ׂ͢ΔͨΊʹ͸ઌಡΈͯ͠͸͡ΊͯτʔΫϯछผΛ൑ఆͰ͖Δ͜ͱ͕ଟ͍ • ઌಡΈʹٻΊΒΕΔཁ͕݅ෳࡶͰ͋Ε͹͋Δ΄Ͳࣈ۟ղੳͷ࣮૷͕೉͘͠ͳΔʢײ૝ʣ •

    ҰํͰ • έʔεʹΑͬͯ͸εϖʔε۠੾ΓͰτʔΫϯʹ෼ׂͰ͖ΔͳΒॳख strings.Split ͰࡁΉ৔߹ ΋͋Δ • έʔεʹΑͬͯ͸શͯͷจࣈྻΛτʔΫϯʹ͢Δඞཁ͸ͳ͍ʢۭന΍վߦΛແࢹ͢Δͷͱ ಉ͡ʣ
  10. 29 ͜͜·Ͱͷֶͼ bufio.ScannerʢVS bufio.Reader VS []rune VS []byteʣ • Ұ౓ʹಡΈࠐΉόΠτ਺ͷ্ݶ͕Θ͔͍ͬͯΔ৔߹ʢ΋͘͠͸͋Δఔ౓খ͍͞ͱΘ͔͍ͬͯΔ৔

    ߹ʣbufio.Scannerͷ࢖͍উख͕ྑ͍ • ؆қͳτʔΫϯ෼ׂʹ͓͍ͯ͸bufio.SplitFuncͷ֦ு͕ޮ͘ • ಡΈࠐΉόΠτ਺ͷ্ݶ͕Θ͔͍ͬͯΔͳΒ͹ bufio.Scanner.Buffer() Ͱ্ݶΛઃఆͰ͖Δ • ಡΈࠐΉόΠτ਺͕Θ͔Βͳ͍৔߹ʢେ͖͘ͳΔՄೳੑ͕େ͖͍৔߹ʣ͸ bufio.Reader • จࣈͱͯ͠ύʔε͢Δ৔߹͸[]runeͰʢྫ: github.com/goccy/go-yaml/scannerʣ • ϓϩτίϧͷϔομͳͲαΠζ͕໌Β͔ͳ΋ͷͷ৔߹͸[]byteͰ
  11. 31 ֤छίʔυΧόϨοδϨϙʔτ͔Β৘ใΛऔಘ͢Δ github.com/k1LoW/octocov • ίʔυΧόϨοδΛऩूɾϨϙʔτ͢Δπʔϧ • πʔϧͷಛੑ্ɺαϙʔτ͢ΔΧόϨοδϑΥʔϚοτ͸ଟ͚Ε͹ଟ͍΄Ͳྑ͍ • v0.54.6Ͱͷαϙʔτ …

    Go coverageɺLCOVɺSimpleCovɺCloverɺCoberturaɺJaCoCo • ॲཧͷಛ௃ • େ఍1ͭͷϑΝΠϧ • ΋ͱ΋ͱΧόϨοδϨϙʔτϑΝΠϧ͸ϚγϯϦʔμϒϧͳߏ଄ʹͳ͍ͬͯΔ
  12. 35 YAMLϑΝΠϧͷҰ෦Ͱม਺ల։Λ͢Δ github.com/k1LoW/expand • YAMLϑΝΠϧͷҰ෦Ͱม਺ల։Λ͢ΔػೳΛ࡞Δ͜ͱ͕Ͱ͖ΔϥΠϒϥϦ • ؀ڥม਺ల։ͱ͔ɺGitHub ActionsͷWorkflow YAMLͷΑ͏ͳม਺ల։ͱ͔ •

    ॲཧͷಛ௃ • ม਺ల։ͨ͋͠ͱͷߏจղੳ͸YAMLύʔαʔʹ೚͍ͤͨ • env DEBUG=true ͷจࣈྻ true Λͦͷ··YAMLύʔαʔʹ౉ͯ͠bool஋ͱ͍ͨ͠ • JSONΛ౉ͨ͠ΒYAMLͷߏจͱͯ͠ల։͞Εͯཉ͍͠ • ͱ͸͍͑ɺల։ͨ͠஋ͷߏ଄Ͱల։લͷYAMLͷߏจ໦͕ݩͱେ͖ͣ͘ΕΔΑ͏ͳ࣮૷Ͱ ͸͋ͬͯ͸͍͚ͳ͍ʢ text/template ͳͲʹΑΔม਺ల։Ͱ͸೉͍͠ͱ൑அ ʣ
  13. 41 ੜ੒͞ΕͨGoͷίʔυΛ͞Βʹॻ͖׵͑Δ ύʔεͷ࣮૷ʢRewrite(ctx context.Context, p string, w io.Writer) errorʣ •

    astutil.Apply() Λ࢖ͬͯ௚઀ASTΛࠩ͠ସ͑ ͍ͯΔ • ੜ੒͞ΕͨGoͷίʔυͳͷͰɺ͋Δఔ౓ ϧʔϧԽ͞Ε͍ͯΔͨΊɺࠩ͠ସ͑ର৅Λ ಛఆ͢ΔͨΊͷ൑ఆ৚݅΋ͦ͜·Ͱ࡞Γࠐ Ήඞཁ͕ͳ͍
  14. 43 ຊηογϣϯʹ͓͚Δύʔε ߏจղੳ͕ඞཁͩͬͨΓৄࡉͳࣈ۟ղੳ͕ඞཁͳύʔε͸࣮૷ίετ͕ߴ͍ • ղੳ͍ͨ͠ߏจʹΑ࣮ͬͯ૷͕େ͖͘ҟͳΔʢͦΕ͸ͦ͏ʣ • ϓϩάϥϛϯάݴޠ΍ͦΕʹྨ͢Δ΋ͷ͸ந৅ߏจ໦ʢASTʣʹͳΔ͜ͱ͕ଟ͍ • ͦΕʹྨ͢Δ΋ͷ …

    JSON΍YAML͸ͦͷ࠷ͨΔྫ • ߏจղੳͷख๏͸ଟ͘ఏҊ͞Ε͍ͯΔ • ʢಛʹAST͕ඞཁͳϨϕϧͷʣύʔαΛಠࣗʹ։ൃ͢Δͷ͸͔ͳΓίετ͕ߴ͍ͨΊɺαʔυ ύʔςΟ΋ؚΊద੾ͳύοέʔδʹ೚ͤΔͷ͕ྑ͍
  15. 45 ͦͷଞ: ύοέʔδΛ׆༻͍ͯ͠Δέʔε έʔε͝ͱʹ׆༻͍ͯ͠Δศརύοέʔδ • .ssh/config Λ࢖༻ͨ͠SSH઀ଓʢgithub.com/k1LoW/sshcʣ • ssh_configͷύʔε …

    github.com/kevinburke/ssh_config • .protoϑΝΠϧͷύʔεͱͦΕΛར༻ͨ͠gRPC௨৴ʢgithub.com/k1LoW/grpcstubʣ • .protoϑΝΠϧͷύʔε … github.com/bufbuild/protocompile • OpenAPI SpecʹΑΔHTTPϦΫΤετ/ϨεϙϯεͷόϦσʔγϣϯʢgithub.com/k1LoW/runnʣ • OpenAPIυΩϡϝϯτͷύʔε … github.com/getkin/kin-openapi • άϧʔϐϯάͰ͖ΔProcess exporter for Prometheusʢgithub.com/k1LoW/grouped_process_exporterʣ • /procʢprocfsʣ ͷύʔε … github.com/prometheus/procfs
  16. • ຊηογϣϯͰ͸ʮύʔεʯͷ࣮૷ʹ͍༷ͭͯʑͳέʔεΛ঺հͨ͠ɻ • ࠓճ͸ΑΓଟ͘ͷέʔεΛ঺հ͢Δ͜ͱͰԿ͔͠Βͷؾ͖͕ͮಘΒΕΕ͹ͱ͍͏ظ଴Λࠐ Ίͯߏ੒ͨ͠ɻ • Ұݴʹʮύʔεʯͱ͍ͬͯ΋έʔε͝ͱʹಛ௃͕͋Γɺ࣮૷ํ๏͕͋Δɻ • ҰํͰέʔεΛ၆ᛌͯ͠ΈΔͱڞ௨͍ͯ͠Δύʔλϯ΋͋ΔΑ͏ʹݟ͑Δɻ •

    ύʔεύοέʔδΛఏڙͯ͘͠Ε͍ͯΔOSSϝϯςφͷํʹ͸ײँ͔͠ͳ͍ɻ • ֤έʔεͷ۩ମతͳ࣮૷Λ֬ೝ͍ͨ͠৔߹ɺ঺հͨ͠έʔε͸શͯOSSʹͳ͍ͬͯΔͷͰɺ֤Ϧ ϙδτϦͰࢀরՄೳʢPull Request଴ͬͯ·͢ʂʣɻ 47 ͓ΘΓʹ ͓ΘΓʹ