ゼロから作る Protocol Buffer のパーサーとレキサー / Writing Protocol Buffer Parser/Lexer in Go from scratch

Slide 1

Slide 1 text

Yohei Yoshimuta θϩ͔Β࡞Δ Protocol Buffer ͷ ύʔαʔͱϨΩαʔ Go Conference 23 April 2022 ϋογϡλά: #gocon #goconB

Slide 2

Slide 2 text

ࣗݾ঺հ

Slide 3

Slide 3 text

ࣗݾ঺հʢॴଐʣ ύϥϨϧ: • Ϣʔβʔͷ 70 % ͕ Z ੈ୅ • 1 ೔ͷ௨࿩࣌ؒ͸ฏۉ 3 ࣌ؒ Go ͷར༻࣮੷: • WebSocket: github.com/kuiperbelt/kuiperbelt • σʔλసૹ: Cloud DataFlow Ͱ MySQL ͔Β BigQuery ΁

Slide 4

Slide 4 text

σʔλߏ଄ͷෆҰக ଟ͘ͷϓϩάϥϛϯά՝୊͸ɺ͋ΔσʔλΛผͷߏ଄Λ࣋ͬͨσʔλʹஔ͖ ׵͑Δ͜ͱͰ͋Δ • ςΩετΛߦʹ෼ׂ • όΠφϦͷγϦΞϥΠζɾσγϦΞϥΠζ • ਖ਼نදݱ • ϨΩαʔ • ύʔαʔ ग़య: Lexical Scanning in Go: https://talks.golang.org/2011/lex.slide

Slide 5

Slide 5 text

ϨΩαʔʢࣈ۟ղੳثʣ จࣈετϦʔϜΛτʔΫϯʹஔ͖׵͑Δ enum MachineState { option allow_alias = true ; STOPPED = 0 ; RUNNING = 1 ; } enum MachineState { option allow_alias = true; STOPPED = 0; RUNNING = 1; }

Slide 6

Slide 6 text

Slide 7

Slide 7 text

ύʔαʔʢߏจղੳثʣ τʔΫϯΛ໦ߏ଄ͷσʔλʹஔ͖׵͑Δ enum MachineState { option allow_alias = true ; STOPPED = 0 ; RUNNING = 1 ; } Enum Option Field Field Name = MachineState Name = allow_alias Value = true Name =RUNNING Value = 0 Name =STOPPED Value = 1

Slide 8

Slide 8 text

Slide 9

Slide 9 text

࣮૷͸೉͍͜͠ͱ΋ ෳࡶͳςΩετ͸ͦΕࣗମಠࣗͷϧʔϧ΍ߏจ͕͋ΓɺυΩϡϝϯτԽ͞Ε͍ͯ ͳ͍ߏจ΋͠͹͠͹ ్தͰखʹෛ͑ͳ͘ͳΔ͜ͱ΋͋Δ Go ͸ύʔαʔΛࣗ࡞͢Δे෼ͳ؀ڥΛఏڙͯ͘͠Ε͍ͯΔʂ …ͨͩ͠ɺ৻ॏͳઃܭ͸ඞཁ Protocol Buffer εΩʔϚʹߜͬͯύʔαʔͷઃܭͱ࣮૷Λݟ͍͖ͯ·͠ΐ͏

Slide 10

Slide 10 text

ࠓ೔࿩͢͜ͱ • ύʔαʔͷϢʔεέʔε • ύʔαʔΛͳͥࣗ࡞͠Α͏ͱࢥͬͨͷ͔ • Ͳ͏͍͏ઃܭɾΞϓϩʔνΛऔͬͨͷ͔ • ࣮૷ৄࡉ • Ԡ༻తͳ՝୊ • ϨΩαʔͷόοϑΝઃܭ • ந৅ߏจ໦ͷΠϯλʔϑΣʔεઃܭ

Slide 11

Slide 11 text

ϨΩαʔͱύʔαʔ͸ॏཁͳߏ੒ཁૉ • ίϯύΠϥ ίϯύΠϥͷϑϩϯτΤϯυ ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

Slide 12

Slide 12 text

ίϯύΠϥͷϑϩϯτΤϯυ ϨΩαʔͱύʔαʔ͸ॏཁͳߏ੒ཁૉ • ίϯύΠϥ • ੩తղੳ ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

Slide 13

Slide 13 text

ίϯύΠϥͷϑϩϯτΤϯυ ϨΩαʔͱύʔαʔ͸ॏཁͳߏ੒ཁૉ • ίϯύΠϥ • ੩తղੳ • ίʔυੜ੒ ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

Slide 14

Slide 14 text

ίϯύΠϥͷϑϩϯτΤϯυ ϨΩαʔͱύʔαʔ͸ॏཁͳߏ੒ཁૉ • ίϯύΠϥ • ੩తղੳ • ίʔυੜ੒ • σʔλϑΝΠϧ ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

Slide 15

Slide 15 text

Protocol Buffer εΩʔϚ Protocol Buffers ͸ Google ͕։ൃͨ͠ݴޠɾϓϥοτϑΥʔϜʹґଘ͠ͳ͍ɺߏ ଄Խ͞ΕͨσʔλΛγϦΞϥΠζ͢Δ࢓૊Έ XML ΍ JSON ͷΑ͏ʹ࢖͑Δ͕ɺΑΓখ͘͞ɾૣ͘ɾ؆୯ ཉ͍͠σʔλͷߏ଄ʢεΩʔϚʣΛࣄલʹఆ͓ٛͯ͘͠ message Person { required string name = 1; required int32 id = 2; optional string email = 3; } ग़య: Protocol Bu ff ers | Google Developers: https://developers.google.com/protocol-bu ff ers

Slide 16

Slide 16 text

ݹ͍ύʔαʔ ਖ਼نදݱΛ࢖ͬͨύʔαʔ ϓϩδΣΫτʹಛԽͨ͠੩తղੳπʔϧͷͨΊʹ։ൃͨ͠ • Ϧϯλʔ

Slide 17

Slide 17 text

ݹ͍ύʔαʔ ਖ਼نදݱΛ࢖ͬͨύʔαʔ ϓϩδΣΫτʹಛԽͨ͠੩తղੳπʔϧͷͨΊʹ։ൃͨ͠ • Ϧϯλʔ • ίʔυδΣωϨʔλʔʢόϦσʔγϣϯʣ

Slide 18

Slide 18 text

ݹ͍ύʔαʔ ਖ਼نදݱΛ࢖ͬͨύʔαʔ ϓϩδΣΫτʹಛԽͨ͠੩తղੳπʔϧͷͨΊʹ։ൃͨ͠ • Ϧϯλʔ • ίʔυδΣωϨʔλʔʢόϦσʔγϣϯʣ • υΩϡϝϯτδΣωϨʔλʔ

Slide 19

Slide 19 text

৽͍͠ύʔαʔ ݹ͍ύʔαʔΛஔ͖׵͔͑ͨͬͨ ଟ͘ͷ໰୊Λ๊͍͑ͯͨ: • ͢΂ͯͷจ๏نଇʹରԠ͍ͯ͠ͳ͍

Slide 20

Slide 20 text

৽͍͠ύʔαʔ ݹ͍ύʔαʔΛஔ͖׵͔͑ͨͬͨ ଟ͘ͷ໰୊Λ๊͍͑ͯͨ: • ͢΂ͯͷจ๏نଇʹରԠ͍ͯ͠ͳ͍ • ίʔυ͕֦ுͮ͠Β͍

Slide 21

Slide 21 text

ϨΩαʔɺύʔαʔɺΞφϥΠ βʔͷ໾ׂΛ໌֬ʹͯ͠ɺઃܭ ͠௚ͨ͠ go-protoparser ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Protocol Buffer ? ? ? ? ? ? ? ? ? ? ? ? ? ? go-protoparser protolint

Slide 22

Slide 22 text

Ͳ͏͍͏ઃܭɾΞϓϩʔνΛऔͬ ͨͷ͔

Slide 23

Slide 23 text

ࠓ೔͸ Protocol Buffer εΩʔϚͷύʔαʔʹয఺Λ౰ͯΔ syntax = "proto3"; package tutorial; message Outer { // A message is an aggregate a set of typed fi elds. message Inner { // Level 2 int64 ival = 1; } repeated Inner inner_message = 2; EnumAllowingAlias enum_ fi eld =3; map my_map = 4; }

Slide 24

Slide 24 text

Parse ؔ਺ ͜ͷؔ਺͸จࣈετϦʔϜΛҾ਺ʹऔͬͯɺந৅ߏจ໦Λฦ͢ io.Reader ΠϯλʔϑΣʔεΛ࢖͑͹ɺ༷ʑͳೖྗʹରԠͰ͖Δ func Parse(input io.Reader) (*parser.Proto, error)

Slide 25

Slide 25 text

Proto ܕ εΩʔϚ͸࣍ͷจ๏نଇʹै͏: • syntax จ͔Β͸͡·Δ • ͦͷޙ͸ɺimport จɺpackage จɺoption จɺmessage จɺenum จɺ service จɺemptyStatement จͷͲΕ͔Λ܁Γฦ͢ // Proto represents a protocol buffer de fi nition. type Proto struct { Syntax *Synta x ProtoBody []interface{} }

Slide 26

Slide 26 text

Proto ܕ syntax = "proto3"; package tutorial; message Outer { message Inner { // Level 2 int64 ival = 1; } repeated Inner inner_message = 2; EnumAllowingAlias enum_ fi eld =3; map my_map = 4; } Proto Syntax Package Message Message Field Field MapField Field ProtoBody

Slide 27

Slide 27 text

Ͳ͏΍ͬͯύʔε͢Δʁ Ξϓϩʔν͸ෳ਺͋Δ: • yacc/lex ͷΑ͏ͳπʔϧΛ࢖͏ • ਖ਼نදݱΛ࢖͏

Slide 28

Slide 28 text

πʔϧ πʔϧΛ࢖͏͜ͱࣗମ͸໰୊ͳ͍͕: • ͦΕઐ༻ͷݴޠΛֶͿඞཁ͕ൃੜ͕ͪ͠

Slide 29

Slide 29 text

πʔϧ πʔϧΛ࢖͏͜ͱࣗମ͸໰୊ͳ͍͕: • ͦΕઐ༻ͷݴޠΛֶͿඞཁ͕ൃੜ͕ͪ͠ • ༻్͕߹Θͳ͍͜ͱ΋͋Δ • ͲͷΑ͏ͳจ๏نଇʹͰ΋ରԠͰ͖Δͱ͍͏Θ͚Ͱ͸ͳ͍

Slide 30

Slide 30 text

ਖ਼نදݱ ࠷ॳʹ࡞ͬͨύʔαʔͷΞϓϩʔν • Մಡੑ͕௿͍

Slide 31

Slide 31 text

ਖ਼نදݱ ࠷ॳʹ࡞ͬͨύʔαʔͷΞϓϩʔν • Մಡੑ͕௿͍ • ͢΂ͯͷจ๏نଇΛ໢ཏͰ͖ͳ͚Ε͹݁ہɺঢ়ଶ؅ཧ͕ඞཁʹͳΔ

Slide 32

Slide 32 text

ਖ਼نදݱ ࠷ॳʹ࡞ͬͨύʔαʔͷΞϓϩʔν • Մಡੑ͕௿͍ • ͢΂ͯͷจ๏نଇΛ໢ཏͰ͖ͳ͚Ε͹݁ہɺঢ়ଶ؅ཧ͕ඞཁʹͳΔ • ࠶ؼతͳϚονϯά͸೉͍͠

Slide 33

Slide 33 text

ϨΩαʔͱύʔαʔΛࣗ࡞͠Α͏ ϨΩαʔͱύʔαʔʹ෼͚࣮ͯ૷͢Δͱɺෳࡶͳจ๏نଇ΋ѻ͍΍͘͢ ͳΔ͜ͱ͕஌ΒΕ͍ͯΔ จ๏نଇͷ؍఺͔Β͸ɺϨΩαʔ͸τʔΧφΠβʔͱ΋ݺ͹ΕΔɻύʔ αʔ͸γϯλοΫεΞφϥΠβʔͱ΋ݺ͹ΕΔɻ ίϯύΠϥ΍ΠϯλϓϦλ͸௨ৗ͜ΕΒΛผʑͷϓϩηεͱͯ͠ѻ͏ɻ͜ ͷઃܭ͸ͦͷޙͷ࣮૷Λָʹͯ͘͠ΕΔ

Slide 34

Slide 34 text

Parser io.Reader Lexer Proto ύʔαʔͷॳظԽ ύʔαʔ͸ϨΩαʔʹґଘ͢Δ ϨΩαʔ͸จࣈετϦʔϜʹґଘ͢Δ // Parse parses a Protocol Buffer fi le. func Parse(input io.Reader) (*parser.Proto, error) { p := parser.NewParser( lexer.NewLexer( input, ), ) return p.ParseProto() }

Slide 35

Slide 35 text

࣮૷ৄࡉʢϨΩαʔʣ

Slide 36

Slide 36 text

Parser io.Reader Lexer Proto

Slide 37

Slide 37 text

ࣈ۟ղੳʹΑͬͯಘΒΕͨΞΠςϜ ࣈ۟ղੳͷ݁Ռ͸ 2 ͭͷϑΟʔϧυͰද͢ • ܕ: ྫ͑͹ Number • ஋: ྫ͑͹ “100” // Lexer is a tokenizer. type Lexer struct { Token Token // Token is the lexical type. Text string // Text is the lexical value. It has a cool name "lexeme."

Slide 38

Slide 38 text

Token ܕ τʔΫϯ͸ͨͩͷ Integer ఆ਺ iota Λ࢖ͬͯɺ͢΂ͯͷτʔΫϯΛྻڍ͢Δ type Token int // Token represents a lexical type. const ( // The result of Scan is one of these tokens. TILLEGAL Token = iota // Special tokens TEO F TIDENT // Identi fi ers

Slide 39

Slide 39 text

TINTLIT // Literals TFLOATLI T TBOOLLI T TSTRLI T // Misc characters TSEMICOLON // ; TCOLON // : TEQUALS // = TQUOTE // " or ' ... TSYNTAX // Keywords TPACKAG E TMESSAG E ...

Slide 40

Slide 40 text

ϨΩαʔͷΤϯτϦʔؔ਺ ύʔαʔ͸ lexer.Next() ͷݺͼग़͠ͱ lexer ͷΞΠςϜͷࢀরɺΛ܁Γ ฦ͠ߦ͏ // Next scans the read buffer. func (lex *Lexer) Next() { var err error lex.Token, lex.Text, lex.Pos, err = lex.scan() if err != nil { log.Printf(`lexer encountered the error "%v"`, err) } }

Slide 41

Slide 41 text

bufio.Reader Lexer ͸จࣈετϦʔϜΛ rune ୯ҐͰૢ࡞ ͢Δ ͦͷͨΊʹɺLexer ͸ io.Reader Λ bufio.Reader Ͱϥοϓ͍ͯ͠Δ bufio.Reader ͸όοϑΝϦϯάͱςΩετ I/O ͷͨΊͷϔϧύʔؔ਺Λఏڙͯ͘͠ΕΔ // Lexer is a tokenizer. type Lexer struct { ... r *bu fi o.Reade r } // NewLexer creates a new lexer. func NewLexer(r io.Reader) *Lexer { l := &Lexer{ r: bu fi o.NewReader(r), } return l }

Slide 42

Slide 42 text

read ؔ਺ var eof = rune(0) func (l *Lexer) read() rune { ch, _, err := l.r.ReadRune() if err != nil { return eo f } return c h }

Slide 43

Slide 43 text

peek ؔ਺ func (l *Lexer) peek() rune { ch := l.read() l.r.UnreadRune() return c h }

Slide 44

Slide 44 text

࣮ࡍʹࣈ۟ղੳΛߦ͏ؔ਺ εςʔτϚγϯ͸ switch จΛ ࢖࣮ͬͯ૷Ͱ͖Δ • ࠷ॳͷ rune Λ peek ͢Δ func (l *Lexer) scan() (Token, string, error) { ch := l.peek()

Slide 45

Slide 45 text

࣮ࡍʹࣈ۟ղੳΛߦ͏ؔ਺ εςʔτϚγϯ͸ switch จΛ ࢖࣮ͬͯ૷Ͱ͖Δ • ࠷ॳͷ rune Λ peek ͢Δ • ࣍ͷΞΫγϣϯΛܾΊΔ func (l *Lexer) scan() (Token, string, error) { ch := l.peek() switch { case unicode.IsSpace(ch): case ch == eof: case isQuote(ch):

Slide 46

Slide 46 text

࣮ࡍʹࣈ۟ղੳΛߦ͏ؔ਺ εςʔτϚγϯ͸ switch จΛ ࢖࣮ͬͯ૷Ͱ͖Δ • ࠷ॳͷ rune Λ peek ͢Δ • ࣍ͷΞΫγϣϯΛܾΊΔ • ͦͷΞΫγϣϯΛ࣮ߦ͢Δ func (l *Lexer) scan() (Token, string, error) { ch := l.peek() switch { case unicode.IsSpace(ch): l.read() return l.scan() case ch == eof: return TEOF, "", nil case isQuote(ch): lit, err := l.scanStrLit() if err != nil { return TILLEGAL, "", err } return TSTRLIT, lit, nil

Slide 47

Slide 47 text

ࣝผࢠΛಡΈऔΔ // ident = letter { letter | decimalDigit | "_" } func (l *Lexer) scanIdent() string { ident := string(l.read()) for { next := l.peek() switch { case isLetter(next), isDecimalDigit(next), next == '_': ident += string(s.read()) default: return iden t }

Slide 48

Slide 48 text

จࣈͱ਺ࣈΛಡΈऔΔ // letter = "A" … "Z" | "a" … "z" func isLetter(r rune) bool { // ref. https://en.wikipedia.org/wiki/List_of_Unicode_characters#Basic_Latin if r < 'A' { return false } if r > 'z' { return false } if r > 'Z' && r < 'a' { return false } return true } // decimalDigit = "0" … "9" func isDecimalDigit(r rune) bool { return '0' <= r && r <= '9 ' }

Slide 49

Slide 49 text

σϞ: εΩʔϚΛࣈ۟ղੳ “Lex” ͢Δ import ( "fmt" "os" "github.com/yoheimuta/go-protoparser/v4/lexer" ) func main() { lex := lexer.NewLexer(os.Stdin) for !lex.IsEOF() { lex.Next() fmt.Println("[", lex.Text, "]") } }

Slide 50

Slide 50 text

Slide 51

Slide 51 text

࣮૷ৄࡉʢύʔαʔʣ

Slide 52

Slide 52 text

Parser io.Reader Lexer Proto

Slide 53

Slide 53 text

ύʔεΛ͸͡ΊΔ ࠷ॳʹ syntax จΛύʔε͠ ͯɺͦͷޙʹ body ͦΕͧΕΛผͷؔ਺ʹ෼͚ͯ ͓͘ͱϢχοτςετ͕ॻ͖ ΍͘͢ͳΔ // ParseProto parses the proto. // proto = syntax { import | package | option | topLevelDef | emptyStatement } func (p *Parser) ParseProto() (*Proto, error) { syntax, err := p.ParseSyntax() if err != nil { return nil, err } protoBody, err := p.parseProtoBody() if err != nil { return nil, err } return &Proto{ Syntax: syntax, ProtoBody: protoBody, }, nil }

Slide 54

Slide 54 text

Syntax Protocol Buffer εΩʔϚʹ͸ 2 ͭͷόʔδϣϯ͕͋Δ • syntax = "proto3" • syntax = "proto2" // Syntax is used to de fi ne the protobuf version. type Syntax struct { ProtobufVersion string }

Slide 55

Slide 55 text

4 ͭͷεςοϓͰεΩʔϚΛύʔε͢Δ 1. ܁Γฦ͠ࣈ۟ղੳ “Lex” ͢Δ

Slide 56

Slide 56 text

func (p *Parser) ParseSyntax() (*Syntax, error) { p.lex.Next() if p.lex.Token != lexer.TSYNTAX { return nil, p.unexpected("syntax") // unexpected is a helper function to report the error } p.lex.Next() if p.lex.Token != lexer.TEQUALS { return nil, p.unexpected("=") } p.lex.Next() if p.lex.Token != lexer.TQUOTE { return nil, p.unexpected("quote") } p.lex.Next() if p.lex.Text != "proto3" && p.lex.Text != "proto2" { return nil, p.unexpected("proto3 or proto2") } version := p.lex.Tex t p.lex.Next() if p.lex.Token != lexer.TQUOTE { return nil, p.unexpected("quote") } p.lex.Next() if p.lex.Token != lexer.TSEMICOLON { return nil, p.unexpected(";") } return &Syntax{ ProtobufVersion: version }, nil }

Slide 57

Slide 57 text

4 ͭͷεςοϓͰεΩʔϚΛύʔε͢Δ 1. ܁Γฦ͠ࣈ۟ղੳ “Lex” ͢Δ 2. ઌಡΈ “Lookahead” ͢Δ

Slide 58

Slide 58 text

ઌಡΈʢϧοΫΞϔουʣύʔαʔ yacc ͕ੜ੒͢ΔύʔαʔΛؚΊɺଟ͘ͷύʔα͸ 1 ͭτʔΫϯΛઌಡ Έ͢Δ͜ͱͰɺ࣍ʹ࣮ߦ͢ΔΞΫγϣϯΛܾఆ͢Δ ϓϩάϥϛϯάݴޠͷઃܭऀ͸จ๏Λఆٛ͢Δͱ͖ʹɺ͜ΕΛߟྀ͠ ͍ͯΔ ޮ཰తͳ্ʹɺ࣮૷΋؆୯ʂ

Slide 59

Slide 59 text

parseProtoBody 1 ͭτʔΫϯΛઌಡΈ͢Δ // protoBody = { import | package | option | topLevelDef | emptyStatement } // topLevelDef = message | enum | service | extend func (p *Parser) parseProtoBody() ([]interface{}, error) { var protoBody []interface{} for { if p.IsEOF() { return protoBody, nil } var stmt interface{} p.lex.Next() token := p.lex.Toke n switch token {

Slide 60

Slide 60 text

4 ͭͷεςοϓͰεΩʔϚΛύʔε͢Δ 1. ܁Γฦ͠ࣈ۟ղੳ “Lex” ͢Δ 2. ઌಡΈ “Lookahead” ͢Δ 3. εςʔτΛભҠ͢Δ

Slide 61

Slide 61 text

parseProtoBody 1 ͭτʔΫϯΛઌಡΈ͢Δ εςʔτ͕֬ఆͨ͠ΒɺͦΕ ʹରԠ͢Δؔ਺ΛݺͿ case scanner.TIMPORT: importValue, err := p.ParseImport() if err != nil { return nil, er r } stmt = importValu e case scanner.TPACKAGE: packageValue, err := p.ParsePackage() if err != nil { return nil, er r } stmt = packageValue

Slide 62

Slide 62 text

parseProtoBody 1 ͭτʔΫϯΛઌಡΈ͢Δ εςʔτ͕֬ఆͨ͠ΒɺͦΕ ʹରԠ͢Δؔ਺ΛݺͿ case scanner.TOPTION: option, err := p.ParseOption() if err != nil { return nil, er r } stmt = optio n case scanner.TMESSAGE: message, err := p.ParseMessage() if err != nil { return nil, er r } stmt = message

Slide 63

Slide 63 text

parseProtoBody 1 ͭτʔΫϯΛઌಡΈ͢Δ εςʔτ͕֬ఆͨ͠ΒɺͦΕ ʹରԠ͢Δؔ਺ΛݺͿ case scanner.TENUM: enum, err := p.ParseEnum() if err != nil { return nil, er r } stmt = enu m case scanner.TSERVICE: service, err := p.ParseService() if err != nil { return nil, er r } stmt = service

Slide 64

Slide 64 text

4 ͭͷεςοϓͰεΩʔϚΛύʔε͢Δ 1. ܁Γฦ͠ࣈ۟ղੳ “Lex” ͢Δ 2. ઌಡΈ “Lookahead” ͢Δ 3. εςʔτΛભҠ͢Δ 4. ந৅ߏจ໦ͷϊʔυΛ૊ΈཱͯΔ

Slide 65

Slide 65 text

parseProtoBody ͦΕͧΕͷจʹରԠ͢Δϊʔ υ͕ἧͬͨΒ Proto ߏ଄ମʹ ֨ೲ͢Δ case scanner.TEXTEND: extend, err := p.ParseExtend() if err != nil { return nil, er r } stmt = exten d } protoBody = append(protoBody, extend) return &Proto{ Syntax: syntax, ProtoBody: protoBody, }, nil

Slide 66

Slide 66 text

σϞ: εΩʔϚΛύʔε͢Δ syntax = "proto3"; package tutorial; message Outer { message Inner { // Level 2 int64 ival = 1; } repeated Inner inner_message = 2; EnumAllowingAlias enum_ fi eld =3; map my_map = 4; }

Slide 67

Slide 67 text

Slide 68

Slide 68 text

·ͱΊ ϨΩαʔͱύʔαʔΛ෼͚ͯઃܭͨ͜͠ͱͰɺෳࡶͳจ๏نଇͰ΋ѻ ͍΍͘͢ͳͬͨ ϨΩαʔͱύʔαʔͦΕͧΕͷεςʔτϚγϯ͸খ͘͞ɺݟ௨͠Α͘ ࣮૷Ͱ͖ΔΑ͏ʹͳͬͨ ͦΕͧΕͷΞΫγϣϯ΋খ͘͞ɺςετΛॻ͖΍͘͢ͳͬͨ

Slide 69

Slide 69 text

Ԡ༻తͳ՝୊

Slide 70

Slide 70 text

໰୊ ࠷ॳʹެ։ͨ͠ go-protoparser ʹ͸ 2 ͭͷ໰୊͕͋ͬͨ: 1. όοΫτϥοΩϯά͕Ͱ͖ͳ͍ͱίʔυ͕ෳࡶʹͳΓ͕ͪ • ϨΩαʔʹόοϑΝػߏΛ࣋ͨͤͯɺઌಡΈ͕ؒҧͬͯͨΒ࠷ॳ ͔Β΍Γ௚ͤΔΑ͏ʹͨ͠ɻৄࡉ͸ϦϙδτϦΛࢀর

Slide 71

Slide 71 text

໰୊ ࠷ॳʹެ։ͨ͠ go-protoparser ʹ͸ 2 ͭͷ໰୊͕͋ͬͨ: 1. όοΫτϥοΩϯά͕Ͱ͖ͳ͍ͱίʔυ͕ෳࡶʹͳΓ͕ͪ • ϨΩαʔʹόοϑΝػߏΛ࣋ͨͤͯɺઌಡΈ͕ؒҧͬͯͨΒ࠷ॳ ͔Β΍Γ௚ͤΔΑ͏ʹͨ͠ɻৄࡉ͸ϦϙδτϦΛࢀর 2. ந৅ߏจ໦͕࢖͍ͮΒ͍ • ੩తղੳثΛΧελϚΠζ͢Δͱ͖ʹ΋େࣄͳࢹ఺ • ࠓ೔͸ͪ͜ΒΛৄ͘͠ݟ͍ͯ͘

Slide 72

Slide 72 text

Parser io.Reader Lexer Proto Analyzer

Slide 73

Slide 73 text

ந৅ߏจ໦ͷΠϯλʔϑΣʔεͷ໰୊ Ϧϯλʔ͸ ProtoBody Λ૸ࠪͯ͠ɺཁૉͷܕΛಛఆ͔ͯ͠ΒϦϯτ ϧʔϧΛద༻͢Δඞཁ͕͋Δ // Proto represents a protocol buffer de fi nition. type Proto struct { Syntax *Synta x // ProtoBody is a slice of sum type consisted of *Import, *Package, *Option, *Message, *Enum, *Service, *Extend and *EmptyStatement. ProtoBody []interface{} }

Slide 74

Slide 74 text

ܕ Switch ܕ Switch Ͱܕ Assertion Λ௚ྻ ʹࢦఆ͢Δ͜ͱ͕Ͱ͖Δ • ϘΠϥʔϓϨʔτ͕ଟ͍ • ؒҧ͑΍͍͢ for _, s := range src { switch t := s.(type) { case *parser.Import: case *parser.Package: case *parser.Option: case *parser.Message:

Slide 75

Slide 75 text

Visitor ύλʔϯ σβΠϯύλʔϯͷҰͭ • Visitor ύλʔϯ͸ɺଟ͘ͷΫϥε ͔Β੒ΔΦϒδΣΫτͰߏ੒͞Ε ͨߏ଄͕͋ͬͯɺͦΕʹର͢Δ৽ ͍͠ॲཧΛఆ͍ٛͨ͠ͱ͖ʹ࢖͏ Proto Syntax Package Message Message Field Field MapField Field ProtoBody

Slide 76

Slide 76 text

Visitor ύλʔϯ σβΠϯύλʔϯͷҰͭ • Visitor ύλʔϯ͸ɺଟ͘ͷΫϥε ͔Β੒ΔΦϒδΣΫτͰߏ੒͞Ε ͨߏ଄͕͋ͬͯɺͦΕʹର͢Δ৽ ͍͠ॲཧΛఆ͍ٛͨ͠ͱ͖ʹ࢖͏ • Visitor ͸ΦϒδΣΫτߏ଄ͷཁૉ ʹର͢ΔॲཧΛද͢ Visitor VisitSyntax(Syntax) VisitMessage(Message) LintingVisitor VisitSyntax(Syntax) VisitMessage(Message) FormatingVisitor VisitSyntax(Syntax) VisitMessage(Message) Visitee Accept(Visitor) Syntax Accept(Visitor) Message Accept(Visitor)

Slide 77

Slide 77 text

Visitor ύλʔϯΛಋೖ͢Δ 4 εςοϓ 1. ύʔαʔͰ Visitor ΠϯλʔϑΣʔεΛఆٛ͢Δ Visitor VisitSyntax(Syntax) VisitMessage(Message) LintingVisitor VisitSyntax(Syntax) VisitMessage(Message) FormatingVisitor VisitSyntax(Syntax) VisitMessage(Message) Visitee Accept(Visitor) Syntax Accept(Visitor) Message Accept(Visitor)

Slide 78

Slide 78 text

Visitor ΠϯλʔϑΣʔε Ϧϯλʔ͸͜ͷ Visitor Λ࣮૷͢Δ͜ͱʹͳΔ // Visitor is for dispatching Protocol Buffer elements. type Visitor interface { VisitComment(*Comment) VisitEnum(*Enum) (next bool) VisitField(*Field) (next bool) VisitImport(*Import) (next bool) VisitMessage(*Message) (next bool) VisitPackage(*Package) (next bool) VisitSyntax(*Syntax) (next bool)

Slide 79

Slide 79 text

Visitor ύλʔϯΛಋೖ͢Δ 4 εςοϓ 1. ύʔαʔͰ Visitor ΠϯλʔϑΣʔεΛఆٛ͢Δ 2. ύʔαʔͰ Visitee ΠϯλʔϑΣʔεΛఆٛ͢Δ Visitor VisitSyntax(Syntax) VisitMessage(Message) LintingVisitor VisitSyntax(Syntax) VisitMessage(Message) FormatingVisitor VisitSyntax(Syntax) VisitMessage(Message) Visitee Accept(Visitor) Syntax Accept(Visitor) Message Accept(Visitor)

Slide 80

Slide 80 text

Visitee ΠϯλʔϑΣʔε ந৅ߏจ໦͸͜ͷ Visitee Λ࣮૷͢Δ͜ͱʹͳΔ // Visitee is implemented by all Protocol Buffer elements. type Visitee interface { Accept(v Visitor) } // Proto represents a protocol buffer de fi nition. type Proto struct { Syntax *Synta x // ProtoBody is a slice of sum type consisted of *Import, *Package, *Option, *Message, *Enum, *Service, *Extend and *EmptyStatement. ProtoBody []Visite e }

Slide 81

Slide 81 text

Visitor ύλʔϯΛಋೖ͢Δ 4 εςοϓ 1. ύʔαʔͰ Visitor ΠϯλʔϑΣʔεΛఆٛ͢Δ 2. ύʔαʔͰ Visitee ΠϯλʔϑΣʔεΛఆٛ͢Δ 3. ந৅ߏจ໦͕ Visitee ΠϯλʔϑΣʔεΛ࣮૷͢Δ Visitor VisitSyntax(Syntax) VisitMessage(Message) LintingVisitor VisitSyntax(Syntax) VisitMessage(Message) FormatingVisitor VisitSyntax(Syntax) VisitMessage(Message) Visitee Accept(Visitor) Syntax Accept(Visitor) Message Accept(Visitor)

Slide 82

Slide 82 text

Visitee ࣮૷ ϧʔτϊʔυʢProtoʣͷ Accept ؔ਺ ͕ɺϦϯλʔʹΑͬͯݺ͹ΕΔ Accept ؔ਺Λ௨ͯ͠ Visitor ͕఻ൖ͞ ΕΔ // Accept dispatches the call to the visitor. func (p *Proto) Accept(v Visitor) { p.Syntax.Accept(v) for _, body := range p.ProtoBody { body.Accept(v) } }

Slide 83

Slide 83 text

// Accept dispatches the call to the visitor. func (s *Syntax) Accept(v Visitor) { if !v.VisitSyntax(s) { return } for _, comment := range s.Comments { comment.Accept(v) } } ͦΕͧΕͷϊʔυ͸ରԠ͢Δ Visit ؔ ਺ΛݺͿ Ҿ͖ଓ͖ɺAccept ؔ਺Λ௨ͯ͠ Visitor ͕఻ൖ͞ΕΔ Visitee ࣮૷

Slide 84

Slide 84 text

Visitor ύλʔϯΛಋೖ͢Δ 4 εςοϓ 1. ύʔαʔͰ Visitor ΠϯλʔϑΣʔεΛఆٛ͢Δ 2. ύʔαʔͰ Visitee ΠϯλʔϑΣʔεΛఆٛ͢Δ 3. ந৅ߏจ໦͕ Visitee ΠϯλʔϑΣʔεΛ࣮૷͢Δ 4. ੩తղੳثʢex. Ϧϯλʔʣ͕ Visitor Πϯλʔ ϑΣʔεΛ࣮૷͢Δ Visitor VisitSyntax(Syntax) VisitMessage(Message) LintingVisitor VisitSyntax(Syntax) VisitMessage(Message) FormatingVisitor VisitSyntax(Syntax) VisitMessage(Message) Visitee Accept(Visitor) Syntax Accept(Visitor) Message Accept(Visitor)

Slide 85

Slide 85 text

Slide 86

Slide 86 text

// LintingVisitor represents a visitor representing various lint operations. type LintingVisitor struct{} func (v * LintingVisitor) VisitSyntax(s *parser.Syntax) bool { if s.ProtobufVersion != 3 { v.AddFailuref(s.Meta.Pos, "Syntax should be 3 but was %q.", s.ProtobufVersion) } return false } func (v * LintingVisitor) VisitMessage(message *parser.Message) bool { name := message.MessageNam e if !strs.IsUpperCamelCase(name) { expected := strs.ToUpperCamelCase(name) v.AddFailuref(message.Meta.Pos, "Message name %q must be UpperCamelCase like %q", name, expected) } return true }

Slide 87

Slide 87 text

Visit! proto := parser.Parse(reader) linter := LintingVisitor{} proto.Accept(linter) formatter := FormatingVisitor{} proto.Accept(formatter)

Slide 88

Slide 88 text

Visit! proto := parser.Parse(reader) linter := LintingVisitor{} proto.Accept(linter) formatter := FormatingVisitor{} proto.Accept(formatter)

Slide 89

Slide 89 text

͜ΕͰ͓͠·͍ ศརͳந৅ߏจ໦ͷΠϯλʔϑΣʔεΛඋ͑ͨɺຊ֨తͳύʔαʔͱ ϨΩαʔ͕Ͱ͖ͨ Go ඪ४ϥΠϒϥϦ͚ͩͰɺ֎෦ϥΠϒϥϦ͸࢖ͬͯͳ͍ Visitor ύλʔϯ͸ύʔαʔΛΑΓ࢖͍΍ͯ͘͘͢͠ΕΔ ந৅ߏจ໦Λ࢖ͬͨॲཧͷ௥Ճ͸৽͍͠ Visitor ͷ࣮૷ͰࡁΉͷͰɺ ύʔαʔͷίʔυ͸ม͑ͳ͍͍ͯ͘

Slide 90

Slide 90 text

ৄ͍͠৘ใ ϨΩαʔͱύʔαʔ: github.com/yoheimuta/go-protoparser Ϧϯλʔ: github.com/yoheimuta/protolint ίϯύΠϥ: ίϯύΠϥ―ݪཧɾٕ๏ɾπʔϧ Visitor ύλʔϯ: ΦϒδΣΫτࢦ޲ʹ͓͚Δ࠶ར༻ͷͨΊͷσβΠϯύλʔϯ

Slide 91

Slide 91 text

Thank you @yoheimuta great@parallelcorp.com The Go gopher was designed by Renée French. Illustrations by tottie.