Getting along with YAML comments with Psych

by Masaki Hara

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

© 2024 Wantedly, Inc. Psych-comments Psych-comments s = Psych::Comments.parse_stream(input) env = s.children[0].children[0].children[1] env.children << Psych::Nodes::Scalar.new("NEW_TOKEN") output = Psych::Comments.emit_yaml(s) bundle add psych-comments require "psych/comments"

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

© 2024 Wantedly, Inc. YAML family tree XML Perl Marshaller YAML 1.0 YAML 1.2 YAML as “simple XML” YAML as “Portable Marshaller” YAML as “JSON upper-compat” JSON 1998 Data::Denter, Around 2001 2004 2001 (Launch of json.org) 2009

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

© 2024 Wantedly, Inc. Three-fold YAML processing Ruby Object Node Graph Event Tree YAML "&a [1, true, *a]" [...] "1" "true" *a !!seq [...] !!int "1" !!str "true" * [1, true].tap { |a| a << a } Representation Serialization Presentation Native

Slide 29

Slide 29 text

© 2024 Wantedly, Inc. Three-fold YAML processing Ruby Object Node Graph Event Tree YAML Process aliases and anchors (Cycles etc.) Process Tags (Custom Objects) Representation Serialization Presentation Native

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

© 2024 Wantedly, Inc. YAML Kind and Tags Sequence Mapping Scalar !!seq !!omap !!pairs !!map !!set !ruby/object !!str !!binary !!null !!bool !!int !!float !!timestamp !!yaml !ruby/symbol !!merge !!value

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

© 2024 Wantedly, Inc. YAML is… ● YAML is (in a way) a portable Marshal ● Abstraction layers to sort out complexity ○ Parsing: remove syntax details ○ Deserializing: connect anchors and aliases ○ Interpreting: resolve and realize tags

Slide 38

Slide 38 text

Slide 39

Slide 39 text

Slide 40

Slide 40 text

© 2024 Wantedly, Inc. Psych’s API levels Ruby Object Node Graph Event Tree YAML Psych.load Psych.dump Psych.parse #to_yaml Psych::Parser Psych::Emitter High-level API Mid-level API Low-level API Representation Serialization Presentation Native

Slide 41

Slide 41 text

© 2024 Wantedly, Inc. Psych’s API levels and psych-comments Ruby Object Node Graph Event Tree YAML Psych.parse #to_yaml Mid-level API psych-comments’ API Representation Serialization Presentation Native

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Slide 44

Slide 44 text

© 2024 Wantedly, Inc. Psych (YAML) mid-level API Psych (a.k.a. YAML): mid-level API s = Psych.parse_stream(input) env = s.children[0].children[0].children[1] env.children << Psych::Nodes::Scalar.new("NEW_TOKEN") output = s.to_yaml

Slide 45

Slide 45 text

© 2024 Wantedly, Inc. Recap: Psych-comments Recap: Psych-comments s = Psych::Comments.parse_stream(input) env = s.children[0].children[0].children[1] env.children << Psych::Nodes::Scalar.new("NEW_TOKEN") output = Psych::Comments.emit_yaml(s)

Slide 46

Slide 46 text

Slide 47

Slide 47 text

Slide 48

Slide 48 text

Slide 49

Slide 49 text

© 2024 Wantedly, Inc. Extending the parser 2-pass parser Psych::Nodes::Node libyaml YAML text without comments Source location Psych::Nodes::Node with comments psych- comments Parse nodes Parse comments

Slide 50

Slide 50 text

Slide 51

Slide 51 text

Slide 52

Slide 52 text

Slide 53

Slide 53 text

Slide 54

Slide 54 text

Slide 55

Slide 55 text

Slide 56

Slide 56 text

Slide 57

Slide 57 text

Slide 58

Slide 58 text

Slide 59

Slide 59 text

© 2024 Wantedly, Inc. Comment scanning edge case 3 Edge case 3: comments on a key-value pair # foo foo: 1 bar: # bar 2 NOTE: Psych lacks a node type for key-value pairs, instead hanging keys and values alternatingly in a flat array

Slide 60

Slide 60 text

Slide 61

Slide 61 text

Slide 62

Slide 62 text

Slide 63

Slide 63 text

© 2024 Wantedly, Inc. Comment scanning edge case 4 Edge case 4 solution: attach to the whole element root: # foo - foo: 1 - # bar bar: 2 NOTE: “foo: 1” implicitly generates a Mapping node, which the comment attaches to.

Slide 64

Slide 64 text

Slide 65

Slide 65 text

Slide 66

Slide 66 text

Slide 67

Slide 67 text

Slide 68

Slide 68 text

© 2024 Wantedly, Inc. Internal commands for YAML formatting Prepared utilities for formatting print "foo" space! newline! indented do … end Print after generating reserved spaces and indentation Reserve spaces or indentation Bump indent level NOTE: there is one more “virtual indentation” util for bullets

Slide 69

Slide 69 text

Slide 70

Slide 70 text

Slide 71

Slide 71 text

Slide 72

Slide 72 text

Slide 73

Slide 73 text

Slide 74

Slide 74 text

Slide 75

Slide 75 text

© 2024 Wantedly, Inc. Generating bullet comments Hoisting map comments above bullets → ● Lookahead the tree and generate comments ● Then avoid duplication via a queue ○ Note that we should not mutate the input

Slide 76

Slide 76 text

Slide 77

Slide 77 text

Slide 78

Slide 78 text

Slide 79

Slide 79 text

Slide 80

Slide 80 text

Slide 81

Slide 81 text

© 2024 Wantedly, Inc. config/locales: translation mistakes Solution 1: check for correspondence Synchronizing locales... Generating en.our_new_service.try_it_out Generating ja.our_new_service.try_if_out

Slide 82

Slide 82 text

Slide 83

Slide 83 text

Slide 84

Slide 84 text

© 2024 Wantedly, Inc. config/locales: intentional absense Solution 2: (ab)use YAML tags en: # No data due to # translation # cost ja: japan_only: start: !only:ja "開始" cf. https://github.com/creasty/i18n_flow

Slide 85

Slide 85 text

Slide 86

Slide 86 text

Slide 87

Slide 87 text

Slide 88

Slide 88 text

Slide 89

Slide 89 text

Slide 90

Slide 90 text

Slide 91

Slide 91 text

Slide 92

Slide 92 text

Slide 93

Slide 93 text

Slide 94

Slide 94 text

Slide 95

Slide 95 text

Slide 96

Slide 96 text

Slide 97

Slide 97 text

Slide 98

Slide 98 text

Slide 99

Slide 99 text

© 2024 Wantedly, Inc. Recap: Three-fold YAML processing Ruby Object Node Graph Event Tree YAML "&a [1, true, *a]" [...] "1" "true" *a !!seq [...] !!int "1" !!str "true" * [1, true].tap { |a| a << a } Representation Serialization Presentation Native

Slide 100

Slide 100 text

© 2024 Wantedly, Inc. Levels of abstraction Presentation Serialization Representation Anchors & aliases Non-specific tags Scalar content formatting Directives Node style Comments Spacing Key ordering Tag style Escapes Tags Node links Scalar content

Slide 101

Slide 101 text

© 2024 Wantedly, Inc. Levels of abstraction – Psych Presentation Serialization Representation Anchors & aliases Non-specific tags Scalar content formatting Directives Node style Comments Spacing Key ordering Tag style Escapes Tags Node links Scalar content Psych Mid-level API

Slide 102

Slide 102 text

Slide 103

Slide 103 text

© 2024 Wantedly, Inc. Recap: Psych-comments layer Presentation Serialization Representation Anchors & aliases Non-specific tags Scalar content formatting Directives Node style Comments Spacing Key ordering Tag style Escapes Tags Node links Scalar content Psych-comments

Slide 104

Slide 104 text

Slide 105

Slide 105 text

Slide 106

Slide 106 text

© 2024 Wantedly, Inc. Wrap up ● I made psych-comments gem. ● It processes YAML comments. ● It neatly solves your problem, partly reusing Psych’s own algorithms. ● Thankfully people are interested in expanding it, but as a responsible maintainer, I’m going to limit its scope.