Making your own Code Formatter in Swift

Slide 1

Slide 1 text

Making your ownɹɹɹɹ Code Formatter in Swiftɹɹɹɹɹ 2019/01/19 iOSConf SG 2019 Yasuhiro Inami / @inamiy

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

2016 → 2019 4 → 14ɹɹ ! developers ɹɹ ɹ ɹ

Slide 5

Slide 5 text

! 14 developers ! 14 coding styles

Slide 6

Slide 6 text

! Programming Styles 1. Language (Swift, Objective-C, C++, etc) 2. Typing (Object-oriented, functional) or untyping (oh my!) 3. Architecture (MVC, MVVM, Reactive Programming, etc) 4. Variable / Function Naming 5. Comments / Documentation 6. Whitespaces (tabs V.S. spaces, newline)

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

We need the Law 1. ! Coding Style Guide • https://github.com/raywenderlich/swift-style-guide • https://google.github.io/swift/ 2. " Linter / Formatter • realm/SwiftLint • nicklockwood/SwiftFormat • Both solves syntactic problems

Slide 10

Slide 10 text

SwiftLint / SwiftFormat • realm/SwiftLint • Powered by SourceKit(ten) • Syntax structure = untyped dictionary tree • Some rules are autocorrect-able • nicklockwood/SwiftFormat • Scope-aware, built-in tokenizer • Syntax structure = token array

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

SwiftSyntax https://github.com/apple/swift-syntax

Slide 14

Slide 14 text

SwiftSyntax • Swift bindings for apple/swift's libSyntax • Type-safe, AST-level structured editing library • Includes trivias (whitespaces & comments) • Attaches trivias to leading / trailing of token • Similar to .NET (Roslyn) / TypeScript Syntax API • Internally calls swiftc -frontend -emit-syntax and passes JSON for "C++ to Swift" bridging (in Swift 4.2)

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

Swift AST Explorer https://swift-ast-explorer.kishikawakatsumi.com

Slide 17

Slide 17 text

Q. Where do Swift syntaxes come from?

Slide 18

Slide 18 text

Swift 4.2 Formal grammar About the Language Reference — The Swift Programming Language (Swift 4.2)

Slide 19

Slide 19 text

Slide 20

Slide 20 text

// Note: Pseudo-code struct TopLevelDeclaration { let statements: [Statement] } // Simple statements, compiler control, and control flow. enum Statement { case expression(Expression, Semicolon?) case declaration(Declaration, Semicolon?) case loop(LoopStatement, Semicolon?) case branch(BranchStatement, Semicolon?) case labeled(LabeledStatement, Semicolon?) case controlTransfer(ControlTransferStatement, Semicolon?) case `defer`(DeferStatement, Semicolon?) case `do`(DoStatement, Semicolon?) case compilerControl(CompilerControlStatement) }

Slide 21

Slide 21 text

Slide 22

Slide 22 text

/// Introduces a new name or construct, e.g. functions, structs, variables. enum Declaration { case ìmport`(ImportDeclaration) case constant(ConstantDeclaration) case variable(VariableDeclaration) case `typealias`(TypealiasDeclaration) case function(FunctionDeclaration) case ènum`(EnumDeclaration) case `struct`(StructDeclaration) case `class`(ClassDeclaration) case `protocol`(ProtocolDeclaration) case initializer(InitializerDeclaration) case deinitializer(DeinitializerDeclaration) case èxtension`(ExtensionDeclaration) case `subscript`(SubscriptDeclaration) case òperator`(OperatorDeclaration) case precedenceGroup(PrecedenceGroupDeclaration) }

Slide 23

Slide 23 text

expression ! [try-operator] prefix-expression [binary-expressions] prefix-expression ! [prefix-operator] postfix-expression | in-out-expression postfix-expression ! primary-expression | function-call-expression | initializer-expression | l binary-expressions ! binary-expression [binary-expressions] binary-expression ! binary-operator prefix-expression | assignment-operator [try-operator] prefix-expression | conditional-operator [try-operator] prefix-expression | type-casting-operator

Slide 24

Slide 24 text

/// Can evaluate & return a value and/or cause a side effect. struct Expression { let tryOperator: TryOperator? let prefixExpression: PrefixExpression let binaryExpressions: [BinaryExpression] } enum PrefixExpression { case prefixOperator(PrefixOperator?, PostfixExpression) case `inout`(InoutExpression) } enum PostfixExpression { case primary(PrimaryExpression) case functionCall(FunctionCallExpression) case initializer(InitializerExpression) ... }

Slide 25

Slide 25 text

whitespace ! whitespace-item [whitespace] whitespace-item ! line-break | comment | multiline-comment | U+0000, U+0009, U+000B, U+000C, or U+0020 line-break ! U+000A | U+000D | U+000D followed by U+000A comment ! // comment-text line-break multiline-comment ! /* multiline-comment-text */ comment-text ! comment-text-item [comment-text] comment-text-item ! Any Unicode scalar value except U+000A or U+000D multiline-comment-text ! multiline-comment-text-item [multiline-comment-text] multiline-comment-text-item ! multiline-comment | comment-text-item | Any Unicode scalar value except /* or */

Slide 26

Slide 26 text

identifier ! identifier-head [identifier-characters] | ` identifier-head [identifier-characters] ` | implicit-parameter-name identifier-list ! identifier | identifier , identifier-list identifier-head ! A-Z | a-z | _ | U+00A8 | U+00AA | ... identifier-character ! 0-9 | U+0300–U+036F | U+1DC0–U+1DFF | ... | identifier-head identifier-characters ! identifier-character [identifier-characters]

Slide 27

Slide 27 text

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

SwiftSyntax Example

Slide 30

Slide 30 text

iOSConfSG = 2019; ↓ iOSConfSG = 2019

Slide 31

Slide 31 text

open class SemicolonTrimmer: SyntaxRewriter { open override func visit(_ syntax: CodeBlockItemSyntax) -> Syntax { if let nextToken = syntax.nextToken, nextToken.leadingTrivia.contains(where: { $0.isNewline }) || syntax.isLastChild { let noSemicolon = syntax.withSemicolon(nil) return super.visit(noSemicolon) } else { return super.visit(syntax) } } }

Slide 32

Slide 32 text

iOSConfSG = 2019 ↓ iOSConfSG = 2_019

Slide 33

Slide 33 text

open override func visit(_ token: TokenSyntax) -> Syntax { guard case var .integerLiteral(text) = token.tokenKind else { return token } if text.hasPrefix("0x") || text.hasPrefix("0b") || text.hasPrefix("0o") { return token } text = text.replacingOccurrences(of: "_", with: "") let index1 = text.index(after: text.startIndex) var index = text.endIndex while text.formIndex(&index, offsetBy: -3, limitedBy: index1) { text.insert("_", at: index) } return token.withKind(.integerLiteral(text)) }

Slide 34

Slide 34 text

struct Foo { - struct Bar { - struct Baz {} - } } + +extension Foo { + struct Bar {} +} + +extension Foo.Bar { + struct Baz {} +}

Slide 35

Slide 35 text

func foo(_ value: Int?) throws -> Int { - if let value = value { - if value >= 0 { - return value - } else { - throw MyError.negative - } - } else { - throw MyError.none - } + guard let value = value else { throw MyError.none } + guard value >= 0 else { throw MyError.negative } + return value }

Slide 36

Slide 36 text

SwiftSyntax types • SyntaxRewriter / SyntaxVisitor • Trivia (leading / trailing) • Comments & whitespaces are stored in TokenSyntax • SyntaxFactory • Creates a new Syntax nodes • Diagnostic • Data types for showing diagnosed Note & FixIt

Slide 37

Slide 37 text

try! Swift NYC 2017 - Improving Swift Tools with libSyntax apple/swift/lib/Syntax

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

Speeding up SwiftSyntax by using the parser directly - Swift Forums apple/swift#21762 apple/swift-syntax#59

Slide 40

Slide 40 text

SwiftSyntax + Optics

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

If visiting descendants manually... // Get grandchild (2 getters). var rightParen = syntax.parameters.rightParen // Modify grandchild (e.g. changing trivia). rightParen = modify(rightParen) // Set new child. let parameters = syntax.parameters .withRightParen(rightParen) // Set new `syntax` ( ❌ many more code needed if deeply nested) return syntax.withParameters(parameters)

Slide 45

Slide 45 text

Use ! Lens! override func visit(_ syntax: InitializerDeclSyntax) -> DeclSyntax { return _modify(lens: .parameters >>> .rightParen, syntax: syntax) } private func _modify( lens: Lens, syntax: Whole ) -> Whole where Whole: Syntax, Part: Syntax { let part = lens.getter(syntax) let part2 = modify(part) return lens.setter(syntax, part2) }

Slide 46

Slide 46 text

Lens.parameters >>> Lens.rightParen

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

.wowɹɹ ɹ >>> .suchɹ >>> .nestɹ >>> .muchɹ >>> .deepɹ

Slide 50

Slide 50 text

! Lens: Functional getter & setter struct Lens { let getter: (Whole) -> Part let setter: (Whole, Part) -> Whole static func >>> ( l: Lens, r: Lens ) -> Lens { return Lens( getter: { r.getter(l.getter($0)) }, setter: { a, c in l.setter(a, r.setter(l.getter(a), c)) } ) } }

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

! Prism: Lens for sum types struct Prism { let tryGet: (Whole) -> Part? let inject: (Part) -> Whole static func >>> ( l: Prism, r: Prism ) -> Prism { return Prism( tryGet: { l.tryGet($0).flatMap(r.tryGet) }, inject: { l.inject(r.inject($0)) } ) } }

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

>>> : (Lens, Lens) -> Lens >>> : (Prism , Prism) -> Prism >>> : (Lens , Prism) -> ??? >>> : (Prism, Lens) -> ??? Q. How to compose Lens and Prism together?

Slide 59

Slide 59 text

>>> : (Lens, Lens) -> Lens >>> : (Prism , Prism) -> Prism >>> : (Lens , Prism) -> ??? >>> : (Prism, Lens) -> ??? ɹɹɹɹɹˣˣˣɹLift to AffineTraversal!ɹˣˣˣ init(lens:) : (Lens) -> AffineTraversal init(prism:) : (Prism ) -> AffineTraversal >>> : (AffineTraversal , AffineTraversal) -> AffineTraversal

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

Lens.trailingClosure >>> Prism.some >>> Lens.rightParen

Slide 62

Slide 62 text

↔ Afﬁne Traversal: Lens + Prism struct AffineTraversal { let tryGet: (Whole) -> Part? let setter: (Whole, Part) -> Whole init(lens: Lens) { self.init(tryGet: lens.getter, setter: lens.setter) } init(prism: Prism) { self.init(tryGet: prism.tryGet, setter: { prism.inject($1) }) } }

Slide 63

Slide 63 text

SwiftRewriter https://github.com/inamiy/SwiftRewriter

Slide 64

Slide 64 text

Recap • SwiftSyntax: Type-safe & lossless AST editing library • Many incoming improvements toward Swift 5.0 or later • Swift AST Explorer: Useful AST visualizer for debugging • Formal grammar: Syntax production rules • Optics: Functional approach to traverse data structure • Let's make your own code formatter using SwiftSyntax!

Slide 65

Slide 65 text

References (Swift AST) • try! Swift NYC 2017 - Improving Swift Tools with libSyntax • try! Swift Tokyo 2018 - AST Meta-programming • apple/swift/lib/Syntax • Speeding up SwiftSyntax by using the parser directly • About the Language Reference (Swift 4.2) • SwiftFormat (Part 1 of 3) - Schibsted Bytes • google/swift (format branch)

Slide 66

Slide 66 text

References (Optics) • Lenses in Swift - Chris Eidhof • Brandon Williams - Lenses in Swift • Lenses and Prisms in Swift: a pragmatic approach | Fun iOS • Lenses, Folds, and Traversals - Edward Kmett • Oleg's gists - Glassery • Profunctor Optics: Modular Data Accessors

Slide 67

Slide 67 text

Thanks! Yasuhiro Inami @inamiy