Source code as an input

Talk by Paul Taykalo (originally posted here: https://speakerdeck.com/paultaykalo/source-code-as-input).

Паша расскажет об инструментах, которые участвуют в обработке исходного кода - SourceKit, SourceKitten, libSyntax. Вас ожидает увлекательное интро, в ходе которого вы узнаете:
- о роли и ответственностях каждого из инструментов
- практическое применение тех или иных возможностей в работе
- текущий статус разработки, каких возможностей можно ожидать от инструментов в будущем
- чего ожидать вообще не стоит :)

This talk was made for CocoaFriday #1 ( https://cocoaheads.org.ua/cocoafriday/1 ) which took place Mar 1, 2019.

Video is available here: https://youtu.be/0NH-G77BivU

Transcript

1. Source Code
   as an the input
   1 — Source Code as Input, by Paul Taykalo, MacPaw
   

   View Slide

2. When just writing a code is not
   enough
   2 — Source Code as Input, by Paul Taykalo, MacPaw
   

   View Slide

3. A lot of source - not a problem
   - Code style
   ❓
   - Documentation
   ❓
   - Refactoring
   ❓
   - Code generation
   ❓
   - Error detection
   ❓
   - Obsfuscation
   ❓
   3 — Source Code as Input, by Paul Taykalo, MacPaw
   

   View Slide

4. Code Style
   4 — Source Code as Input, by Paul Taykalo, MacPaw
   

   View Slide

5. Documentation
   5 — Source Code as Input, by Paul Taykalo, MacPaw
   

   View Slide

6. Refactoring
   6 — Source Code as Input, by Paul Taykalo, MacPaw
   

   View Slide

7. Code generation
   7 — Source Code as Input, by Paul Taykalo, MacPaw
   

   View Slide

8. Error detection / Linting
   8 — Source Code as Input, by Paul Taykalo, MacPaw
   

   View Slide

9. Obfuscation
   9 — Source Code as Input, by Paul Taykalo, MacPaw
   

   View Slide

10. Processing source
    10 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

11. 11 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

12. Source
    Obvious, we'll work with Swi! and ObjC
    12 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

13. Destination
    13 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

14. Destination
    - Docs
    - More code!
    - Code Checks
    - Images
    14 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

15. Internal Representation
    15 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

16. Rules
    disabled_rules: # rule identifiers to exclude from running
    - colon
    - comma
    - control_statement
    opt_in_rules: # some rules are only opt-in
    - empty_count
    # Find all the available rules by running:
    # swiftlint rules
    included: # paths to include during linting. `--path` is ignored if present.
    - Source
    excluded: # paths to ignore during linting. Takes precedence over `included`.
    - Carthage
    - Pods
    - Source/ExcludedFolder
    - Source/ExcludedFile.swift
    - Source/*/ExcludedFile.swift # Exclude files with a wildcard
    16 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

17. Tools for processing Code
    17 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

18. Tools for processing Code
    Regex-based solutions
    - IR: Strings
    - Somewhat unreliable
    18 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

19. Tools for processing Code
    Custom Parser Solutions
    - IR: (DSL)
    - Trying to parse code on its own
    - Using own AST
    19 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

20. Tools for processing Code
    Sourcekit based solutions
    - IR: AST, Binary, JSON*
    - Using Sourcekit library
    - Always up to date
    20 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

21. SourceKit
    21 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

22. SourceKit is ...
    22 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

23. SourceKit is ...
    23 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

24. SourcekitService is ...
    24 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

25. SourceKit is...
    Welcome to SourceKit! SourceKit is a framework
    for supporting IDE features like indexing, syntax-
    coloring, code-completion, etc. In general it
    provides the infrastructure that an IDE needs for
    excellent language support.
    SourceKit currently only supports the Swi!
    language.
    25 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

26. SourceKit
    - Code Completion
    - Cursor Info
    - Demangling/Demangling
    - Documentation
    - Module interface generation
    - Indexing*
    26 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

27. SourceKit options
    - C Interface
    - sourcekitd.framework
    - libsourcekitdInProc.dylib
    - Official Python Binding
    - Unofficial Swi" Binding: SourceKitten
    27 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

28. SourceKitten
    28 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

29. SourceKitten
    -
    ✅
    Command Line Util
    -
    ✅
    Framework
    -
    ✅
    Wrapper over SourceKit
    -
    ✅
    Writtent in Swi!
    29 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

30. Tools with Sourcekitten
    -
    ✅
    Swi!Lint: Swi! style and conventions.
    -
    ✅
    Jazzy: Docs for Swi! & Objective-C.
    -
    ✅
    Sourcery: Meta-programming for Swi!
    -
    ✅
    Cuckoo: Mocking framework for Swi!.
    30 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

31. Tools with Sourcekitten
    -
    ✅
    Swi!yMocky: Framework for mock genertion.
    -
    ✅
    SourceKittenDaemon: Auto Completions .
    -
    ✅
    SourceDocs: Markdown documentation.
    -
    ✅
    Taylor: Measure Swi! code metrics
    31 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

32. SourceKitten uses SwiftLint,
    which is based on SourceKitten
    32 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

33. libSyntax
    33 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

34. libsyntax - C library
    SwithSyntax - Swi! wrapper library
    34 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

35. return 1
    35 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

36. http://bit.ly/2SBJtS4
    36 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

37. 37 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

38. libSyntax and
    Sourcekit
    38 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

39. Thank you
    39 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

40. Q & A
    40 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

41. Source Code as Input
    by Paul Taykalo, MacPaw
    @TT_Kilew
    41 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide

42. Links
    https://github.com/apple/swi!/blob/master/
    tools/SourceKit/docs/Protocol.md
    https://forums.swi!.org/t/speeding-up-
    swi!syntax-by-using-the-parser-directly/18493
    https://academy.realm.io/posts/appbuilders-jp-
    simard-sourcekit/
    https://github.com/apple/swi!/blob/master/
    tools/SourceKit
    42 — Source Code as Input, by Paul Taykalo, MacPaw
    

    View Slide