The Voice of Code - COSCUP 2018

Nowadays, the screen readers are used for the blind people to read the text on the screen. The speech speed of the screen reader is usually incredibly fast, and it reads almost everything on the screen. Also, the generated speech is not so friendly for the users.

However, it's important for people to quickly get the meaningful summary of the current source code navigation, even for people with average eye-sight.

This talk introduces you the design of the navigation assistant, and the technical implementation details of the tool.

The assistant is implemented in Go, and its language parser is used to generate a short comprehensive summary, and the text will be converted to voice speech to improve your source code navigation experience.

Transcript

1. The Voice of Code
   Building Source Code Navigation Assistant
   Yo-An Lin (@c9s)
   

   View Slide

2. Did you ever wanted to
   change the world?
   

   View Slide

3. What if there is a new
   way ...
   

   View Slide

4. to change the world for
   some people?
   

   View Slide

5. And light up their
   worlds?
   

   View Slide

6. View Slide

7. View Slide

8. Blind Coding Competition
   

   View Slide

9. We did this game for
   fun,
   

   View Slide

10. but for some people, the true
    difficulties and inconvenience
    are with them in the whole life.
    

    View Slide

11. How can developers code if they can’t see
    the screen?
    

    View Slide

12. Voice Over
    

    View Slide

13. View Slide

14. View Slide

15. View Slide

16. If left paren x equals five right paren
    left brace print left paren quote hello
    world exclaim quote right paren right
    brace.
    if (x == 5) { print("Hello World!") }
    

    View Slide

17. View Slide

18. View Slide

19. The Fact
    

    View Slide

20. The Difficulties
    

    View Slide

21. Difficulties
    • You need to confirm every input.
    • You need to confirm the current location on the screen.
    • You need to search the components that is displayed on the
    screen.
    

    View Slide

22. Difficulties for Coders
    • How do you read through huge amount of console logs?
    • How do you read JSON?
    • How do you separate shell prompts and the current input
    without seeing it?
    

    View Slide

23. Unreadable logs
    

    View Slide

24. MongoDB shell version v3.4.10
    connecting to: mongodb://127.0.0.1:27017
    MongoDB server version: 3.4.10
    Server has startup warnings:
    2018-07-30T15:02:31.783+0800 I CONTROL [initandlisten]
    2018-07-30T15:02:31.783+0800 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the
    database.
    2018-07-30T15:02:31.783+0800 I CONTROL [initandlisten] ** Read and write access to data and
    configuration is unrestricted.
    2018-07-30T15:02:31.783+0800 I CONTROL [initandlisten]
    >
    

    View Slide

25. Unreadable Interface
    

    View Slide

26. View Slide

27. View Slide

28. View Slide

29. View Slide

30. View Slide

31. View Slide

32. View Slide

33. Apple Accessibility
    

    View Slide

34. View Slide

35. Does Your Software Support Accessibility?
    

    View Slide

36. 沒有，因為你只想到你⾃自⼰己
    

    View Slide

37. How Can We Improve
    This?
    

    View Slide

38. The UX of Voice
    

    View Slide

39. Speech Recognition Speech Synthesis
    

    View Slide

40. Outline describes the shape
    

    View Slide

41. Details are later
    

    View Slide

42. 1. Resolution
    

    View Slide

43. func TestProxy(t *testing.T) {
    func test proxy left paren T 星號
    testing dot T right paren left brace
    

    View Slide

44. func TestProxy(t *testing.T) {
    Function TestProxy, with 1 parameter t, which type
    is a pointer of testing.T
    

    View Slide

45. If left paren x equals five right paren
    left brace print left paren quote hello
    world exclaim quote right paren right
    brace.
    

    View Slide

46. If left paren x equals five right paren
    left brace print left paren quote hello
    world exclaim quote right paren right
    brace.
    

    View Slide

47. We only need these
    characters when we need
    to edit them.
    

    View Slide

48. 2. Key Point
    

    View Slide

49. File /Users/c9s/src/github.com/c9s/
    gracula/cmd/spokencode/main.go is
    opened
    

    View Slide

50. File /Users/c9s/src/github.com/c9s/
    gracula/cmd/spokencode/main.go is
    opened
    

    View Slide

51. File main.go is opened
    

    View Slide

52. File main.go is opened
    Space
    

    View Slide

53. File main.go is opened
    from the directory "cmd/gracula"
    Space
    

    View Slide

54. 3. Outline
    

    View Slide

55. What's The Outline of a
    File?
    

    View Slide

56. You might want to know
    the shape of the file.
    

    View Slide

57. View Slide

58. There are ~1k lines of code in this file.
    2 Struct Types, 12 Public Functions, 3 Private Functions and 14
    Constants.
    

    View Slide

59. Speech Synthesis Manager
    

    View Slide

60. say "Hello World"
    

    View Slide

61. Oh [[emph +; rate 165]][[volm +3]]
    Long Long Johnson
    

    View Slide

62. func TestProxy
    [[char LTRL]]
    (*
    [[char NORM]]
    testing
    [[char LTRL]]
    .
    [[char NORM]]
    T
    [[char LTRL]]
    )
    [[char NORM]]
    

    View Slide

63. Transcript Generator
    

    View Slide

64. File /path/to/main.go is opened.
    

    View Slide

65. Variable numOfFunctions is an int type value.
    

    View Slide

66. Gracula
    Voice Assistant for Coder
    

    View Slide

67. Gracula
    • Run as a JSON-RPC server
    • Generate summary for files, types and statements.
    • Generate readable transcripts for voice synthesizer to read.
    • Integrate voice synthesizers for different platforms.
    • Ability of adjusting resolution.
    • Support LSP (Language Server Protocol)
    

    View Slide

68. Language Server
    Protocol
    語⾔言伺服器通訊協定
    

    View Slide

69. The Problem Matrix
    

    View Slide

70. The Solution
    

    View Slide

71. Language Server Protocol
    

    View Slide

72. Language Server Protocol
    • JSON-RPC based protocol.
    • Major languages are supported.
    • Major IDEs and Editors are supported.
    • Supported by Microsoft.
    

    View Slide

73. Supported IDEs and Editors
    

    View Slide

74. Go As An
    Example
    Language Information
    Query
    

    View Slide

75. Go
    • Pure Go Parser for Go since Go 1.5.
    • AST structures in Go.
    • JSON RPC server is ready.
    • Mature language.
    • Tool Guru
    

    View Slide

76. What's Guru
    

    View Slide

77. A Go tool provides
    command for querying the
    type information
    

    View Slide

78. Guru Commands
    • callees - show possible targets of selected function call
    • callers - show possible callers of selected function
    • callstack - show path from callgraph root to selected function
    • definition - show declaration of selected identifier
    • describe - describe selected syntax: definition, methods, etc
    • what - show basic information about the selected syntax node
    

    View Slide

79. Language Query
    guru -json describe /Users/c9s/src/golang.org/x/tools/cmd/guru/callers.go:#5096
    {
    "desc": "identifier",
    "pos": "/Users/c9s/src/golang.org/x/tools/cmd/guru/callers.go:186:6",
    "detail": "value",
    "value": {
    "type": "[]golang.org/x/tools/cmd/guru/serial.Caller",
    "objpos": "/Users/c9s/src/golang.org/x/tools/cmd/guru/callers.go:186:6"
    }
    }
    

    View Slide

80. Using Go Parser API
    

    View Slide

81. go/parser
    filepath := "src/golang.org/x/tools/cmd/guru/main.go"
    fset := token.NewFileSet()
    f, err := parser.ParseFile(fset, filepath, nil, parser.ParseComments)
    if err != nil {
    log.Fatal(err)
    return
    }
    

    View Slide

82. go/parser
    for k, object := range f.Scope.Objects {
    fmt.Printf("object: %s %+v", k, object)
    }
    

    View Slide

83. go/parser
    for _, decl := range f.Decls {
    switch e := decl.(type) {
    case *ast.FuncDecl:
    case *ast.GenDecl:
    for _, spec := range e.Specs {
    switch s := spec.(type) {
    case *ast.TypeSpec:
    fmt.Printf(" type spec: name=%+v type=%v\n", s.Name, s.Type)
    switch t := s.Type.(type) {
    case *ast.StructType:
    case *ast.FuncType:
    case *ast.InterfaceType:
    case *ast.ArrayType:
    }
    case *ast.ValueSpec:
    fmt.Printf(" value spec: names=%v values=%v\n", s.Names, s.Values)
    case *ast.ImportSpec:
    fmt.Printf(" import spec: path=%v\n", s.Path.Value)
    }
    }
    default:
    fmt.Printf(" decl: %+v\n", e)
    }
    }
    

    View Slide

84. If we try to change the
    world
    

    View Slide

85. We don’t have to always
    do the fashionable things
    

    View Slide

86. We can do the change.
    We can change their lives.
    

    View Slide

87. A new world.
    A new experience.
    

    View Slide

88. Thank You
    

    View Slide

89. Q&A
    

    View Slide