The Voice of Code - COSCUP 2018

7490b4e3e9cb85a1f7dc0c8ea01a86e5?s=47 Yo-An Lin
August 12, 2018

The Voice of Code - COSCUP 2018

Nowadays, the screen readers are used for the blind people to read the text on the screen. The speech speed of the screen reader is usually incredibly fast, and it reads almost everything on the screen. Also, the generated speech is not so friendly for the users.

However, it's important for people to quickly get the meaningful summary of the current source code navigation, even for people with average eye-sight.

This talk introduces you the design of the navigation assistant, and the technical implementation details of the tool.

The assistant is implemented in Go, and its language parser is used to generate a short comprehensive summary, and the text will be converted to voice speech to improve your source code navigation experience.


Yo-An Lin

August 12, 2018


  1. The Voice of Code Building Source Code Navigation Assistant Yo-An

    Lin (@c9s)
  2. Did you ever wanted to change the world?

  3. What if there is a new way ...

  4. to change the world for some people?

  5. And light up their worlds?

  6. None
  7. None
  8. Blind Coding Competition

  9. We did this game for fun,

  10. but for some people, the true difficulties and inconvenience are

    with them in the whole life.
  11. How can developers code if they can’t see the screen?

  12. Voice Over

  13. None
  14. None
  15. None
  16. If left paren x equals five right paren left brace

    print left paren quote hello world exclaim quote right paren right brace. if (x == 5) { print("Hello World!") }
  17. None
  18. None
  19. The Fact

  20. The Difficulties

  21. Difficulties • You need to confirm every input. • You

    need to confirm the current location on the screen. • You need to search the components that is displayed on the screen.
  22. Difficulties for Coders • How do you read through huge

    amount of console logs? • How do you read JSON? • How do you separate shell prompts and the current input without seeing it?
  23. Unreadable logs

  24. MongoDB shell version v3.4.10 connecting to: mongodb:// MongoDB server version:

    3.4.10 Server has startup warnings: 2018-07-30T15:02:31.783+0800 I CONTROL [initandlisten] 2018-07-30T15:02:31.783+0800 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database. 2018-07-30T15:02:31.783+0800 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted. 2018-07-30T15:02:31.783+0800 I CONTROL [initandlisten] >
  25. Unreadable Interface

  26. None
  27. None
  28. None
  29. None
  30. None
  31. None
  32. None
  33. Apple Accessibility

  34. None
  35. Does Your Software Support Accessibility?

  36. 沒有,因為你只想到你⾃自⼰己

  37. How Can We Improve This?

  38. The UX of Voice

  39. Speech Recognition Speech Synthesis

  40. Outline describes the shape

  41. Details are later

  42. 1. Resolution

  43. func TestProxy(t *testing.T) { func test proxy left paren T

    星號 testing dot T right paren left brace
  44. func TestProxy(t *testing.T) { Function TestProxy, with 1 parameter t,

    which type is a pointer of testing.T
  45. If left paren x equals five right paren left brace

    print left paren quote hello world exclaim quote right paren right brace.
  46. If left paren x equals five right paren left brace

    print left paren quote hello world exclaim quote right paren right brace.
  47. We only need these characters when we need to edit

  48. 2. Key Point

  49. File /Users/c9s/src/ gracula/cmd/spokencode/main.go is opened

  50. File /Users/c9s/src/ gracula/cmd/spokencode/main.go is opened

  51. File main.go is opened

  52. File main.go is opened Space

  53. File main.go is opened from the directory "cmd/gracula" Space

  54. 3. Outline

  55. What's The Outline of a File?

  56. You might want to know the shape of the file.

  57. None
  58. There are ~1k lines of code in this file. 2

    Struct Types, 12 Public Functions, 3 Private Functions and 14 Constants.
  59. Speech Synthesis Manager

  60. say "Hello World"

  61. Oh [[emph +; rate 165]][[volm +3]] Long Long Johnson

  62. func TestProxy [[char LTRL]] (* [[char NORM]] testing [[char LTRL]]

    . [[char NORM]] T [[char LTRL]] ) [[char NORM]]
  63. Transcript Generator

  64. File <File>/path/to/main.go</File> is opened.

  65. Variable <Var>numOfFunctions</Var> is an int type value.

  66. Gracula Voice Assistant for Coder

  67. Gracula • Run as a JSON-RPC server • Generate summary

    for files, types and statements. • Generate readable transcripts for voice synthesizer to read. • Integrate voice synthesizers for different platforms. • Ability of adjusting resolution. • Support LSP (Language Server Protocol)
  68. Language Server Protocol 語⾔言伺服器通訊協定

  69. The Problem Matrix

  70. The Solution

  71. Language Server Protocol

  72. Language Server Protocol • JSON-RPC based protocol. • Major languages

    are supported. • Major IDEs and Editors are supported. • Supported by Microsoft.
  73. Supported IDEs and Editors

  74. Go As An Example Language Information Query

  75. Go • Pure Go Parser for Go since Go 1.5.

    • AST structures in Go. • JSON RPC server is ready. • Mature language. • Tool Guru
  76. What's Guru

  77. A Go tool provides command for querying the type information

  78. Guru Commands • callees - show possible targets of selected

    function call • callers - show possible callers of selected function • callstack - show path from callgraph root to selected function • definition - show declaration of selected identifier • describe - describe selected syntax: definition, methods, etc • what - show basic information about the selected syntax node
  79. Language Query guru -json describe /Users/c9s/src/ { "desc": "identifier", "pos":

    "/Users/c9s/src/", "detail": "value", "value": { "type": "[]", "objpos": "/Users/c9s/src/" } }
  80. Using Go Parser API

  81. go/parser filepath := "src/" fset := token.NewFileSet() f, err :=

    parser.ParseFile(fset, filepath, nil, parser.ParseComments) if err != nil { log.Fatal(err) return }
  82. go/parser for k, object := range f.Scope.Objects { fmt.Printf("object: %s

    %+v", k, object) }
  83. go/parser for _, decl := range f.Decls { switch e

    := decl.(type) { case *ast.FuncDecl: case *ast.GenDecl: for _, spec := range e.Specs { switch s := spec.(type) { case *ast.TypeSpec: fmt.Printf(" type spec: name=%+v type=%v\n", s.Name, s.Type) switch t := s.Type.(type) { case *ast.StructType: case *ast.FuncType: case *ast.InterfaceType: case *ast.ArrayType: } case *ast.ValueSpec: fmt.Printf(" value spec: names=%v values=%v\n", s.Names, s.Values) case *ast.ImportSpec: fmt.Printf(" import spec: path=%v\n", s.Path.Value) } } default: fmt.Printf(" decl: %+v\n", e) } }
  84. If we try to change the world

  85. We don’t have to always do the fashionable things

  86. We can do the change. We can change their lives.

  87. A new world. A new experience.

  88. Thank You

  89. Q&A