The Voice of Code - COSCUP 2018

Nowadays, the screen readers are used for the blind people to read the text on the screen. The speech speed of the screen reader is usually incredibly fast, and it reads almost everything on the screen. Also, the generated speech is not so friendly for the users.

However, it's important for people to quickly get the meaningful summary of the current source code navigation, even for people with average eye-sight.

This talk introduces you the design of the navigation assistant, and the technical implementation details of the tool.

The assistant is implemented in Go, and its language parser is used to generate a short comprehensive summary, and the text will be converted to voice speech to improve your source code navigation experience.

Transcript

1. The Voice of Code Building Source Code Navigation Assistant Yo-An Lin (@c9s)
2. Did you ever wanted to change the world?
3. What if there is a new way ...
4. to change the world for some people?
5. And light up their worlds?
8. Blind Coding Competition
9. We did this game for fun,
10. but for some people, the true difficulties and inconvenience are with them in the whole life.
11. How can developers code if they can’t see the screen?
12. Voice Over
16. If left paren x equals five right paren left brace print left paren quote hello world exclaim quote right paren right brace. if (x == 5) { print("Hello World!") }
19. The Fact
20. The Difficulties
21. Difficulties • You need to confirm every input. • You need to confirm the current location on the screen. • You need to search the components that is displayed on the screen.
22. Difficulties for Coders • How do you read through huge amount of console logs? • How do you read JSON? • How do you separate shell prompts and the current input without seeing it?
23. Unreadable logs
24. MongoDB shell version v3.4.10 connecting to: mongodb://127.0.0.1:27017 MongoDB server version: 3.4.10 Server has startup warnings: 2018-07-30T15:02:31.783+0800 I CONTROL [initandlisten] 2018-07-30T15:02:31.783+0800 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database. 2018-07-30T15:02:31.783+0800 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted. 2018-07-30T15:02:31.783+0800 I CONTROL [initandlisten] >
25. Unreadable Interface
33. Apple Accessibility
35. Does Your Software Support Accessibility?
36. 沒有，因為你只想到你⾃自⼰己
37. How Can We Improve This?
38. The UX of Voice
39. Speech Recognition Speech Synthesis
40. Outline describes the shape
41. Details are later
42. 1. Resolution
43. func TestProxy(t *testing.T) { func test proxy left paren T 星號 testing dot T right paren left brace
44. func TestProxy(t *testing.T) { Function TestProxy, with 1 parameter t, which type is a pointer of testing.T
45. If left paren x equals five right paren left brace print left paren quote hello world exclaim quote right paren right brace.
46. If left paren x equals five right paren left brace print left paren quote hello world exclaim quote right paren right brace.
47. We only need these characters when we need to edit them.
48. 2. Key Point
49. File /Users/c9s/src/github.com/c9s/ gracula/cmd/spokencode/main.go is opened
50. File /Users/c9s/src/github.com/c9s/ gracula/cmd/spokencode/main.go is opened
51. File main.go is opened
52. File main.go is opened Space
53. File main.go is opened from the directory "cmd/gracula" Space
54. 3. Outline
55. What's The Outline of a File?
56. You might want to know the shape of the file.
58. There are ~1k lines of code in this file. 2 Struct Types, 12 Public Functions, 3 Private Functions and 14 Constants.
59. Speech Synthesis Manager
60. say "Hello World"
61. Oh [[emph +; rate 165]][[volm +3]] Long Long Johnson
62. func TestProxy [[char LTRL]] (* [[char NORM]] testing [[char LTRL]] . [[char NORM]] T [[char LTRL]] ) [[char NORM]]
63. Transcript Generator
64. File <File>/path/to/main.go</File> is opened.
65. Variable <Var>numOfFunctions</Var> is an int type value.
66. Gracula Voice Assistant for Coder
67. Gracula • Run as a JSON-RPC server • Generate summary for files, types and statements. • Generate readable transcripts for voice synthesizer to read. • Integrate voice synthesizers for different platforms. • Ability of adjusting resolution. • Support LSP (Language Server Protocol)
68. Language Server Protocol 語⾔言伺服器通訊協定
69. The Problem Matrix
70. The Solution
71. Language Server Protocol
72. Language Server Protocol • JSON-RPC based protocol. • Major languages are supported. • Major IDEs and Editors are supported. • Supported by Microsoft.
73. Supported IDEs and Editors
74. Go As An Example Language Information Query
75. Go • Pure Go Parser for Go since Go 1.5. • AST structures in Go. • JSON RPC server is ready. • Mature language. • Tool Guru
76. What's Guru
77. A Go tool provides command for querying the type information
78. Guru Commands • callees - show possible targets of selected function call • callers - show possible callers of selected function • callstack - show path from callgraph root to selected function • definition - show declaration of selected identifier • describe - describe selected syntax: definition, methods, etc • what - show basic information about the selected syntax node
79. Language Query guru -json describe /Users/c9s/src/golang.org/x/tools/cmd/guru/callers.go:#5096 { "desc": "identifier", "pos": "/Users/c9s/src/golang.org/x/tools/cmd/guru/callers.go:186:6", "detail": "value", "value": { "type": "[]golang.org/x/tools/cmd/guru/serial.Caller", "objpos": "/Users/c9s/src/golang.org/x/tools/cmd/guru/callers.go:186:6" } }
80. Using Go Parser API
81. go/parser filepath := "src/golang.org/x/tools/cmd/guru/main.go" fset := token.NewFileSet() f, err := parser.ParseFile(fset, filepath, nil, parser.ParseComments) if err != nil { log.Fatal(err) return }
82. go/parser for k, object := range f.Scope.Objects { fmt.Printf("object: %s %+v", k, object) }
83. go/parser for _, decl := range f.Decls { switch e := decl.(type) { case *ast.FuncDecl: case *ast.GenDecl: for _, spec := range e.Specs { switch s := spec.(type) { case *ast.TypeSpec: fmt.Printf(" type spec: name=%+v type=%v\n", s.Name, s.Type) switch t := s.Type.(type) { case *ast.StructType: case *ast.FuncType: case *ast.InterfaceType: case *ast.ArrayType: } case *ast.ValueSpec: fmt.Printf(" value spec: names=%v values=%v\n", s.Names, s.Values) case *ast.ImportSpec: fmt.Printf(" import spec: path=%v\n", s.Path.Value) } } default: fmt.Printf(" decl: %+v\n", e) } }
84. If we try to change the world
85. We don’t have to always do the fashionable things
86. We can do the change. We can change their lives.
87. A new world. A new experience.
88. Thank You
89. Q&A