Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Voice of Code - COSCUP 2018

Yo-An Lin
August 12, 2018

The Voice of Code - COSCUP 2018

Nowadays, the screen readers are used for the blind people to read the text on the screen. The speech speed of the screen reader is usually incredibly fast, and it reads almost everything on the screen. Also, the generated speech is not so friendly for the users.

However, it's important for people to quickly get the meaningful summary of the current source code navigation, even for people with average eye-sight.

This talk introduces you the design of the navigation assistant, and the technical implementation details of the tool.

The assistant is implemented in Go, and its language parser is used to generate a short comprehensive summary, and the text will be converted to voice speech to improve your source code navigation experience.

Yo-An Lin

August 12, 2018
Tweet

More Decks by Yo-An Lin

Other Decks in Technology

Transcript

  1. The Voice of Code
    Building Source Code Navigation Assistant
    Yo-An Lin (@c9s)

    View full-size slide

  2. Did you ever wanted to
    change the world?

    View full-size slide

  3. What if there is a new
    way ...

    View full-size slide

  4. to change the world for
    some people?

    View full-size slide

  5. And light up their
    worlds?

    View full-size slide

  6. Blind Coding Competition

    View full-size slide

  7. We did this game for
    fun,

    View full-size slide

  8. but for some people, the true
    difficulties and inconvenience
    are with them in the whole life.

    View full-size slide

  9. How can developers code if they can’t see
    the screen?

    View full-size slide

  10. If left paren x equals five right paren
    left brace print left paren quote hello
    world exclaim quote right paren right
    brace.
    if (x == 5) { print("Hello World!") }

    View full-size slide

  11. The Difficulties

    View full-size slide

  12. Difficulties
    • You need to confirm every input.
    • You need to confirm the current location on the screen.
    • You need to search the components that is displayed on the
    screen.

    View full-size slide

  13. Difficulties for Coders
    • How do you read through huge amount of console logs?
    • How do you read JSON?
    • How do you separate shell prompts and the current input
    without seeing it?

    View full-size slide

  14. Unreadable logs

    View full-size slide

  15. MongoDB shell version v3.4.10
    connecting to: mongodb://127.0.0.1:27017
    MongoDB server version: 3.4.10
    Server has startup warnings:
    2018-07-30T15:02:31.783+0800 I CONTROL [initandlisten]
    2018-07-30T15:02:31.783+0800 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the
    database.
    2018-07-30T15:02:31.783+0800 I CONTROL [initandlisten] ** Read and write access to data and
    configuration is unrestricted.
    2018-07-30T15:02:31.783+0800 I CONTROL [initandlisten]
    >

    View full-size slide

  16. Unreadable Interface

    View full-size slide

  17. Apple Accessibility

    View full-size slide

  18. Does Your Software Support Accessibility?

    View full-size slide

  19. 沒有,因為你只想到你⾃自⼰己

    View full-size slide

  20. How Can We Improve
    This?

    View full-size slide

  21. The UX of Voice

    View full-size slide

  22. Speech Recognition Speech Synthesis

    View full-size slide

  23. Outline describes the shape

    View full-size slide

  24. Details are later

    View full-size slide

  25. 1. Resolution

    View full-size slide

  26. func TestProxy(t *testing.T) {
    func test proxy left paren T 星號
    testing dot T right paren left brace

    View full-size slide

  27. func TestProxy(t *testing.T) {
    Function TestProxy, with 1 parameter t, which type
    is a pointer of testing.T

    View full-size slide

  28. If left paren x equals five right paren
    left brace print left paren quote hello
    world exclaim quote right paren right
    brace.

    View full-size slide

  29. If left paren x equals five right paren
    left brace print left paren quote hello
    world exclaim quote right paren right
    brace.

    View full-size slide

  30. We only need these
    characters when we need
    to edit them.

    View full-size slide

  31. 2. Key Point

    View full-size slide

  32. File /Users/c9s/src/github.com/c9s/
    gracula/cmd/spokencode/main.go is
    opened

    View full-size slide

  33. File /Users/c9s/src/github.com/c9s/
    gracula/cmd/spokencode/main.go is
    opened

    View full-size slide

  34. File main.go is opened

    View full-size slide

  35. File main.go is opened
    Space

    View full-size slide

  36. File main.go is opened
    from the directory "cmd/gracula"
    Space

    View full-size slide

  37. What's The Outline of a
    File?

    View full-size slide

  38. You might want to know
    the shape of the file.

    View full-size slide

  39. There are ~1k lines of code in this file.
    2 Struct Types, 12 Public Functions, 3 Private Functions and 14
    Constants.

    View full-size slide

  40. Speech Synthesis Manager

    View full-size slide

  41. say "Hello World"

    View full-size slide

  42. Oh [[emph +; rate 165]][[volm +3]]
    Long Long Johnson

    View full-size slide

  43. func TestProxy
    [[char LTRL]]
    (*
    [[char NORM]]
    testing
    [[char LTRL]]
    .
    [[char NORM]]
    T
    [[char LTRL]]
    )
    [[char NORM]]

    View full-size slide

  44. Transcript Generator

    View full-size slide

  45. File /path/to/main.go is opened.

    View full-size slide

  46. Variable numOfFunctions is an int type value.

    View full-size slide

  47. Gracula
    Voice Assistant for Coder

    View full-size slide

  48. Gracula
    • Run as a JSON-RPC server
    • Generate summary for files, types and statements.
    • Generate readable transcripts for voice synthesizer to read.
    • Integrate voice synthesizers for different platforms.
    • Ability of adjusting resolution.
    • Support LSP (Language Server Protocol)

    View full-size slide

  49. Language Server
    Protocol
    語⾔言伺服器通訊協定

    View full-size slide

  50. The Problem Matrix

    View full-size slide

  51. The Solution

    View full-size slide

  52. Language Server Protocol

    View full-size slide

  53. Language Server Protocol
    • JSON-RPC based protocol.
    • Major languages are supported.
    • Major IDEs and Editors are supported.
    • Supported by Microsoft.

    View full-size slide

  54. Supported IDEs and Editors

    View full-size slide

  55. Go As An
    Example
    Language Information
    Query

    View full-size slide

  56. Go
    • Pure Go Parser for Go since Go 1.5.
    • AST structures in Go.
    • JSON RPC server is ready.
    • Mature language.
    • Tool Guru

    View full-size slide

  57. A Go tool provides
    command for querying the
    type information

    View full-size slide

  58. Guru Commands
    • callees - show possible targets of selected function call
    • callers - show possible callers of selected function
    • callstack - show path from callgraph root to selected function
    • definition - show declaration of selected identifier
    • describe - describe selected syntax: definition, methods, etc
    • what - show basic information about the selected syntax node

    View full-size slide

  59. Language Query
    guru -json describe /Users/c9s/src/golang.org/x/tools/cmd/guru/callers.go:#5096
    {
    "desc": "identifier",
    "pos": "/Users/c9s/src/golang.org/x/tools/cmd/guru/callers.go:186:6",
    "detail": "value",
    "value": {
    "type": "[]golang.org/x/tools/cmd/guru/serial.Caller",
    "objpos": "/Users/c9s/src/golang.org/x/tools/cmd/guru/callers.go:186:6"
    }
    }

    View full-size slide

  60. Using Go Parser API

    View full-size slide

  61. go/parser
    filepath := "src/golang.org/x/tools/cmd/guru/main.go"
    fset := token.NewFileSet()
    f, err := parser.ParseFile(fset, filepath, nil, parser.ParseComments)
    if err != nil {
    log.Fatal(err)
    return
    }

    View full-size slide

  62. go/parser
    for k, object := range f.Scope.Objects {
    fmt.Printf("object: %s %+v", k, object)
    }

    View full-size slide

  63. go/parser
    for _, decl := range f.Decls {
    switch e := decl.(type) {
    case *ast.FuncDecl:
    case *ast.GenDecl:
    for _, spec := range e.Specs {
    switch s := spec.(type) {
    case *ast.TypeSpec:
    fmt.Printf(" type spec: name=%+v type=%v\n", s.Name, s.Type)
    switch t := s.Type.(type) {
    case *ast.StructType:
    case *ast.FuncType:
    case *ast.InterfaceType:
    case *ast.ArrayType:
    }
    case *ast.ValueSpec:
    fmt.Printf(" value spec: names=%v values=%v\n", s.Names, s.Values)
    case *ast.ImportSpec:
    fmt.Printf(" import spec: path=%v\n", s.Path.Value)
    }
    }
    default:
    fmt.Printf(" decl: %+v\n", e)
    }
    }

    View full-size slide

  64. If we try to change the
    world

    View full-size slide

  65. We don’t have to always
    do the fashionable things

    View full-size slide

  66. We can do the change.
    We can change their lives.

    View full-size slide

  67. A new world.
    A new experience.

    View full-size slide