Foundations of a Modern IDE

Developers may take for granted many features in a modern IDE, but it was not long ago when there was just the command line. Today with the growing complexity of software stacks and frameworks, the importance of effective developer tools has only followed. This talk will cover the basics of static analysis, parsing, debugging, code comprehension, and how we can leverage all these features to develop powerful, integrated language tools and plugins.

Transcript

1. Breandan Considine JetBrains Foundations of a Modern IDE Houston TechFest 9/13
2. Welcome to Houston TechFest • Please turn off all electronic devices or set them to vibrate. • If you must take a phone call, please do so in the lobby so as not to disturb others. • Thanks to our Diamond Sponsors: Thank you for being a part of the 8th Annual Houston TechFest!
4. What is your ideal IDE?
5. Traditional IDE • Rapid navigation (declarations, find usages) • Code assistance (completions, refactoring) • Program analysis (validation, inspections) • Build automation (dependencies, compiling)
6. Today’s IDE • Plugin support • Debugging & profiling • Version control • Framework support • Cloud enabled • Visual programming • Device simulation
7. Back to the basics! Why use an IDE? • Minimizes ad-hoc configuration • Provides scaffolding for common tasks • Allows you to focus on writing new code • Accelerates feedback by detecting issues early • Assembles and deploys complex applications
8. Languages and Grammars Platform/UI Core: Lexer, Parser, Indexer, Threading Code Awareness Plugins
9. Usability requirements • Keystroke to response under 100ms • Handles arbitrarily large projects smoothly • Robust parsing, most of the time code is broken • Any loss of work is unacceptable
10. http://www.ibm.com/developerworks/opensource/tutorials/os-ecl-commplgin1/ Editor Debugger Build System Symbol Table Incremental lexer Incremental Parser
11. http://www.eclipse.org/articles/article.php?file=Article-JavaCodeManipulation_AST/index.html Language parsing • Error recovery is key • Incremental parsing ◦ Parsing deltas is much more efficient ◦ Queues enable seamless editing and analysis of code ◦ Once we have an AST, we can expose the code to IDE
12. Platform/UI Core: Lexer, Parser, Indexer, Threading Code Awareness Plugins Languages and Grammars
13. From [E]BNF to [A]ST <try> ::= try [resources] <block> <catches>; <resources> ::= ( <resource> {';' <resource>} ); <resource> ::= [modifier] <res type> <id> '=' <exprs>; <catches> ::= <catch> | <catch> <catches> | <finally>; <catch> ::= catch( <throwable> <identifier> ) <block>; <throwable> ::= <throw type> {'|' <throw type>}; <finally> ::= finally <block>; <block> ::= {…} *<exprs> ::= extra credit
14. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} __________
15. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} ___________try_____________ / | \ ___resources____ block \ / | \ ? catches ( ___resource__ ? ? / | | \ res type: ? ? ? Connection ? ? ? <try> ::= try [resources] <block> <catches>;
16. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} ___________try_____________ / | \ ___resources____ block \ / | \ ? catches ( ___resource__ ? ? ? <resources> ::= ( <resource> {';' <resource>} );
17. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} ___________try_____________ / | \ ___resources____ block \ / | \ ? catches ( ___resource__ ? ? / | | \ res type: ?d: ? exp?s: Connection c getC() <resource> ::= [modifier] <res type> <id> '=' <exprs>;
18. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} ___________try_____________ / | \ ___resources____ block \ / | \ ? catches ( ___resource__ ? ? / | | \ res type: id: ? exp?s: Connection c getC() <resource> ::= [modifier] <res type> <id> '=' <exprs>;
19. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} ___________try_____________ / | \ ___resources____ block \ / | \ ? catches ( ___resource__ ? ? / | | \ res type: id: = exp?s: Connection c <resource> ::= [modifier] <res type> <id> '=' <exprs>;
20. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} ___________try_____________ / | \ ___resources____ block \ / | \ ? catches ( ___resource__ ? ? / | | \ res type: id: = exprs: Connection c getC() <resource> ::= [modifier] <res type> <id> '=' <exprs>;
21. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} ___________try_____________ / | \ ___resources____ block \ / | \ ? catches ( ___resource__ ) ? / | | \ res type: id: = exprs: Connection c getC() <resources> ::= ( <resource> {';' <resource>} );
22. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} ___________try_____________ / | \ ___resources____ block \ / | \ {…} catches ( ___resource__ ) ? / | | \ res type: id: = exprs: Connection c getC() <try> ::= try [resources] <block> <catches>;
23. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} ___________try_____________ / | \ ___resources____ block \ / | \ {…} catches ( ___resource__ ) ? / | | \ res type: id: = exprs: Connection c getC() <try> ::= try [resources] <block> <catches>;
24. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} ___________try_____________ / | \ ___resources____ block \ / | \ {…} catches ( ___resource__ ) ? / | | \ res type: id: = exprs: Connection c getC() <catches> ::= <catch> | <catch> <catches> | <finally>;
25. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} ___________try_____________ / | \ ___resources____ block \ / | \ {…} ____catches ( ___resource__ ) / / | | \ ______catch________ finally res type: id: = exprs: / | | \ \ Connection c getC() ? throwable id: ? block block ? ? ? <catch> ::= catch( <throwable> <identifier> ) <block>;
26. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} ___________try_____________ / | \ ___resources____ block \ / | \ {…} ____catches ( ___resource__ ) / / | | \ ______catch________ finally res type: id: = exprs: / | | \ \ Connection c getC() ? throwable id: ? block block ? ? ? <catches> ::= <catch> | <catch> <catches> | <finally>;
27. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} ___________try_____________ / | \ ___resources____ block \ / | \ {…} ____catches ( ___resource__ ) / / | | \ ______catch________ finally res type: id: = exprs: / | | \ \ Connection c getC() ( throwable id: ? block block ? ? ? <catch> ::= catch( <throwable> <identifier> ) <block>;
28. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} ___________try_____________ / | \ ___resources____ block \ / | \ {…} ____catches ( ___resource__ ) / / | | \ ______catch________ finally res type: id: = exprs: / | | \ \ Connection c getC() ( throwable id: ? block block / ? ? throw type: SQLException <throwable> ::= <throw type> {'|' <throw type>};
29. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} ___________try_____________ / | \ ___resources____ block \ / | \ {…} ____catches ( ___resource__ ) / / | | \ ______catch________ finally res type: id: = exprs: / | | \ \ Connection c getC() ( throwable id: ? block block / e ? throw type: SQLException <catch> ::= catch( <throwable> <identifier> ) <block>;
30. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} ___________try_____________ / | \ ___resources____ block \ / | \ {…} ____catches ( ___resource__ ) / / | | \ ______catch________ finally res type: id: = exprs: / | | \ \ Connection c getC() ( throwable id: ) block block / e ? throw type: SQLException <catch> ::= catch( <throwable> <identifier> ) <block>;
31. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} ___________try_____________ / | \ ___resources____ block \ / | \ {…} ____catches ( ___resource__ ) / / | | \ ______catch________ finally res type: id: = exprs: / | | \ \ Connection c getC() ( throwable id: ) block block / e {…} throw type: SQLException <catch> ::= catch( <throwable> <identifier> ) <block>;
32. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} ___________try_____________ / | \ ___resources____ block \ / | \ {…} ____catches ( ___resource__ ) / / | | \ ______catch________ finally res type: id: = exprs: / | | \ \ Connection c getC() ( throwable id: ) block block / e {…} throw type: SQLException <catches> ::= <catch> | <catch> <catches> | <finally>;
33. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} ___________try_____________ / | \ ___resources____ block \ / | \ {…} ____catches____ ( ___resource__ ) / \ / | | \ ______catch________ finally res type: id: = exprs: / | | \ \ \ Connection c getC() ( throwable id: ) block block / e {…} {?} throw type: SQLException <finally> ::= finally <block>;
34. try (Connection c = getC()) {...} catch (SQLException e) {...} finally {...} ___________try_____________ / | \ ___resources____ block \ / | \ {…} ____catches____ ( ___resource__ ) / \ / | | \ ______catch________ finally res type: id: = exprs: / | | \ \ \ Connection c getC() ( throwable id: ) block block / e {…} {…} throw type: SQLException <finally> ::= finally <block>;
36. Parser generators • Writing parsers is fun • Debugging them is not • Most plugins just use generators • Correct by construction • Easier to maintain
37. Platform/UI Core: Lexer, Parser, Indexer, Threading Code Awareness Plugins Languages and Grammars
38. Code Awareness and Comprehension • Understanding how your code is organized ◦ Code Quality: documentation, cleanliness, complexity ◦ Static Analysis: runtime, dataflow, security, concurrency • Understands how to diagnose and fix issues ◦ Debugging, refactoring, intentions ◦ Framework support and device/platform specific
39. Static analysis and debugging • Bugs are difficult to detect ◦ Obvious bugs are the most harmless ◦ Many bugs are hidden in plain sight • Static analysis catches runtime errors ◦ Checks for syntax then employs more advanced heuristics • Hot swap debugging
40. Code completion • Predictive suggestions • Chain completions • Context sensitive • Type aware • Did you mean...?
42. Dependency Resolution • Directed acyclic graph • Each dependency is represented by an edge • How do we build an application? • Using a topological sort!
43. commons-logging slf4j spring-core sitemesh spring-asm glassfish spring-web javassist jackson dom4j joda-time hudson jetty asm-commons
44. commons-logging slf4j spring-core sitemesh spring-asm glassfish spring-web javassist jackson dom4j joda-time hudson jetty asm-commons
45. spring-core spring-asm glassfish spring-web javassist jackson dom4j joda-time hudson jetty asm-commons commons-logging slf4j sitemesh
46. spring-core spring-asm glassfish spring-web javassist jackson dom4j joda-time hudson jetty asm-commons commons-logging slf4j sitemesh
47. glassfish spring-web javassist jackson dom4j joda-time hudson jetty asm-commons commons-logging slf4j sitemesh spring-asm spring-core
48. glassfish spring-web javassist jackson dom4j joda-time hudson jetty asm-commons commons-logging slf4j sitemesh spring-asm spring-core
49. glassfish javassist jackson dom4j joda-time hudson jetty asm-commons commons-logging slf4j sitemesh spring-asm spring-core spring-web
50. glassfish javassist jackson dom4j joda-time hudson jetty asm-commons commons-logging slf4j sitemesh spring-asm spring-core spring-web
51. javassist jackson dom4j joda-time hudson jetty asm-commons commons-logging slf4j sitemesh spring-asm spring-core spring-web glassfish
52. javassist jackson dom4j joda-time hudson jetty asm-commons commons-logging slf4j sitemesh spring-asm spring-core spring-web glassfish
53. javassist dom4j joda-time hudson asm-commons commons-logging slf4j sitemesh spring-asm spring-core spring-web glassfish jackson jetty
54. javassist dom4j joda-time hudson asm-commons commons-logging slf4j sitemesh spring-asm spring-core spring-web glassfish jackson jetty
55. joda-time hudson asm-commons commons-logging slf4j sitemesh spring-asm spring-core spring-web glassfish jackson jetty javassist dom4j
56. joda-time hudson asm-commons commons-logging slf4j sitemesh spring-asm spring-core spring-web glassfish jackson jetty javassist dom4j
57. asm-commons commons-logging slf4j sitemesh spring-asm spring-core spring-web glassfish jackson jetty javassist dom4j hudson joda-time
58. asm-commons commons-logging slf4j sitemesh spring-asm spring-core spring-web glassfish jackson jetty javassist dom4j hudson joda-time
59. commons-logging slf4j sitemesh spring-asm spring-core spring-web glassfish jackson jetty javassist dom4j hudson joda-time asm-commons
60. Core: Lexer, Parser, Indexer, Threading Code Awareness Plugins Languages and Grammars Platform/UI
66. Core: Lexer, Parser, Indexer, Threading Code Awareness Plugins Languages and Grammars Platform/UI
67. An IDE’s work is never done • There are uncountably many possible features • Languages & frameworks are constantly evolving • Hardware & platforms are constantly evolving • Programmers are constantly iterating software • Static analysis ⇔ Halting problem, cf. Rice
68. Too many features, too little time • Each feature has uncountably many use cases • Each feature needs to be tested for side effects • Each feature must be supported indefinitely • Hurray for plugin developers!
69. Semantic Merge • Language-aware diff/merge tool • Code is merged according to syntax tree
70. • Records a complete stack at each instant • Review the full trace after program has run
71. • Headless indexing and code review • Static analysis tools • Works with Maven/IDEA builds • Repository browser •
72. Core: Lexer, Parser, Indexer, Threading Code Awareness Plugins Languages and Grammars Platform/UI
73. Questions Comments
74. Thanks to all our Sponsors!