NLP + SE = ❤️

We present an overview of the current opportunities and challenges of applying NLP techniques to solve software engineering problems.

Transcript

1. NLP + SE = ❤ Georgios Gousios TU Delft
2. function setTimeout(callBack, delay){...}; browserSingleton.startPoller(100, function(delay, fn) { setTimeout(delay,fn); } ); Can you spot the bug?
3. Finding bugs function setTimeout(callBack, delay) {...}; browserSingleton.startPoller(100, function(delay, fn) { setTimeout(delay, fn); });
4. Finding bugs function setTimeout(callBack, delay) {...}; browserSingleton.startPoller(100, function(delay, fn) { setTimeout(delay, fn); }); Name that denotes a function Name that denotes a function
5. Finding bugs function setTimeout(callBack, delay) {...}; browserSingleton.startPoller(100, function(delay, fn) { setTimeout(delay, fn); }); Name that denotes a function Name that denotes a function Order of application Order of application
6. — Earl Barr, UCL “Source code is bimodal: it combines a formal, algorithmic channel and a natural language channel of identifiers and comments.”
7. Finding bugs function setTimeout(callBack, delay) {...}; browserSingleton.startPoller(100, function(delay, fn) { setTimeout(delay,fn); });
8. Finding bugs function setTimeout(callBack, delay) {...}; browserSingleton.startPoller(100, function(delay, fn) { setTimeout(delay,fn); }); Natural language channel Natural language channel
9. Finding bugs function setTimeout(callBack, delay) {...}; browserSingleton.startPoller(100, function(delay, fn) { setTimeout(delay,fn); }); Natural language channel Natural language channel Code semantics channel Code semantics channel
10. Static Analysis • Static analysis only captures the semantics channel • Bug detection and other forms of static analysis is pattern matching on increasingly precise semantics • Most static bug detectors find a subset of bugs (Habib and Pradel, ASE 2018) • Humans need to identify the patterns • As the semantics relax, static analysis becomes unsound • Almost impossible for dynamic languages (“stringly typed”)
11. function setTimeout(callBack: a -> b, delay: int){…}; browserSingleton.startPoller(100, function(delay, fn) { setTimeout(delay,fn); } );
12. function setTimeout(callBack: a -> b, delay: int){…}; browserSingleton.startPoller(100, function(delay, fn) { setTimeout(delay,fn); } ); The compiler can only help if humans add semantic information
13. How can NLP help? NLP approaches to software analysis aim to exploit the natural language information channel to help with tasks such as: • Bug finding • Type annotations • Inconsistencies • Source code summarisation • …
14. The Naturalness hypothesis “Software is a form of human communication; software corpora have similar statistical properties to natural language corpora; and these properties can be exploited to build better software engineering tools.” Hindle et al. On the naturalness of software. ICSE 2012
15. Naturalness showcased Hindle et al. On the naturalness of software. ICSE 2012 Code n-grams are less “surprising” to a language model than English
16. Naturalness showcased Hindle et al. On the naturalness of software. ICSE 2012 Code n-grams are less “surprising” to a language model than English We can train language models to predict next tokens better in code
17. Finding bugs Pradel and Shen. Deepbugs: A Learning Approach to Name-based Bug Detection. OOPSLA 2018 Training models to distinguish correct from buggy code Buggy code Correct code Buggy Correct
18. Finding bugs Pradel and Shen. Deepbugs: A Learning Approach to Name-based Bug Detection. OOPSLA 2018 How to produce buggy code? • Swap function arguments foo(a, b) -> foo(b, a) • Replace binary operators i <= length -> i % length • Replace binary operand i <= length -> i <= foo
19. Finding bugs Pradel and Shen. Deepbugs: A Learning Approach to Name-based Bug Detection. OOPSLA 2018 Training on 150k Javascript files Swapped arguments Wrong binary operator Wrong binary operand Accuracy 94% 92% 89%
20. Predicting types def bigger_number(a, b): if a > b: return a else: return b Python 2.7 code
21. Predicting types def bigger_number(a, b): if a > b: return a else: return b Python 2.7 code Geen Python 2.7 na 2019!
22. Predicting types def bigger_number(a: ???, b: ???) -> ???: if a > b: return a else: return b Python 3.5+ code. Can you guess the types?
23. Predicting types def bigger_number(a: ???, b: ???) -> ???: if a > b: return a else: return b Python 3.5+ code. Can you guess the types? How can we automatically annotate JavaScript/Python code with types?
24. Predicting types def is_bigger(a:int, b:int) -> boolean: “”” Returns True if a is number a is bigger than b, else False “”” return a > b Learning from existing code annotations
25. Predicting types def is_bigger(a:int, b:int) -> boolean: “”” Returns True if a is number a is bigger than b, else False “”” return a > b Learning from existing code annotations embedding
26. Predicting types def is_bigger(a:int, b:int) -> boolean: “”” Returns True if a is number a is bigger than b, else False “”” return a > b Learning from existing code annotations embedding
27. Predicting types def is_bigger(a:int, b:int) -> boolean: “”” Returns True if a is number a is bigger than b, else False “”” return a > b Learning from existing code annotations embedding sequence learning
28. Predicting types def is_bigger(a:int, b:int) -> boolean: “”” Returns True if a is number a is bigger than b, else False “”” return a > b Learning from existing code annotations embedding sequence learning
29. Predicting types def is_bigger(a:int, b:int) -> boolean: “”” Returns True if a is number a is bigger than b, else False “”” return a > b Learning from existing code annotations embedding sequence learning
30. Predicting types def is_bigger(a:int, b:int) -> boolean: “”” Returns True if a is number a is bigger than b, else False “”” return a > b Learning from existing code annotations embedding sequence learning concat +
31. Predicting types def is_bigger(a:int, b:int) -> boolean: “”” Returns True if a is number a is bigger than b, else False “”” return a > b Learning from existing code annotations embedding sequence learning concat prediction +
32. Predicting types Results on 500 partially annotated GitHub projects return type argument type combined precision .67 .61 .65 recall .62 .57 .59 precision .76 .77 .80 recall .70 .70 .71 Top - 1 Top -3
33. Finding inconsistencies How to find inconsistent function/variable names? Liu et al. Learning to Spot and Refactor Inconsistent Method Names. ICSE 2019
34. Finding inconsistencies How to find inconsistent function/variable names? Liu et al. Learning to Spot and Refactor Inconsistent Method Names. ICSE 2019
35. Finding inconsistencies How to find inconsistent function/variable names? Liu et al. Learning to Spot and Refactor Inconsistent Method Names. ICSE 2019
36. Finding inconsistencies How to find inconsistent function/variable names? Liu et al. Learning to Spot and Refactor Inconsistent Method Names. ICSE 2019
37. Finding inconsistencies How to find inconsistent function/variable names? Liu et al. Learning to Spot and Refactor Inconsistent Method Names. ICSE 2019
38. Finding inconsistencies How to find inconsistent function/variable names? Liu et al. Learning to Spot and Refactor Inconsistent Method Names. ICSE 2019 Methods with the similar names should have similar bodies
39. Finding inconsistencies Liu et al. Learning to Spot and Refactor Inconsistent Method Names. ICSE 2019
40. Finding inconsistencies Liu et al. Learning to Spot and Refactor Inconsistent Method Names. ICSE 2019 1. Build embeddings of function names and body vectors 2. For each function body: 1. Find functions close to it in vector space 2. Check their respective name distance
41. Code summarization Wan et al. Improving automatic source code summarization via deep reinforcement learning. ASE 2018 def add(a, b): return a + b def ???(a, b): return a + b
42. Code summarization Wan et al. Improving automatic source code summarization via deep reinforcement learning. ASE 2018 def add(a, b): return a + b “”” Adds two numbers “”” def ???(a, b): return a + b
43. Code summarization Wan et al. Improving automatic source code summarization via deep reinforcement learning. ASE 2018 def add(a, b): return a + b “”” Adds two numbers “”” def ???(a, b): return a + b add
44. Code summarization Wan et al. Improving automatic source code summarization via deep reinforcement learning. ASE 2018 Use critic network to re-adjust model weights BLEU score of 0.35
45. Code summarization Wan et al. Improving automatic source code summarization via deep reinforcement learning. ASE 2018 NL channel Semantics channel Use critic network to re-adjust model weights BLEU score of 0.35
46. Code summarization Wan et al. Improving automatic source code summarization via deep reinforcement learning. ASE 2018 NL channel Semantics channel Use critic network to re-adjust model weights BLEU score of 0.35
47. Main challenges • Developers like to invent names • Code vocabularies are 10x the size of NL ones • Compression techniques (e.g. BPE) to the rescue • How to feed code to a network without loosing info from either the NL or the semantics channel? • Code2Vec, TreeLSTMs, GGNNs,… • Keeping up with evolution • Making tools — not just research papers
48. ML4SE course! • Student presentations of course projects, including poster sessions • Oct 30, 13:45 - 17:00, Pulse-Hall 7