Katie Bell - The computer science of marking computer science assignments

When writing systems to test if beginner programmers' code was correct, I didn't expect to need numpy, scipy a custom C module and a whole lot of cool geometry algorithms. Giving actionable feedback on tasks (in this case logo/turtle vector drawings), is necessary for the learning process and goes some fun places. Take this as a case study of writing efficient geometry number crunching in Python.

https://us.pycon.org/2016/schedule/presentation/2247/

Transcript

1. The Computer Science of Marking Computer Science Assignments Katie Bell Software Engineer - Grok Learning
2. I’ve never heard of this ‘Grok Learning’ Image: CC0 Public Domain
3. Background
4. Let me tell you a story... Image: The Princess Bride (1987)
5. Turtles “It’s turtles all the way down!” 1
6. Example from turtle import * left(60) forward(100) right(120) forward(100) right(120) forward(100) left(90) forward(100) left(90) forward(100) left(90) forward(100)
7. Example challenges
8. ◦ Simple, graphical, engaging ◦ Built-in module in Python ◦ Teachers were asking for it ◦ Ties in well with maths teaching Why Turtle?
9. True to built-in Python Turtle in the browser Frontend Sandboxed Python Browser Code and stdin Realtime animation events SVG animation Modified turtle.py
10. The marking system Because feedback is important. 2
11. “You’re missing this line: {diagram}” “The angle between these two lines should be {angle} but it’s {angle}°.” “This line is the wrong colour. It was {colour1} but it should be {colour2}.” “There’s an extra line here {diagram}, that shouldn’t be there.” Meaningful feedback goals
12. 1. Rendered pixel-level differences 2. Vector comparison Marking Options:
13. Comparing pixel by pixel angle = 360/7 angle = int(360/7)
14. Vector Comparison Down the rabbit hole... 3
15. Visually distinguishable differences between drawings should matter. ◦ Additional lines, Missing lines ◦ Lines of different lengths/positions So we have two drawings: Expected and Actual
16. ◦ Fuzzy floating point matching ◦ Drawing order should not matter. ◦ Overlapping lines should not matter. ◦ Translation of the whole drawing should not matter Additional Requirements:
17. Connected & Overlapping Lines
18. Simplifying Raw line segments from turtle drawing Minimised line segments
19. Where we’re at:
20. Filled areas! “It’s complicated.” - Facebook 4
21. from turtle import * fillcolor('red') begin_fill() for i in range(3): forward(100) left(120) end_fill() Fills with turtle
22. The fun of fills
23. The fun of fills
24. The fun of fills White rectangle drawn over the top
25. The fun of fills
26. “Resolving overlapping polygons is a solved problem right? There should be a ready made algorithm and library for this.” - Me
27. Convex Simple Polygon Types of polygons Simple Polygon Complex Polygon (Self intersecting)
28. Triangulate Image Source: http://hyperboleandahalf.blogspot.com/
29. Delaunay Triangulation Image: Public Domain
30. from scipy.spatial import Delaunay Conveniently in Scipy
31. 1. Resolve shapes to non-overlapping triangles 2. Work out what colour each triangle is 3. Stitch the triangles together to form an outline of the filled area 4. Compare the vectors of filled area outlines for each colour Here’s the plan:
32. Applying triangulation
33. Applying triangulation
34. Applying triangulation
35. Take the centroid of each triangle. Determine the highest (drawing order) shape which contains that point. Colour each triangle
36. Testing if a point is in a complex polygon
37. Testing if a point is in a complex polygon
38. Testing if a point is in a complex polygon
39. Testing if a point is in a complex polygon
40. INTERSECTING LINES Trickier than originally thought.
41. In order for the triangulation to work
42. In order for the triangulation to work
43. Resolve all the shapes in one triangulation
44. Resolve all the shapes in one triangulation
45. Stitching together
46. Stitching together
47. Stitching together
48. Stitching together
49. Stitching together
50. Stitching together
51. Stitching together
52. Simplify the vectors and we're done!
53. Sweet, it works! But there's one small problem... 5
54. Birthday Banner
55. 6-8 seconds Per test case, for 6 test cases.
56. Triangulation of Birthday Banner:
57. $ nosetests test_checker_logo_diff.py --with-profile ................................................ 13327123 function calls (13321330 primitive calls) in 22.131 seconds Ordered by: cumulative time ncalls cumtime percall filename:lineno(function) 9/1 22.131 22.131 site-packages/nose/suite.py:176(__call__) 9/1 22.131 22.131 site-packages/nose/suite.py:197(run) 48 22.129 0.461 site-packages/nose/case.py:44(__call__) 48 22.129 0.461 site-packages/nose/case.py:115(run) 48 22.124 0.461 site-packages/nose/case.py:142(runTest) 48 22.124 0.461 unittest/case.py:394(__call__) 48 22.124 0.461 unittest/case.py:297(run) 38 22.091 0.581 checker_logo_diff.py:1094(check) 146 21.554 0.148 checker_logo_diff.py:925(resolve_fill_triangles) 90280 20.810 0.000 checker_logo_diff.py:140(is_intersection) 538466 14.157 0.000 site-packages/numpy/core/numeric.py:2276(isclose) 467962 13.619 0.000 site-packages/numpy/core/numeric.py:2212(allclose) 76 11.475 0.151 checker_logo_diff.py:773(collapse_shapes_to_lines) Profiling
58. Attempt 1: Bentley–Ottmann algorithm
59. Conditions: 1. No two line segment endpoints or crossings have the same x-coordinate 2. No line segment endpoint lies upon another line segment 3. No three line segments intersect at a single point. Attempt 1: Bentley–Ottmann algorithm
60. ◦ 2 days of development time ◦ 84% of test cases passing ◦ Speed roughly the same as before Conclusion: Not working Attempt 1: Bentley–Ottmann algorithm
61. Attempt 2: CGAL The Computational Geometry Algorithms Library (C++)
62. github.com/CGAL/cgal-swig-bindings “This project is still experimental and more packages will be added.” So uh… Python bindings?
63. But it was too precise...
64. ◦ 305 lines of horrific SWIG and C++ typedefs ◦ ~1 day of development time ◦ Test suite even slower Conclusion: Not working Attempt 2: CGAL
65. is_intersection
66. Attempt 3: Write a custom C Module def is_intersection(seg1, seg2, check_dir=False, ends=True): return fastlogo.fast_is_intersection( tuple(seg1[0]), tuple(seg1[1]), tuple(seg2[0]), tuple(seg2[1]), check_dir, ends)
67. ~10x speed increase! For both birthday banner and the test suite.
68. Lessons Learned: 1. Geometry is fun! 2. Profiling is the best. 3. If you want more speed, forget algorithms and rewrite it in C.
69. ◦ True-to-builtin Python Turtle in the browser ◦ Good automated feedback to students Todo: ◦ Add yet more detailed feedback ◦ More algorithmic/speed improvements Where we are now:
70. Questions? @notsolonecoder, @groklearning katie@groklearning.com Try a turtle activity (Hour of Code): groklearning.com/hoc-2015