Slide 1

Slide 1 text

The Computer Science of Marking Computer Science Assignments Katie Bell Software Engineer - Grok Learning

Slide 2

Slide 2 text

I’ve never heard of this ‘Grok Learning’ Image: CC0 Public Domain

Slide 3

Slide 3 text

Background

Slide 4

Slide 4 text

Let me tell you a story... Image: The Princess Bride (1987)

Slide 5

Slide 5 text

Turtles “It’s turtles all the way down!” 1

Slide 6

Slide 6 text

Example from turtle import * left(60) forward(100) right(120) forward(100) right(120) forward(100) left(90) forward(100) left(90) forward(100) left(90) forward(100)

Slide 7

Slide 7 text

Example challenges

Slide 8

Slide 8 text

○ Simple, graphical, engaging ○ Built-in module in Python ○ Teachers were asking for it ○ Ties in well with maths teaching Why Turtle?

Slide 9

Slide 9 text

True to built-in Python Turtle in the browser Frontend Sandboxed Python Browser Code and stdin Realtime animation events SVG animation Modified turtle.py

Slide 10

Slide 10 text

The marking system Because feedback is important. 2

Slide 11

Slide 11 text

“You’re missing this line: {diagram}” “The angle between these two lines should be {angle} but it’s {angle}°.” “This line is the wrong colour. It was {colour1} but it should be {colour2}.” “There’s an extra line here {diagram}, that shouldn’t be there.” Meaningful feedback goals

Slide 12

Slide 12 text

1. Rendered pixel-level differences 2. Vector comparison Marking Options:

Slide 13

Slide 13 text

Comparing pixel by pixel angle = 360/7 angle = int(360/7)

Slide 14

Slide 14 text

Vector Comparison Down the rabbit hole... 3

Slide 15

Slide 15 text

Visually distinguishable differences between drawings should matter. ○ Additional lines, Missing lines ○ Lines of different lengths/positions So we have two drawings: Expected and Actual

Slide 16

Slide 16 text

○ Fuzzy floating point matching ○ Drawing order should not matter. ○ Overlapping lines should not matter. ○ Translation of the whole drawing should not matter Additional Requirements:

Slide 17

Slide 17 text

Connected & Overlapping Lines

Slide 18

Slide 18 text

Simplifying Raw line segments from turtle drawing Minimised line segments

Slide 19

Slide 19 text

Where we’re at:

Slide 20

Slide 20 text

Filled areas! “It’s complicated.” - Facebook 4

Slide 21

Slide 21 text

from turtle import * fillcolor('red') begin_fill() for i in range(3): forward(100) left(120) end_fill() Fills with turtle

Slide 22

Slide 22 text

The fun of fills

Slide 23

Slide 23 text

The fun of fills

Slide 24

Slide 24 text

The fun of fills White rectangle drawn over the top

Slide 25

Slide 25 text

The fun of fills

Slide 26

Slide 26 text

“Resolving overlapping polygons is a solved problem right? There should be a ready made algorithm and library for this.” - Me

Slide 27

Slide 27 text

Convex Simple Polygon Types of polygons Simple Polygon Complex Polygon (Self intersecting)

Slide 28

Slide 28 text

Triangulate Image Source: http://hyperboleandahalf.blogspot.com/

Slide 29

Slide 29 text

Delaunay Triangulation Image: Public Domain

Slide 30

Slide 30 text

from scipy.spatial import Delaunay Conveniently in Scipy

Slide 31

Slide 31 text

1. Resolve shapes to non-overlapping triangles 2. Work out what colour each triangle is 3. Stitch the triangles together to form an outline of the filled area 4. Compare the vectors of filled area outlines for each colour Here’s the plan:

Slide 32

Slide 32 text

Applying triangulation

Slide 33

Slide 33 text

Applying triangulation

Slide 34

Slide 34 text

Applying triangulation

Slide 35

Slide 35 text

Take the centroid of each triangle. Determine the highest (drawing order) shape which contains that point. Colour each triangle

Slide 36

Slide 36 text

Testing if a point is in a complex polygon

Slide 37

Slide 37 text

Testing if a point is in a complex polygon

Slide 38

Slide 38 text

Testing if a point is in a complex polygon

Slide 39

Slide 39 text

Testing if a point is in a complex polygon

Slide 40

Slide 40 text

INTERSECTING LINES Trickier than originally thought.

Slide 41

Slide 41 text

In order for the triangulation to work

Slide 42

Slide 42 text

In order for the triangulation to work

Slide 43

Slide 43 text

Resolve all the shapes in one triangulation

Slide 44

Slide 44 text

Resolve all the shapes in one triangulation

Slide 45

Slide 45 text

Stitching together

Slide 46

Slide 46 text

Stitching together

Slide 47

Slide 47 text

Stitching together

Slide 48

Slide 48 text

Stitching together

Slide 49

Slide 49 text

Stitching together

Slide 50

Slide 50 text

Stitching together

Slide 51

Slide 51 text

Stitching together

Slide 52

Slide 52 text

Simplify the vectors and we're done!

Slide 53

Slide 53 text

Sweet, it works! But there's one small problem... 5

Slide 54

Slide 54 text

Birthday Banner

Slide 55

Slide 55 text

6-8 seconds Per test case, for 6 test cases.

Slide 56

Slide 56 text

Triangulation of Birthday Banner:

Slide 57

Slide 57 text

$ nosetests test_checker_logo_diff.py --with-profile ................................................ 13327123 function calls (13321330 primitive calls) in 22.131 seconds Ordered by: cumulative time ncalls cumtime percall filename:lineno(function) 9/1 22.131 22.131 site-packages/nose/suite.py:176(__call__) 9/1 22.131 22.131 site-packages/nose/suite.py:197(run) 48 22.129 0.461 site-packages/nose/case.py:44(__call__) 48 22.129 0.461 site-packages/nose/case.py:115(run) 48 22.124 0.461 site-packages/nose/case.py:142(runTest) 48 22.124 0.461 unittest/case.py:394(__call__) 48 22.124 0.461 unittest/case.py:297(run) 38 22.091 0.581 checker_logo_diff.py:1094(check) 146 21.554 0.148 checker_logo_diff.py:925(resolve_fill_triangles) 90280 20.810 0.000 checker_logo_diff.py:140(is_intersection) 538466 14.157 0.000 site-packages/numpy/core/numeric.py:2276(isclose) 467962 13.619 0.000 site-packages/numpy/core/numeric.py:2212(allclose) 76 11.475 0.151 checker_logo_diff.py:773(collapse_shapes_to_lines) Profiling

Slide 58

Slide 58 text

Attempt 1: Bentley–Ottmann algorithm

Slide 59

Slide 59 text

Conditions: 1. No two line segment endpoints or crossings have the same x-coordinate 2. No line segment endpoint lies upon another line segment 3. No three line segments intersect at a single point. Attempt 1: Bentley–Ottmann algorithm

Slide 60

Slide 60 text

○ 2 days of development time ○ 84% of test cases passing ○ Speed roughly the same as before Conclusion: Not working Attempt 1: Bentley–Ottmann algorithm

Slide 61

Slide 61 text

Attempt 2: CGAL The Computational Geometry Algorithms Library (C++)

Slide 62

Slide 62 text

github.com/CGAL/cgal-swig-bindings “This project is still experimental and more packages will be added.” So uh… Python bindings?

Slide 63

Slide 63 text

But it was too precise...

Slide 64

Slide 64 text

○ 305 lines of horrific SWIG and C++ typedefs ○ ~1 day of development time ○ Test suite even slower Conclusion: Not working Attempt 2: CGAL

Slide 65

Slide 65 text

is_intersection

Slide 66

Slide 66 text

Attempt 3: Write a custom C Module def is_intersection(seg1, seg2, check_dir=False, ends=True): return fastlogo.fast_is_intersection( tuple(seg1[0]), tuple(seg1[1]), tuple(seg2[0]), tuple(seg2[1]), check_dir, ends)

Slide 67

Slide 67 text

~10x speed increase! For both birthday banner and the test suite.

Slide 68

Slide 68 text

Lessons Learned: 1. Geometry is fun! 2. Profiling is the best. 3. If you want more speed, forget algorithms and rewrite it in C.

Slide 69

Slide 69 text

○ True-to-builtin Python Turtle in the browser ○ Good automated feedback to students Todo: ○ Add yet more detailed feedback ○ More algorithmic/speed improvements Where we are now:

Slide 70

Slide 70 text

Questions? @notsolonecoder, @groklearning [email protected] Try a turtle activity (Hour of Code): groklearning.com/hoc-2015