Slide 10
Slide 10 text
Python lexer
1,4-1,5: NAME 'i'
1,0-1,3: NAME 'for'
1,6-1,8: NAME 'in'
1,9-1,10: OP '['
1,10-1,11: NUMBER '1'
1,11-1,12: OP ','
1,12-1,13: NUMBER '2'
…
2,0-2,0: ENDMARKER ''
python_line = “i for in [1,2]"
for token in tokens:
handle_token(token)