UP Lecture 06

Compilers
Lexer Testing
(202402)

Transcript

1. jgs Compilers Lecture 06: Lexer Implementation Dr. Javier Gonzalez-Sanchez jgonzalezs@up.edu.mx
2. Dr. Javier Gonzalez-Sanchez | Compilers | 2 jgs Start With a DFA B,b 0 1 .. . Delimiter, operator, whitespace, quotation mark S0 SE S1 SE SE Stop S1 S2 SE SE SE Stop S2 SE S3 S3 SE Stop S3 SE S3 S3 SE Stop SE SE SE SE SE Stop
3. Compilers | Dr. Javier Gonzalez-Sanchez | 3 jgs s0/0 = s1 s1/B = s2 s1/b = s2 s2/0 = s3 s2/1 = s3 s3/0 = s3 s3/1 = s3
4. Dr. Javier Gonzalez-Sanchez | Compilers | 4 jgs Class Token
5. Dr. Javier Gonzalez-Sanchez | Compilers | 5 jgs Class Lexer B,b 0 1 .. . Delimiter, operator, whitespace, quotation mark S0 SE S1 SE SE Stop S1 S2 SE SE SE Stop S2 SE S3 S3 SE Stop S3 SE S3 S3 SE Stop SE SE SE SE SE Stop
6. Dr. Javier Gonzalez-Sanchez | Compilers | 6 jgs Algorithm 1. Initialize currentState to “s0” 2. Create an empty string to store words, 3. Index to track position in the input line. 4. Loop through each character in the input line. ▪ if (the character is not an operator, delimiter, or space) { - get the next state from the DFA, - append the character to the current token, - and update currentState. ▪ } otherwise { - If currentState is an accepting state, store the token with its state name; Otherwise, store it as an error. - If the currentCharacter is an operator, store it as an “OPERATOR” token; - if the currentCharacter is a delimiter, store it as a “DELIMITER” token. - Reset currentState to “s0” and clear the string storage. } 5. After processing all characters, check the last string/word and store it accordingly.
7. Compilers | Dr. Javier Gonzalez-Sanchez | 7 jgs
8. Dr. Javier Gonzalez-Sanchez | Compilers | 8 jgs Class Lexer
9. Dr. Javier Gonzalez-Sanchez | Compilers | 9 jgs Class Main
10. Dr. Javier Gonzalez-Sanchez | Compilers | 10 jgs Run
11. Dr. Javier Gonzalez-Sanchez | Compilers | 11 jgs Homework Review this code: https://github.com/javiergs/TheLexer
12. jgs Next Step
13. Dr. Javier Gonzalez-Sanchez | Compilers | 13 jgs Homework Teams of Three Members
14. Dr. Javier Gonzalez-Sanchez | Compilers | 14 jgs Homework Programming Assignment Develop a Lexical Analyzer by coding a DFA
15. Dr. Javier Gonzalez-Sanchez | Compilers | 15 jgs Question
16. Dr. Javier Gonzalez-Sanchez | Compilers | 16 jgs Github Classroom https://classroom.github.com/a/N9Wx8bTE
17. Dr. Javier Gonzalez-Sanchez | Compilers | 17 jgs Step 1
18. Dr. Javier Gonzalez-Sanchez | Compilers | 18 jgs Step 2
19. Dr. Javier Gonzalez-Sanchez | Compilers | 19 jgs Step 3
20. Dr. Javier Gonzalez-Sanchez | Compilers | 20 jgs Step 4
21. Dr. Javier Gonzalez-Sanchez | Compilers | 21 jgs Step 5
22. Dr. Javier Gonzalez-Sanchez | Compilers | 22 jgs Step 6
23. Dr. Javier Gonzalez-Sanchez | Compilers | 23 jgs Step 7
24. Dr. Javier Gonzalez-Sanchez | Compilers | 24 jgs Step 8
25. Dr. Javier Gonzalez-Sanchez | Compilers | 25 jgs Step 9
26. Dr. Javier Gonzalez-Sanchez | Compilers | 26 jgs Step 10
27. Dr. Javier Gonzalez-Sanchez | Compilers | 27 jgs Step 11
28. Dr. Javier Gonzalez-Sanchez | Compilers | 28 jgs
29. Dr. Javier Gonzalez-Sanchez | Compilers | 29 jgs
30. Dr. Javier Gonzalez-Sanchez | Compilers | 30 jgs Test
31. Dr. Javier Gonzalez-Sanchez | Compilers | 31 jgs One Last Thing
32. jgs Testing
33. Dr. Javier Gonzalez-Sanchez | Compilers | 33 jgs Code | input.txt hello;world cse340 up 2013/05/31 // end boolean $xx= ((((((((23WE + 44 - 3 / 2 % 45 <=17) > 0xfffff.34.45;
34. Javier Gonzalez-Sanchez | SER332 | Spring 2018 | 34 jgs DELIMITER ( DELIMITER ( DELIMITER ( DELIMITER ( ERROR 23WE OPERATOR + INTEGER 44 OPERATOR - INTEGER 3 OPERATOR / INTEGER 2 OPERATOR % INTEGER 45 OPERATOR < OPERATOR = INTEGER 17 DELIMITER ) OPERATOR > ERROR 0xfffff.34.45 DELIMITER ; IDENTIFIER hello DELIMITER ; IDENTIFIER world IDENTIFIER cse340 IDENTIFIER up INTEGER 2013 OPERATOR / OCTAL 05 OPERATOR / INTEGER 31 OPERATOR / OPERATOR / IDENTIFIER end KEYWORD boolean IDENTIFIER $xx OPERATOR = DELIMITER ( DELIMITER ( DELIMITER ( DELIMITER ( Code | output.txt
35. Dr. Javier Gonzalez-Sanchez | Compilers | 35 jgs Questions
36. Dr. Javier Gonzalez-Sanchez | Compilers | 36 jgs Homework 1. Finish Your Lexer. Team Project. Due in a week.
37. jgs Compilers Javier Gonzalez-Sanchez, Ph.D. jgonzalezs@up.edu.mx Spring 2024 Copyright. These slides can only be used as study material for the Compilers course at Universidad Panamericana. They cannot be distributed or used for another purpose.