Mit Schema-Validierung APIs lahmlegen

Das "JSON-Schema Specification Draft" ist der aktuelle Standard zur Verifizierung von JSON-Datenstrukturen. Leider enthält die Spezifikation eine Schwachstelle, mit der man eine Applikation komplett blockieren kann. In meinem Vortrag zeige ich, wie man reguläre Ausdrücke für eine einfache DOS-Attacke ausnutzt, und warum ein Schutz davor immer die Spezifikation verletzt.

Der Zuhörer lernt auch, weshalb die Situation bei JSON-Schema noch schwieriger ist als beim älteren XML-Standard. Denn XML-Schema-Bibliotheken sind anfällig für dieselbe Attacke, aber nur deshalb, weil sie von der W3C-Spezifikation abweichen.

Die Code-Snippets für XML- und JSON-Validierung sind online unter https://github.com/rkeytacked/java-redos .

Weitere Links aus den Slides:
- XML-Schema Regular Expression definition, https://www.w3.org/TR/xmlschema-2/#dt-regex
- Russ Cox' paper on regular expression engines, https://swtch.com/~rsc/regexp/regexp1.html
- ReDoS on OWASP, https://www.owasp.org/index.php/Regular_expression_Denial_of_Service_-_ReDoS
- .net Regular Expressions, https://docs.microsoft.com/en-us/dotnet/standard/base-types/backtracking-in-regular-expressions
- JSON Schema Definition of Regular Expressions, http://json-schema.org/latest/json-schema-validation.html#rfc.section.4.3
- java.util.regex.Pattern, https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html
- RE2/J, https://github.com/google/re2j
- PR to fix json-schema, https://github.com/everit-org/json-schema/pull/148
- PR to fix nakadi, https://github.com/zalando/nakadi/pull/853
- json-schema handling of Regular Expressions, https://github.com/everit-org/json-schema#regexp-implementations

Transcript

1. Mit Schema-Validierung APIs lahmlegen EIN AUSFLUG IN DIE WELT DER REGULÄREN AUSDRÜCKE UND DAS VERTRAUEN IN KOMPLIZIERTE LIBRARIES PETER LISKE 15-10-2018
2. Regular Expressions ❤ RegExp Engines Schema Validation ReDoS Attacks © Disney
3. 3 CHOMSKY HIERARCHY ⇔ AUTOMATA THEORY finite-state automaton
4. 4 BEAUTIFULLY SIMPLE REGULAR EXPRESSIONS Regular Expression due to W3C XML Schema specification a|b|c a* b{3,5} c? xyz (...) BRANCHES CONCATENATION QUANTIFIERS GROUPING . [a-z] \d CHARACTER SETS © Disney
5. REGULAR EXPRESSIONS FOR XML SCHEMA VALIDATION CAN BE EVALUATED IN LINEAR TIME LESSON #1 © Disney
6. 6 PATTERN MATCHING IN XML VALIDATION LIBRARIES Did you ever look into XML libraries’ code to see how they do RegExp matching? Example: https://github.com/rkeytacked/java-redos
7. ALL MAJOR XML SCHEMA VALIDATION LIBRARIES DO NOT CONFORM TO THE SPECS AND HAVE EXPONENTIAL WORST-CASE RUNTIME LESSON #2 © Disney
8. 8 match pattern against input REALITY CHECK (Russ Cox, 2007) Source: https://swtch.com/~rsc/regexp/regexp1.html
9. 9 HISTORY OF REGULAR EXPRESSIONS Kleene’s Regular Sets Chomsky Hierarchy usage in compiler design 1970 2000 Thompson: qed ed grep 2010 1960 1950 1980 1990 more UNIX tools: vi lex sed awk expr emacs Perl 2 even more features in Reg Exps Spencer’s Advanced Reg Exps used in specifications POSIX.2 standard Hazel’s PCRE library XML Schema definitions (XSD) JSON Schema definitions ReDoS attacks known ECMA- Script standards Russ Cox’ RE2
10. 10 MICROSOFT .NET https://docs.microsoft.com/en-us/dotnet/standard/base-types/backtracking-in-regular-expressions
11. 11 MICROSOFT .NET https://docs.microsoft.com/en-us/dotnet/standard/base-types/backtracking-in-regular-expressions Pattern needs backtracking?!?!
12. “ADVANCED” REGULAR EXPRESSION ENGINES ARE SO COMPLICATED THAT IT’S EASIER TO BLAME THE DEVELOPER THAN TO WRITE BETTER ENGINES LESSON #3 © Disney
13. 13 SIMPLE REGULAR EXPRESSIONS Regular Expression due to W3C XML Schema specification a|b|c a* b{3,5} c? xyz (...) BRANCHES CONCATENATION QUANTIFIERS GROUPING . [a-z] \d CHARACTER SETS
14. 14 ADVANCED FEATURES Anchors $ END OF A LINE OR TEXT ^ START OF A LINE OR TEXT
15. 15 ADVANCED FEATURES Lookarounds (?= … ) (?! … ) (?<! … ) LOOK AHEAD NEGATIVE LOOK AHEAD NEGATIVE LOOK BEHIND (?<= … ) LOOK BEHIND
16. 16 ADVANCED FEATURES Atomic Groups (?> … ) ATOMIC MATCH
17. 17 ADVANCED FEATURES Inline Modifiers (?i … ) (?x … ) (?m … ) CASE INSENSITIVE FREE SPACING MODE MULTILINE MODE (?s … ) SINGLE LINE MODE
18. 18 ADVANCED FEATURES Sub-Routines (?N) (?&NAME) REPEAT Nth PATTERN REPEAT NAMED PATTERN (?(DEFINE) (<FOO> … ) (<BAR> … )) DEFINITION BLOCK
19. 19 ADVANCED FEATURES Conditionals (?(A) … ) IF CONDITION A MATCHED THEN … (?(A)…|…) IF CONDITION A MATCHED THEN … ELSE …
20. 20 ADVANCED FEATURES Inline Comments (?# … ) JUST A COMMENT IN THE MIDDLE OF YOUR REGEX (?x) # some # comment COMMENTS IN FREE-SPACING MODE
21. 21 ADVANCED FEATURES Inline Code (?{ … }) PERL CODE CAPSULE
22. 22 JSON SCHEMA REGULAR EXPRESSIONS Regular Expression due to JSON Schema specification ^ … $ ECMA-262 Regular Expression Syntax LIBRARY DEVELOPER, you SHOULD implement a full backtracking engine. SCHEMA AUTHOR, you SHOULD NOT use any of these creepy features. simple Reg Exps ?
23. JSON SCHEMA SPECIFICATION ALLOWS BACKTRACKING IN PATTERN MATCHING HENCE ENFORCES EXPONENTIAL WORST-CASE RUNTIME LESSON #4 © Disney
24. 24 PATTERN MATCHING IN JSON VALIDATION LIBRARIES Did you ever look into JSON libraries’ code to see how they do RegExp matching? Example: https://github.com/rkeytacked/java-redos
25. 25 HOW TO FIX PATTERN MATCHING FOR YOUR JSON-VALIDATING API JSON-Schema Your JSON schema validating API e.g. zalando/nakadi JSON schema lib e.g. json-schema RegExp engine e.g. java.util.regex or RE2/J “… and they lived happily ever after?” “No.”
26. YOU CAN SECURE YOUR SCHEMA-VALIDATING API BUT IT’S NOT CONFORM TO JSON SCHEMA VALIDATION DRAFT LESSON #5 © Disney
27. 27 BRAINSTORMING: WHAT CAN WE IMPROVE? Kleene’s Regular Sets Chomsky Hierarchy usage in compiler design 1970 2000 Thompson: qed ed grep 2010 1960 1950 1980 1990 more UNIX tools: vi lex sed awk expr emacs Perl 2 even more features in Reg Exps Spencer’s Advanced Reg Exps used in specifications POSIX.2 standard Hazel’s PCRE library XML Schema definitions (XSD) JSON Schema definitions ReDoS attacks known ECMA- Script standards Russ Cox’ RE2
28. 28 THE END ALMOST NO WEB SERVICES WERE HARMED DURING THE MAKING OF THIS TALK © Disney