Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mit Schema-Validierung APIs lahmlegen

Mit Schema-Validierung APIs lahmlegen

Das "JSON-Schema Specification Draft" ist der aktuelle Standard zur Verifizierung von JSON-Datenstrukturen. Leider enthält die Spezifikation eine Schwachstelle, mit der man eine Applikation komplett blockieren kann. In meinem Vortrag zeige ich, wie man reguläre Ausdrücke für eine einfache DOS-Attacke ausnutzt, und warum ein Schutz davor immer die Spezifikation verletzt.

Der Zuhörer lernt auch, weshalb die Situation bei JSON-Schema noch schwieriger ist als beim älteren XML-Standard. Denn XML-Schema-Bibliotheken sind anfällig für dieselbe Attacke, aber nur deshalb, weil sie von der W3C-Spezifikation abweichen.

Die Code-Snippets für XML- und JSON-Validierung sind online unter https://github.com/rkeytacked/java-redos .

Weitere Links aus den Slides:
- XML-Schema Regular Expression definition, https://www.w3.org/TR/xmlschema-2/#dt-regex
- Russ Cox' paper on regular expression engines, https://swtch.com/~rsc/regexp/regexp1.html
- ReDoS on OWASP, https://www.owasp.org/index.php/Regular_expression_Denial_of_Service_-_ReDoS
- .net Regular Expressions, https://docs.microsoft.com/en-us/dotnet/standard/base-types/backtracking-in-regular-expressions
- JSON Schema Definition of Regular Expressions, http://json-schema.org/latest/json-schema-validation.html#rfc.section.4.3
- java.util.regex.Pattern, https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html
- RE2/J, https://github.com/google/re2j
- PR to fix json-schema, https://github.com/everit-org/json-schema/pull/148
- PR to fix nakadi, https://github.com/zalando/nakadi/pull/853
- json-schema handling of Regular Expressions, https://github.com/everit-org/json-schema#regexp-implementations

Peter Liske

October 15, 2018
Tweet

More Decks by Peter Liske

Other Decks in Programming

Transcript

  1. Mit Schema-Validierung APIs lahmlegen EIN AUSFLUG IN DIE WELT DER

    REGULÄREN AUSDRÜCKE UND DAS VERTRAUEN IN KOMPLIZIERTE LIBRARIES PETER LISKE 15-10-2018
  2. 4 BEAUTIFULLY SIMPLE REGULAR EXPRESSIONS Regular Expression due to W3C

    XML Schema specification a|b|c a* b{3,5} c? xyz (...) BRANCHES CONCATENATION QUANTIFIERS GROUPING . [a-z] \d CHARACTER SETS © Disney
  3. 6 PATTERN MATCHING IN XML VALIDATION LIBRARIES Did you ever

    look into XML libraries’ code to see how they do RegExp matching? Example: https://github.com/rkeytacked/java-redos
  4. ALL MAJOR XML SCHEMA VALIDATION LIBRARIES DO NOT CONFORM TO

    THE SPECS AND HAVE EXPONENTIAL WORST-CASE RUNTIME LESSON #2 © Disney
  5. 8 match pattern against input REALITY CHECK (Russ Cox, 2007)

    Source: https://swtch.com/~rsc/regexp/regexp1.html
  6. 9 HISTORY OF REGULAR EXPRESSIONS Kleene’s Regular Sets Chomsky Hierarchy

    usage in compiler design 1970 2000 Thompson: qed ed grep 2010 1960 1950 1980 1990 more UNIX tools: vi lex sed awk expr emacs Perl 2 even more features in Reg Exps Spencer’s Advanced Reg Exps used in specifications POSIX.2 standard Hazel’s PCRE library XML Schema definitions (XSD) JSON Schema definitions ReDoS attacks known ECMA- Script standards Russ Cox’ RE2
  7. “ADVANCED” REGULAR EXPRESSION ENGINES ARE SO COMPLICATED THAT IT’S EASIER

    TO BLAME THE DEVELOPER THAN TO WRITE BETTER ENGINES LESSON #3 © Disney
  8. 13 SIMPLE REGULAR EXPRESSIONS Regular Expression due to W3C XML

    Schema specification a|b|c a* b{3,5} c? xyz (...) BRANCHES CONCATENATION QUANTIFIERS GROUPING . [a-z] \d CHARACTER SETS
  9. 14 ADVANCED FEATURES Anchors $ END OF A LINE OR

    TEXT ^ START OF A LINE OR TEXT
  10. 15 ADVANCED FEATURES Lookarounds (?= … ) (?! … )

    (?<! … ) LOOK AHEAD NEGATIVE LOOK AHEAD NEGATIVE LOOK BEHIND (?<= … ) LOOK BEHIND
  11. 17 ADVANCED FEATURES Inline Modifiers (?i … ) (?x …

    ) (?m … ) CASE INSENSITIVE FREE SPACING MODE MULTILINE MODE (?s … ) SINGLE LINE MODE
  12. 18 ADVANCED FEATURES Sub-Routines (?N) (?&NAME) REPEAT Nth PATTERN REPEAT

    NAMED PATTERN (?(DEFINE) (<FOO> … ) (<BAR> … )) DEFINITION BLOCK
  13. 19 ADVANCED FEATURES Conditionals (?(A) … ) IF CONDITION A

    MATCHED THEN … (?(A)…|…) IF CONDITION A MATCHED THEN … ELSE …
  14. 20 ADVANCED FEATURES Inline Comments (?# … ) JUST A

    COMMENT IN THE MIDDLE OF YOUR REGEX (?x) # some # comment COMMENTS IN FREE-SPACING MODE
  15. 22 JSON SCHEMA REGULAR EXPRESSIONS Regular Expression due to JSON

    Schema specification ^ … $ ECMA-262 Regular Expression Syntax LIBRARY DEVELOPER, you SHOULD implement a full backtracking engine. SCHEMA AUTHOR, you SHOULD NOT use any of these creepy features. simple Reg Exps ?
  16. 24 PATTERN MATCHING IN JSON VALIDATION LIBRARIES Did you ever

    look into JSON libraries’ code to see how they do RegExp matching? Example: https://github.com/rkeytacked/java-redos
  17. 25 HOW TO FIX PATTERN MATCHING FOR YOUR JSON-VALIDATING API

    JSON-Schema Your JSON schema validating API e.g. zalando/nakadi JSON schema lib e.g. json-schema RegExp engine e.g. java.util.regex or RE2/J “… and they lived happily ever after?” “No.”
  18. YOU CAN SECURE YOUR SCHEMA-VALIDATING API BUT IT’S NOT CONFORM

    TO JSON SCHEMA VALIDATION DRAFT LESSON #5 © Disney
  19. 27 BRAINSTORMING: WHAT CAN WE IMPROVE? Kleene’s Regular Sets Chomsky

    Hierarchy usage in compiler design 1970 2000 Thompson: qed ed grep 2010 1960 1950 1980 1990 more UNIX tools: vi lex sed awk expr emacs Perl 2 even more features in Reg Exps Spencer’s Advanced Reg Exps used in specifications POSIX.2 standard Hazel’s PCRE library XML Schema definitions (XSD) JSON Schema definitions ReDoS attacks known ECMA- Script standards Russ Cox’ RE2
  20. 28 THE END ALMOST NO WEB SERVICES WERE HARMED DURING

    THE MAKING OF THIS TALK © Disney