Validating Data Everywhere with Rx

Data validation is a real pain, but it's a huge help in preventing bugs. By using one system for validating data everywhere, the burden of validation is greatly simplified.

Rx is a schema language that can be used to validate data structures anywhere in your Perl programs. It integrates with Moose types, is extremely extensible, and is portable across languages: Rx implementations also exist in Python, Ruby, PHP, and JavaScript.

Transcript

1. Rx
3. SPAM
4. Store Data Spam Storage
5. Store Data Spam Storage pobox web server http
6. { “cur_page”: 1, “per_page”: 10, “total”: 902, “items”: [ { “id”: 1234, “hdr_subject”: “rock hard math tests”, “discarded_by”: “dnsbl/rbl.wolfram.com” }, ... ], }
7. --- cur_page: 1 per_page: 10 total : 902 items : - id: 1234 hdr_subject : rock hard math tests discarded_by: dnsbl/rbl.wolfram.com ...
8. Store Data Spam Storage pobox web server
9. Store Data Spam Storage ajax web code pobox web server
10. Store Data Spam Storage ajax web code pobox web server your cool app
11. { “cur_page”: 1, “per_page”: 10, “total”: 902, “items”: [ { “id”: 1234, “hdr_subject”: “rock hard math tests”, “discarded_by”: “dnsbl/rbl.wolfram.com” }, ... ], }
12. XML
13. --- cur_page: 1 per_page: 10 total : 902 items : - id: 1234 hdr_subject : rock hard math tests discarded_by: dnsbl/rbl.wolfram.com ...
14. { “cur_page”: 1, “per_page”: 10, “total”: 902, “items”: [ { “id”: 1234, “hdr_subject”: “rock hard math tests”, “discarded_by”: “dnsbl/rbl.wolfram.com” }, ... ], }
15. <spam:page spam:curPageNum=”1” spam:perPage=”10” spam:totalItems=”902”><spam:item spam:id=”1234”><spam:headers> <spam:header spam:header=”subject”> rock hard math tests</spam:header> </spam:headers><spam:filterInfo> <spam:discardedBy><spam:filterType> dnsbl</spam:filterType> <spam:filterArgs>rbl.wolfram.com </spam:filterArgs></spam:discardedBy> </spam:filterInfo></spam:item> <elipsis /></spam:page>
16. XSD
18. RELAX NG
19. XML isn’t perlish
20. ...but schemas are good
21. total : 902 cur_page: 1 per_page: 10 items : - id: 1234 hdr_subject : rock hard math tests discarded_by: dnsbl/rbl.wolfram.com
22. total : 902 cur_page: 1 per_page: 10 items : - id: 1234 hdr_subject : rock hard math tests discarded_by: dnsbl/rbl.wolfram.com type: //rec required: total : //int per_page: //int cur_page: { type: //int, range: { min: 1 } } items : { type: //rec required: { ... }
23. type: //rec required: total : //int per_page: //int cur_page: { type: //int, range: { min: 1 } } items : { type: //rec required: { ... }
24. Perl
25. my $rx = Data::Rx->new; Perl
26. my $rx = Data::Rx->new; Perl
27. my $rx = Data::Rx->new; my $data = { type => ‘//rec’, ... }; Perl
28. my $rx = Data::Rx->new; my $data = { type => ‘//rec’, ... }; Perl
29. my $rx = Data::Rx->new; my $data = { type => ‘//rec’, ... }; my $schema = $rx->make_schema($data); Perl
30. my $rx = Data::Rx->new; my $data = { type => ‘//rec’, ... }; my $schema = $rx->make_schema($data); Perl
31. my $rx = Data::Rx->new; my $data = { type => ‘//rec’, ... }; my $schema = $rx->make_schema($data); if ($schema->check( $input )) { Perl
32. my $rx = Data::Rx->new; my $data = { type => ‘//rec’, ... }; my $schema = $rx->make_schema($data); if ($schema->check( $input )) { ... Perl
33. my $rx = Data::Rx->new; my $data = { type => ‘//rec’, ... }; my $schema = $rx->make_schema($data); if ($schema->check( $input )) { ... } Perl
34. my $rx = Data::Rx->new; my $data = decode_json( $json ); my $schema = $rx->make_schema($data); if ($schema->check( $input )) { ... } Perl
35. var rx = new Rx(); var schema = rx.makeSchema( schemaFromJSON ); if (schema.check(input) { ... } JavaScript
36. rx = Rx.Factory() schema = rx.makeSchema( schemaFromJSON ) if schema.check(input): ... Python
37. rx = Rx.new schema = rx.makeSchema( schemaFromJSON ) if schema.check(input) then ... end Ruby
38. $rx = new Rx(); $schema = $rx.makeSchema( $schemaFromJSON ) if ($schema.check( $input )) { ... } PHP
39. $rx = new Rx(); $schema = $rx.makeSchema( $schemaFromJSON ) if ($schema.check( $input )) { ... } PHP (forgive me)
40. --- type: //str “Isn’t it awfully nice to have a schema?”
41. --- type: //int 42
42. --- type : //int range: { min: 3, max: 18 } 17
43. --- type : //int value: 10 10
44. --- type : //any undef, 1, [ ... ], CGI->new
45. --- type : //any of : - //int - //str - { type: //num, value: 3.141 } “String type, I choose you!”
46. Simple Core Rx Types
47. Simple Core Rx Types •//nil
48. Simple Core Rx Types •//nil •//def
49. Simple Core Rx Types •//nil •//def •//num
50. Simple Core Rx Types •//nil •//def •//num •//int
51. Simple Core Rx Types •//nil •//def •//num •//int •//bool
52. Simple Core Rx Types •//nil •//def •//num •//int •//bool •//str
53. Simple Core Rx Types •//nil •//def •//num •//int •//bool •//str •//one
54. Simple Core Rx Types •//nil •//def •//num •//int •//bool •//str •//one •//fail
55. the //arr type --- type: //arr length: { min: 6, max: 6 } contents: type : //int range: { min: 3, max: 18 } [ 7, 3, 4, 11, 17, 9 ]
56. the //seq type --- type: //seq contents: - //int - //bool - //str [ 10, true, “awesome” ]
57. the //map type --- type : //map values: //int { x => 1, y => 2, z => 3 }
58. the //rec type --- type : //rec required: first: //str last : //str optional: { middle: //str } { first => ‘Rico’, last => ‘Signes’ }
59. the //all type --- type : //all of: - { type: //int, range: { min: 3 } } - { type: //num, range: { max: 18 } } 10
60. //str
61. taguri
62. tag: AUTHORITY , DATE : STRING
63. tag:
64. tag:rjbs.manxome.org
65. tag:rjbs.manxome.org,2008-10-11
66. tag:rjbs.manxome.org,2008-10-11:rx
67. tag:rjbs.manxome.org,2008-10-11:rx/slide
69. //str
70. //str
71. //str tag:codesimply.com,2008:rx/core/str
72. --- type: //str
73. --- type: tag:codesimply.com,2008:rx/core/str
74. --- type: tag:codesimply.com,2008:rx/core/str --- type: //str
75. /prefix/rest
76. /prefix/rest (empty) tag:codesimply.com,2008:rx/core/ .meta tag:codesimply.com,2008:rx/meta/ local tag:you@cpan.org:2009:RxType,
77. /prefix/rest (empty) tag:codesimply.com,2008:rx/core/ .meta tag:codesimply.com,2008:rx/meta/ local tag:you@cpan.org:2009:RxType,
78. /prefix/rest (empty) tag:codesimply.com,2008:rx/core/ .meta tag:codesimply.com,2008:rx/meta/ local tag:you@cpan.org:2009:RxType,
79. type: tag:you@cpan.org:2009:RxType,Account type: /local/Account type: tag:codesimply.com,2008:rx/core/str type: //str type: tag:codesimply.com,2008:rx/meta/str type: /.meta/str
80. Making Types
82. •decide what it does
83. •decide what it does •give it a name (tag)
84. •decide what it does •give it a name (tag) •write tests
85. •decide what it does •give it a name (tag) •write tests •implement it
86. •decide what it does •give it a name (tag) •write tests •implement it
87. use Test::More tests => 2; my $rx = Data::Rx->new({ prefix => { moose => ‘tag:rjbs.manxome.org,2008-10-04:rx/moose/’, }, type_plugins => [ ‘Data::Rx::Type::MooseTC’ ] }); my $array_of_int = $rx->make_schema({ type => ‘/moose/tc’, moose_type => ‘ArrayRef[Int]’, }); ok($array_of_int->check([1]), “[1] is an ArrayRef[Int]”); ok(! $array_of_int->check( 1 ), “1 is not an ArrayRef[Int]”); t/moose-types.t
88. •...but then you’d have to rewrite your tests in every language. •We already have Rx running on a bunch of platforms... •...with a shared test suite... •...and we can re-use it.
89. { “empty” : “[]”, “str-1” : “[\”string\”, 1 ]”, “0” : “[ 0 ]”, “0-1” : “[ 0, 1 ]”, “0-1-1” : “[ 0, 1, 1 ]”, “0-s1-1” : “[ 0, \”1\”, 1 ]”, “0-1-1-2”: “[ 0, 1, 1, 2 ]”, “0-1-1-2”: “[ 0, 1, 1, 2 ]”, “0-str” : “[ 0, \”string\” ]”, “0-str-3”: “[ 0, \”string\”, 3 ]”, “0-str-3-18” : “[ 0, \”string\”, 3, 18 ]”, “0-str-3-T” : “[ 0, \”string\”, 3, true ]”, “0-str-3-T-F” : “[ 0, \”string\”, 3, true, false ]”, “0-str-3-T-str”: “[ 0, \”string\”, 3, true, \”false\” ]”, “0-str-3-T-F-T”: “[ 0, \”string\”, 3, true, false, true ]” } spec/data/arr.json
90. { “schema”: { “type” : “//arr”, “contents”: { “type”: “//int” }, “length” : { “max”: 3 } }, “pass”: { “arr”: [ “empty”, “0”, “0-1”, “0-1-1” ] }, “fail”: { “arr” : [ “str-1”, “0-s1-1”, “0-1-1-2”, “0-str” ], “bool”: “*”, “null”: “*”, “num” : “*”, “obj” : “*”, “str” : “*” } } spec/schemata/array-3-int.json
91. { “schema”: { “type” : “tag:rx-foundation.biz/mtfnpy”, “hardness”: 8, “ronpaul” : false }, “pass”: { “mtf”: { “garbled”: true }, “arr”: [ “empty”, “0”, “0-1”, “0-1-1” ] }, “fail”: { “bool”: “*”, “null”: “*”, “num” : “*”, “obj” : “*”, “str” : “*” } } your-schema.json
92. Rx validates data
93. Rx validates data (not strings)
94. Rx validates data (not strings) •Rx isn’t about JSON
95. Rx validates data (not strings) •Rx isn’t about JSON •...or YAML
96. Rx validates data (not strings) •Rx isn’t about JSON •...or YAML •it validates data in memory
97. Rx validates data (not strings) •Rx isn’t about JSON •...or YAML •it validates data in memory •how you transmit it over the wire is your problem
98. Core Types are Portable
99. Core Types are Portable •you won’t see regex in Rx core
100. Core Types are Portable •you won’t see regex in Rx core •because regex don’t work the same everywhere
101. Core Types are Portable •you won’t see regex in Rx core •because regex don’t work the same everywhere •(but there is a PCRE type outside of core!)
102. Too Portable for Perl!
103. Too Portable for Perl! •//int means an integer
104. Too Portable for Perl! •//int means an integer •but in many/most languages, 5 !== “5”
105. Too Portable for Perl! •//int means an integer •but in many/most languages, 5 !== “5” •perl actually can’t distinguish these
106. Too Portable for Perl! •//int means an integer •but in many/most languages, 5 !== “5” •perl actually can’t distinguish these •...that’s why the test suite has “todo”
107. Non-portable Types
108. Non-portable Types type: /perl/obj isa : DateTime
109. Non-portable Types type: /perl/obj isa : DateTime type: /perl/obj does: Throwable
110. Non-portable Types
111. Non-portable Types type : /pcre/str regex: q/\A867-[530]{9}\z/
112. Non-portable Types type : /pcre/str regex: q/\A867-[530]{9}\z/ type: /moose/tc does: ArrayRef[DateTime]
113. Rx Everywhere
114. Rx Everywhere •perl6
115. Rx Everywhere •perl6 •Haskell
116. Rx Everywhere •perl6 •Haskell •C#
117. Rx Everywhere •perl6 •Haskell •C# •ML
118. Any Questions?
119. Thank You!