ASN.1 for Perl 6

Brief explanation of code for supporting a subset of ASN.1 in Perl 6: ASN::Grammar, ASN::META, ASN::BER.

Transcript

1. ASN.1 for Perl 6 with elegance and meta(-OP)compilation or “Yo, I heard you like compilers so we added a type generator into your compilation…”
2. The talk is mainly about an application of Meta- Object Protocol (MOP) in a “cool” way Not about “metacompilation” as in “supercompilation” Think of “meta(object protocol)compilation” Or simply “types’n’code generation using MOP” No supercompilation today
3. Instead, today… [‘Metaprogramming’, ‘Compile Time’]
4. Instead, today… [‘Metaprogramming’, ‘Compile Time’]
5. Instead, today… [+] [‘Metaprogramming’, ‘Compile Time’]
6. Instead, today… [+] [‘Metaprogramming’, ‘Compile Time’] == Metaprogramming at compile time!
7. Instead, today… [+] [‘Metaprogramming’, ‘Compile Time’] == Metaprogramming at compile time! NEED TO TRY IT AT HOME! - Anonymous
8. Instead, today… [+] [‘Metaprogramming’, ‘Compile Time’] == Metaprogramming at compile time! MY KIDS LOVE IT! - Anonymous NEED TO TRY IT AT HOME! - Anonymous
9. Instead, today… [+] [‘Metaprogramming’, ‘Compile Time’] == Metaprogramming at compile time! MY KIDS LOVE IT! - Anonymous FOR FUN AND PROFIT! - Anonymous NEED TO TRY IT AT HOME! - Anonymous
10. Instead, today… [+] [‘Metaprogramming’, ‘Compile Time’] == Metaprogramming at compile time! MY KIDS LOVE IT! - Anonymous FOR FUN AND PROFIT! - Anonymous SOUNDS KINDA COOL - Anonymous NEED TO TRY IT AT HOME! - Anonymous
11. Worry a bit about this “metaprogramming”, “meta-object protocol” things? WORRY NOT! But before that, a long time ago (in 1984) in a nice, nice galaxy called Milky Way…
12. The Problem (1/*) Data types are structures. But RAM is numbers! Different OS, different hardware, different bits… We want to define a type and work with it everywhere What to do? For example, ASN.1. (works in space (satellites), your cell phone (LTE), your laptop (X.509) too…)
13. The Problem (1/*) Data types are structures. But RAM is numbers! Different OS, different hardware, different bits… We want to define a type and work with it everywhere What to do? For example, ASN.1. (works in space (satellites), your cell phone (LTE), your laptop (X.509) too…)
14. The Problem (1/*) ASN.1 consists of 1. A generic language for type descriptions 2. Many sets rules for encoding and decoding of the types
15. The Problem (2/*) World-Schema DEFINITIONS AUTOMATIC TAGS ::= BEGIN Rocket ::= SEQUENCE { name UTF8String (SIZE(1..16)), message UTF8String DEFAULT "Hello World", fuel ENUMERATED {solid, liquid, gas}, speed CHOICE { mph INTEGER, kmph INTEGER } OPTIONAL, payload SEQUENCE OF UTF8String } END value Rocket ::= { name "Falcon", fuel solid, speed mph : 18000, payload { "Car", "GPS" } } 301D8006 46616C63 6F6E8201 00A30480 024650A4 0A0C0343 61720C03 475053
16. The Problem (2/*) World-Schema DEFINITIONS AUTOMATIC TAGS ::= BEGIN Rocket ::= SEQUENCE { name UTF8String (SIZE(1..16)), message UTF8String DEFAULT "Hello World", fuel ENUMERATED {solid, liquid, gas}, speed CHOICE { mph INTEGER, kmph INTEGER } OPTIONAL, payload SEQUENCE OF UTF8String } END value Rocket ::= { name "Falcon", fuel solid, speed mph : 18000, payload { "Car", "GPS" } } 301D8006 46616C63 6F6E8201 00A30480 024650A4 0A0C0343 61720C03 475053 Type
17. The Problem (2/*) World-Schema DEFINITIONS AUTOMATIC TAGS ::= BEGIN Rocket ::= SEQUENCE { name UTF8String (SIZE(1..16)), message UTF8String DEFAULT "Hello World", fuel ENUMERATED {solid, liquid, gas}, speed CHOICE { mph INTEGER, kmph INTEGER } OPTIONAL, payload SEQUENCE OF UTF8String } END value Rocket ::= { name "Falcon", fuel solid, speed mph : 18000, payload { "Car", "GPS" } } 301D8006 46616C63 6F6E8201 00A30480 024650A4 0A0C0343 61720C03 475053 Value
18. The Problem (2/*) World-Schema DEFINITIONS AUTOMATIC TAGS ::= BEGIN Rocket ::= SEQUENCE { name UTF8String (SIZE(1..16)), message UTF8String DEFAULT "Hello World", fuel ENUMERATED {solid, liquid, gas}, speed CHOICE { mph INTEGER, kmph INTEGER } OPTIONAL, payload SEQUENCE OF UTF8String } END value Rocket ::= { name "Falcon", fuel solid, speed mph : 18000, payload { "Car", "GPS" } } 301D8006 46616C63 6F6E8201 00A30480 024650A4 0A0C0343 61720C03 475053 Bytes
19. Rules • There are a lot of ways to encode a value of a type described in ASN.1. • Various binary encodings (BER, DER etc) and text ones (XML, JSON etc) are defined.
20. Quick JSON { "name":"Falcon", "message":"Hello World", "fuel":“liquid", "speed":{ "mph":18000 }, "payload":["Car", "GPS unit"] }
21. Rules • There are a lot of ways to encode a value of a type described in ASN.1. • Various binary encodings (BER, DER etc) and text ones (XML, JSON etc) are defined. • I was focused on BER (Basic Encoding Rules), as LDAP protocol uses it. • Other encoding can be added.
22. How to implement?
23. First approach (Write a compiler!)
24. + && - Good things… Bad things… Possible to recompile updates as much as needed Creates a maintenance burden: additional sources just to generate, compile and throw away Target is source code – easy to generate and debug The source code must be immutable and so any changes add technical debt Can tune generated code to a certain degree Complicates overall build process Still great for linters, code minimizers etc.
25. Second approach (Write a lib!)
26. + && - Good things… Bad things… Easy to implement Creates a maintenance burden: any change to the spec itself require manual updates done by a programmer Can handle any case one can imagine More prone to errors Any changes are preserved Way harder to reason about correctness Easy to mix with the main application code With a large spec, writing out types manually is an anti-pattern (still good for early prototypes)
27. ASN::BER • So we took the second approach! – We can represent ASN.1 types using Perl 6 ones – We can serialize/parse to/from bytes
28. ASN::BER • So we took the second approach! – We can represent ASN.1 types using Perl 6 ones – We can serialize/parse to/from bytes It does its job!
29. ASN::BER • So we took the second approach! – We can represent ASN.1 types using Perl 6 ones – We can serialize/parse to/from bytes It does its job! It is tedious to use too 
30. ASN::BER • So we took the second approach! – We can represent ASN.1 types using Perl 6 ones – We can serialize/parse to/from bytes It does its job! It is tedious to use too  In Perl 6 we want and can do better!
31. Goals overview We don’t want to generate high-level sources We don’t want to spend our time as an ASN.1 compiler (can as well create something else!) Make machine program a machine We don’t want to add a maintenance burden
32. The main application code Initially, we want to get types we can work with from our ASN.1 specification (X in XY problem) How can the code look like?
33. The main application code give me Types;
34. The main application code # We want to get specific types # of the particular ASN.1 spec give me Types <file ldap.asn>;
35. The main application code # To import a compilation unit, # `use` statement is used in Perl 6 give me Types <file ldap.asn>; use Types <file ldap.asn>;
36. The main application code # To import a compilation unit, # `use` statement is used in Perl 6 give me Types <file ldap.asn>; use Types <file ldap.asn>; # Use ASN::META library use ASN::META <file ldap.asn>; # Just use types here…
37. HOW?! In Perl 6, we have a number of language constructs that are executed at compile time Any “normal” Perl 6 code is fine No “templates” or anything like that! No pain!
38. HOW?! In Perl 6, we have a number of language constructs that are executed at compile time Any “normal” Perl 6 code is fine No “templates” or anything like that! No pain!
39. HOW?! In Perl 6, we have a number of language constructs that are executed at compile time Any “normal” Perl 6 code is fine No “templates” or anything like that! No pain!
40. BUT WHY?!
41. ASN::META
42. ASN::META internals (0/*) At Compile Time { parse  load plugins  create types and code  export! } Metaprogramming sub EXPORT(*@params) { my $keys = @params.Map; my $ASN = parse-ASN slurp $keys<file>; my @*PLUGINS; with $keys<plugin> -> $plugin-name { require ::($plugin-name); @*PLUGINS.push(::($plugin-name)); } my $*POOL = TypePool.new; compile-types($_) with $ASN; $*POOL.export.Map; }
43. Parsing stage • A Perl 6 grammar • Complete enough to process LDAP ASN.1 spec • (very incomplete in other areas) • (very open for pull requests) • Visit github.com/Altai-man/ASN-Grammar for more details
44. Exporting stage • Just Perl 6 “EXPORT” subroutine.
45. Exporting stage • Just Perl 6 “EXPORT” subroutine. • That’s all, really. Now let us peek just a little at the logic code…
46. multi sub compile-complex-builtin('SEQUENCE', $type, $name) { my $new-type = Metamodel::ClassHOW.new_type(:$name); my @ASN-order = create-order-of-fields(…); $new-type.^add_method('ASN-order', method { @ASN-order }); for $type.params<fields>-> $field { my $attr = Attribute.new( name => prepare-name(…), type => get-type(…), package => $new-type, :has_accessor); # more things happen here… $new-type.^add_attribute($attr); } $new-type.^add_role(ASNSequence); $new-type.^compose; $*POOL.add(ASNType.new( :$name, base-type => 'SEQUENCE', type => $new-type)); }
47. multi sub compile-complex-builtin('SEQUENCE', $type, $name) { my $new-type = Metamodel::ClassHOW.new_type(:$name); my @ASN-order = create-order-of-fields(…); $new-type.^add_method('ASN-order', method { @ASN-order }); for $type.params<fields>-> $field { my $attr = Attribute.new( name => prepare-name(…), type => get-type(…), package => $new-type, :has_accessor); # more things happen here… $new-type.^add_attribute($attr); } $new-type.^add_role(ASNSequence); $new-type.^compose; $*POOL.add(ASNType.new( :$name, base-type => 'SEQUENCE', type => $new-type)); }
48. multi sub compile-complex-builtin('SEQUENCE', $type, $name) { my $new-type = Metamodel::ClassHOW.new_type(:$name); my @ASN-order = create-order-of-fields(…); $new-type.^add_method('ASN-order', method { @ASN-order }); for $type.params<fields>-> $field { my $attr = Attribute.new( name => prepare-name(…), type => get-type(…), package => $new-type, :has_accessor); # more things happen here… $new-type.^add_attribute($attr); } $new-type.^add_role(ASNSequence); $new-type.^compose; $*POOL.add(ASNType.new( :$name, base-type => 'SEQUENCE', type => $new-type)); }
49. multi sub compile-complex-builtin('SEQUENCE', $type, $name) { my $new-type = Metamodel::ClassHOW.new_type(:$name); my @ASN-order = create-order-of-fields(…); $new-type.^add_method('ASN-order', method { @ASN-order }); for $type.params<fields>-> $field { my $attr = Attribute.new( name => prepare-name(…), type => get-type(…), package => $new-type, :has_accessor); # more things happen here… $new-type.^add_attribute($attr); } $new-type.^add_role(ASNSequence); $new-type.^compose; $*POOL.add(ASNType.new( :$name, base-type => 'SEQUENCE', type => $new-type)); }
50. multi sub compile-complex-builtin('SEQUENCE', $type, $name) { my $new-type = Metamodel::ClassHOW.new_type(:$name); my @ASN-order = create-order-of-fields(…); $new-type.^add_method('ASN-order', method { @ASN-order }); for $type.params<fields>-> $field { my $attr = Attribute.new( name => prepare-name(…), type => get-type(…), package => $new-type, :has_accessor); # more things happen here… $new-type.^add_attribute($attr); } $new-type.^add_role(ASNSequence); $new-type.^compose; $*POOL.add(ASNType.new( :$name, base-type => 'SEQUENCE', type => $new-type)); }
51. multi sub compile-complex-builtin('SEQUENCE', $type, $name) { my $new-type = Metamodel::ClassHOW.new_type(:$name); my @ASN-order = create-order-of-fields(…); $new-type.^add_method('ASN-order', method { @ASN-order }); for $type.params<fields>-> $field { my $attr = Attribute.new( name => prepare-name(…), type => get-type(…), package => $new-type, :has_accessor); # more things happen here… $new-type.^add_attribute($attr); } $new-type.^add_role(ASNSequence); $new-type.^compose; $*POOL.add(ASNType.new( :$name, base-type => 'SEQUENCE', type => $new-type)); }
52. multi sub compile-complex-builtin('SEQUENCE', $type, $name) { my $new-type = Metamodel::ClassHOW.new_type(:$name); my @ASN-order = create-order-of-fields(…); $new-type.^add_method('ASN-order', method { @ASN-order }); for $type.params<fields>-> $field { my $attr = Attribute.new( name => prepare-name(…), type => get-type(…), package => $new-type, :has_accessor); # more things happen here… $new-type.^add_attribute($attr); } $new-type.^add_role(ASNSequence); $new-type.^compose; $*POOL.add(ASNType.new( :$name, base-type => 'SEQUENCE', type => $new-type)); }
53. ASN::META internals (2/*) We create ASN.1-compatible types using meta-object protocol implementation in Perl 6 (or, more generally, metaprogramming)
54. ASN::META internals (3/*) We create ASN.1-compatible types using meta-object protocol implementation in Perl 6 (or, more generally, metaprogramming) Including simple types (integers, strings)
55. ASN::META internals (4/*) We create ASN.1-compatible types using meta-object protocol implementation in Perl 6 (or, more generally, metaprogramming) Including simple types (integers, strings) Including complex types (sequences, choices)
56. ASN::META internals (5/*) We create ASN.1-compatible types using meta-object protocol implementation in Perl 6 (or, more generally, metaprogramming) Including simple types (integers, strings) Including complex types (sequences, choices) Including recursive types with reference resolution…
57. ASN::META internals (6/*) We create ASN.1-compatible types using meta-object protocol implementation in Perl 6 (or, more generally, metaprogramming) Including simple types (integers, strings) Including complex types (sequences, choices) Including recursive types with reference resolution… At compile time!
58. Client side • For the client code it looks like ASN::META just exports needed types as if they were defined there • Except ASN::META does not have any except compiler-related ones • Do not need to use ASN::BER
59. Client side • For the client code it looks like ASN::META just exports needed types as if they were defined there • Except ASN::META does not have any except compiler-related ones • Do not need to use ASN::BER (“a lot”, at least)
60. Goals check We don’t want to generate high-level sources We don’t want to spend time as an ASN.1 compiler Make machine program a machine We don’t want to add a maintenance burden
61. Not a silver bullet (Of course it is not!)
62. Not a silver bullet (Of course it is not!) There are great use cases for the first approach
63. Not a silver bullet (Of course it is not!) There are great use cases for the first approach There are great use cases for the second approach
64. Not a silver bullet (Of course it is not!) There are great use cases for the first approach There are great use cases for the second approach See a suitable case? Consider utilizing metaprogramming!
65. Not a silver bullet (Of course it is not!) There are great use cases for the first approach There are great use cases for the second approach See a suitable case? Consider utilizing metaprogramming! (maybe even at compile time)
66. Not a silver bullet (Of course it is not!) There are great use cases for the first approach There are great use cases for the second approach See a suitable case? Consider utilizing metaprogramming! (maybe even at compile time) Perl 6 is a very handy tool to do so!
67. Conclusions
68. Conclusions • Nobody wants to implement ASN.1
69. Conclusions • Nobody wants to implement ASN.1
70. Conclusions • Implementing recursive types is not fun (but we still need them)
71. Conclusions • Implementing recursive types is not fun (but we still need them)
72. Conclusions • Metaprogramming can be very helpful and clever (almost as a fox)
73. Conclusions • Metaprogramming can be very helpful and clever (almost as a fox) • Compile time code execution can be very helpful and clever (almost as a fox)
74. Conclusions • Metaprogramming can be very helpful and clever (almost as a fox) • Compile time code execution can be very helpful and clever (almost as a fox) • Perl 6 delivers!
75. QUESTIONS?