Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A variant of recursive descent parsing

A variant of recursive descent parsing

Showing how C++17 std::variant<>, std::visit<> and overloads of lambdas can help make a simple parser very easy to read and understand.

Presentation given at the distributed SwedenCPP Stockholm + London C++ UG meetup.

Björn Fahller

October 26, 2017
Tweet

More Decks by Björn Fahller

Other Decks in Programming

Transcript

  1. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    A variant of recursive descent parsing
    Björn Fahller
    E --> E "+" E
    | E "-" E
    | "-" E
    | E "*" E
    | E "/" E
    | "(" E ")"
    | v

    View full-size slide

  2. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    Parser generators suck
    %token NUM
    %left '-' '+'
    %left '*' '/'
    %precedence NEG /* negation--unary minus */
    %right '^' /* exponentiation */
    %% /* The grammar follows. */
    exp:
    NUM { $$ = $1; }
    | exp '+' exp { $$ = $1 + $3; }
    | exp '-' exp { $$ = $1 - $3; }
    | exp '*' exp { $$ = $1 * $3; }

    View full-size slide

  3. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    Lexers suck
    using token = enum { number, identifier, lparen, …};
    union yystype { double num; const char* id; };
    yystype* yylval;
    token yylex();

    if (yylex() == number) { … blah(yylval->num); … }

    View full-size slide

  4. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    std::variant
    using v = std::variantstd::string,
    bool,
    std::list>;

    View full-size slide

  5. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    std::variant
    A very brief intro.
    using v = std::variantstd::string,
    bool,
    std::list>;
    v var{std::string(”foo”)};
    Construct holding a std::string

    View full-size slide

  6. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    std::variant
    A very brief intro.
    using v = std::variantstd::string,
    bool,
    std::list>;
    v var{std::string(”foo”)};
    std::get(v) = ”bar”;
    Access the std::string
    and assign it a new value

    View full-size slide

  7. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    std::visit(func_obj, v...)
    visitor visitor_obj;
    int i = std::visit(visitor_obj, variant_var);

    View full-size slide

  8. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    std::visit(func_obj, v...)
    Matt Kline
    ”std::visit is everything wrong with modern C++”
    https://bitbashing.io/std-visit.html
    visitor visitor_obj;
    int i = std::visit(visitor_obj, variant_var);

    View full-size slide

  9. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    std::visit(func_obj, v...)
    class visitor
    {
    public:
    int operator()(std::string) { /* str things */ }
    int operator()(int) { /* int things */ }
    int operator()(auto) { /* other things */ }
    };
    visitor visitor_obj;
    int i = std::visit(visitor_obj, variant_var);

    View full-size slide

  10. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    class visitor
    {
    public:
    int operator()(std::string) { /* str things */ }
    int operator()(int) { /* int things */ }
    int operator()(auto) { /* other things */ }
    };
    visitor visitor_obj;
    int i = std::visit(visitor_obj, variant_var);
    std::visit(func_obj, v...)

    Uncool! Implementation far away from use.

    View full-size slide

  11. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    class visitor
    {
    public:
    int operator()(std::string) { /* str things */ }
    int operator()(int) { /* int things */ }
    int operator()(auto) { /* other things */ }
    };
    visitor visitor_obj;
    int i = std::visit(visitor_obj, variant_var);
    std::visit(func_obj, v...)

    Uncool! Implementation far away from use.

    A lambda is an object of
    class type with operator()

    View full-size slide

  12. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    class visitor
    {
    public:
    int operator()(std::string) { /* str things */ }
    int operator()(int) { /* int things */ }
    int operator()(auto) { /* other things */ }
    };
    visitor visitor_obj;
    int i = std::visit(visitor_obj, variant_var);
    std::visit(func_obj, v...)

    Uncool! Implementation far away from use.

    A lambda is an object of
    class type with operator()

    A lambda can be a base class

    View full-size slide

  13. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    class visitor
    {
    public:
    int operator()(std::string) { /* str things */ }
    int operator()(int) { /* int things */ }
    int operator()(auto) { /* other things */ }
    };
    visitor visitor_obj;
    int i = std::visit(visitor_obj, variant_var);
    std::visit(func_obj, v...)

    Uncool! Implementation far away from use.

    A lambda is an object of
    class type with operator()

    A lambda can be a base class

    View full-size slide

  14. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    overload
    template
    class overload : T...
    {
    public:
    overload(T... t) : T(t)... {}
    using T::operator()...;
    };
    auto x = overload(
    [](int) {},
    [](std::string){}
    );

    View full-size slide

  15. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    overload
    template
    class overload : T...
    {
    public:
    overload(T... t) : T(t)... {}
    using T::operator()...;
    };
    auto x = overload(
    [](int) {},
    [](std::string){}
    );
    C++17 variadic using, to bring
    operator() from each T into view

    View full-size slide

  16. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    overload
    template
    class overload : T...
    {
    public:
    overload(T... t) : T(t)... {}
    using T::operator()...;
    };
    auto x = overload(
    [](int) {},
    [](std::string){}
    );
    C++17 automatic template
    type deduction from constructor.

    View full-size slide

  17. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    std::variant
    using v = std::variantstd::string,
    bool,
    std::list>;
    void func(v obj) {
    std::visit(overload(
    [](int) { /* int stuff */ },
    [](std::string) { /* string stuff */ },
    [](const auto&) { /* other stuff */ }),
    obj);
    }

    View full-size slide

  18. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    std::variant
    using v = std::variantstd::string,
    bool,
    std::list>;
    void func(v obj) {
    std::visit(overload(
    [](int) { /* int stuff */ },
    [](std::string) { /* string stuff */ },
    [](const auto&) { /* other stuff */ }),
    obj);
    }
    Now the logic is where
    we use it. Cool!

    View full-size slide

  19. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    Representing tokens
    Tokens for a very simple calculator.
    template
    struct C {};
    struct number { double value; };
    struct ident { std::string_view value; };
    struct eof {};
    struct remember {};
    struct forget {};

    View full-size slide

  20. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    Representing tokens
    Tokens for a very simple calculator.
    template
    struct C {};
    struct number { double value; };
    struct ident { std::string_view value; };
    struct eof {};
    struct remember {};
    struct forget {};
    For single character tokens like
    + - * / (

    View full-size slide

  21. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    Representing tokens
    Tokens for a very simple calculator.
    template
    struct C {};
    struct number { double value; };
    struct ident { std::string_view value; };
    struct eof {};
    struct remember {};
    struct forget {};
    Let numeric literals hold the value

    View full-size slide

  22. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    Representing tokens
    Tokens for a very simple calculator.
    template
    struct C {};
    struct number { double value; };
    struct ident { std::string_view value; };
    struct eof {};
    struct remember {};
    struct forget {};
    And identifiers the name.

    View full-size slide

  23. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    Representing tokens
    Tokens for a very simple calculator.
    template
    struct C {};
    struct number { double value; };
    struct ident { std::string_view value; };
    struct eof {};
    struct remember {};
    struct forget {};
    Keywords for storing
    and dropping variables

    View full-size slide

  24. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    Representing tokens
    Tokens for a very simple calculator.
    template
    struct C {};
    struct number { double value; };
    struct ident { std::string_view value; };
    struct eof {};
    struct remember {};
    struct forget {};
    and for lexer errors an
    exception is thrown

    View full-size slide

  25. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    Representing tokens
    Tokens for a very simple calculator.
    template
    struct C {};
    struct number { double value; };
    struct ident { std::string_view value; };
    struct eof {};
    struct remember {};
    struct forget {};
    using token = std::variant<
    number,ident,
    C<'-'>, C<'+'>, C<'/'>, C<'*'>,
    C<'('>, C<')'>, C<'='>,
    eof, remember, forget
    >;

    View full-size slide

  26. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    Representing tokens
    Tokens for a very simple calculator.
    template
    struct C {};
    struct number { double value; };
    struct ident { std::string_view value; };
    struct eof {};
    struct remember {};
    struct forget {};
    struct other {};
    using token = std::variant<
    number,ident,
    C<'-'>, C<'+'>, C<'/'>, C<'*'>,
    C<'('>, C<')'>, C<'='>,
    eof, remember, forget
    >;
    class lex
    {
    public:
    lex(std::string s);
    token next_token(); // consume
    token peek(); // look ahead
    void drop(); // consume
    private:
    ...
    };

    View full-size slide

  27. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    double calculator::parse(std::string s)
    {
    lexer = std::make_unique(move(s));
    auto t = lexer->peek();
    auto rv = std::visit(
    overload{
    [=](remember) { lexer->drop();return parse_remember_var();},
    [=](forget) { lexer->drop();return parse_forget_var();},
    [=](auto) { return parse_expr();}
    },
    t);
    if (!std::holds_alternative(lexer next_token()))

    throw "garbage after expr";
    return rv;
    }
    double calculator::parse_expr()
    {
    auto v = parse_term();
    for (bool done = false; !done;) {
    auto t = lexer->peek();
    v = std::visit(
    overload{
    [=](C<'+'>) { lexer->drop(); return v+parse_term();},
    [=](C<'-'>) { lexer->drop(); return v-parse_term();},
    [&](auto) { done = true; return v; }
    },
    t);
    }
    return v;
    }
    double calculator::parse_term()
    {
    auto v = parse_factor();
    for (bool done = false; !done;) {
    auto t = lexer->peek();
    v = std::visit(
    overload{
    [=](C<'/'>) { lexer->drop(); return v/parse_factor();},
    [=](C<'*'>) { lexer->drop(); return v*parse_factor();},
    [&](auto) { done = true; return v; }
    },
    t);
    }
    return v;
    }
    double calculator::parse_factor()
    {
    auto t = lexer->next_token();
    return std::visit(
    overload{
    [=](ident var) { return lookup(var.value); },
    [=](number n) { return n.value; },
    [=](C<'+'>) { return parse_term(); },
    [=](C<'-'>) { return -parse_term(); },
    [=](C<'('>) { return parse_paren(); },
    [=](auto) -> double { throw "unexpected"; }
    },
    t);
    }
    double calculator::parse_paren()
    {
    auto v = parse_expr();
    auto t = lexer->next_token();
    if (!std::holds_alternative>(t)) throw "expected ')'";
    return v;
    }
    double calculator::parse_forget_var()
    {
    auto t = lexer->next_token();
    return std::visit(
    overload{
    [=](ident i) { return drop_variable(i.value);},
    [=](auto) -> double { throw "expected variable name";}
    },
    t);
    }
    double calculator::parse_remember_var()
    {
    auto var = lexer->next_token();
    if (!std::holds_alternative(var)) throw "expected variable name";
    if (!std::holds_alternative>(lexer->next_token())) throw "expected =";
    auto v = parse_expr();
    memory[std::get(var).value] = v;
    return v;
    }

    View full-size slide

  28. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    double calculator::parse(std::string s)
    {
    lexer = std::make_unique(move(s));
    auto t = lexer->peek();
    auto rv = std::visit(
    overload{
    [=](remember) { lexer->drop();return parse_remember_var();},
    [=](forget) { lexer->drop();return parse_forget_var();},
    [=](auto) { return parse_expr();}
    },
    t);
    if (!std::holds_alternative(lexer next_token()))

    throw "garbage after expr";
    return rv;
    }
    double calculator::parse_expr()
    {
    auto v = parse_term();
    for (bool done = false; !done;) {
    auto t = lexer->peek();
    v = std::visit(
    overload{
    [=](C<'+'>) { lexer->drop(); return v+parse_term();},
    [=](C<'-'>) { lexer->drop(); return v-parse_term();},
    [&](auto) { done = true; return v; }
    },
    t);
    }
    return v;
    }
    double calculator::parse_term()
    {
    auto v = parse_factor();
    for (bool done = false; !done;) {
    auto t = lexer->peek();
    v = std::visit(
    overload{
    [=](C<'/'>) { lexer->drop(); return v/parse_factor();},
    [=](C<'*'>) { lexer->drop(); return v*parse_factor();},
    [&](auto) { done = true; return v; }
    },
    t);
    }
    return v;
    }
    double calculator::parse_factor()
    {
    auto t = lexer->next_token();
    return std::visit(
    overload{
    [=](ident var) { return lookup(var.value); },
    [=](number n) { return n.value; },
    [=](C<'+'>) { return parse_term(); },
    [=](C<'-'>) { return -parse_term(); },
    [=](C<'('>) { return parse_paren(); },
    [=](auto) -> double { throw "unexpected"; }
    },
    t);
    }
    double calculator::parse_paren()
    {
    auto v = parse_expr();
    auto t = lexer->next_token();
    if (!std::holds_alternative>(t)) throw "expected ')'";
    return v;
    }
    double calculator::parse_forget_var()
    {
    auto t = lexer->next_token();
    return std::visit(
    overload{
    [=](ident i) { return drop_variable(i.value);},
    [=](auto) -> double { throw "expected variable name";}
    },
    t);
    }
    double calculator::parse_remember_var()
    {
    auto var = lexer->next_token();
    if (!std::holds_alternative(var)) throw "expected variable name";
    if (!std::holds_alternative>(lexer->next_token())) throw "expected =";
    auto v = parse_expr();
    memory[std::get(var).value] = v;
    return v;
    }

    View full-size slide

  29. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    double calculator::parse(std::string s)
    {
    lexer = std::make_unique(move(s));
    auto t = lexer->peek();
    auto rv = std::visit(
    overload{
    [=](remember) { lexer->drop();return parse_remember_var();},
    [=](forget) { lexer->drop();return parse_forget_var();},
    [=](auto) { return parse_expr();}
    },
    t);
    if (!std::holds_alternative(lexer next_token()))

    throw "garbage after expr";
    return rv;
    }
    double calculator::parse_expr()
    {
    auto v = parse_term();
    for (bool done = false; !done;) {
    auto t = lexer->peek();
    v = std::visit(
    overload{
    [=](C<'+'>) { lexer->drop(); return v+parse_term();},
    [=](C<'-'>) { lexer->drop(); return v-parse_term();},
    [&](auto) { done = true; return v; }
    },
    t);
    }
    return v;
    }
    double calculator::parse_term()
    {
    auto v = parse_factor();
    for (bool done = false; !done;) {
    auto t = lexer->peek();
    v = std::visit(
    overload{
    [=](C<'/'>) { lexer->drop(); return v/parse_factor();},
    [=](C<'*'>) { lexer->drop(); return v*parse_factor();},
    [&](auto) { done = true; return v; }
    },
    t);
    }
    return v;
    }
    double calculator::parse_factor()
    {
    auto t = lexer->next_token();
    return std::visit(
    overload{
    [=](ident var) { return lookup(var.value); },
    [=](number n) { return n.value; },
    [=](C<'+'>) { return parse_term(); },
    [=](C<'-'>) { return -parse_term(); },
    [=](C<'('>) { return parse_paren(); },
    [=](auto) -> double { throw "unexpected"; }
    },
    t);
    }
    double calculator::parse_paren()
    {
    auto v = parse_expr();
    auto t = lexer->next_token();
    if (!std::holds_alternative>(t)) throw "expected ')'";
    return v;
    }
    double calculator::parse_forget_var()
    {
    auto t = lexer->next_token();
    return std::visit(
    overload{
    [=](ident i) { return drop_variable(i.value);},
    [=](auto) -> double { throw "expected variable name";}
    },
    t);
    }
    double calculator::parse_remember_var()
    {
    auto var = lexer->next_token();
    if (!std::holds_alternative(var)) throw "expected variable name";
    if (!std::holds_alternative>(lexer->next_token())) throw "expected =";
    auto v = parse_expr();
    memory[std::get(var).value] = v;
    return v;
    }
    double calculator::parse_factor()
    {
    auto t = lexer->next_token();
    return std::visit(
    overload{
    [=](ident var) { return lookup(var.value); },
    [=](number n) { return n.value; },
    [=](C<'+'>) { return parse_term(); },
    [=](C<'-'>) { return -parse_term(); },
    [=](C<'('>) { return parse_paren(); },
    [=](auto) -> double { throw "unexpected"; }
    },
    t);
    }

    View full-size slide

  30. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    double calculator::parse(std::string s)
    {
    lexer = std::make_unique(move(s));
    auto t = lexer->peek();
    auto rv = std::visit(
    overload{
    [=](remember) { lexer->drop();return parse_remember_var();},
    [=](forget) { lexer->drop();return parse_forget_var();},
    [=](auto) { return parse_expr();}
    },
    t);
    if (!std::holds_alternative(lexer next_token()))

    throw "garbage after expr";
    return rv;
    }
    double calculator::parse_expr()
    {
    auto v = parse_term();
    for (bool done = false; !done;) {
    auto t = lexer->peek();
    v = std::visit(
    overload{
    [=](C<'+'>) { lexer->drop(); return v+parse_term();},
    [=](C<'-'>) { lexer->drop(); return v-parse_term();},
    [&](auto) { done = true; return v; }
    },
    t);
    }
    return v;
    }
    double calculator::parse_term()
    {
    auto v = parse_factor();
    for (bool done = false; !done;) {
    auto t = lexer->peek();
    v = std::visit(
    overload{
    [=](C<'/'>) { lexer->drop(); return v/parse_factor();},
    [=](C<'*'>) { lexer->drop(); return v*parse_factor();},
    [&](auto) { done = true; return v; }
    },
    t);
    }
    return v;
    }
    double calculator::parse_factor()
    {
    auto t = lexer->next_token();
    return std::visit(
    overload{
    [=](ident var) { return lookup(var.value); },
    [=](number n) { return n.value; },
    [=](C<'+'>) { return parse_term(); },
    [=](C<'-'>) { return -parse_term(); },
    [=](C<'('>) { return parse_paren(); },
    [=](auto) -> double { throw "unexpected"; }
    },
    t);
    }
    double calculator::parse_paren()
    {
    auto v = parse_expr();
    auto t = lexer->next_token();
    if (!std::holds_alternative>(t)) throw "expected ')'";
    return v;
    }
    double calculator::parse_forget_var()
    {
    auto t = lexer->next_token();
    return std::visit(
    overload{
    [=](ident i) { return drop_variable(i.value);},
    [=](auto) -> double { throw "expected variable name";}
    },
    t);
    }
    double calculator::parse_remember_var()
    {
    auto var = lexer->next_token();
    if (!std::holds_alternative(var)) throw "expected variable name";
    if (!std::holds_alternative>(lexer->next_token())) throw "expected =";
    auto v = parse_expr();
    memory[std::get(var).value] = v;
    return v;
    }
    double calculator::parse_factor()
    {
    auto t = lexer->next_token();
    return std::visit(
    overload{
    [=](ident var) { return lookup(var.value); },
    [=](number n) { return n.value; },
    [=](C<'+'>) { return parse_term(); },
    [=](C<'-'>) { return -parse_term(); },
    [=](C<'('>) { return parse_paren(); },
    [=](auto) -> double { throw "unexpected"; }
    },
    t);
    }

    View full-size slide

  31. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    double calculator::parse(std::string s)
    {
    lexer = std::make_unique(move(s));
    auto t = lexer->peek();
    auto rv = std::visit(
    overload{
    [=](remember) { lexer->drop();return parse_remember_var();},
    [=](forget) { lexer->drop();return parse_forget_var();},
    [=](auto) { return parse_expr();}
    },
    t);
    if (!std::holds_alternative(lexer next_token()))

    throw "garbage after expr";
    return rv;
    }
    double calculator::parse_expr()
    {
    auto v = parse_term();
    for (bool done = false; !done;) {
    auto t = lexer->peek();
    v = std::visit(
    overload{
    [=](C<'+'>) { lexer->drop(); return v+parse_term();},
    [=](C<'-'>) { lexer->drop(); return v-parse_term();},
    [&](auto) { done = true; return v; }
    },
    t);
    }
    return v;
    }
    double calculator::parse_term()
    {
    auto v = parse_factor();
    for (bool done = false; !done;) {
    auto t = lexer->peek();
    v = std::visit(
    overload{
    [=](C<'/'>) { lexer->drop(); return v/parse_factor();},
    [=](C<'*'>) { lexer->drop(); return v*parse_factor();},
    [&](auto) { done = true; return v; }
    },
    t);
    }
    return v;
    }
    double calculator::parse_factor()
    {
    auto t = lexer->next_token();
    return std::visit(
    overload{
    [=](ident var) { return lookup(var.value); },
    [=](number n) { return n.value; },
    [=](C<'+'>) { return parse_term(); },
    [=](C<'-'>) { return -parse_term(); },
    [=](C<'('>) { return parse_paren(); },
    [=](auto) -> double { throw "unexpected"; }
    },
    t);
    }
    double calculator::parse_paren()
    {
    auto v = parse_expr();
    auto t = lexer->next_token();
    if (!std::holds_alternative>(t)) throw "expected ')'";
    return v;
    }
    double calculator::parse_forget_var()
    {
    auto t = lexer->next_token();
    return std::visit(
    overload{
    [=](ident i) { return drop_variable(i.value);},
    [=](auto) -> double { throw "expected variable name";}
    },
    t);
    }
    double calculator::parse_remember_var()
    {
    auto var = lexer->next_token();
    if (!std::holds_alternative(var)) throw "expected variable name";
    if (!std::holds_alternative>(lexer->next_token())) throw "expected =";
    auto v = parse_expr();
    memory[std::get(var).value] = v;
    return v;
    }

    View full-size slide

  32. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    double calculator::parse(std::string s)
    {
    lexer = std::make_unique(move(s));
    auto t = lexer->peek();
    auto rv = std::visit(
    overload{
    [=](remember) { lexer->drop();return parse_remember_var();},
    [=](forget) { lexer->drop();return parse_forget_var();},
    [=](auto) { return parse_expr();}
    },
    t);
    if (!std::holds_alternative(lexer next_token()))

    throw "garbage after expr";
    return rv;
    }
    double calculator::parse_expr()
    {
    auto v = parse_term();
    for (bool done = false; !done;) {
    auto t = lexer->peek();
    v = std::visit(
    overload{
    [=](C<'+'>) { lexer->drop(); return v+parse_term();},
    [=](C<'-'>) { lexer->drop(); return v-parse_term();},
    [&](auto) { done = true; return v; }
    },
    t);
    }
    return v;
    }
    double calculator::parse_term()
    {
    auto v = parse_factor();
    for (bool done = false; !done;) {
    auto t = lexer->peek();
    v = std::visit(
    overload{
    [=](C<'/'>) { lexer->drop(); return v/parse_factor();},
    [=](C<'*'>) { lexer->drop(); return v*parse_factor();},
    [&](auto) { done = true; return v; }
    },
    t);
    }
    return v;
    }
    double calculator::parse_factor()
    {
    auto t = lexer->next_token();
    return std::visit(
    overload{
    [=](ident var) { return lookup(var.value); },
    [=](number n) { return n.value; },
    [=](C<'+'>) { return parse_term(); },
    [=](C<'-'>) { return -parse_term(); },
    [=](C<'('>) { return parse_paren(); },
    [=](auto) -> double { throw "unexpected"; }
    },
    t);
    }
    double calculator::parse_paren()
    {
    auto v = parse_expr();
    auto t = lexer->next_token();
    if (!std::holds_alternative>(t)) throw "expected ')'";
    return v;
    }
    double calculator::parse_forget_var()
    {
    auto t = lexer->next_token();
    return std::visit(
    overload{
    [=](ident i) { return drop_variable(i.value);},
    [=](auto) -> double { throw "expected variable name";}
    },
    t);
    }
    double calculator::parse_remember_var()
    {
    auto var = lexer->next_token();
    if (!std::holds_alternative(var)) throw "expected variable name";
    if (!std::holds_alternative>(lexer->next_token())) throw "expected =";
    auto v = parse_expr();
    memory[std::get(var).value] = v;
    return v;
    }
    double calculator::parse_paren()
    {
    auto v = parse_expr();
    auto t = lexer->next_token();
    if (!std::holds_alternative>(t)) throw "expected ')'";
    return v;
    }

    View full-size slide

  33. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    double calculator::parse(std::string s)
    {
    lexer = std::make_unique(move(s));
    auto t = lexer->peek();
    auto rv = std::visit(
    overload{
    [=](remember) { lexer->drop();return parse_remember_var();},
    [=](forget) { lexer->drop();return parse_forget_var();},
    [=](auto) { return parse_expr();}
    },
    t);
    if (!std::holds_alternative(lexer next_token()))

    throw "garbage after expr";
    return rv;
    }
    double calculator::parse_expr()
    {
    auto v = parse_term();
    for (bool done = false; !done;) {
    auto t = lexer->peek();
    v = std::visit(
    overload{
    [=](C<'+'>) { lexer->drop(); return v+parse_term();},
    [=](C<'-'>) { lexer->drop(); return v-parse_term();},
    [&](auto) { done = true; return v; }
    },
    t);
    }
    return v;
    }
    double calculator::parse_term()
    {
    auto v = parse_factor();
    for (bool done = false; !done;) {
    auto t = lexer->peek();
    v = std::visit(
    overload{
    [=](C<'/'>) { lexer->drop(); return v/parse_factor();},
    [=](C<'*'>) { lexer->drop(); return v*parse_factor();},
    [&](auto) { done = true; return v; }
    },
    t);
    }
    return v;
    }
    double calculator::parse_factor()
    {
    auto t = lexer->next_token();
    return std::visit(
    overload{
    [=](ident var) { return lookup(var.value); },
    [=](number n) { return n.value; },
    [=](C<'+'>) { return parse_term(); },
    [=](C<'-'>) { return -parse_term(); },
    [=](C<'('>) { return parse_paren(); },
    [=](auto) -> double { throw "unexpected"; }
    },
    t);
    }
    double calculator::parse_paren()
    {
    auto v = parse_expr();
    auto t = lexer->next_token();
    if (!std::holds_alternative>(t)) throw "expected ')'";
    return v;
    }
    double calculator::parse_forget_var()
    {
    auto t = lexer->next_token();
    return std::visit(
    overload{
    [=](ident i) { return drop_variable(i.value);},
    [=](auto) -> double { throw "expected variable name";}
    },
    t);
    }
    double calculator::parse_remember_var()
    {
    auto var = lexer->next_token();
    if (!std::holds_alternative(var)) throw "expected variable name";
    if (!std::holds_alternative>(lexer->next_token())) throw "expected =";
    auto v = parse_expr();
    memory[std::get(var).value] = v;
    return v;
    }
    double calculator::parse_paren()
    {
    auto v = parse_expr();
    auto t = lexer->next_token();
    if (!std::holds_alternative>(t)) throw "expected ')'";
    return v;
    }

    View full-size slide

  34. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    double calculator::parse(std::string s)
    {
    lexer = std::make_unique(move(s));
    auto t = lexer->peek();
    auto rv = std::visit(
    overload{
    [=](remember) { lexer->drop();return parse_remember_var();},
    [=](forget) { lexer->drop();return parse_forget_var();},
    [=](auto) { return parse_expr();}
    },
    t);
    if (!std::holds_alternative(lexer next_token()))

    throw "garbage after expr";
    return rv;
    }
    double calculator::parse_expr()
    {
    auto v = parse_term();
    for (bool done = false; !done;) {
    auto t = lexer->peek();
    v = std::visit(
    overload{
    [=](C<'+'>) { lexer->drop(); return v+parse_term();},
    [=](C<'-'>) { lexer->drop(); return v-parse_term();},
    [&](auto) { done = true; return v; }
    },
    t);
    }
    return v;
    }
    double calculator::parse_term()
    {
    auto v = parse_factor();
    for (bool done = false; !done;) {
    auto t = lexer->peek();
    v = std::visit(
    overload{
    [=](C<'/'>) { lexer->drop(); return v/parse_factor();},
    [=](C<'*'>) { lexer->drop(); return v*parse_factor();},
    [&](auto) { done = true; return v; }
    },
    t);
    }
    return v;
    }
    double calculator::parse_factor()
    {
    auto t = lexer->next_token();
    return std::visit(
    overload{
    [=](ident var) { return lookup(var.value); },
    [=](number n) { return n.value; },
    [=](C<'+'>) { return parse_term(); },
    [=](C<'-'>) { return -parse_term(); },
    [=](C<'('>) { return parse_paren(); },
    [=](auto) -> double { throw "unexpected"; }
    },
    t);
    }
    double calculator::parse_paren()
    {
    auto v = parse_expr();
    auto t = lexer->next_token();
    if (!std::holds_alternative>(t)) throw "expected ')'";
    return v;
    }
    double calculator::parse_forget_var()
    {
    auto t = lexer->next_token();
    return std::visit(
    overload{
    [=](ident i) { return drop_variable(i.value);},
    [=](auto) -> double { throw "expected variable name";}
    },
    t);
    }
    double calculator::parse_remember_var()
    {
    auto var = lexer->next_token();
    if (!std::holds_alternative(var)) throw "expected variable name";
    if (!std::holds_alternative>(lexer->next_token())) throw "expected =";
    auto v = parse_expr();
    memory[std::get(var).value] = v;
    return v;
    }
    std::variant
    std::visit(visitor, v…)
    and overload of lambdas
    makes for simple type safe lexers
    and easy to read parser code

    View full-size slide

  35. Variant of Parsing, C++ Stockholm 0x07, Björn Fahller @bjorn_fahller
    Björn Fahller
    https://github.com/rollbear/variant_parse
    [email protected]
    @bjorn_fahller
    @rollbear cpplang, swedencpp
    A variant of recursive descent parsing

    View full-size slide