Using rules and tokens

Grammars in Perl 6 offers a very useful way to split the grammar elements into parts. Let us use it to clarify the grammar elements.

The complex regex \s* (.*? ';') ['#' <-[\n]>* ]? contains two parts—the regex to extract a statement and the regex for comments. We are extracting them into separate rules. A single rule describes a small piece of the grammar and can refer to other rules. Examine the following example:

grammar G {
    rule TOP {
        ^ 
            [ <statement> \s* <comment>? ]*
        $
    }

    rule statement {
        .*? ';'
    }

    rule comment {
        '#' <-[\n]>*
    }
}

Now the TOP rule is much clearer and you immediately see that the program is a sequence of statements with optional comments after them (our grammar does not allow a comment without a statement).

So far, the grammar parses the refer.pl file completely but we can go further. We can extract statements and the next task is to understand them. Let us...