Before we get started there are some basic things you need to know for the examples to make sense.

$self->[4];         # Huh?  What's that then?

$self->[EXPR];      # Oh right, it's the expression slot

[META, NEXT, TOKEN POS]

use Template::TT3::Constants ':elements';

package Template::TT3::SomeModule;

use Template::TT3::Class
    version   => 3.14,
    constants => ':elements';

my $left_precedence = $self->[META]->[LPREC];

my $factory = $self->[META]->[ELEMS];

# $element is an element object
$token = \$element;

$$token = $$token->[NEXT];

$$token->parse_expr($token, $scope, $prec, $force);

foo = fill bar

$$token->parse_infix($lhs, $token, $scope, $parent, $follow)

$$token->parse_body($token, $scope, $parent, $follow)

The Template::TT3::Scanner is responsible for scanning the template source and identifying the blocks of plain text and embedded template tags. In this first scanning phase, the emphasis is on identifying the separate tokens in the document and creating elements to represent them. A simple ordered list of element objects is created. Each element object contains the source token (i.e. the literal text that was scanned) and its position (character offset) in the source template. It also has a reference to the token that immediately follows it. In other words, we create a single linked list of tokens with each token pointing forwards to the next token.

For example, consider this simple template:

Hello [% name %]

The scanner first recognises the chunk of text at the beginning and creates a text element ( Template::TT3::Element::Text ) to represent it. Elements are implemented as blessed array references (rather than the traditional blessed hash reference) for the sake of memory efficiency and execution speed. For the purpose of demonstration we'll show a simplified representation of this text element as:

# type, token, position 
[ text => 'Hello ', 0 ]         # Template::TT3::Element::Text

Following the text we have the [% start token. The scanner will recognise any number of different tags in a template. It effectively constructs a regular expression that matches a chunk of text up to the first tag start token. In pseudo-regex form it looks like this:

/ \G ( .*? ) ( $start1 | $start2 | ... | $startn ) /

Note the \G marker at the start of the pattern. This indicates that we always want to start scanning at the current global regex matching point. In other words, we always continue from where we left off. This allows us to scan the template source non-destructively.

Now that it has identified a tag start token, it looks up the tag object (a subclass of Template::TT3::Tag ) corresponding to that token and asks it to continue scanning from that point on. Note that we must pass it a reference to the source text rather than a copy of the source text. This is so that the tag can use the same \G marker to continue parsing from the current point. Passing around a reference to the text is also more efficient as it avoids copy the string each time.

So the tag gets a copy of the string and takes over the tokenising process. Different tags can implement different tokenising rules, but we'll just concentrate on what the default TT3 tag does for now. In this example, there is just a single word in the tag, name. However, we don't overlook the start tag, end tag and any intervening whitespace. We also create tokens to represent them:

[ tag_start  => '[%',     6 ]   # Template::TT3::Element::TagStart
[ whitespace => ' ',      8 ]   # Template::TT3::Element::Whitespace
[ word       => 'name',   9 ]   # Template::TT3::Element::Word
[ whitespace => ' ',     13 ]   # Template::TT3::Element::Whitespace
[ tag_end    => '%]',    14 ]   # Template::TT3::Element::TagEnd

Once the tag has identified its own end token it returns control to the scanner. We're now at the end of the template so there's nothing else for the scanner to add. Before it returns, it adds a final token to the end of the list to indicate the end of file. The full set of elements generated to represent this is template is:

[ text       => 'Hello ', 0 ]   # Template::TT3::Element::Text
[ tag_start  => '[%',     6 ]   # Template::TT3::Element::TagStart
[ whitespace => ' ',      8 ]   # Template::TT3::Element::Whitespace
[ word       => 'name',   9 ]   # Template::TT3::Element::Word
[ whitespace => ' ',     13 ]   # Template::TT3::Element::Whitespace
[ tag_end    => '%]',    14 ]   # Template::TT3::Element::TagEnd
[ EOF        => 'EOF',   16 ]   # Template::TT3::Element::Eof

Remember that each token also has a reference to the next token in the list, although that isn't show here.

The benefit of creating elements to represent all the tokens in the source, including all the things that we normally don't care about like whitespace, comments, tag start/end tokens, and so on, is that we can regenerate the original template source at any time. This is invaluable for debugging purposes. It also makes it possible to transform the template in various of interesting ways. For example, we can easily generate HTML complete with syntax highlighting. Or we can automatically parse a template, refactor it in some way (e.g. renaming a variable) and then save the modified template source back to disk.

TODO: There should be an optimisation option to ignore whitespace tokens if you're in a real hurry and know you're never going to need them.

TODO: I'm also planning to automatically consume whitespace into the tag start/end tokens (perhaps as an optional thing if I can think of any reason why someone might not want that). You would end up with a shorter token list:

[ text       => 'Hello ', 0 ]   # Template::TT3::Element::Text
[ tag_start  => '[% ',    6 ]   # Template::TT3::Element::TagStart
[ word       => 'name',   9 ]   # Template::TT3::Element::Word
[ tag_end    => ' %]',   14 ]   # Template::TT3::Element::TagEnd
[ EOF        => 'EOF',   16 ]   # Template::TT3::Element::Eof

Once we have a list of tokens we can parse them into expressions. But first let's clear up some terminology.

When we talk about tokens we're referring to the individual blocks of characters that we scanned from the template document: 'Hello ', '[%', ' ', and so on. However, they're represented internally as element objects. So tokens are elements and elements are tokens.

When we talk about expressions we're referring to particular combinations of tokens that have a specific meaning. This could be a complex expression like the following:

a + b * c + (d / e * 100)

Or it could be something degeneratively simple like this:

a

We also treat raw text chunks as simple expressions. In fact, we treat everything as an expression. A template is little more than a sequence of expressions. To process the template, we simply evaluate each expression and glue the resulting text into a single string.

Expressions are represented as trees of tokens. In the simple case an expression can be a solitary token. So expressions are tokens, which means they are also elements. Expressions are tokens are elements. They're just organised in a different way. Organising the raw stream of tokens into trees of expressions is the process of parsing the template. Note that we don't change the original linear sequence of tokens. Rather, we add additional links to describe the way that different tokens are connected together.

The first thing we need to consider is how we get rid of the whitespace tokens. Well that's easy. If you have an $element you can call its skip_ws() method.

$element = $element->skip_ws;

Of if you're using a reference to the element in $token:

$$token = $$token->skip_ws;

If the $element isn't a whitespace token then the method will return $self. In other words, no change. If it is a whitespace token, then the token calls skip_ws() on the token immediately following it (i.e. the next token). Thus, the skip_ws() method will merrily skip through any number of whitespace tokens and return a reference to the first non-whitespace token.

You can also pass a reference to the current token as an argument. In this case the skip_ws() method will automatically update your $element variable to reference the first non-whitespace token following.

$element->skip_ws(\$element);

Many of the element parsing methods expect a reference to the current token as the first argument. This is effectively the "current token" pointer and parsing methods will automatically advance it as they consume tokens. It's optional for the skip_ws() method and one or two other related methods, but it's mandatory for anything else that can advance the current parsing position. Because of this, all the methods assume that $token is already a reference to an element, so the code looks more like this:

$$token->skip_ws($token);

Having called skip_ws() to skip over any whitespace, we can now call parse_expr() on the token that follows in. We'll explain it in detail in a moment, but for now you just need to know that it returns an expression.

$expr = $$token->skip_ws->parse_expr($token, $scope)

As it happens, you don't need to explicitly call the skip_ws() method before calling parse_expr() . All whitespace tokens implement a parse_expr() method that does something like this:

sub parse_expr {
    my $self = shift;
    $self->next->parse_expr(@_);
}

Whitespace tokens "know" that whitespace is allowed before expressions, so they simply skip over themselves and call parse_expr() on the next token. If that's also a whitespace token then it will delegate onto its next token, and so on. So given any token, we can simply ask it to parse an expression and let it take care of skipping over whitespace:

$expr = $$token->parse_expr($token, $scope);

Lets look at our simple template again:

Hello [% name %]

The first token is a simple text token. It has an equally simple parse_expr() method that advances the $token reference and returns itself.

sub parse_expr {
    my ($self, $token) = @_;
    $$token = $self->[NEXT];
    return $self;
}

Yay! We've got our first expression. Furthermore, our $token reference now points at the token immediately following the text token. So we can ask that to return itself as an expression.

$expr = $$token->parse_expr($token, $scope);

In this case the token is the [% tag start token. But that's just a special kind of whitespace so it immediately skips over itself and asks the next token to parse_expr(). Guess what? That's whitespace too! But it all gets taken care of automatically. The next non-whitespace token is the name word. When words appear in expressions like this they're treated as variable names. So the word token also "knows" that it's an expression and can advanced the token pointer and return itself.

So to parse a sequence of expressions we can write something like this:

while ($expr = $$token->parse_expr($token, $scope)) {
    push(@exprs, $expr);
}

Well, almost. We also have to look out for delimiters like ; that are allowed (but not necessarily required) between statements:

[% foo; bar %]

To avoid complicating matters, we'll just skip to the fact that there's a parse_exprs() method which will parse a sequence of expressions.

$exprs = $$token->parse_exprs($token, $scope);

There are a few element that aren't expressions. We already mentioned the ; delimiter. Tag end tokens like %] are also considered a special kind of delimiter. Other terminator tokens like ), ], } and end are also non-expressions. If one of these is encountered then parse_exprs() will stop and return all the expressions that it's collected so far.

Operator precedence is used to ensure that binary operators are correctly grouped together. Consider this simple expression:

a + b * c

The mathematical convention is that * has a higher precedence than +. In practice that means that you do the multiplication first and then the addition:

a + (b * c)

Let's look at what happens when you call the parse_expr() method against the a token. The word token is a valid expression by itself, but instead of just returning itself, it calls the parse_infix() method on the token following it. It passes itself as the first argument followed by the usual $token and $scope arguments. The third argument is a precedence level. This is the key to operator precedence parsing.

sub parse_expr {
    my ($self, $token, $scope, $prec) = @_;

    # advance token
    $$token = $self->[NEXT];

    return $token->parse_infix($self, $token, $scope, $prec);
}

Let's first consider what happens if the next token isn't an infix operator (or a whitespace token which we'll assume gets skipped automatically). In this case, the default parse_infix() method returns $lhs. In other words, we get the a word token passed back as an expression.

If the next token is an infix operator, like + in our example, then it stores the left hand expression in its LHS slot. It then calls the parse_expr() on the next token (b) to return an expression to become its RHS.

sub parse_infix {
    my ($self, $lhs, $token, $scope) = @_;

    # save the LHS
    $self->[LHS] = $lhs;

    # advance token passed the operator
    $$token = $self->[NEXT];

    # parse an expression on the RHS
    $self->[RHS] = $$token->parse_expr($token, $scope);

    return $self;
}

To implement operator precedence we add an extra argument.

sub parse_infix {
    my ($self, $lhs, $token, $scope, $prec) = @_;

Each operator has it's own precedence level (defined in $self->[META]->[LPREC]). If the $prec argument is specified and it is higher that the operator's own precedence then the method returns the $lhs immediately.

return $lhs
    if $prec && $prec > $self->[META]->[LPREC];

If the operator's precedence is higher, or if $prec is undefined then the method continues. When it calls the parse_expr() method to fetch an expression for the RHS it passes its own precedence level as the extra argument.

$self->[RHS] = $$token->parse_expr(
    $token, $scope, $self->[META]->[LPREC]
);

The end result is that if we have a low precedence operator followed by a higher precedence operator, like this:

a + b * c

Then the higher precedence * completes itself first before returning back to the + operator.

a + (b * c)

If instead we have a high precedence operator coming first:

a * b + c

Then the second, lower precedence operator, will return immediately allow the first operator to complete itself first.

(a * b) + c

To complete things, we need to change the final return to be an additional call to parse_infix() in case there are any other lower-precedence infix operators following that have yielded to let us go first.

TODO: more on this

Simple constructor blesses all arguments into list based object.

Internal method for initialising the element metadata that is referenced via the META slot.

Returns a constructor function.

Stub configuration method for subclasses to redefine if they need to

Used to change an element into a different type.

Used to append a new element to the current element. You can either pass a reference to an element object, or the arguments required to construct one.

$element->append( text => 'Hello', 42 );      # type, token, position

This is similar to the append() method but attaches the new token to its BRANCH slot instead of the usual NEXT slot.

$element->branch( text => 'Hello', 42 );      # type, token, position

Used to append additional content to the scanned token.

my $element = $tokens->append( text => 'Hello ' );
$element->extend('World');

print $element->token;      # Hello World

This returns the metadata entry in the META slot.

This returns the element in the NEXT slot that indicate the next token in the template source. If a token reference is passed as an argument then it will also be updated to reference the next token. The following are equivalent:

$element = $element->next;

$element->next(\$element);

If $token is a reference to an element (as is usually the case) then you would write:

$$token->next($token);

This returns the original text of the element token.

print $element->token;

This return the character offset of the start of the token in the template source.

print " at offset ", $element->pos;

This method simply returns the first argument, the element object itself. It exists only to provide a useful no-op method to create aliases to.

Returns the leftward precedence of the element. This is the value stored in the LPREC slot of the META data. The leftward precedence is used for all postfix and infix operators.

Returns the rightward precedence of the element. This is the value stored in the RPREC slot of the META data. The rightward precedence is used for all prefix operators.

This method simply returns undef. Like self() it exists only for creating no-op method aliases. For example, the default parse_word() method is an alias to null(), effectively indicating that the element can't be parsed as a word. Elements that are words, or can be interpreted as words should define their own parse_word() method to do something meaningful.

This method of convenience simply returns the first argument. It can be used as syntactic sugar for elements that decline a particular role. For example, elements that aren't infix operators should implement a parse_infix() method like this:

sub parse_infix {
    my ($self, $lhs, $token, $scope, $prec) = @_;

    # we're not an infix operator so we just return the $lhs
    return $lhs;
}

However, it's a lot easier to define parse_infix() as an alias to the reject() method.

package Template::TT3::Element::SomeElement;

use Template::TT3::Class
    base  => 'Template::TT3::Element',
    alias = {
        parse_infix => 'reject',
    };

The alias() method will locate the reject() method defined in the Template::TT3::Element base class and define parse_infix() as a local alias to it.

In fact, you don't need to do this at all because the Template::TT3::Element base class already defines parse_infix() as an alias to reject() . However, you might be using another base class that defines a different parse_infix() method that you want to over-ride with the reject() method.

This is another simple method of convenience which advances the token reference to the next element and returns itself.

sub parse_expr {
    my ($self, $token) = @_;
    return $self->accept($token);
}

This is short-hand for:

sub parse_expr {
    my ($self, $token) = @_;
    $$token = $self->[NEXT];
    return $self;
}

This method can be used to test if an element's token matches a particular value. So instead of writing something like this:

if ($element->token eq 'hello') {
    # ...
}

You can write:

if ($element->is('hello')) {
    # ...
}

When you've matched successfully a token you usually want to do something meaningful and move onto the next token. In this example, $token is a reference to the current element:

    if ($$token->is('hello')) {
        # do something meaningful
        print "Hello back to you!";

        # advance to the next token
        $$token->next($token);
    }
    else {
        die "Sorry, I don't understand '", $$token->token, "\n";
    }

The is() method accepts a reference to the current element as an optional second argument. If the match is successful then the reference will be advanced to point to the next token. Thus the above example can be written more succinctly as:

if ($$token->is('hello', $token)) {
    # do something meaningful
    print "Hello back to you!";
}
else {
    die "Sorry, I don't understand '", $$token->token, "\n";
}

This method is similar to in() but allows you to specify a a reference to a hash array of potential matches that you're interested in. If the token matches one of the keys in the hash array then the corresponding value will be returned.

    my $matches = {
        hello => 'Hello back to you',
        hey   => 'Hey, wazzup?',
        hi    => 'Hello',
        yo    => 'Yo Dawg',
    };

    if ($response = $$token->in($matches)) {
        print $response;
    }
    else {
        die "Sorry, I don't understand '", $$token->token, "\n";
    }

As with is(), you can pass a reference to the current token as the optional second argument. If the match is successful then the reference will be advanced to point to the next token.

Used to skip over any whitespace tokens.

$expr = $$token->skip_ws->parse_expr($token, $scope);

Advances to the next token and then skips any whitespace

$expr = $$token->next_skip_ws->parse_expr($token, $scope);

This is equivalent to:

$expr = $$token->next->skip_ws->parse_expr($token, $scope);

Used to skip over any delimiter tokens. These include the semi-colon ; and any tag end tokens.

This method parses an expression. It returns undef if the element cannot be parsed as an expression.

my $expr = $$token->parse_expr($token);

This method parses a sequence of expressions separated by optional delimiters (semi-colons and tag end tokens).

my $list = $$token->parse_exprs($token, $scope, $prec)
    || die "No expressions!";

The method returns a reference to a list of parsed expression elements. If no expressions can be parsed then it returns undef. The optional $force argument can be set to a true value to force it to always return a list reference, even if no expressions were parsed. This is required to parse expressions inside constructs that can be empty. e.g. empty lists ([ ]), hash arrays ({ }), argument lists (foo()), etc.

# always returns a list reference with the $force option set
my $list = $$token->parse_exprs($token, $scope, $prec, 1);

foreach my $expr (@$list) {
    print "parsed expression: ", $expr->source, "\n";
}

This method is a wrapper around parse_exprs() that creates a block element ( Template::TT3::Element::Block ) to represent the list of expressions.

my $block = $$token->parse_block($token, $scope, $prec, 1);

print "parsed block: ", $block->source, "\n";

This method is the one usually called by command elements that expect a block or a single expression to follow them. For example, the if command can be written as any of the following:

    [% if x %][% y %][% end %]

    [% if x; y; end %]

    [% if x { y } %]

    [% if x y %]

The delimiter elements, %] and ;, respond to the parse_body() method call by parsing a block of expressions up to the terminating end token. The left brace element, {, responds by parsing a block of expressions up to the terminating }. All other elements response by calling their own parse_expr() method and returning themselves as a single expression (or undef if they don't yield an expression).

This method parses the element as a word. If an element can represent a word then it should return itself ($self). Otherwise it should return undef. The default method in the base class returns undef.

This method is called to parse a token immediately after a dotop. In the usual case we would expect to find a word after a dotop. Hence, the word element ( Template::TT3::Element::Word ) responds to this method. Other elements that can appear after a dotop include numbers and the $ and @ sigils.

[% foo.bar       %]
[% items.0       %]
[% users.$uid    %]
[% album.@tracks %]

All other elements inherit the default method which returns undef

This method parses the elements as an argument list. The only element that responds to this method is the left parenthesis (( - implemented in Template::TT3::Element::Construct::Parens ). All other elements inherit the default method which returns undef.

This is a wrapper around the parse_args() method which is used in place of parse_args() when a function appears on the left side of an assignment or in a named block/sub declaration.

[% bold(text) = "$text" %]

[% sub add(x,y) { x + y } %]

[% sub html_elem(name, %attrs, @content) {
      '<' name attrs.html_attrs '>'
        content.join
      ''
   }
%]

This method is called to parse an element as a filename. Words, keywords and quoted strings all constitute valid filenames (or filename parts) and implement methods that respond accordingly. The dot (.) and slash (/) may also appear in filenames.

Any of these elements will scan forwards for any other filename tokens and combine them into a single filename string. Consider the site/header.tt3 filename specified for the fill command in the example below:

[% fill site/header.tt3 %]

The filename is initially scanned as five separate tokens: 'site', '/', 'header', '.', and 'tt3'. The parse_filename() method will reconstitute them into a single string.

Elements that do not represent valid filename parts inherit the default method which returns undef

This method is called to parse postfix operators that can appear after a variable name. This includes the -- post-decrement and ++ post-increment operators, the () parenthesis that denote function arguments, and the [] and {} operators for accessing list and hash items.

foo--
bar++
bar(10,20)
score[n]
user{name}

Technically speaking, those last three are post-circumfix operators, but we're not too worried about technical details here. The important thing is that postfix operators in this implementation are those that can only appear directly after a variable name with no intervening whitespace.

Most elements are not postfix operators. The default parse_postfix() method simply returns the $lhs expression.

This method is called to parse binary infix operators. The element on the left of the operator is passed as the first argument, followed by the usual element reference (\$element / $token), scope ($scope) and optional precedence level ($prec).

Elements that represent infix operators will parse their right hand side from the following expression and return themselves. All other elements inherit the default method which returns the $lhs argument.

This is used to parse follow-on blocks, e.g. elsif or else following an if. It is probably going to be renamed as parse_branch().

This is a temporary hack. It may no longer be a temporary hack by the time you read this, but I just didn't get around to updating the documentation.

Evaluates the element expression in the context of $context and returns the text generated by it. Any non-text values will be coerced to text. This is the method that is called against expressions that are embedded in template blocks:

[% foo %]               # $foo_element->text($context)

Evaluates the element expression in the context of $context and returns the raw value generated by it. This is the method that is called on expression that yield their value to some other expression. e.g. values on the right hand side of an assignment such as the variable bar in the following example:

[% foo = bar %]         # $bar_element->value($context)

Evaluates the element expression in the context of $context and returns the raw value or values generated by it. This is the method that is called on expression that yield their values to some other expression. e.g. values passed as arguments to a function or method call.

[% foo(bar) %]          # $bar_element->values($context)
[% foo(@bar) %]         # $at_bar_element->values($context)

Most elements return the same thing from values() as they do for value() (one is usually an alias for the other).

TODO: more on this

TODO: more on this

TODO: more on this