This method takes a quoted string like "foo $bar" and tokenises it into chunks of text and variable references.

NOTE: The argument list is something of a monstrosity which is likely to get pared down Real Soon Now.

The first three arguments are the regular $input, $output and $scope. Then we have the complete $token including any quote marks. We create a literal dquote element to represent this complete token in case we need to regenerate the original source code. The next argument, $pos, tells us the source position it was parsed at. That goes into the token element too. The next argument is the string content without the enclosing quotes. The final argument is the delimiter, usually '"', but it can be something else, e.g. in the case of qq:/foo $bar/.

We tokenise the string and create a sub-stream of tokens representing the original chunks. We don't want to inject these into the normal output stream because it would then look like they were read from the source, messing up any regeneration. So we hang them off the $token->[BRANCH] pointer.

+-------------------+  BRANCH   +-------------+
| dquote:"foo $bar" |---------->| text:"foo " |
+-------------------+           +-------------+
          |                            |  NEXT
         \|/                          \|/
         TBA                    +--------------+
                                | word:bar     |
                                +--------------+