HTML::TokeParser

Gisle Aas (gisle@aas.no)
08 Jul 1998 15:09:45 +0200 

I am uploading the HTML-Parser-2.20 now, which has this addition:

NAME
    HTML::TokeParser - Alternative HTML::Parser interface

SYNOPSIS
     require HTML::TokeParser;
     $p = HTML::TokeParser->new("index.html") || die "Can't open: $!";
     while (my $token = $p->get_token) {
         #...
     }

DESCRIPTION
    The HTML::TokeParser is an alternative interface to the
    HTML::Parser class. It basically turns the HTML::Parser inside
    out. You associate a file (or any IO::Handle object) with the
    parser at construction time and then repeatedly call $parser-
    >get_token to obtain the tags and text found in the parsed
    document. No need to make a subclass to make the parser do
    anything.

    Calling the methods defined by the HTML::Parser base class will
    be confusing, so don't do that. Use the following methods
    instead:

    $p = HTML::TokeParser->new( $file );
        The object constructor needs a file name or a reference to
        some file handle object as argument. If a file name (plain
        scalar) is passed to the constructor and the file can't be
        opened for reading, then the constructor will return an
        undefined value.

    $p->get_token
        This method will return the next *token* found in the HTML
        document, or `undef' at the end of the document. The token
        is returned as an array reference. The first element of the
        array will be a single character string denoting the type of
        this token; "S" for start tag, "E" for end tag, "T" for
        text, "C" for comment, and "D" for declaration. The rest of
        the array is the same as the arguments passed to the
        HTML::Parser callbacks (see the HTML::Parser manpage). This
        summarize the tokens that can occur:

          ["S", $tag, %$attr, @$attrseq, $origtext]
          ["E", $tag, $origtext]
          ["T", $text]
          ["C", $text]
          ["D", $text]

    $p->unget_token($token,...)
        If you find out you have read too many tokens you can push
        them back, so that they are returned the next time $p-
        >get_token is called.

    $p->get_tag( [$tag] )
        This method return the next tag (skipping any other tokens),
        or undef if there is no more tags in the document. If an
        argument is given, then we skip tokens until the specified
        tag is found. The tags are returned as a hash reference of
        the same form as for $p->get_token above, but the type code
        (first element) is missing and the name of end tags is
        prefixed with "/". This means that the tags returned look
        like this:

          [$tag, %$attr, @$attrseq, $origtext]
          ["/$tag", $origtext]

    $p->get_text( [$endtag] )
        This method returns all text found at the current position.
        It might return a zero length string if there is no text.
        The optional $endtag argument specify that any text
        occurring before the given tag is to be returned. Any
        entities will be expanded to their corresponding character.

        The $p->{textify} attribute is a hash that define how
        certain tags can be treated as text. If the name of a start
        tag match a key in this hash then this tag is converted to
        text. The hash value is used to specify which tag attribute
        to obtain the text from. If this attribute is missing, then
        the upper case name of the tag enclosed in brackets is
        returned, e.g. "[IMG]". The hash value can also be a
        subroutine reference. In this case the routine is called
        with the token content as parameters to obtain the text.

        The default $p->{textify} value is:

          {img => "alt", applet => "alt"}

        This means that <IMG> and <APPLET> tags are treated as text,
        and that the text to substitute can be found as ALT
        attribute.

    $p->get_trimmed_text( [$endtag] )
        Same as $p->get_text above, but will collapse any sequence
        of white space to a single space character. Leading and
        trailing space is removed.

EXAMPLES
    This example extract all links from a document. It will print
    one line for each link, containing the URL and the textual
    description between the <A>...</A> tags:

      use HTML::TokeParser;
      $p = HTML::TokeParser->new(shift||"index.html");

      while (my $token = $p->get_tag("a")) {
          my $url = $token->[1]{href} || "-";
          my $text = $p->get_trimmed_text("/a");
          print "$url\t$text\n";
      }

    This example extract the <TITLE> from the document:

      use HTML::TokeParser;
      $p = HTML::TokeParser->new(shift||"index.html");
      if ($p->get_tag("title")) {
          my $title = $p->get_trimmed_text;
          print "Title: $title\n";
      }

SEE ALSO
    the HTML::Parser manpage

COPYRIGHT
    Copyright 1998 Gisle Aas.

    This library is free software; you can redistribute it and/or
    modify it under the same terms as Perl itself.