SEARCH
NEW RPMS
DIRECTORIES
ABOUT
FAQ
VARIOUS
BLOG
DONATE




YUM REPOSITORY

 
 

HTML::TokeParser::Simple

Section: User Contributed Perl Documentation (3)
Updated: 2005-11-30
Index 

NAME

HTML::TokeParser::Simple - Easy to use "HTML::TokeParser" interface 

SYNOPSIS

 use HTML::TokeParser::Simple; my $p = HTML::TokeParser::Simple->new( $somefile ); while ( my $token = $p->get_token ) {     # This prints all text in an HTML doc (i.e., it strips the HTML)     next unless $token->is_text;     print $token->as_is; }
 

DESCRIPTION

"HTML::TokeParser" is an excellent module that's often used for parsing HTML.However, the tokens returned are not exactly intuitive to parse:

 ["S",  $tag, $attr, $attrseq, $text] ["E",  $tag, $text] ["T",  $text, $is_data] ["C",  $text] ["D",  $text] ["PI", $token0, $text]

To simplify this, "HTML::TokeParser::Simple" allows the user ask moreintuitive (read: more self-documenting) questions about the tokens returned.

You can also rebuild some tags on the fly. Frequently, the attributesassociated with start tags need to be altered, added to, or deleted. Thisfunctionality is built in.

Since this is a subclass of "HTML::TokeParser", all "HTML::TokeParser"methods are available. To truly appreciate the power of this module, pleaseread the documentation for "HTML::TokeParser" and "HTML::Parser". 

CONTRUCTORS

 

new($source)

The constructor for "HTML::TokeParser::Simple" can be used just like"HTML::TokeParser"'s constructor:

  my $parser = HTML::TokeParser::Simple->new($filename);  # or  my $parser = HTML::TokeParser::Simple->new($filehandle);  # or  my $parser = HTML::TokeParser::Simple->new(\$html_string);
 

new($source_type, $source)

If you wish to be more explicit, there is a new style ofconstructor avaiable.

  my $parser = HTML::TokeParser::Simple->new(file   => $filename);  # or  my $parser = HTML::TokeParser::Simple->new(handle => $filehandle);  # or  my $parser = HTML::TokeParser::Simple->new(string => $html_string);

Note that you do not have to provide a reference for the string if using thestring constructor.

As a convenience, you can also attempt to fetch the HTML directly from a URL.

  my $parser = HTML::TokeParser::Simple->new(url => 'http://some.url');

This method relies on "LWP::Simple". If this module is not found or the pagecannot be fetched, the constructor will "croak()". 

PARSER METHODS

 

get_token

This method will return the next token that "HTML::TokeParser::get_token()"method would return. However, it will be blessed into a class appropriatewhich represents the token type. 

get_tag

This method will return the next token that "HTML::TokeParser::get_tag()"method would return. However, it will be blessed into either theHTML::TokeParser::Simple::Token::Tag::Start orHTML::TokeParser::Simple::Token::Tag::End class. 

peek

As of version 3.14, you can now "peek()" at the upcomings tokens withoutaffecting the state of the parser. By default, "peek()" will return the textof the next token, but specifying an integer $count will return the text ofthe next $count tokens.

This is useful when you're trying to debug where you are in a document.

 warn $parser->peek(3); # show the next 3 tokens
 

ACCESSORS

The following methods may be called on the token object which is returned,not on the parser object. 

Boolean Accessors

These accessors return true or false.
*
"is_tag([$tag])"

Use this to determine if you have any tag. An optional ``tag type'' may bepassed. This will allow you to match if it's a particular tag. Thesupplied tag is case-insensitive.

 if ( $token->is_tag ) { ... }

Optionally, you may pass a regular expression as an argument.

*
"is_start_tag([$tag])"

Use this to determine if you have a start tag. An optional ``tag type'' may bepassed. This will allow you to match if it's a particular start tag. Thesupplied tag is case-insensitive.

 if ( $token->is_start_tag ) { ... } if ( $token->is_start_tag( 'font' ) ) { ... }

Optionally, you may pass a regular expression as an argument. To match allheader (h1, h2, ... h6) tags:

 if ( $token->is_start_tag( qr/^h[123456]$/ ) ) { ... }
*
"is_end_tag([$tag])"

Use this to determine if you have an end tag. An optional ``tag type'' may bepassed. This will allow you to match if it's a particular end tag. Thesupplied tag is case-insensitive.

When testing for an end tag, the forward slash on the tag is optional.

 while ( $token = $p->get_token ) {   if ( $token->is_end_tag( 'form' ) ) { ... } }

Or:

 while ( $token = $p->get_token ) {   if ( $token->is_end_tag( '/form' ) ) { ... } }

Optionally, you may pass a regular expression as an argument.

*
"is_text()"

Use this to determine if you have text. Note that this is not to beconfused with the "return_text" (deprecated) method described below!"is_text" will identify text that the user typically sees display in the Webbrowser.

*
"is_comment()"

Are you still reading this? Nobody reads POD. Don't you know you're supposedto go to CLPM, ask a question that's answered in the POD and get flamed? It'sa rite of passage.

Really.

"is_comment" is used to identify comments. See the HTML::Parser documentationfor more information about comments. There's more than you might think.

*
"is_declaration()"

This will match the DTD at the top of your HTML. (You do use DTD's, don'tyou?)

*
"is_process_instruction()"

Process Instructions are from XML. This is very handy if you need to parse outPHP and similar things with a parser.

Currently, there appear to be some problems with process instructions. You canoverride "HTML::TokeParser::Simple::Token::ProcessInstruction" if you need to.

*
"is_pi()"

This is a shorthand for "is_process_instruction()".

 

Data Accessors

Some of these were originally "return_" methods, but that name was not onlyunwieldy, but also went against reasonable conventions. The "get_" methodslisted below still have "return_" methods available for backwardscompatibility reasons, but they merely call their "get_" counterpart. Forexample, calling "return_tag()" actually calls "get_tag()" internally.
*
"get_tag()"

Do you have a start tag or end tag? This will return the type (lower case).Note that this is not the same as the "get_tag()" method on the actualparser object.

*
"get_attr([$attribute])"

If you have a start tag, this will return a hash ref with the attribute namesas keys and the values as the values.

If you pass in an attribute name, it will return the value for just thatattribute.

Returns false if the token is not a start tag.

*
"get_attrseq()"

For a start tag, this is an array reference with the sequence of theattributes, if any.

Returns false if the token is not a start tag.

*
"return_text()"

This method has been heavily deprecated (for a couple of years) in favor of"as_is". Programmers were getting confused over the difference between"is_text", "return_text", and some parser methods such as"HTML::TokeParser::get_text" and friends.

Using this method still succeeds, but will now carp and will be removedin the next major release of this module.

*
"as_is()"

This is the exact text of whatever the token is representing.

*
"get_token0()"

For processing instructions, this will return the token found immediately afterthe opening tag. Example: For <?php, ``php'' will be the start of the returnedstring.

Note that process instruction handling appears to be incomplete in"HTML::TokeParser".

Returns false if the token is not a process instruction.

 

MUTATORS

The "delete_attr()" and "set_attr()" methods allow the programmer to rewritestart tag attributes on the fly. It should be noted that bad HTML will be``corrected'' by this. Specifically, the new tag will have all attributeslower-cased with the values properly quoted.

Self-closing tags (e.g. <hr />) are also handled correctly. Some olderbrowsers require a space prior to the final slash in a self-closed tag. Ifsuch a space is detected in the original HTML, it will be preserved.

Calling a mutator on an token type that does not support that property is ano-op. For example:

 if ($token->is_comment) {    $token->set_attr(foo => 'bar'); # does nothing }
*
"delete_attr($name)"

This method attempts to delete the attribute specified. It will silently failif called on anything other than a start tag. The argument iscase-insensitive, but must otherwise be an exact match of the attribute you areattempting to delete. If the attribute is not found, the method will returnwithout changing the tag.

 # <body bgcolor="#FFFFFF"> $token->delete_attr('bgcolor'); print $token->as_is; # <body>

After this method is called, if successful, the "as_is()", "get_attr()"and "get_attrseq()" methods will all return updated results.

*
"set_attr($name,$value)"

This method will set the value of an attribute. If the attribute is not found,then "get_attrseq()" will have the new attribute listed at the end.

 # <p> $token->set_attr(class => 'some_class'); print $token->as_is; # <p class="some_class"> # <body bgcolor="#FFFFFF"> $token->set_attr('bgcolor','red'); print $token->as_is; # <body bgcolor="red">

After this method is called, if successful, the "as_is()", "get_attr()"and "get_attrseq()" methods will all return updated results.

*
"set_attr($hashref)"

Under the premise that "set_" methods should accept what their corresponding"get_" methods emit, the following works:

  $tag->set_attr($tag->get_attr);

Theoretically that's a no-op and for purposes of rendering HTML, it should be.However, internally this calls "$tag->rewrite_tag", so see that method tounderstand how this may affect you.

Of course, this is useless if you want to actually change the attributes, so youcan do this:

  my $attrs = {    class  => 'headline',    valign => 'top'  };  $token->set_attr($attrs)     if $token->is_start_tag('td') &&  $token->get_attr('class') eq 'stories';
*
"rewrite_tag()"

This method rewrites the tag. The tag name and the name of all attributes willbe lower-cased. Values that are not quoted with double quotes will be. Thismay be called on both start or end tags. Note that both "set_attr()" and"delete_attr()" call this method prior to returning.

If called on a token that is not a tag, it simply returns. Regardless of howit is called, it returns the token.

 # <body alink=#0000ff BGCOLOR=#ffffff class='none'> $token->rewrite_tag; print $token->as_is; # <body alink="#0000ff" bgcolor="#ffffff" class="none">

A quick cleanup of sloppy HTML is now the following:

 my $parser = HTML::TokeParser::Simple->new( string => $ugly_html ); while (my $token = $parser->get_token) {     $token->rewrite_tag;     print $token->as_is; }
 

PARSER VERSUS TOKENS

The parser returns tokens that are blessed into appropriate classes. Somepeople get confused and try to call parser methods on tokens and token methodson the parser. To prevent this, "HTML::TokeParser::Simple" versions 1.4 andabove now bless all tokens into appropriate token classes. Please keep this inmind while using this module (and many thanks to PodMaster<http://www.perlmonks.org/index.pl?node_id=107642> for pointing out this issueto me.) 

EXAMPLES

 

Finding comments

For some strange reason, your Pointy-Haired Boss (PHB) is convinced that thegraphics department is making fun of him by embedding rude things about him inHTML comments. You need to get all HTML comments from the HTML.

 use strict; use HTML::TokeParser::Simple; my @html_docs = glob( "*.html" ); open PHB, "> phbreport.txt" or die "Cannot open phbreport for writing: $!"; foreach my $doc ( @html_docs ) {     print "Processing $doc\n";     my $p = HTML::TokeParser::Simple->new( file => $doc );     while ( my $token = $p->get_token ) {         next unless $token->is_comment;         print PHB $token->as_is, "\n";     } } close PHB;
 

Stripping Comments

Uh oh. Turns out that your PHB was right for a change. Many of the commentsin the HTML weren't very polite. Since your entire graphics department wasjust fired, it falls on you need to strip those comments from the HTML.

 use strict; use HTML::TokeParser::Simple; my $new_folder = 'no_comment/'; my @html_docs  = glob( "*.html" ); foreach my $doc ( @html_docs ) {     print "Processing $doc\n";     my $new_file = "$new_folder$doc";     open PHB, "> $new_file" or die "Cannot open $new_file for writing: $!";     my $p = HTML::TokeParser::Simple->new( $file => doc );     while ( my $token = $p->get_token ) {         next if $token->is_comment;         print PHB $token->as_is;     }     close PHB; }
 

Changing form tags

Your company was foo.com and now is bar.com. Unfortunately, whoever wrote yourHTML decided to hardcode ``http://www.foo.com/'' into the "action" attribute ofthe form tags. You need to change it to ``http://www.bar.com/''.

 use strict; use HTML::TokeParser::Simple; my $new_folder = 'new_html/'; my @html_docs  = glob( "*.html" ); foreach my $doc ( @html_docs ) {     print "Processing $doc\n";     my $new_file = "$new_folder$doc";     open FILE, "> $new_file" or die "Cannot open $new_file for writing: $!";     my $p = HTML::TokeParser::Simple->new( file => $doc );     while ( my $token = $p->get_token ) {         if ( $token->is_start_tag('form') ) {             my $action = $token->get_attr(action);             $action =~ s/www\.foo\.com/www.bar.com/;             $token->set_attr('action', $action);         }         print FILE $token->as_is;     }     close FILE; }
 

CAVEATS

For compatability reasons with "HTML::TokeParser", methods that returnreferences are violating encapsulation and altering the references directlywill alter the state of the object. Subsequent calls to "rewrite_tag()"can thus have unexpected results. Do not alter these references directlyunless you are following behavior described in these docs. In the future,certain methods such as "get_attr", "get_attrseq" and others may return acopy of the reference rather than the original reference. This behavior hasnot yet been changed in order to maintain compatability with previous versionsof this module. At the present time, your author is not aware of anyone takingadvantage of this ``feature,'' but it's better to be safe than sorry.

Use of $HTML::Parser::VERSION which is less than 3.25 may result inincorrect behavior as older versions do not always handle XHTML correctly. Itis the programmer's responsibility to verify that the behavior of this codematches the programmer's needs.

Note that "HTML::Parser" processes text in 512 byte chunks. This sometimeswill cause strange behavior and cause text to be broken into more than onetoken. You can suppress this behavior with the following command:

 $p->unbroken_text( [$bool] );

See the "HTML::Parser" documentation andhttp://www.perlmonks.org/index.pl?node_id=230667 for more information. 

BUGS

There are no known bugs, but that's no guarantee.

Address bug reports and comments to: <eop_divo_sitrucAATTyahoo.com>. Whensending bug reports, please provide the version of "HTML::Parser","HTML::TokeParser", "HTML::TokeParser::Simple", the version of Perl, and theversion of the operating system you are using.

Reverse the name to email the author. 

SUBCLASSING

You may wish to change the behavior of this module. You probably do not wantto subclass "HTML::TokeParser::Simple". Instead, you'll want to subclass oneof the token classes. "HTML::TokeParser::Simple::Token" is the base class forall tokens. Global behavioral changes should go there. Otherwise, see theappropriate token class for the behavior you wish to alter. 

SEE ALSO

HTML::TokeParser::Simple::Token

HTML::TokeParser::Simple::Token::Tag

HTML::TokeParser::Simple::Token::Text

HTML::TokeParser::Simple::Token::Comment

HTML::TokeParser::Simple::Token::Declaration

HTML::TokeParser::Simple::Token::ProcessInstruction 

COPYRIGHT

Copyright (c) 2004 by Curtis ``Ovid'' Poe. All rights reserved. This program isfree software; you may redistribute it and/or modify it under the same terms asPerl itself 

AUTHOR

Curtis ``Ovid'' Poe <eop_divo_sitrucAATTyahoo.com>

Reverse the name to email the author.


 

Index

NAME
SYNOPSIS
DESCRIPTION
CONTRUCTORS
new($source)
new($source_type, $source)
PARSER METHODS
get_token
get_tag
peek
ACCESSORS
Boolean Accessors
Data Accessors
MUTATORS
PARSER VERSUS TOKENS
EXAMPLES
Finding comments
Stripping Comments
Changing form tags
CAVEATS
BUGS
SUBCLASSING
SEE ALSO
COPYRIGHT
AUTHOR

This document was created byman2html,using the manual pages.