Section: User Contributed Perl Documentation (1)
Updated: perl 5.005, patch 03Index
sitescooper - download news from web sites and convert it automaticallyinto one of several formats suitable for viewing on a Palm handheld.
sitescooper [options] [ [-site sitename] ...]
sitescooper [options] [-sites sitename ...]
sitescooper [options] [-name nm] [-levels n] [-storyurl regexp] [-set sitefileparam value] url [...]
Options: [-debug] [-refresh] [-fullrefresh] [-config file][-install dir] [-instapp app] [-dump] [-dumpprc] [-nowrite] [-nodates][-quiet] [-admin cmd] [-nolinkrewrite] [-stdout-to file] [-badcache][-keep-tmps] [-fromcache] [-noheaders] [-nofooters][-outputtemplate file.tmpl][-grep] [-profile file.nhp] [-profiles file.nhp file2.nhp ...][-filename template] [-prctitle template] [-parallel] [-disc][-limit numkbytes] [-maxlinks numlinks] [-maxstories numstories]
[-text | -html | -mhtml | -doc | -plucker | -mplucker | -isilo| -misilo | -richreader | -pipe fmt command][-bw | -color][-cvtargs args_for_converter]
This script, in conjunction with its configuration file and its set ofsite
files, will download news stories from several top news sites intotext format and/or onto your Palm handheld (with the aid of themakedoc
Alternatively URLs can be supplied on the command line, in which casethose URLs will be downloaded and converted using a reasonable set ofdefault settings.
HTTP and local files, using the file:/// protocol, are both supported.
Multiple types of sites are supported:1-level sites, where the text to be converted is all present on one page(such as Slashdot, Linux Weekly News, BluesNews, NTKnow, Ars Technica);
2-level sites, where the text to be converted is linked to from a Tableof Contents page (such as Wired News, BBC News, and I, Cringely);
3-level sites, where the text to be converted is linked to from a Tableof Contents page, which in turned is linked to from a list of issuespage (such as PalmPower).
In addition sites that post news as items on one big page, such asSlashdot, Ars Technica, and BluesNews, are supported using diff.
Note that at this moment in time, the URLs-on-the-command-line invocationformat does not support 2- or 3-level sites.
The script is portable to most UNIX variants that support perl, as wellas the Win32 platform (tested with ActivePerl 5.00502 build 509).
sitescooper maintains a cache in its temporary directory; files are keptin this cache for a week at most. Ditto for the text output directory(set with TextSaveDir in the built-in configuration).
If a password is required for the site, and the current sitescooper sessionis interactive, the user will be prompted for the username and password.This authentication token will be saved for later use. This way a sitethat requires login can be set up as a .site -- just log in once, and yourpassword is saved for future non-interactive runs.
Note however that the encryption used to hide the password in thesitescooper configuration is pretty transparent; I recommend that ratherthan using your own username and password to log in to passworded sites, adedicated, sitescooper account is used instead.
- Refresh all links -- ignore the already_seen file, do not diff pages,and always fetch links. If a cached page is available, it will beused.
- Refresh all links -- ignore the already_seen file, do not diff pages,and always fetch links, even if they are available in the cache.
- -config file
- Read the configuration from file instead of using the built-in one.
- -limit numkbytes
- Set the limit for output file size to numkbytes kilobytes, instead ofthe default 200K. A limit of 0 means unlimited, any amount of output.
- -maxlinks numlinks
- Stop retrieving web pages after numlinks have been traversed. This isnot used to specify how ``deep'' a site should be scooped -- it is thenumber of links followed in total.
- -maxstories numstories
- Stop retrieving web pages after numstories stories have been retrieved.
- -install dir
- The directory to save PDB files to once they've been converted, in orderto have them installed to your Palm handheld.
- -instapp app
- The application to run to install PDB files onto your Palm, once they'vebeen converted.
- -site sitename
- Limit the run to the site named in the sitename argument. Normally allavailable sites will be downloaded. To limit the run to 2 or more sites,provide multiple -site arguments like so:
-site ntk.site -site tbtf.site
- -sites sitename [...]
- Limit the run to multiple sites; an easier way to specify multiple sitesthan using the -site argument for each file.
- Use James Brown's NewsHound profile searching code. Any sites that do notcontain IgnoreProfiles: 1 will then be searched for the active profiles.Active profiles are loaded from the ProfileDir specified in the sitescooperconfiguration file, or specified using the -profile or -profilesarguments.
- -profile file.nhp
- Limit the run to the site named in the file.nhp argument. Normally allavailable sites will be downloaded. To limit the run to 2 or more sites,provide multiple -profile arguments like so:
-profile ntk.site -profile tbtf.site
- -profiles file.nhp [...]
- Limit the run to multiple sites; an easier way to specify multiple sitesthan using the -profile argument for each file.
- -name name
- When specifying a URL on the command-line, this provides the name thatshould be used when installing the site to the Pilot. It acts exactlythe same way as the Name: field in a site file.
- -levels n
- When specifying a URL on the command-line, this indicates how many levelsa site has. Not needed when using .site files.
- -storyurl regexp
- When specifying a URL on the command-line, this indicates the regularexpression which links to stories should conform to. Not needed when using.site files.
- Convert the page(s) downloaded into DOC format, with all the articleslisted in full, one after the other.
- Convert the page(s) downloaded into plain text format, with all thearticles listed in full, one after the other.
- Convert the page(s) downloaded into HTML format, on one big page, witha table of contents (taken from the site if possible), followed by allthe articles one after another.
- Convert the page(s) downloaded into HTML format, but retain themultiple-page format. This will create the output in a directorycalled site_name; in conjunction with the -dump argument,it will output the path of this directory on standard output beforeexiting.
- Convert the page(s) downloaded into Plucker format (seehttp://plucker.gnu-designs.com/ ), on one big page. The page(s) will bedisplayed with a table of contents (taken from the site if possible), followedby all the articles one after another.
- Convert the page(s) downloaded into iSilo format (seehttp://www.isilo.com/ ), on one big page. This is the default. Thepage(s) will be displayed with a table of contents (taken from the site ifpossible), followed by all the articles one after another.
- Convert the page(s) downloaded into iSilo format (seehttp://www.isilo.com/ ), with one iSilo document per site, with each storyon a separate page. The iSilo document will have a table-of-contentspage, taken from the site if possible, with each article on a separatepage.
- Convert the page(s) downloaded into RichReader format using HTML2Doc.exe(see http://users.erols.com/arenakm/palm/RichReader.html ). The page(s)will be displayed with a table of contents (taken from the site ifpossible), followed by all the articles one after another.
- -pipe fmt command
- Convert the page(s) downloaded into an arbitrary format, using the commandprovided. Sitescooper will still rewrite the page(s) according to thefmt argument, which should be one of:
- Plain text format.
- HTML in one big page.
- HTML in multiple pages.
The command argument can contain __SCOOPFILE__, which will be replacedwith the filename of the file containing the rewritten pages in the aboveformat, __SYNCFILE__, which will be replaced with a suitable filenamein the Palm synchronization folder, and __TITLE__, which will bereplaced by the title of the file (generally a string containing the dateand site name).
Note that for the -mhtml switch, __SCOOPFILE__ will be replacedwith the name of the file containing the table-of-contents page. It's upto the conversion utility to follow the href links to the other filesin that directory.
- Arguments for the conversion utility. For example, Plucker will display imagesbetter on some Palms using ``-cvtargs --bpp=2'' or ``-cvtargs --bpp=4''.
- Indicate that the target can display only 2-bit images, black and white only.This is generally the default for iSilo and Plucker.
- Indicate that the target can display colour images.
- Rewrite links to external sites or unscooped pages as underlined text, todifferentiate them from links to scooped pages. This is the default behaviourfor most formats apart from -plucker or -mplucker.
- Do not rewrite links to external sites or unscooped pages; leave them pointingoutside the current scoop. However, links to other pages that are included inthe current scoop, are rewritten to point to the scooped pages instead of thesource URL. This is the default for Plucker (-plucker or -mpluckerarguments).
- Do not rewrite links on scooped documents -- leave them exactly as they are.This includes even links to other scooped pages. See also -keeplinks).
- Output the page(s) downloaded directly to stdout in text or HTML format,instead of writing them to files and converting each one. This option NO LONGERimplies -text, like it used to, so to dump text, use -dump -text.
- Output the page(s) downloaded directly to stdout, in converted format as a PDBfile (note: not PRC format!), suitable for installation to a Palm handheld.
- Test mode -- do not write to the cache or already_seen file, instead writewhat would be written normally to a directory called new_cache and anew_already_seen file. This is very handy when writing a new site file.
- Send some HTTP headers to bypass web caching proxy servers. This is generallyuseful if a web caching proxy server somewhere between sitescooper and thetarget site is returning out-of-date files.
- Enable debugging output. This output is in addition to the usual progressmessages.
- Process sites quietly, without printing the usual progress messages toSTDERR. Warnings about incorrect site files and system errors will stillbe output, however.
- -admin cmd
- Perform an administrative command. This is intended to ease the task ofwriting scripts which use sitescooper output. The following admincommands are available:
- List the sites which would be scooped on a scooping run, and their URLs.Instead of scooping any sites, sitescooper will exit after performing thistask. The format is one site per line, with the site file name first, atab, the site's URL, a tab, the site name, a tab, and the output filenamethat would be generated without path or extension. For example:
foobar.site http://www.foobar.com/ Foo Bar 1999_01_01_Foo_Bar
- Write a journal with dumps of the documents as they pass through theformatting and stripping steps of the scooping process. This iswritten to a file called journal in the sitescooper temporarydirectory.
- import-cookies file
- Import a Netscape cookies file into sitescooper, so that certain siteswhich require them, can use them. For example, the siteeconomist_full.site requires this. Here's how to import cookies on aUNIX machine:
sitescooper.pl -admin import-cookies ~/.netscape/cookies
and on Windows:
perl sitescooper.pl -admin import-cookies ``C:\Program Files\Netscape\Users\Default\cookies.txt''
Unfortunately, MS Internet Explorer cookies are currently unsupported.If you wish to write a patch to support them, that'd be great.
- Do not attach the sitescooper header (URL, site name, and navigationlinks) to each page.
- Do not attach the sitescooper footer ("copyright retained by originalauthors'' blurb) to each page.
- -outputtemplate file.tmpl
- Read the output formatting template from the file file.tmpl. This overridesthe settings of the -noheaders and -nofooters flags.See the OUTPUT TEMPLATES section below for details on this.
- Do not perform any network access, retrieve everything from the cache orthe shared cache.
- -filename template
- Change the format of output filenames. template contains the followingkeyword strings, which are substituted as follows:
- The current year, in 4-digit format.
- The current month number (from 01 to 12), in 2-digit format.
- The current month name (from Jan to Dec), in 3-letter format.
- The current day of the month (from 01 to 31), in 2-digit format.
- The current day of the week (from Sun to Sat), in 3-letter format.
- The current hour (from 00 to 23), in 2-digit format.
- The current minute (from 00 to 59), in 2-digit format.
- The current site's name.
- The section of the current site (now obsolete).
The default filename template is YYYY_MM_DD_Site.
- -prctitle template
- Change the format of the titles of the resulting PDB files. templatemay contain the same keyword strings as -filename.
The default PDB title template is YYYY-Mon-DD: Site.
- Do not put the date in the installable file's filename. This allows you toautomatically overwrite old files with new ones when you HotSync. It'sa compatibility shortcut for -filename Site -prctitle ``Site''.
- -preload preload_method
- Preload pages using the given preload method. Currently supportedpreload methods are:
- Use the Perl LWP module to load pages. This is the default, and issingle-threaded; in other words, each page needs to load fully beforethe next page can be requested.
- Use a number of subprocesses running LWP requests to load pages. This ismulti-threaded, and several pages can be loaded at once; however you pay incosts of network bandwidth, CPU time and memory used. The optional nargument instructs sitescooper to use that number of processes; the defaultn is 4. This is only available on UNIX at the moment.
- Disconnect a PPP connection once the scooping has finished. Currently thiscode is experimental, and will probably only work on Macintoshes. This is offby default.
- -stdout-to file
- Redirect the output of sitescooper into the named file. This is needed onWindows NT and 95, where certain combinations of perl and Windows do notseem to support the > operator.
- Keep temporary files after conversion. Normally the .txt or .htmlrendition of a site is deleted after conversion; this option keepsit around.
You can control exactly what HTML or text is written to the output file usingthe -outputtemplate
argument. This argument takes the name of a file, whichis read and parsed to provide replacement templates for sitescooper.
The file is read as a HTML- or XML-style tagged format; so for example thetemplate for the main page in HTML format is read from between the<htmlmainpage> and </htmlmainpage> tags. The templatesthat can be defined are as follows:
- The main page, in HTML format; this is used when the -html output format, orone based on it (such as -plucker or -isilo), is used. It is also used for the-mhtml format's main (top-level) page.
- Sub-page, in HTML format; this is used for the -mhtml output format's subpages, ie. pages other than the top-level one.
- The snippet of HTML encapsulating each story. This is included for eachpiece of snarfed text, in all HTML files.
- The main page, in text format; this is used when the -text output format, orone based on it (such as -doc), is used.
- Sub-page, in text format; this is currently unused.
- The snippet of text encapsulating each story. This is included for eachpiece of snarfed text, in all text-format or DOC-format files.
A sample template file is provided in the file default_templates.html; thismay have been installed in the sitescooper install directory,/usr/share/sitescooper, or /usr/local/share/sitescooper. Note that the actualtemplates used are not loaded from this file; instead they are incorporatedinside the sitescooper script, so changing this file will have no effect.
To install, edit the script and change the #! line. You may also need to(a) change the Pilot install dir if you plan to use the pilot installationfunctionality, and (b) edit the other parameters marked with CUSTOMISE incase they need to be customised for your site. They should be set toacceptable defaults (unless I forgot to comment out the proxy server linesI use ;).
To snarf the ever-cutting NTKnow newsletter.
sitescooper.pl -refresh -html http://www.ntk.net/
To snarf NTKnow, ignoring any previously-read text, and producing HTMLoutput.
sitescooper.pl -refresh -html -site site_samples/tech/ntk.site
To snarf NTKnow using the site file provided with the main distribution,producing HTML output.
makes use of the $http_proxy
environment variable, if itis set.
Justin Mason <jm /at/ jmason.org>
Copyright (C) 1999-2000 Justin Mason
This program is free software; you can redistribute it and/or modify itunder the terms of the GNU General Public License as published by the FreeSoftware Foundation; either version 2 of the License, or (at your option)any later version.
This program is distributed in the hope that it will be useful, butWITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITYor FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public Licensefor more details.
You should have received a copy of the GNU General Public License alongwith this program; if not, write to the Free Software Foundation, Inc., 59Temple Place - Suite 330, Boston, MA 02111-1307, USA, or read it onthe web at http://www.gnu.org/copyleft/gpl.html .
The CPAN script category for this script is Web
All these can be picked up from CPAN at http://www.cpan.org/ . Note thatHTML::Entities is actually included in one of the previous packages, soyou do not need to install it separately.
will be used, if running on a Win32 platform, to find thePilot Desktop software's installation directory. Algorithm::Diff
to supportdiffing sites without running an external diff
application (this is requiredon Mac systems).
Sitescooper downloads news stories from the web and converts them to Palmhandheld iSilo, DOC or text format for later reading on-the-move. Sitefiles and full documentation can be found athttp://sitescooper.org/
into one of several formats suitable for viewing on a Palm handheld."
- OUTPUT TEMPLATES
- SCRIPT CATEGORIES
This document was created byman2html,using the manual pages.