MAN page from RedHat Other wget-man-1.4.5-3.i386.rpm

wget

Section: User Commands (1)
Updated: 1996 Nov 11
Index

NAME

wget - a utility to retrieve files from the World Wide Web

SYNOPSIS

wget [options] [URL-list]

WARNING

The information in this man page is an extract from the fulldocumentation ofWget.It is well out of date. Please refer to the info page for full,up-to-date documentation. You can view the info documentation withthe Emacs info subsystem or the standalone info program.

DESCRIPTION

Wgetis a utility designed for retrieving binary documents across the Web,through the use of HTTP (Hyper Text Transfer Protocol) andFTP (File Transfer Protocol), and saving them to disk.Wgetis non-interactive, which means it can work in the background, whilethe user is not logged in, unlike most of web browsers (thus you maystart the program and log off, letting it do its work). Analysingserver responses, it distinguishes between correctly and incorrectlyretrieved documents, and retries retrieving them as many times asnecessary, or until a user-specified limit is reached. REST isused in FTP on hosts that support it. Proxy servers aresupported to speed up the retrieval and lighten network load.

Wgetsupports a full-featured recursion mechanism, through which you canretrieve large parts of the web, creating local copies of remotedirectory hierarchies. Of course, maximum level of recursion and otherparameters can be specified. Infinite recursion loops are alwaysavoided by hashing the retrieved data. All of this works for bothHTTP and FTP.

The retrieval is conveniently traced with printing dots, each dotrepresenting one kilobyte of data received. Builtin features offermechanisms to tune which links you wish to follow (cf. -L, -D and -H).

URL CONVENTIONS

Most of the URL conventions described in RFC1738 are supported. Twoalternative syntaxes are also supported, which means you can use threeforms of address to specify a file:

Normal URL (recommended form):

http://host[:port]/pathhttp://fly.cc.fer.hr/ftp://ftp.xemacs.org/pub/xemacs/xemacs-19.14.tar.gzftp://username:password@host/dir/file

FTP only (ncftp-like):hostname:/dir/file

HTTP only (netscape-like):hostname(:port)/dir/file

You may encode your username and/or password to URL using the form:

ftp://user:password@host/dir/file

If you do not understand these syntaxes, just use the plain ordinarysyntax with which you would call lynx or netscape. Notethat the alternative forms are deprecated, and may cease beingsupported in the future.

OPTIONS

There are quite a few command-line options forwget.Note that you do not have to know or to use them unless you wish tochange the default behaviour of the program. For simple operations youneed no options at all. It is also a good idea to put frequently usedcommand-line options in .wgetrc, where they can be stored in a morereadable form.

This is the complete list of options with descriptions, sorted indescending order of importance:

-h --help: Print a help screen. You will also get help if you do not supplycommand-line arguments.

-V --version: Display version ofwget.

-v --verbose: Verbose output, with all the available data. The default outputconsists only of saving updates and error messages. If the output isstdout, verbose is default.

-q --quiet: Quiet mode, with no output at all.

-d --debug: Debug output, and will work only ifwgetwas compiled with -DDEBUG. Note that when the program is compiled withdebug output, it is not printed unless you specify -d.

-i filename --input-file=filename: Read URL-s fromfilename,in which case no URL-s need to be on the command line. If there areURL-s both on the command line and in a filename, those on thecommand line are first to be retrieved. The filename need not be anHTML document (but no harm if it is) - it is enough if the URL-sare just listed sequentially.
However, if you specify --force-html, the document will be regarded asHTML. In that case you may have problems with relative links,which you can solve either by adding <base href="url"> to the documentor by specifying --base=url on the command-line.

-o logfile --output-file=logfile: Log messages to logfile, instead of default stdout. Verboseoutput is now the default at logfiles. If you do not wish it, use -nv(non-verbose).

-a logfile --append-output=logfile: Append to logfile - same as -o, but appends to a logfile (or creatinga new one if the old does not exist) instead of rewriting the old logfile.

-t num --tries=num: Set number of retries tonum.Specify 0 for infinite retrying.

--follow-ftp: Follow FTP links from HTML documents.

-c --continue-ftp: Continue retrieval of FTP documents, from where it was left off. Ifyou specify "wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z", and thereis already a file named ls-lR.Z in the current directory,wgetcontinue retrieval from the offset equal to the length of the existingfile. Note that you do not need to specify this option if the onlything you want iswgetto continue retrieving where it left off when the connection is lost -wgetdoes this by default. You need this option when you want to continueretrieval of a file already halfway retrieved, saved by other FTPsoftware, or left bywget being killed.

-g on/off --glob=on/off: Turn FTP globbing on or off. By default, globbing will be turned on ifthe URL contains a globbing characters (an asterisk, e.g.). Globbingmeans you may use the special characters (wildcards) to retrieve morefiles from the same directory at once, like wgetftp://gnjilux.cc.fer.hr/*.msg. Globbing currently works only on UNIX FTPservers.

-e command --execute=command: Execute command, as if it were a part of .wgetrc file. Acommand invoked this way will take precedence over the same commandin .wgetrc, if there is one.

-N --timestamping: Use the so-called time-stamps to determine whether to retrieve afile. If the last-modification date of the remote file is equal to,or older than that of local file, and the sizes of files are equal,the remote file will not be retrieved. This option is useful forweekly mirroring ofHTTPorFTPsites, since it will not permit downloading of the same file twice.

-F --force-html: When input is read from a file, force it to be HTML. Thisenables you to retrieve relative links from existing HTML fileson your local disk, by adding <base href> to HTML, or using--base.

-B base_href --base=base_href: Use base_href as base reference, as if it were in the file, inthe form <base href="base_href">. Note that the base in the file willtake precedence over the one on the command-line.

-r --recursive

Recursive web-suck. According to the protocol of the URL, this canmean two things. Recursive retrieval of a HTTP URL means thatWgetwill download the URL you want, parse it as an HTML document (ifan HTML document it is), and retrieve the files this document isreferring to, down to a certain depth (default 5; change it with -l).Wgetwill create a hierarchy of directories locally, corresponding to theone found on the HTTP server.

This option is ideal for presentations, where slow connections shouldbe bypassed. The results will be especially good if relative linkswere used, since the pages will then work on the new location withoutchange.

When using this option with an FTP URL, it will retrieve all thedata from the given directory and subdirectories, similar toHTTP recursive retrieval.

You should be warned that invoking this option may cause graveoverloading of your connection. The load can be minimized by loweringthe maximal recursion level (see -l) and/or by lowering the number ofretries (see -t).

-m --mirror: Turn on mirroring options. This will set recursion and time-stamping,combining -r and -N.

-l depth --level=depth: Set recursion depth level to the specified level. Default is 5.After the given recursion level is reached, the sucking will proceedfrom the parent. Thus specifying -r -l1 should equal a recursion-lessretrieve from file. Setting the level to zero makes recursion depth(theoretically) unlimited. Note that the number of retrieved documentswill increase exponentially with the depth level.

-H --span-hosts: Enable spanning across hosts when doing recursive retrieving. See-r and -D. Refer toFOLLOWING LINKSfor a more detailed description.

-L --relative: Follow only relative links. Useful for retrieving a specific homepagewithout any distractions, not even those from the same host. Refer toFOLLOWING LINKSfor a more detailed description.

-D domain-list --domains=domain-list: Set domains to be accepted and DNS looked-up, where domain-list is acomma-separated list. Note that it does not turn on -H. This speedsthings up, even if only one host is spanned. Refer toFOLLOWING LINKSfor a more detailed description.

-A acclist / -R rejlist --accept=acclist / --reject=rejlist: Comma-separated list of extensions to accept/reject. For example, ifyou wish to download only GIFs and JPEGs, you will use -A gif,jpg,jpeg.If you wish to download everything except cumbersome MPEGs and .AUfiles, you will use -R mpg,mpeg,au.
-X list --exclude-directories list: Comma-separated list of directories to exclude from FTP fetching.

-P prefix --directory-prefix=prefix: Set directory prefix ("." by default) toprefix. The directory prefix is the directory where all otherfiles and subdirectories will be saved to.

-T value --timeout=value: Set the read timeout to a specified value. Whenever a read is issued,the file descriptor is checked for a possible timeout, which couldotherwise leave a pending connection (uninterrupted read). The defaulttimeout is 900 seconds (fifteen minutes).

-Y on/off --proxy=on/off: Turn proxy on or off. The proxy is on by default if the appropriateenvironmental variable is defined.

-Q quota[KM] --quota=quota[KM]: Specify download quota, in bytes (default), kilobytes ormegabytes. More useful for rc file. See below.

-O filename --output-document=filename: The documents will not be written to the appropriate files, but allwill be appended to a unique file name specified by this option. Thenumber of tries will be automatically set to 1. If this filename is`-', the documents will be written to stdout, and --quiet will beturned on. Use this option with caution, since it turns off all thediagnosticsWgetcan otherwise give about various errors.

-S --server-response: Print the headers sent by the HTTP server and/or responses sentby the FTP server.

-s --save-headers: Save the headers sent by the HTTP server to the file, before theactual contents.

--header=additional-header: Define an additional header. You can define more than additionalheaders. Do not try to terminate the header with CR or LF.

--http-user --http-passwd: Use these two options to set username and passwordWgetwill send to HTTP servers. Wget supports only the basicWWW authentication scheme.

-nc: Do not clobber existing files when saving to directory hierarchywithin recursive retrieval of several files. This option isextremelyuseful when you wish to continue where you left off with retrieval.If the files are .html or (yuck) .htm, it will be loaded fromthe disk, and parsed as if they have been retrieved from the Web.

-nv: Non-verbose - turn off verbose without being completely quiet (use-q for that), which means that error messages and basic informationstill get printed.

-nd: Do not create a hierarchy of directories when retrievingrecursively. With this option turned on, all files will getsaved to the current directory, without clobbering (ifa name shows up more than once, the filenames will getextensions .n).

-x: The opposite of -nd -- Force creation of a hierarchy of directorieseven if it would not have been done otherwise.

-nh: Disable time-consuming DNS lookup of almost all hosts. Refer toFOLLOWING LINKSfor a more detailed description.

-nH: Disable host-prefixed directories. By default, http://fly.cc.fer.hr/will produce a directory named fly.cc.fer.hr in which everything elsewill go. This option disables such behaviour.

--no-parent: Do not ascend to parent directory.

-k --convert-links: Convert the non-relative links to relative ones locally.

FOLLOWING LINKS

Recursive retrieving has a mechanism that allows you to specify whichlinkswgetwill follow.

Only relative links: When only relative links are followed (option -L), recursiveretrieving will never span hosts.will never get called, and the process will be very fast, with theminimum strain of the network. This will suit your needs most of thetime, especially when mirroring the output the output of *2htmlconverters, which generally produce only relative links.

Host checking: The drawback of following the relative links solely is that humansoften tend to mix them with absolute links to the very same host,and the very same page. In this mode (which is the default), allURL-s that refer to the same host will be retrieved.
The problem with this options are the aliases of the hosts and domains.Thus there is no way forwgetto know that regoc.srce.hr and www.srce.hr are the samehosts, or that fly.cc.fer.hr is the same as fly.cc.etf.hr.Whenever an absolute link is encountered, gethostbyname iscalled to check whether we are really on the same host. Althoughresults of gethostbyname are hashed, so that it will never getcalled twice for the same host, it still presents a nuisance e.g. inthe large indexes of difference hosts, when each of them has to belooked up. You can use -nh to prevent such complex checking, and thenwgetwill just compare the hostname. Things will run much faster, butalso much less reliable.

Domain acceptance: With the -D option you may specify domains that will be followed.The nice thing about this option is that hosts that are not fromthose domains will not get DNS-looked up. Thus you may specify-Dmit.edu,just to make sure that nothing outside .mit.edu gets looked up.This is very important and useful. It also means that -D doesnot imply -H (it must be explicitly specified). Feel free to usethis option, since it will speed things up greatly, with almost allthe reliability of host checking of all hosts.
Of course, domain acceptance can be used to limit the retrieval toparticular domains, but freely spanning hosts within the domain,but then you must explicitly specify -H.

All hosts: When -H is specified without -D, all hosts are being spanned. It isuseful to set the recursion level to a small value in those cases.Such option is rarely useful.

FTP: The rules forFTPare somewhat specific, since they have to be. To haveFTPlinks followed fromHTMLdocuments, you must specify -f (follow_ftp). If you do specify it,FTPlinks will be able to span hosts even if span_hosts is not set.Option relative_only (-L) has no effect onFTP.However, domain acceptance (-D) and suffix rules (-A/-R) still apply.

STARTUP FILE

Wgetsupports the use of initialization file.wgetrc.First a system-wide init file will be looked for(/usr/local/lib/wgetrc by default) and loaded. Then the user's filewill be searched for in two places: In the environmental variableWGETRC (which is presumed to hold the full pathname) and$HOME/.wgetrc.Note that the settings in user's startup file may override the systemsettings, which includes the quota settings (he he).

The syntax of each line of startup file is simple:

variable = value

Valid values are different for different variables. The complete setof commands is listed below, the letter after equation-sign denotingthe value the command takes. It is on/off for on oroff (which can also be 1 or 0), string for anystring or N for positive integer. For example, you may specify"use_proxy = off" to disable use of proxy servers by default. You mayuse inf for infinite value (the role of 0 on the commandline), where appropriate. The commands are case-insensitive andunderscore-insensitive, thus DIr__Prefix is the same asdirprefix. Empty lines, lines consisting of spaces, or linesbeginning with '#' are skipped.

Most of the commands have their equivalent command-line option,except some more obscure or rarely used ones. A sample init file isprovided in the distribution, named sample.wgetrc.

accept/reject = string: Same as -A/-R.
add_hostdir = on/off: Enable/disable host-prefixed hostnames. -nH disables it.
always_rest = on/off: Enable/disable continuation of the retrieval, the same as -c.
base = string: Set base for relative URL-s, the same as -B.
convert links = on/off: Convert non-relative links locally. The same as -k.
debug = on/off: Debug mode, same as -d.
dir_mode = N: Set permission modes of created subdirectories (default is 755).
dir_prefix = string: Top of directory tree, the same as -P.
dirstruct = on/off: Turning dirstruct on or off, the same as -x or -nd, respectively.
domains = string: Same as -D.
follow_ftp = on/off: FollowFTPlinks fromHTMLdocuments, the same as -f.
force_html = on/off: If set to on, force the input filename to be regarded as an HTMLdocument, the same as -F.
ftp_proxy = string: Use the string as FTP proxy, instead of the one specified inenvironment.
glob = on/off: Turn globbing on/off, the same as -g.
header = string: Define an additional header, like --header.
http_passwd = string: Set HTTP password.
http_proxy = string: Use the string as HTTP proxy, instead of the one specified inenvironment.
http_user = string: Set HTTP user.
input = string: Read the URL-s from filename, like -i.
kill_longer = on/off: Consider data longer than specified in content-length headeras invalid (and retry getting it). The default behaviour is to saveas much data as there is, provided there is more than or equalto the value in content-length.
logfile = string: Set logfile, the same as -o.
login = string: Your user name on the remote machine, forFTP.Defaults to "anonymous".
mirror = on/off: Turn mirroring on/off. The same as -m.
noclobber = on/off: Same as -nc.
no_parent = on/off: Same as --no-parent.
no_proxy = string: Use the string as the comma-separated list of domains to avoid inproxy loading, instead of the one specified in environment.
num_tries = N: Set number of retries per URL, the same as -t.
output_document = string: Set the output filename, the same as -O.
passwd = string: Your password on the remote machine, forFTP.Defaults tousernameAATThostname.domainname.
quiet = on/off: Quiet mode, the same as -q.
quota = quota: Specify the download quota, which is useful to put in/usr/local/lib/wgetrc. When download quota is specified,wgetwill stop retrieving after the download sum has become greater thanquota. The quota can be specified in bytes (default), kbytes ('k'appended) or mbytes ('m' appended). Thus "quota = 5m" will set thequota to 5 mbytes. Note that the user's startup file overrides systemsettings.
reclevel = N: Recursion level, the same as -l.
recursive = on/off: Recursive on/off, the same as -r.
relative_only = on/off: Follow only relative links (the same as -L). Refer to sectionFOLLOWING LINKSfor a more detailed description.
robots = on/off: Use (or not) robots.txt file.
server_response = on/off: Choose whether or not to print the HTTP and FTP serverresponses, the same as -S.
simple_host_check = on/off: Same as -nh.
span_hosts = on/off: Same as -H.
timeout = N: Set timeout value, the same as -T.
timestamping = on/off: Turn timestamping on/off. The same as -N.
use_proxy = on/off: Turn proxy support on/off. The same as -Y.
verbose = on/off: Turn verbose on/off, the same as -v/-nv.

SIGNALS

Wgetwill catch the SIGHUP (hangup signal) and ignore it. If theoutput was on stdout, it will be redirected to a file namedwget-log. This is also convenient when you wish to redirectthe output of Wget interactively.

$ wget http://www.ifi.uio.no/~larsi/gnus.tar.gz &$ kill -HUP %%       # to redirect the output

Wget will not try to handle any signals other thanSIGHUP. Thus you may interrupt Wget using ^C orSIGTERM.

EXAMPLES

Get URL http://fly.cc.fer.hr/:wget http://fly.cc.fer.hr/Force non-verbose output:wget -nv http://fly.cc.fer.hr/Unlimit number of retries:wget -t0 http://www.yahoo.com/Create a mirror image of fly's web (with the same directory structurethe original has), up to six recursion levels, with only one try perdocument, saving the verbose output to log file 'log':wget -r -l6 -t1 -o log http://fly.cc.fer.hr/Retrieve from yahoo host only (depth 50):wget -r -l50 http://www.yahoo.com/

ENVIRONMENT

http_proxy,ftp_proxy,no_proxy,WGETRC,HOME

FILES

/usr/local/lib/wgetrc,$HOME/.wgetrc

UNRESTRICTIONS

Wgetis free; anyone may redistribute copies of Wget to anyone under the terms stated in the General Public License, a copyof which accompanies each copy ofWget.

AUTHOR

Hrvoje Niksic <hniksicAATTsrce.hr> is the author of Wget. Thanks to thebeta testers and all the other people who helped with usefulsuggestions.

This document was created byman2html,using the manual pages.