MAN page from openSUSE Tumbleweed perl-Convert-UUlib-1.800.0-2.2.x86_64.rpm

UUlib

Section: User Contributed Perl Documentation (3)
Updated: 2025-05-29
Index

NAME

Convert::UUlib - decode uu/xx/b64/mime/yenc/etc-encoded data from a massive number of files

SYNOPSIS

 use Convert::UUlib ':all';  # read all the files named on the commandline and decode them # into the CURRENT directory. See below for a longer example. LoadFile $_ for @ARGV; for my $uu (GetFileList) {    if ($uu->state & FILE_OK) {      $uu->decode;      print $uu->filename, "\n";    } }

DESCRIPTION

This module started as an interface to the uulib/uudeview library by FrankPilhofer that can be used to decode all kinds of usenet (and other)binary messages.

After upstream abondoned the project, th library was continuously bugfixedand improved in this module, with major focuses on security fixes,correctness and speed (that does not mean that this library is consideredsafe with untrusted data, but it surely is safer than the poriginaluudeview).

Read the file doc/library.pdf from the distribution for in-depthinformation about the C-library used in this interface, and the rest ofthis document and especially the non-trivial decoder program at the end.

EXPORTED CONSTANTS

Action code constants

  ACT_IDLE      we don't do anything  ACT_SCANNING  scanning an input file  ACT_DECODING  decoding into a temp file  ACT_COPYING   copying temp to target  ACT_ENCODING  encoding a file

Message severity levels

  MSG_MESSAGE   just a message, nothing important  MSG_NOTE      something that should be noticed  MSG_WARNING   important msg, processing continues  MSG_ERROR     processing has been terminated  MSG_FATAL     decoder cannot process further requests  MSG_PANIC     recovery impossible, app must terminate

Options

  OPT_VERSION   version number MAJOR.MINORplPATCH (ro)  OPT_FAST      assumes only one part per file  OPT_DUMBNESS  switch off the program's intelligence  OPT_BRACKPOL  give numbers in [] higher precendence  OPT_VERBOSE   generate informative messages  OPT_DESPERATE try to decode incomplete files  OPT_IGNREPLY  ignore RE:plies (off by default)  OPT_OVERWRITE whether it's OK to overwrite ex. files  OPT_SAVEPATH  prefix to save-files on disk  OPT_IGNMODE   ignore the original file mode  OPT_DEBUG     print messages with FILE/LINE info  OPT_ERRNO     get last error code for RET_IOERR (ro)  OPT_PROGRESS  retrieve progress information  OPT_USETEXT   handle text messages  OPT_PREAMB    handle Mime preambles/epilogues  OPT_TINYB64   detect short B64 outside of Mime  OPT_ENCEXT    extension for single-part encoded files  OPT_REMOVE    remove input files after decoding (dangerous)  OPT_MOREMIME  strict MIME adherence  OPT_DOTDOT    ".."-unescaping has not yet been done on input files  OPT_RBUF      set default read I/O buffer size in bytes  OPT_WBUF      set default write I/O buffer size in bytes  OPT_AUTOCHECK automatically check file list after every loadfile

Result/Error codes

  RET_OK        everything went fine  RET_IOERR     I/O Error - examine errno  RET_NOMEM     not enough memory  RET_ILLVAL    illegal value for operation  RET_NODATA    decoder didn't find any data  RET_NOEND     encoded data wasn't ended properly  RET_UNSUP     unsupported function (encoding)  RET_EXISTS    file exists (decoding)  RET_CONT      continue -- special from ScanPart  RET_CANCEL    operation canceled

File States

 This code is zero, i.e. "false":  UUFILE_READ   Read in, but not further processed The following state codes are or'ed together:  FILE_MISPART  Missing Part(s) detected  FILE_NOBEGIN  No 'begin' found  FILE_NOEND    No 'end' found  FILE_NODATA   File does not contain valid uudata  FILE_OK       All Parts found, ready to decode  FILE_ERROR    Error while decoding  FILE_DECODED  Successfully decoded  FILE_TMPFILE  Temporary decoded file exists

Encoding types

  UU_ENCODED    UUencoded data  B64_ENCODED   Mime-Base64 data  XX_ENCODED    XXencoded data  BH_ENCODED    Binhex encoded  PT_ENCODED    Plain-Text encoded (MIME)  QP_ENCODED    Quoted-Printable (MIME)  YENC_ENCODED  yEnc encoded (non-MIME)

EXPORTED FUNCTIONS

Initializing and cleanup

Initialize is automatically called when the module is loaded and allocatesquite a small amount of memory for todays machines ;) CleanUp releases thatagain.

On my machine, a fairly complete decode with DBI backend needs about 10MBRSS to decode 20000 files.

CleanUp: Release memory, file items and clean up files. Should be called after adecoidng run, if you want to start a new one.

Setting and querying options

$option = GetOption OPT_xxx
SetOption OPT_xxx, opt-value

See the "OPT_xxx" constants above to see which options exist.

Setting various callbacks

SetMsgCallback [callback-function]
SetBusyCallback [callback-function]
SetFileCallback [callback-function]
SetFNameFilter [callback-function]

Call the currently selected FNameFilter

$file = FNameFilter $file

Loading sourcefiles, optionally fuzzy merge and start decoding

($retval, $count) = LoadFile $fname, [$id, [$delflag, [$partno]]]

Load the given file and scan it for encoded contents. Optionally tag itwith the given id, and if $delflag is true, delete the file after itis no longer necessary. If you are certain of the part number, you canspecify it as the last argument.

A better (usually faster) way of doing this is using the "SetFNameFilter"functionality.

$retval = Smerge $pass

If you are desperate, try to call "Smerge" with increasing $passvalues, beginning at 0, to try to merge parts that usually would nothave been merged.

Most probably this will result in garbled files, so never do this bydefault, except:

If the "OPT_AUTOCHECK" option has been disabled (by default it isenabled) to speed up file loading, then you have to call "Smerge -1"after loading all files as an additional pre-pass (which is normally doneby "LoadFile").

$item = GetFileListItem $item_number

Return the $item structure for the $item_number'th found file, or"undef" of no file with that number exists.

The first file has number 0, and the series has no holes, so you caniterate over all files by starting with zero and incrementing until youhit "undef".

This function has to walk the linear list of fils on each access, soif you want to iterate over all items, it is usually faster to use"GetFileList".

@items = GetFileList

Similar to "GetFileListItem", but returns all files in one go, which isvery much faster for large number of items, and has no drawbacks when usedfor a small number of items.

Decoding files

$retval = $item->rename ($newname): Change the ondisk filename where the decoded file will be saved.
$retval = $item->decode_temp: Decode the file into a temporary location, use "$item->infile" toretrieve the temporary filename.
$retval = $item->remove_temp: Remove the temporarily decoded file again.
$retval = $item->decode ([$target_path]): Decode the file to its destination, or the given target path.
$retval = $item->info (callback-function)

Querying (and setting) item attributes

$state = $item->state
$mode = $item->mode ([newmode])
$uudet = $item->uudet
$size = $item->size
$filename = $item->filename ([newfilename})
$subfname = $item->subfname
$mimeid = $item->mimeid
$mimetype = $item->mimetype
$binfile = $item->binfile

Information about source parts

$parts = $item->parts

Return information about all parts (source files) used to decode the fileas a list of hashrefs with the following structure:

 {   partno   => <integer describing the part number, starting with 1>,   # the following member sonly exist when they contain useful information   sfname   => <local pathname of the file where this part is from>,   filename => <the ondisk filename of the decoded file>,   subfname => <used to cluster postings, possibly the posting filename>,   subject  => <the subject of the posting/mail>,   origin   => <the possible source (From) address>,   mimetype => <the possible mimetype of the decoded file>,   mimeid   => <the id part of the Content-Type>, }

Usually you are interested mostly the "sfname" and possibly the "partno"and "filename" members.

Functions below are not documented and not very well tested - feedback welcome

  QuickDecode  EncodeMulti  EncodePartial  EncodeToStream  EncodeToFile  E_PrepSingle  E_PrepPartial

EXTENSION FUNCTIONS

Functions found in this module but not documented in the uulib documentation:

$msg = straction ACT_xxx

Return a human readable string representing the given action code.

$msg = strerror RET_xxx

Return a human readable string representing the given error code.

$str = strencoding xxx_ENCODED

Return the name of the encoding type as a string.

$str = strmsglevel MSG_xxx

Returns the message level as a string.

SetFileNameCallback $cb

Sets (or queries) the FileNameCallback, which is called whenever thedecoding library can't find a filename and wants to extract a filenamefrom the subject line of a posting. The callback will be called withtwo arguments, the subject line and the current candidate for thefilename. The latter argument can be "undef", which means that nofilename could be found (and likely no one exists, so it is safe to alsoreturn "undef" in this case). If it doesn't return anything (not even"undef"!), then nothing happens, so this is a no-op callback:

   sub cb {      return ();   }

If it returns "undef", then this indicates that no filename could befound. In all other cases, the return value is taken to be the filename.

This is a slightly more useful callback:

  sub cb {     return unless $_[1]; # skip "Re:"-plies et al.     my ($subject, $filename) = @_;     # if we find some *.rar, take it     return $1 if $subject =~ /(\w+\.rar)/;     # otherwise just pass what we have     return ();  }

LARGE EXAMPLE DECODER

The general workflow for decoding is like this:

1. Configure options with "SetOption" or "SetXXXCallback".
2. Load all source files with "LoadFile".
3. Optionally "Smerge".
4. Iterate over all "GetFileList" items (i.e. result files).
5. "CleanUp" to delete files and free items.

What follows is the file "example-decoder" from the distribution thatillustrates the above worklfow in a non-trivial example.

   #!/usr/bin/perl   # decode all the files in the directory uusrc/ and copy   # the resulting files to uudst/   use Convert::UUlib ':all';   sub namefilter {      my ($path) = @_;      $path=~s/^.*[\/\\]//;      $path   }   sub busycb {      my ($action, $curfile, $partno, $numparts, $percent, $fsize) = @_;      $_[0]=straction($action);      print "busy_callback(", (join ",",@_), ")\n";      0   }   SetOption OPT_RBUF, 128*1024;   SetOption OPT_WBUF, 1024*1024;   SetOption OPT_IGNMODE, 1;   SetOption OPT_IGNMODE, 1;   SetOption OPT_VERBOSE, 1;   SetOption OPT_AUTOCHK, 0;   # show the three ways you can set callback functions. I normally   # prefer the one with the sub inplace.   SetFNameFilter \&namefilter;   SetBusyCallback "busycb", 333;   SetMsgCallback sub {      my ($msg, $level) = @_;      print uc strmsglevel $_[1], ": $msg\n";   };   # the following non-trivial FileNameCallback takes care   # of some subject lines not detected properly by uulib:   SetFileNameCallback sub {      return unless $_[1]; # skip "Re:"-plies et al.      local $_ = $_[0];      # the following rules are rather effective on some newsgroups,      # like alt.binaries.games.anime, where non-mime, uuencoded data      # is very common      # if we find some *.rar, take it as the filename      return $1 if /(\S{3,}\.(?:[rstuvwxyz]\d\d|rar))\s/i;      # one common subject format      return $1 if /- "(.{2,}?\..+?)" (?:yenc )?\(\d+\/\d+\)/i;      # - filename.par (04/55)      return $1 if /- "?(\S{3,}\.\S+?)"? (?:yenc )?\(\d+\/\d+\)/i;      # - (xxx) No. 1 sayuri81.jpg 756565 bytes      # - (20 files) No.17 Roseanne.jpg [2/2]      return $1 if /No\.[ 0-9]+ (\S+\....) (?:\d+ bytes )?\[/;      # try to detect some common forms of filenames      return $1 if /([a-z0-9_\-+.]{3,}\.[a-z]{3,4}(?:.\d+))/i;      # otherwise just pass what we have      ()   };   # now read all files in the directory uusrc/*   for (<uusrc/*>) {      my ($retval, $count) = LoadFile ($_, $_, 1);      print "file($_), status(", strerror $retval, ") parts($count)\n";   }   Smerge -1;   SetOption OPT_SAVEPATH, "uudst/";   # now wade through all files and their source parts   for my $uu (GetFileList) {      print "file ", $uu->filename, "\n";      print " state ", $uu->state, "\n";      print " mode ", $uu->mode, "\n";      print " uudet ", strencoding $uu->uudet, "\n";      print " size ", $uu->size, "\n";      print " subfname ", $uu->subfname, "\n";      print " mimeid ", $uu->mimeid, "\n";      print " mimetype ", $uu->mimetype, "\n";      # print additional info about all parts      print " parts";      for ($uu->parts) {         for my $k (sort keys %$_) {            print " $k=$_->{$k}";         }         print "\n";      }      $uu->remove_temp;      if (my $err = $uu->decode) {         print " ERROR ", strerror $err, "\n";      } else {         print " successfully saved as uudst/", $uu->filename, "\n";      }   }   print "cleanup...\n";   CleanUp;

PERLMULTICORE SUPPORT

This module supports the perlmulticore standard (see<http://perlmulticore.schmorp.de/> for more info) for the followingfunctions - generally these are functions accessing the disk and/or usingconsiderable CPU time:

   LoadFile   $item->decode   $item->decode_temp   $item->remove_temp   $item->info

The perl interpreter will be reacquired/released on every callbackinvocation, so for performance reasons, callbacks should be avoided ifthat is costly.

Future versions might enable multicore support for more functions.

BUGS AND LIMITATIONS

The original uulib library this module uses was written at a time wheremain memory of measured in megabytes and buffer overflows as a securitythign didn't exist. While a lot of security fixes have been applied overthe years (includign some defense in depth mechanism that can shieldagainst a lot of as-of-yet undetected bugs), using this library forsecurity purposes requires care.

Likewise, file sizes when the uulib library was written were tiny comparedto today, so do not expect this library to handle files larger than 2GB.

Lastly, this module uses a very "C-like" interface, which means it doesn'tprotect you from invalid points as you might expect from "more perlish"modules - for example, accessing a file item object after callinbg"CleanUp" will likely result in crashes, memory corruption, or worse.

AUTHOR

Marc Lehmann <schmorpAATTschmorp.de>, the original uulib library was writtenby Frank Pilhofer <fpAATTinformatik.uni-frankfurt.de>, and later heavilybugfixed by Marc Lehmann.

Index

NAME

SYNOPSIS

DESCRIPTION

EXPORTED CONSTANTS

Action code constants
Message severity levels
Options
Result/Error codes
File States
Encoding types

EXPORTED FUNCTIONS

Initializing and cleanup
Setting and querying options
Setting various callbacks
Call the currently selected FNameFilter
Loading sourcefiles, optionally fuzzy merge and start decoding
Decoding files
Querying (and setting) item attributes
Information about source parts
Functions below are not documented and not very well tested - feedback welcome
EXTENSION FUNCTIONS

LARGE EXAMPLE DECODER

PERLMULTICORE SUPPORT

BUGS AND LIMITATIONS

AUTHOR

SEE ALSO

This document was created byman2html,using the manual pages.