MAN page from Old RedHat 5.X cleanfeed-0.95.7b-1.noarch.rpm

cleanfeed

Section: Cleanfeed - Because spam sucks (8)
Updated: Version 0.95.7b
Index

NAME

Cleanfeed - spam filter for Usenet news servers

SYNOPSIS

INN: Installed as filter_innd.pl, location is configured intoINN at compile time.

Highwind servers: <command line> -program cleanfeed -body

NNTPRelay: ExternalFilter=c:/perl/bin/perl.exe c:/news/cleanfeed.pl

DESCRIPTION

A spam filter for Usenet servers. Cleanfeed blocks spam on the wayinto your server, before it is written to disk or propagated to outboundfeeds. It can also block binaries in non-binary newsgroups and includesseveral other features to keep your newsfeed clean.

Cleanfeed currently works with INN, Cyclone, Typhoon, Breeze, andNNTPRelay servers. See my webpage (listed at the end of this document)for pointers to information about using Cleanfeed with CNews, Diablo,Collabra, or INN versions earlier than 1.5.1.

USAGE

For all versions, place the cleanfeed.conf configuration filesomewhere, then edit the Cleanfeed source file and change the$config_dir option at the top to point to the directory wherethe config file lives.

INN

Install the filter file (called cleanfeed) as filter_innd.pl, andcleanfeed.conf, in the location you specified in config.data (INN1.7.2 and earlier) or when configuring INN 2.x (usually the bin/filterdirectory under the installation root). Make sure both files are readableby the news user. Once in place, the filter is loaded with the commandctlinnd reload filter.perl meow. Filtering can be turned on withctlinnd perl y and turned off with ctlinnd perl n.

Cyclone/Typhoon/Breeze

Add the -program <file> and -body options to the bin/startscript, where <file> is the location and name of the Cleanfeedprogram. Restart the server. Cleanfeed will run as an external process(standalone mode). IMPORTANT: make sure both cleanfeed and cleanfeed.confare readable by the news user! Double-check the permissions as this isa fairly common mistake!

NNTPRelay

Find the ExternalFilter directive in config.txt and make it look like:

ExternalFilter=c:/perl/bin/perl.exe c:/news/cleanfeed.pl

Cleanfeed will run as an external process (standalone mode).

More detailed installation instructions are provided later in thisdocument.

CONFIGURATION OPTIONS

Configuration is accomplished by setting the various options in thecleanfeed.conf configuration file. This file is evaluated as Perlcode, so comments can be included in the usual Perl # syntax. Asample default file is included with the distribution.

If you would rather not use cleanfeed.conf, you can set itslocation to ``undef'' in the source and edit the configurationvariables directly in the source file.

cleanfeed.conf has two sections (which define perl hashes):%config_local and %config_append. Entries in %config_localwill override the default settings of the same name in the Cleanfeedsource. Entries in %config_append can be used to add to most ofthe default regular expressions, for items such as badguys,bin_allowed, poison_groups, etc. Settings in %config_appendfor these items will be appended to the default regexps, seperated by``|'' (or).

If you want to completely override the default regexps for these options,rather than just add to the defaults, you can add an entry for them intothe %config_local section of cleanfeed.conf.

All of this is done quite blindly, so if you do anything odd, be careful.(Cleanfeed will remove the common mistake of including two ``|'' (or) signsin a row.) All config options are exposed to %config_local, includingany that may not be present in the sample file. Only the defined list ofoptions are exposed to %config_append.

Options that are on/off or yes/no should be set to 1 for on/yes, or 0for off/no.

First, you need to tell Cleanfeed which news server software you areusing. At the top of the file, set the appropriate variable to 1. ForINN, set $inn; for Cyclone, Typhoon, or Breeze, set $highwind; andfor NNTPRelay, set $nntprelay. Ensure the other two (the ones you'renot using) are set to 0.

General Settings

aggressive: Set this to 0 to disable all content-based filters. Helpful to pleaseparanoid lawyers, or paranoid customers.
active_file: Set this to the full path to an active file, to allow Cleanfeed to knowwhat groups are moderated. This is normally your server's active file,but it doesn't have to be; it is possible, for example, to run Cyclonewith no active file, but give one to Cleanfeed anyway.

MD5 Body Filter Settings

do_md5: When turned on, the MD5 EMP checks will be done. This should be lefton unless you have a really good reason to turn it off. If you'rerunning Hippo along with Cleanfeed, you might feel Cleanfeed's MD5checks are redundant and want to turn them off, for example. Itwould probably be better to leave it on with the history turneddown, instead.
md5maxmultiposts: Start rejecting articles after we have seen this many copies, accordingto the MD5 checksum filter.
MD5History: How many articles to remember for MD5-based EMP comparison. Since the MD5filter is not prone to false positives, setting this higher is a good ideato catch more spam, if you have the RAM to spare.
MD5maxlife: When a spam is identified by the MD5 EMP filter, it is saved for continualrejection. MD5maxlife specifies how long, in hours, to keep a savedMD5 id which is no longer getting any hits. (A spam id which is stillgetting matches will be saved regardless of age.) 24 hours works well.
fuzzy_md5: When turned on, the message bodies will be munged up a bit before MD5checksums are generated. Whitespace and other non-alphanumericcharacters are stripped and letters are forced to lowercase, as wellas a couple other bits of treachery to try to defeat the ``hashbuster''spam-bots. This adds a bit of ``fuzziness'' to the MD5 filter, andresults in a performance hit as well.
Since the smarter spammers have discovered hashbusting, I recommendthat this be turned on.
fuzzy_max_length: Sets the maximum amount of lines for an article body to be subject tothe fuzzy_md5 munging (above). This keeps extremely large articlesout of those nasty regular expressions.
md5_skips_followups: Determines whether the MD5 filter checks articles with Referencesheaders. The default is to skip them. Setting this option to 0will result in all articles passing through the MD5 filter, whichcan result in a major performance hit, but does close another holein the filter. If you turn this off, you should increase MD5historyas well to avoid shortening your ``window''.
MD5HistSize: The maximum allowed size of the EMP memory for the MD5-checksum EMP filter.Use this as a ``sanity check'' to prevent a sudden burst of spam from eatingup all of your memory. It should be set high enough so that you normallynever hit this number; use the MD5MaxLife to expire the hash instead.

Header-Based EMP Filter Settings

do_phl: Turns on the NNTP-Posting-Host/Lines EMP filter. This filter identifiesspam by identical posting-host headers and article sizes in a short periodof time. You really don't want to turn this off.
do_fsl: Turns on the From/Subject/Lines EMP filter. This filter identifies spamby identical From and Subject headers and article sizes in a short periodof time. This is the one that gets the least number of hits these days,so you won't lose much by shutting it off.
maxmultiposts: Start rejecting articles after we have seen this many copies, accordingto the header-based EMP filter. Since false positives are somewhat morelikely with this filter than with MD5, this should be set appropriatelyhigher to reduce the odds.
ArticleHistory: How many ids to remember for header-based EMP comparison. Setting thishigher will catch more spam because there will be a larger ``window'' tolook at. Larger settings will also consume more memory and have a (small)impact on performance, as well as slightly increase the chance of a falsepositive (since the sample size will be larger). Most articles willactually take up two entries in this history because there are twodifferent header-based filters.
EMPmaxlife: Same as MD5maxlife but for the header-based EMP filter.
EMPHistSize: Same as MD5HistSize but for the header-based EMP filter. If you arerunning the header-based filter but not the MD5 filter for whateverreason, set this high.

Excessive Crosspost Settings

maxgroups: Reject articles crossposted so that followups will be to more thanthis many newsgroups.
low_xpost_maxgroups: Specify a special, lower crosspost limit for certain groups, specifedby regular expression in low_xpost_groups (below). Useful for beingmore strict in groups plagued by crossposting, such as sex, binaries,and jobs groups. (Replaces the old tfjmaxgroups option.)

Misplaced Binaries Filter

block_binaries: Enables blocking of binary posts in non-binary newsgroups. Which newsgroupsallow binaries is configured with bin_allowed (below).
max_encoded_lines: Sets the number of uuencoded or base64-encoded lines to allow beforeconsidering a post to be a binary. This should be set high enough to passregular PGP signatures. (Those satanic Netscape crypto-sigs can die alongwith the other binaries.) Default is 15 lines, which may be a little low ifyou are lenient, which you're not.
binaries_in_mod_groups: If set, binaries are allowed in spite of block_binaries if they areposted only to moderated groups (requires active_file).

HTML

block_mime_html: Enables blocking of MIME-encapsulated HTML posts. This does NOT affectstraight text/html or multipart/alternative posts of the type created bymisconfigured Netscape and Microsoft ``newsreaders'', but ONLY posts whichare MIME-encapsulated HTML, a favorite format of sex spammers whichoften sneaks in under the EMP radar.
block_html: Enables blocking of HTML and multipart/alternative posts. You can specifygroup patterns where HTML is allowed by setting html_allowed (below).

Cancel Message Filtering

block_late_cancels: If turned on, cancels for recently rejected articles will be rejected.Set the window with MIDmaxlife (below). This will result in ahuge number of rejections if you have multiple full feeds and youaren't backlogging. If you are concerned about your downstream sitesreceiving the cancels, leave this off. If you need a performance boost,turn it on.
MIDmaxlife: How long to remember rejected message-ids so cancels for these posts canlater be rejected. Specified in hours. This only has an effect ifblock_late_cancels is enabled (above).

Disabling Other Filters

do_scoring_filter: Enables the (new) ``scoring'' filter. You probably want to leave this on,even if you need to turn of aggressive mode (turning off aggressivemode will disable the content-based parts of the scoring filter).
do_mid_filter (INN only): Enables the message-id filter. This requires an additional patch toINN 1.7.2, which is included with Cleanfeed (but optional). The patchadds a new Perl hook to check message-id's during the NNTP CHECKtransaction, and decide whether to refuse the article. There is apatch for this for INN 2.0 which may get incorporated into the INNdistribution at some point. The default is off.
do_bot_checks: Enables the filters that check for spam bot signatures. The only reasonyou would ever want to turn this off is if you've written your ownversion, or something. Otherwise, leave it on.
do_supersedes_filter: Enables the Excessive Supersedes filter, to catch rogue Supersedesattacks. This filter begins dropping articles with Supersedes headersif too many appear from the same posting-host in a short time. Moderatedgroups are given a higher limit (if active_file is set), as isnews.answers. Default is on.
check_supersedes_path: If set, bad_cancel_paths will also be applied to Supersedes articles.Articles with Supersedes headers, where a path element matches the regexpin bad_cancel_paths, will be dropped. Default is on.
drop_useless_controls: If set, all control messages of types sendsys, senduuname, and versionwill be dropped. These are no longer useful and are a hole fordenial-of-service attacks due to the way INN and some other servershandle them. On by default.
drop_ihave_sendme: If set, control messages of types ihave and sendme will be dropped.See drop_useless_controls. If you use these types of control messages,turn this off. If you're not sure, then you're not using them.
drop_control_with_supersedes: Drops any and all control messages which contain a Supersedes header.Since control messages are not passed through the same filters as regularmessages, a rogue Supersedes attack can use control messages to avoidfiltering; this option closes this hole. Legitimate control messagesdon't have Supersedes headers. On by default.

Hash-Trimming

trimcycles: The EMP memories are trimmed every trimcycles times through the filter.
EMPstarttrimming: Tells the filter not to waste time trimming the EMP memories until theyhave this many entries. Just a minor performance enhancement duringthe first hours the filter is running or when you first start innd.

Logging

verbose: When turned on, verbose logging to news.notice will happen; spam domainswill be listed, etc. When off, only general messages will be logged,making the news.daily summaries less interesting but much shorter andmore to the point. (There is, alas, no way to shut off news.noticelogging entirely.) (news.notice only applies to INN.) Note that thiswill not reduce the number of log entries, but only their verbosity.
logfile (Standalone Mode): If set to the path to a file, this will enable logging of message-idsof all articles processed by the filter. Rejections will be loggedwith the reason for rejection. Note that this will create a very largelogfile which you will need to rotate or delete (see max_log_size,below).
reportfile (Standalone Mode): If set to the path to a file, this will enable generation of a simplereport of articles accepted and rejected. The report file will containone entry per line with the start time, end time, number of articlesaccepted, and number of articles rejected, tab-separated.
log_accepts (Standalone Mode): When using the above logfiles, this setting determines whether articlesaccepted should be logged. When disabled, only rejections will be logged.
max_log_size (Standalone Mode): The size at which to rotate the logfile. This will be replaced bytime-based rotation at some point.
statfile: If this is set to the full path of a file, a crude stats file will bewritten each time the filter is reloaded with ctlinnd reloadfilter.perl meow (for INN) or whenever the Cleanfeed process receives aSIGUSR1 (for standalone mode). The file shows how many entries arepresent in each of the EMP histories, MID history and excessivesupersedes history; timer information if enabled (see timer_info);and the contents of all configuration settings. Posting-hosts in foreach supersedes entry will be listed, along with their counts; theseare not being rejected unless they are over the threshold. Thedefault for this is undef, which disables creation of the stat file.
More comprehensive stats are planned for the future.

Timing Info

timer_info: When enabled, Cleanfeed will generate timing statistics telling youhow many articles per second are being examined by the filter andbeing accepted by the filter. This information will appear in thestatfile if this is enabled, and in the output of INN's ctlinnd modeif the mode.patch is applied to INN. Note that the accepted/secondrate is not necessarily the rate at which your server is acceptingarticles; articles can be rejected by the server after Cleanfeedpasses them, for example if they are posted to groups not in youractive file.
timer_interval: The period over which to average timing information, in seconds. Thedefault is 600 seconds, or 5 minutes.

Debugging

debug_batch_directory: Specifies a directory where debugging ``batchfiles'' can be written.See the Hacker's Guide in this document for more information.
debug_batch_size: The maximum size of a debugging batchfile before it gets rotated.Rotation is done by renaming the file to file.1, file.2, etc.,using the lowest number that doesn't already exist.

Regular Expressions

You can add to most of these regular expressions in the %config_appendsection of cleanfeed.conf; settings you add there will be added tothe defaults, rather than overriding them. If you want to completelyoverride the default settings you can add entries for these to the%config_local section instead.

bin_allowed: This is a regular expression telling the anti-binary filter in whichnewsgroups binaries are allowed. If all groups in the Newsgroups headermatch this pattern, binaries are allowed through the filter. (Thisobviously has no effect when the binary filter is disabled.) If thebinary filter is enabled and this is set to a null string (by overridingthe default in the local config) the result will be blocking all binariesregardless of where they are posted.
poison_groups: If any groups in the Newsgroups header match this regexp, the articlewill be rejected. Thus you can reject crossposts to certain groups evenif they are also posted to groups you carry.
html_allowed: This is a regular expression telling the anti-HTML filter in whichnewsgroups HTML and multipart/alternative posts are allowed. Thisonly has an effect if block_html is turned on (above). The default(to allow HTML in microsoft.* groups) can be added to in cleanfeed.conf.
If you don't want to allow HTML anywhere, not even the microsoft.*groups, override this setting in the local configuration and set itto a null string or undef.
md5exclude: If an article is posted only to groups matching this regexp, the MD5 EMPfilter will not be applied. Useful for ``test'' groups where it's okayfor lots of the posts to be the same.
allexclude: If an article is posted only to groups matching this regexp, NO checksare applied at all.
low_xpost_groups: If a group matches this regular expression, it gets a special crosspostlimit, set in low_xpost_maxgroups, rather than the general crosspostlimit set in maxgroups. This is useful for groups plagued by excessivecrossposting, such as sex, binaries, and jobs groups. The default isto limit crossposts to 6 groups in test, forsale, and jobs groups.Setting this to a null string, or undef, will disable this feature.
badguys: This is a monster regular expression containing domains of known spammers.Only the ``middle'' part of the domains are listed; these are checked asemail addresses in From headers by appending a list of top-level domainsto the end, and as URLs by prepending http:// and an optional ``www.''. Ifyou modify this list, be very careful not to end up with ``||'' in there(two ``or'' signs in a row); this will match every single post that comesthrough, which is Bad.
baddomainpat: If a post contains a URL for a site whose domain name matches thispattern (in .com, .net, and .nu TLDs only) the post will be rejected.For example, there are hundreds of spamming porn sites whose domain namesbegin or end with ``xxx''. This prevents us from having to keep up withtheir nonsense. Yes, it's a little aggressive, but it works.
exempt: Regular expression of NNTP-Posting-Hosts that are exempt from theposting-host-based EMP filter. This is for high-output systems whereall posts contain the same NNTP-Posting-Host header, such as AOL, whichif not exempted would end up hitting the posting-host EMP filter withall of their posts. There aren't many of these out there; a ``regular''multi-user system does not present a problem because the filter doesn'tkick in until it sees a large number of posts from the same posting-hostand also of the same length, in a short period of time.
supersedes_exempt: Regular expression of NNTP-Posting-Hosts that are exempt from theexcessive supersedes filter. Generally this will be systems whichpost a lot of FAQs.
bad_cancel_paths: Cancel messages will be rejected if the Path header contains elementsmatching this regular expression. Also applied to the NNTP-Posting-Host.If check_supersedes_path is set, this will also be checked againstthe Path header of articles with Supersedes headers. This list containssites which are or have recently been the source of rogue cancel attacks.
refuse_messageids (INN only): If you have do_mid_filter (above) enabled, and you have the optionalmessage-id patch applied to INN (or otherwise have obtained the hookfor filter_messageid in INN 2.0), this regular expression will be appliedto message-ids as they are offered to your server, and they will berefused if it matches.
net_abuse_groups
spam_report_groups: These regular expressions are used to exempt certain groups from certainfilters; for example, groups expected to contain spam reports, examplespams, NoCeM notices, etc. These are not in cleanfeed.conf; if youneed to add to them please let me know.

After modifying the filter file, always check for mistakes by typing:

 perl -cw filter_innd.pl (or cleanfeed or whatever you called it)

There should be no errors and no warnings.

You can check cleanfeed.conf with:

 perl -cw cleanfeed.conf

You will get several warnings about variables being used only once;these can be ignored.

If you are running INN, you can modify the file and reload it withctlinnd reload filter.perl meow while the server is running. Theconfiguration in f<cleanfeed.conf> will be reloaded at this time aswell.

With the Highwind servers, modifying the program will require a serverrestart (use the bin/restart script). Note that this will result inall connections (including newsreader clients) being dropped. Thisis not my fault. :)

When in standalone mode, configuration from cleanfeed.conf can bereloaded by sending Cleanfeed a SIGHUP.

I have no idea what NNTPRelay does, but I'm guessing it needs a restartas well.

IMPORTANT NOTE: A common mistake is not setting file permissions oncleanfeed/filter_innd.pl, cleanfeed.conf, and cleanfeed.local so thatthey are readable by the news user. Please double-check your permissions!If Cleanfeed is running, and fails to successfully load cleanfeed.conf,it will use the default settings instead of those you specified in theconfig file.

INSTALLATION - INN

These instructions assume you have the Perl hooks compiled into INN.If you don't, you will need to add them and rebuild the INN distributionbefore proceeding.

With INN, Perl is embedded into the innd program. The filter filedefines subroutines that are called by innd at the appropriate times.

SYSTEM REQUIREMENTS

In order to run Cleanfeed with INN, you will need:

*: INN 1.5.1 or later (1.7.2+insync1.1d or 2.1 recommended)
*: Perl 5.004 or later
*: Perl hooks compiled into INN
*: The MD5 Perl module

INN is available from:
http://www.isc.org/inn.html

The Insync distribution of INN (highly recommended if you aren't runningINN 2.1) is available from:
http://www.insync.net/~aos/inn.html

The MD5 Perl module is available from:
http://www.perl.com/CPAN-local/modules/by-module/MD5/

Perl itself is available from the Perl home page:
http://www.perl.com/

PATCHES AND STUFF

INN 2.0 includes everything you need to run Cleanfeed, except the MD5Perl module.

With earlier versions, Cleanfeed requires some patches to INN in orderto function properly.

If you are running INN 1.7.2+insync1.1d, you already have the originalfilter.patch and the dynamic-load.patch; You need only apply theupgrade.patch.

None of these patches are against INN 2.1; the ``extra feature'' oneslike mode.patch may not apply to 2.1. Ports are always welcome.

filter.patch

This patch provides the basic functionality for Cleanfeed by making someextra headers available to the Perl filter, as well as message bodies.This patch was changed in version 0.95.3. It is against INN 1.7.2 andshould be applied in the innd directory. This patch is included in theinsync ``megapatch'' for INN as of version 1.1c, so if you are running thisversion of INN you need not apply this patch. Not necessary for INN 2.x.

dynamic-load.patch

This patch enables INN's Perl interpreter to load dynamic modules. It isnecessary for MD5 support. The patch is against INN 1.7+insync and shouldbe applied in the lib directory (NOT the innd directory). It applies cleanlyto other versions of INN including 1.5.1 and 1.7.2. This patch is includedin the insync ``megapatch'' for INN as of version 1.1d, so if you are runningthis version of INN you need not apply this patch. Not necessary for INN 2.x.

If you are still using INN 1.5.1, you can use dynamic-1.5.1.patch instead.

In order to compile INN with the new patch, you need to edit the PERL_LIBentry in config.data. Type this command at the shell, and paste its outputinto config.data as PERL_LIB:

    perl -MExtUtils::Embed -e ldopts

Most systems also allow you to simply enter that line in backquotes as PERL_LIB.

This patch requires Perl 5.004 or later! INN will not compile linked withPerl 5.003 after following these instructions!

AIX: There is a problem with Perl dynamic loading from INN under theAIX operating system. In simple terms, it doesn't work. This seems tobe a problem with the gcc compiler. Success has been reported byrebuilding both Perl and INN with IBM's commercial compiler CSet(a.k.a. xlC).

Solaris: There have been multiple reports of Cleanfeed not workingunder Solaris if any part of the system -- INN, Perl, or the MD5 module --are compiled using egcs. Success has been reported by recompilingeverything with gcc, and by upgrading to the very newest egcs.

upgrade.patch

For current users of Cleanfeed, this is a patch for an already-patchedINN, or for 1.7.2+insync1.1d, to bring you up to the new version of theCleanfeed patch. Not applying this patch right now will only lose you acouple of filters, and nothing will break if you don't apply it (nochanges to the filter source or configuration will be required).

messageid.patch

This is a patch which adds a new Perl hook to innd, filter_messageid.This allows you to run a Perl subroutine against each message-id asit is offered to your server, and decide whether to refuse the articlebefore it is even sent to your server. Cleanfeed includes a smallfilter_messageid. This patch is entirely optional.

mode.patch

This patch adds a line to INN's ctlinnd mode output for Perl filterstatus. The output line is generated by the filter_stats subroutine.The default output contains the number of articles accepted, rejectedand refused since the filter started, and the sizes of the EMP,Message-ID, and Excessive Supersedes hashes. If timer_info is enabled,this will also include the rate in articles per second (rounded to thenearest tenth) at which articles were examined (total sent through thefilter) and accepted by the filter, averaged over the timer_intervalnumber of seconds.

After applying the patches, rebuild all of INN and do a ``make update''.The first patch (filter.patch) only requires innd to be rebuilt, butthe dynamic-load.patch requires you to rebuild the whole distribution.Current users upgrading with upgrade.patch need only rebuild innd andreinstall that executable.

Thus:

    cd inn    [to the top-level source directory]    make clean    cd innd    cp wherever/filter.patch .     [from the Cleanfeed distribution]    patch <filter.patch    cd ../lib    cp wherever/dynamic-load.patch   [from the Cleanfeed distribution]    patch <dynamic-load.patch    cd ../config    emacs config.data    [edit the PERL_LIB entry as above]    make all    make update

Finally, you need to install the MD5 Perl module, no matter what version ofINN you are running.

INSTALLING CLEANFEED - INN

In INN 1.7.2 and earlier, the location where INN looks for the Perl filteris set in config.data, as _PATH_PERL_FILTER_INND. By default, thefilename is filter_innd.pl. The Cleanfeed filter program file shouldbe installed in this location. INN comes with an example filter_innd.plfile; move this file (or whatever other filter is in place) out of the wayfirst.

Before putting the filter in place, edit the file, changing $config_dirto the location of your cleanfeed.conf file.

After editing the file, always check for errors with the command:

    perl -cw filter_innd.pl

Once the file is in place, tell innd to reload it:

    ctlinnd reload filter.perl meow

And, if Perl filtering is currently disabled, enable it:

    ctlinnd perl y

Now, you can watch it working by looking at your news.notice log:

    tail -f /var/log/news/news.notice

If your server is running a full feed, you should start seeing aconstant stream of rejections almost immediately.

INSTALLATION - HIGHWIND SERVERS

The various Highwind server packages (Cyclone, Typhoon, and Breeze)all have the same external filter interface. The filter runs asits own process, reading from standard input and writing to standardoutput.

SYSTEM REQUIREMENTS

In order to run Cleanfeed with a Highwind server, you will need:

*: Cyclone, Typhoon or Breeze
*: Perl 5.003 or later
*: The MD5 Perl module

The Highwind servers are commercial products. For more information:
http://www.highwind.com/

The MD5 Perl module is available from:
http://www.perl.com/CPAN-local/modules/by-module/MD5/

Perl itself is available from the Perl home page:
http://www.perl.com/

INSTALLING CLEANFEED - HIGHWIND

The Cleanfeed program file should be installed as ``cleanfeed'' in yournews server's bin directory (cyclone/bin, etc). Make it owned bynews:news and make it executable.

Before putting the filter in place, edit the file, changing $config_dirto the location of your cleanfeed.conf file. Also ensure that theshebang line (the first line of the file, starting with #!) points tothe correct location of your perl executable.

After editing the file, always check for errors with the command:

    perl -cw cleanfeed

There should be no warnings.

Now, edit your bin/start script. You need to add two options to thecommand line that starts up the server process, the -program option totell it what program to use as a filter, and the -body option to tellit to send the bodies as well as the headers.

typhoond -program /typhoon/bin/cleanfeed -body

...along with whatever else you have cluttering up the command line.

(Highwind has indicated that this may/will be a config file optionin a future release.)

Now you can restart the server with the bin/restart script. Checkto make sure Cleanfeed is running, with ``ps -ef'' or ``top''. IfCyclone/Typhoon is unable to start the filter for some reason, it willlog an error via syslog. The error will not be terribly helpful.

You can make Cleanfeed reload its configuration from cleanfeed.confand local code from cleanfeed.local by sending it a SIGHUP.

INSTALLATION - NNTPRELAY

Please note that I do not have an NNTPRelay server, nor access to one,nor much interest in mucking around with Windows NT, and thus I havenot tested the NNTPRelay filtering support myself. The necessary changesand notes were contributed by someone else. Additions and improvementsto this documentation would be most welcome.

The filter interface in NNTPRelay is pretty much the same as in theHighwind servers.

SYSTEM REQUIREMENTS

In order to run Cleanfeed with NNTPRelay, you will need:

*: NNTPRelay version 1.1b4 or later
*: Perl 5.003 or later
*: The MD5 Perl module

NNTPRelay is available from:
http://nntprelay.maxwell.syr.edu/

An NT binary release of Perl 5.004, which apparently includes the MD5module, can be found at:
http://www.perl.com/CPAN/ports/win32/Standard/x86

The MD5 module (in source code) can be found at:
http://www.perl.com/CPAN-local/modules/by-module/MD5/

INSTALLING CLEANFEED - NNTPRELAY

Before putting the filter in place, edit the file, changing $config_dirto the location of your cleanfeed.conf file.

Install the Cleanfeed program file wherever is appropriate onyour system, as ``cleanfeed.pl''. Edit NNTPRelay's config.txtfile, adding an entry like this:

    ExternalFilter=c:/perl/bin/perl.exe c:/news/cleanfeed.pl

Of course, use the correct path to your Perl executable and tothe Cleanfeed program file. Now restart NNTPRelay. If youdefined a logfile in Cleanfeed, it should appear.

THE HACKER'S GUIDE

Cleanfeed will look for a file called cleanfeed.local, in the samedirectory as cleanfeed.conf. If this file exists, it will be loadedand evaluated as Perl code right after the config file. This enablesyou to provide your own local filter code which will survive an upgradeof the main Cleanfeed source.

It will be reloaded when the filter is reloaded with ctlinnd reloadfilter.perl meow (for INN), or when configuration is reloaded with aSIGHUP (in standalone mode). This means that you can modify the runningcode without restarting Cleanfeed.

cleanfeed.local can define a number of different subroutines, which,if defined, will be called at various points in the filter process.Other subroutines can, of course, be defined as required by your code.

The file is simply re-evaluated each time. So, if you remove a subroutinefrom the file completely, that subroutine will remain defined after thereload, because nothing replaced it. You will need instead to define itas an empty subroutine, or explicitely undef it, to make it go away.

STUFF YOU CAN DEFINE

Cleanfeed will call the following subroutines, if they are defined.See the section on return values for instructions on what your codeshould return.

local_config: This is called after configuration is loaded, each time. It will becalled when the filter is reloaded (with INN) or when configurationis reloaded with SIGHUP (running standalone), as well as when thefilter is first run. No return value is expected.
local_filter_before_emp: Called for each (non-control) article, before any other filters.General-purpose spam filters shouldn't go here, because you reallywant to populate the EMP hashes first.
local_filter_after_emp: Called for each (non-control) article, after the EMP filters butbefore any other filters.
local_filter_middle: Called for each (non-control) article, after the ``simple'' filtersbut before the ``expensive'' body checks.
local_filter_scoring: Called during the scoring filter. Return the value, positive ornegative, by which to adjust the article's score.
Warning: Here there be dragons! If you're going to play withthis please examine the existing source, and use the debuggingroutines to watch what you're doing.
local_filter_last: Called for each (non-control) article, after all other filtersare done.
local_filter_cancel: Called for all cancel control messages.
local_filter_newrmgroup: Called for all newgroup and rmgroup control messages.

RETURN VALUES

The general filtering subroutines you can define (local_filter_before_emp,local_filter_after_emp, local_filter_middle, local_filter_last,local_filter_cancel, and local_filter_newrmgroup) are expected toreturn a value indicating whether you want to accept the article beingexamined. If the article is okay, you should return "" (empty string),in which case filtering will proceed as usual. If you want to reject thearticle, you return any other string, which will be used as the reason.

The rejection code actually expects two return values -- the first stringis the ``verbose'' rejection message, and the second is the ``non-verbose''message (see the verbose configuration option). If only one issupplied, it will be used for both purposes.

The scoring filter calls local_filter_scoring, which is expectedto return the value, postive or negative, by which the article's scoreshould be adjusted.

WHAT YOU GET

Your subroutines get information about the article in several variables.

%hdr

A hash containing the article headers. The key is the header name, in``canonical'' case as INN likes them; the value is the content of the header.When running under INN, only headers known to INN will be included in thehash (which includes any header used anywhere in Cleanfeed). In standalonemode, all headers will be present, but only the known headers will be sentin canonical case; others will have the header name (and thus hash key) inwhatever case they are in the article itself, making them difficult to findand use consistently.

The message body is in this hash under the key __BODY__. If running INN2.x with storageapi, it will be provided in wireformat, with linesterminated in \r\n rather than just \n. With the traditional spoolformat (and in all cases with INN prior to 2.x) lines will be terminatedonly with \n.

Examples:

To get the Subject header as a scalar: $hdr{'Subject'}

To get the entire message body as a scalar: $hdr{'__BODY__'}

%lch

A hash containing lowercased versions of some of the article headers.The hash keys are the header names in all lowercase; the values are thecontents of the headers, with all letters forced to lowercase.

Currently, the only headers added to this hash are From, Organization,Subject, Content-Type, X-Newsreader, X-Newsposter, Message-ID, and Sender.

This hash is not availabe to local_filter_before_emp.

@groups

An array containing the newsgroups the article is posted to (from theNewsgroups header). You can find out how many groups the article iscrossposted to with ``scalar @groups''.

@followups

An array containing the newsgroups to which followups are set (from theFollowup-To header). If the article has no Followup-To header, thisarray will be identical to @groups. You can find out how many groupsfollowups are set to with ``scalar @followups''. This is the preferredway to limit crossposting, because limiting only by the Newsgroupsheader will catch FAQs and such.

$lines

The number of lines in the message body. This is not taken from the Linesheader as that can be client-supplied to fool filtering; this is determinedby counting the lines in the message body.

%gr

A hash containing information about the groups the article is postedto. This isn't very straightforward and may not be useful to you, butI'm including it in this documentation for completeness. The followingentries may be present in this hash:

$gr{'net'} - the number of net.* (Usenet II) newsgroups the article isposted to, if any.

$gr{'other'} - the number of non-net.* groups the article is posted to.

$gr{'md5skip'} - true if the article should be exempted from the MD5body checks (if all newsgroups match the regexp in md5exclude).

$gr{'binary'} - true if the article is posted only to groups wherebinaries are allowed (if all newsgroups match bin_allowed).

$gr{'html'} - true if the article is posted only to groups where htmlis allowed (if all newsgroups match html_allowed).

$gr{'poison'} - number of `poison' newsgroups this article is postedto (matching poison_groups). If this is present, you'll only see thisentry in local_filter_before_emp and local_filter_after_emp becauseit will be rejected after that.

$gr{'abuse'} - number of `net abuse' newsgroups this article is postedto (matching net_abuse_groups).

$gr{'reports'} - number of `spam reports' newsgroups this article isposted to (matching spam_report_groups).

$gr{'low_xpost'} - number of `low crosspost limit' groups this articleis posted to (matching low_xpost_groups).

$gr{'mod'} - number of moderated groups this article is posted to(requires that Cleanfeed have an active file).

$gr{'allmod'} - true if this article is posted only to moderated groups.

$gr{'faq'} - true if this article is crossposted to news.answers.

%config

A hash containing all configuration options.

DEBUGGING

When you make filtering changes, you should always check the results forfalse positives. I've provided two subroutines to help you do this:writeheaders() and writefull().

First, make sure debug_batch_directory is set in your configuration.Set this to a directory that is writable by the news user.

Call either of these subroutines with one argument, the basename of thebatch file you want to write the current article to. writeheaderswill dump the article's headers out to the file (with INN this will onlygive you the known headers). writefull will dump the full article,headers (again, known headers with INN) and body. The file will berotated if it becomes larger than debug_batch_size, set in yourconfiguration. The rotation is simple, a number is appended to the endof the file, and incremented until the filename does not exist. You'llhave to delete the old files yourself.

When testing a new filter, simply call writeheaders ("batchfile") orwritefull ("batchfile") when you're going to reject an article.Then you can look at the file to make sure you're doing what you thinkyou're doing.

SIGNALS

When running under Cyclone, Typhoon, Breeze, or NNTPRelay (standalonemode), Cleanfeed will catch SIGHUP, and reload its configuration fromcleanfeed.conf. It will also reload and reevaluate cleanfeed.localif you're using it. Note that, unlike INN, there is no way to reload thefilter code itself without restarting the server.

Cleanfeed in standalone mode will also catch SIGUSR1 and write its crudecurrent-status file (see statfile in the config section) on the nextcycle through the filter.

(I honestly don't know if SIGUSR1 and SIGHUP are things which exist on NTfor NNTPRelay.)

CREDITS

Written by Jeremy Nixon <jeremyAATTexit109.com>.

Originally based on Jeff Garzik's EMP filter.

I can't possibly mention everyone who has submitted ideas or fixesfor the filter, but I'd like to acknowledge the substantialcontributions of several people: Danhiel Baker, Frank Copeland,Brian Moore, John Payne, Russ Allbery, David Riley, and SeokChan LEE.Thanks, guys.

dynamic-load.patch is from Piers Cawley.The body-filtering portion of the INN filter.patch is from Jeff Garzik.messageid.patch is from Ed Mooring.mode.patch is from John Payne.

COPYRIGHT

LICENSE

This software may be distributed freely, provided it is intact (includingall the files from the original archive). You may modify it, and youmay distribute your modified version, provided the original work iscredited to the appropriate authors, and your work is credited to you(don't make changes and pass them off as my work), and that you aren'tcharging for it.

AVAILABILITY

This filter is available at:

http://www.exit109.com/~jeremy/news/antispam.html ftp://ftp.exit109.com/users/jeremy/

Index

NAME
SYNOPSIS
DESCRIPTION
USAGE
CONFIGURATION OPTIONS
General Settings
MD5 Body Filter Settings
Header-Based EMP Filter Settings
Excessive Crosspost Settings
Misplaced Binaries Filter
HTML
Cancel Message Filtering
Disabling Other Filters
Hash-Trimming
Logging
Timing Info
Debugging
Regular Expressions
INSTALLATION - INN
SYSTEM REQUIREMENTS
PATCHES AND STUFF
INSTALLING CLEANFEED - INN
INSTALLATION - HIGHWIND SERVERS
SYSTEM REQUIREMENTS
INSTALLING CLEANFEED - HIGHWIND
INSTALLATION - NNTPRELAY
SYSTEM REQUIREMENTS
INSTALLING CLEANFEED - NNTPRELAY
THE HACKER'S GUIDE
STUFF YOU CAN DEFINE
RETURN VALUES
WHAT YOU GET
DEBUGGING
SIGNALS
CREDITS
COPYRIGHT
LICENSE
AVAILABILITY

This document was created byman2html,using the manual pages.