SEARCH
NEW RPMS
DIRECTORIES
ABOUT
FAQ
VARIOUS
BLOG
DONATE


YUM REPOSITORY

 
 

SA-LEARN

Section: User Contributed Perl Documentation (1)
Updated: perl v5.6.1
Index 

NAME

sa-learn - train SpamAssassin's Bayesian classifier 

SYNOPSIS

sa-learn [options] --file message

sa-learn [options] --mbox mailbox

sa-learn [options] --dir directory

sa-learn [options] --single < message

Options:

 --ham                             Learn messages as ham (non-spam) --spam                            Learn messages as spam --forget                          Forget a message --rebuild                         Rebuild the database if needed --force-expire                    Force an expiry run, rebuild every time -f file, --folders=file           Read list of files/directories from file --dir                             Learn a directory of RFC 822 files --file                            Learn a file in RFC 822 format --mbox                            Learn a file in mbox format --showdots                        Show progress using dots --no-rebuild                      Skip building databases after scan -L, --local                       Operate locally, no network accesses -C file, --config-file=file       Path to standard configuration dir -p prefs, --prefs-file=file       Set user preferences file -D, --debug-level                 Print debugging messages -V, --version                     Print version -h, --help                        Print usage message
 

DESCRIPTION

Given a typical selection of your incoming mail classified as spam or ham(non-spam), this tool will feed each mail to SpamAssassin, allowing itto 'learn' what signs are likely to mean spam, and which are likely tomean ham.

Simply run this command once for each of your mail folders, and it will''learn'' from the mail therein.

Note that globbing in the mail folder names is supported; in other words,listing a folder name as "*" will scan every folder that matches.

SpamAssassin remembers which mail messages it's learnt already, and will notre-learn those messages again, unless you use the ---forget option.

If you make a mistake and scan a mail as ham when it is spam, or viceversa, simply rerun this command with the correct classification, and themistake will be corrected. SpamAssassin will automatically 'forget' theprevious indications. 

INTRODUCTION TO BAYESIAN FILTERING

(Thanks to Michael Bell for this section!)

For a more lengthy description of how this works, go tohttp://www.paulgraham.com and see ``A Plan for Spam''. It's reasonablyreadable, even if statistics make me break out in hives.

The short semi-inaccurate version: Given training, a spam heuristics enginecan take the most ``spammy'' and ``hammy'' words and apply probablisticanalysis. Furthermore, once given a basis for the analysis, the engine cancontinue to learn iteratively by applying both it's non-Bayesian and Bayesianruleset together to create evolving ``intelligence''.

SpamAssassin 2.50 supports Bayesian spam analysis, in the form of theBAYES rules. This is a new feature, quite powerful, and is disableduntil enough messages have been learnt.

The pros of Bayesian spam analysis:

Can greatly reduce false positives and false negatives.
It learns from your mail, so it's tailored to your unique e-mail flow.
Once it starts learning, it can continue to learn from SpamAssassin and improve over time.

And the cons:

A decent number of messages are required before results are useful for ham/spam determination.

It's hard to explain why a message is or isn't marked as spam.
i.e.: a straightforward rule, that matches, say, ``VIAGRA'' iseasy to understand. If it generates a false positive or false negative,it's fairly easy to understand why.

With Bayesian analysis, it's all probabilities - ``because the past saysit's likely as this falls into a probablistic distribution common to pastspam in your systems''. Tell that to your users! Tell that to the clientwhen he asks ``what can I do to change this''. (By the way, the answer inthis case is ``use whitelisting''.)

It will take disk space and memory.
The databases it maintains take quite a lot of resources to store and use.
 

GETTING STARTED

Still interested? Ok, here's the guidelines for getting this working.

First a high-level overview:

Build a significant sample of both ham and spam.
I suggest several thousand of each, placed in SPAM and HAM directories ormailboxes. Yes, you MUST hand-sort this - otherwise the results won't be muchbetter than SpamAssassin on its own. Verify the spamminess/haminess ofEVERY message. You're urged to avoid using a publicly available corpus (sample) -this must be taken from YOUR mail server, if it's to be statistically useful.Otherwise, the results may be pretty skewed.
Use this tool to teach SpamAssassin about these samples, like so:
        sa-learn --spam /path/to/spam/folder        sa-learn --ham /path/to/ham/folder        ...
Let SpamAssassin proceed, learning stuff. When it finds ham and spamit will add the ``interesting tokens'' to the database.
If you need SpamAssassin to forget about specific messages, use the ---forget option.
This can be applied to either ham or spam that has run through thesa-learn processes. It's a bit of a hammer, really, lowering theweighting of the specific tokens in that message (only if that message hasbeen processed before).
Learning from single messages uses a command like this:
        cat mailmessage | sa-learn --ham --no-rebuild --single
This is handy for binding to a key in your mail user agent. It's very fast, asall the time-consuming stuff is deferred until you run with the "--rebuild"option.
Autolearning is enabled by default
If you don't have a corpus of mail saved to learn, you can letSpamAssassin automatically learn the mail that you receive. If you areautolearning from scratch, the amount of mail you receive will determinehow long until the BAYES_* rules are activated.
 

EFFECTIVE TRAINING

Learning filters require training to be effective. If you don't trainthem, they won't work. In addition, you need to train them with newmessages regularly to keep them up-to-date, or their data will becomestale and impact accuracy.

You need to train with both spam and ham mails. One type of mailalone will not have any effect.

Note that if your mail folders contain things like forwarded spam,discussions of spam-catching rules, etc., this will cause trouble. Youshould avoid scanning those messages if possible. (An easy way to do thisis to move them aside, into a folder which is not scanned.)

Another thing to be aware of, is that typically you should aim to trainwith at least 1000 messages of spam, and 1000 ham messages, ifpossible. More is better, but anything over about 5000 messages does notimprove accuracy significantly in our tests.

Be careful that you train from the same source --- for example, if you trainon old spam, but new ham mail, then the classifier will think thata mail with an old date stamp is likely to be spam.

It's also worth noting that training with a very small quantity ofham, will produce atrocious results. You should aim to train with atleast the same amount (or more if possible!) of ham data than spam.

On an on-going basis, it's best to keep training the filter to makesure it has fresh data to work from. There are various ways to dothis:

1. Supervised learning
This means keeping a copy of all or most of your mail, separated into spamand ham piles, and periodically re-training using those. It producesthe best results, but requires more work from you, the user.

(An easy way to do this, by the way, is to create a new folder for'deleted' messages, and instead of deleting them from other folders,simply move them in there instead. Then keep all spam in a separatefolder and never delete it. As long as you remember to move misclassifiedmails into the correct folder set, it's easy enough to keep up to date.)

2. Unsupervised learning from Bayesian classification
Another way to train is to chain the results of the Bayesian classifierback into the training, so it reinforces its own decisions. This is onlysafe if you then retrain it based on any errors you discover.

SpamAssassin does not support this method, due to experimental resultswhich strongly indicate that it does not work well, and since Bayes isonly one part of the resulting score presented to the user (while Bayesmay have made the wrong decision about a mail, it may have been overriddenby another system).

3. Unsupervised learning from SpamAssassin rules
Also called 'auto-learning' in SpamAssassin. Based on statisticalanalysis of the SpamAssassin success rates, we can automatically train theBayesian database with a certain degree of confidence that our trainingdata is accurate.

It should be supplemented with some supervised training in addition, ifpossible.

This is the default, but can be turned off by setting the SpamAssassinconfiguration parameter "auto_learn" to 0.

4. Mistake-based training
This means training on a small number of mails, then only training onmessages that SpamAssassin classifies incorrectly. This works, but ittakes longer to get it right than a full training session would.
 

OPTIONS


---ham
Learn the input message(s) as ham.
---spam
Learn the input message(s) as spam.
---rebuild
Rebuild the databases, typically done after learning with ---no-rebuild,or if you wish to periodically clean the Bayes databases once a day ona busy server.
---force-expire
Forces an expiry run, regardless of whether it may be necessary or not.
---forget
Forget a given message previously learnt.
-h, ---help
Print help message and exit.
-C config, ---config-file=config
Read configuration from config.
-p prefs, ---prefs-file=prefs
Read user score preferences from prefs.
-D, ---debug-level
Produce diagnostic output.
---no-rebuild
Skip the slow rebuilding step which normally takes place after changingdatabase entries. If you plan to scan many folders in a batch, it is faster touse this switch and run "sa-learn --rebuild" once all the folders have beenscanned.
-L, ---local
Do not perform any network accesses while learning details about the mailmessages. This will speed up the learning process, but may result in aslightly lower accuracy.

Note that this is currently ignored, as current versions of SpamAssassin willnot perform network access while learning; but future versions may.

 

FILES

sa-learn and the other parts of SpamAssassin's Bayesian learner,use a set of persistent database files to store the learnt tokens, as follows.
bayes_toks
The database of tokens, containing the tokens learnt, their count ofoccurrences in ham and spam, and the message count of the last messagethey were seen in.

This database also contains some 'magic' tokens, as follows: the number of hamand spam messages learnt, the number of tokens in the database, themessage-count of the last expiry run, the message-count of the oldest token inthe database, and the message-count of the current message (to the nearest5000).

This is a database file, using the first one of the following database modulesthat SpamAssassin can find in your perl installation: "DB_File", "GDBM_File","NDBM_File", or "SDBM_File".

bayes_seen
A map of message-ID to what that message was learnt as. This is usedso that SpamAssassin can avoid re-learning a message it's already seen,and so it can reverse the training if you later decide that messagewas previously learnt incorrectly.

This is a database file, using the first one of the following database modulesthat SpamAssassin can find in your perl installation: "DB_File", "GDBM_File","NDBM_File", or "SDBM_File".

bayes_journal
While SpamAssassin is scanning mails, it needs to track which tokens it uses inits calculations. So that many processes can read the databasessimultaneously, but only one can write at a time, this uses a 'journal' file.

When you run "sa-learn --rebuild", the journal is read, and the tokens thatwere accessed during the journal's lifetime will have their last-access timeupdated in the "bayes_toks" database.

bayes_msgcount
Every time SpamAssassin accesses a mail message for scanning, or every timethe "sa-learn" command is run, the 'message count' is increased by one.This is used to control expiration of old tokens.

Since many processes may be running simultaneously, SpamAssassin does notuse a locked database file for this operation; instead, it uses the sizeof this file as a counter, appending one byte for each message. Once ithits 5000 bytes, the "bayes_toks" database is locked, and the messagecounter entry in that database is increased accordingly.

 

EXPIRATION

Since SpamAssassin auto-learns, the Bayes database files could increaseperpetually until they fill your disk or you run out of memory. To controlthis, SpamAssassin performs expiration.

Every "bayes_expiry_scan_count" / 2 messages, or when "sa-learn --rebuild--force-expire" is run, SpamAssassin will attempt an expiry run, as follows.

SpamAssassin runs through every token in the database. If that token has notbeen used during the scanning of the last "bayes_expiry_scan_count" messages,it is marked for deletion.

Next, if that operation would bring the number of tokens below the"bayes_expiry_min_db_size" threshold, it removes tokens from the for-deletionlist until the resulting database would contain "bayes_expiry_min_db_size"token entries.

It then removes the listed tokens and updates the 'last expiry' setting.

The SpamAssassin configuration settings which control this operation are:

bayes_expiry_min_db_size is part of the SpamAssassin configuration. The default value is 100000, which is roughly equivalent to a 5Mb database file if you're using DB_File.

bayes_expiry_min_db_size is part of the SpamAssassin configuration. The default value is 100000, which is roughly equivalent to a 5Mb database file if you're using DB_File.

bayes_expiry_scan_count is also part of the SpamAssassin configuration. The default value is 5000.

bayes_expiry_scan_count is also part of the SpamAssassin configuration. The default value is 5000.
 

INSTALLATION

The sa-learn command is part of the Mail::SpamAssassin Perl module.Install this as a normal Perl module, using "perl -MCPAN -e shell",or by hand. 

ENVIRONMENT

No environment variables, aside from those used by perl, are required tobe set. 

SEE ALSO

Mail::SpamAssassin(3)spamassassin(1)

http://www.paulgraham.com/ , Paul Graham's ``A Plan For Spam'' paper

http://radio.weblogs.com/0101454/stories/2002/09/16/spamDetection.html , GaryRobinson's f(x) and combining algorithms

http://www.bgl.nu/~glouis/bogofilter/test6000.html , discussion of variousBayes training regimes, including 'train on error' and unsupervised training 

AUTHOR

Justin Mason <jm /at/ jmason.org>


 

Index

NAME
SYNOPSIS
DESCRIPTION
INTRODUCTION TO BAYESIAN FILTERING
GETTING STARTED
EFFECTIVE TRAINING
OPTIONS
FILES
EXPIRATION
INSTALLATION
ENVIRONMENT
SEE ALSO
AUTHOR

This document was created byman2html,using the manual pages.