SEARCH
NEW RPMS
DIRECTORIES
ABOUT
FAQ
VARIOUS
BLOG
DONATE


YUM REPOSITORY

 
 

MAN page from CentOS Other silk-rwflowappend-3.19.1-3.el8.x86_64.rpm

rwflowappend

Section: SiLK Tool Suite (8)
Updated: 2021-01-04
Index 

NAME

rwflowappend - Append incremental SiLK Flow files to hourly files 

SYNOPSIS

  rwflowappend --incoming-directory=DIR_PATH --root-directory=DIR_PATH        --error-directory=DIR_PATH [--archive-directory=DIR_PATH]        [--flat-archive] [--post-command=COMMAND]        [--hour-file-command=COMMAND] [--threads=N]        [--reject-hours-past=NUM] [--reject-hours-future=NUM]        [--no-file-locking] [--polling-interval=NUM]        [--byte-order=ENDIAN] [--pad-header]        [--compression-method=COMP_METHOD]        [--site-config-file=FILENAME]        { --log-destination=DESTINATION          | --log-pathname=FILE_PATH          | --log-directory=DIR_PATH [--log-basename=LOG_BASENAME]            [--log-post-rotate=COMMAND] }        [--log-level=LEVEL] [--log-sysfacility=NUMBER]        [--pidfile=FILE_PATH] [--no-chdir] [--no-daemon]  rwflowappend --help  rwflowappend --version
 

DESCRIPTION

rwflowappend is a daemon that watches a directory for files thatcontain small numbers of SiLK Flow records---these files are calledincremental files---as generated by rwflowpack(8) when it is runwith --output-mode=incremental-files or --output-mode=sending.rwflowappend appends these SiLK Flow records to the hourly filesstored in the SiLK data repository whose directory tree root isspecified by the --root-directory switch.

The directory that rwflowappend watches for incremental files isspecified by --incoming-directory. As rwflowappend scans thisdirectory, it ignores a file if its size is 0 bytes or if its namebegins with a dot ("."). On each scan, if rwflowappend detects afile name that was not present in the previous scan, it records thename and size of the file. If the file has a different size on thenext scan, the new size is recorded. Once the file has the same sizeon two consecutive scans, rwflowappend appends the file to theappropriate hourly file.

After rwflowappend processes an incremental file, the file isdeleted unless the --archive-directory switch is specified, inwhich case the incremental file is moved to that directory or to asubdirectory of that directory depending on whether --flat-archivewas specified. The --post-command switch allows a command to beexecuted on the incremental file after it has been moved to thearchive directory.

If a fatal write error occurs (for example, the disk containing thedata repository becomes full), rwflowappend exits. Before exiting,rwflowappend attempts to truncate the hourly file to the size ithad when it was opened, and rwflowappend moves the incremental fileit was reading to the directory specified by --error-directory.

Running rwflowappend separately from rwflowpack is used whenyou wish to copy the packed SiLK Flow records from the machine doingthe packing to multiple machines for use by analysts. Almost anynetwork file transport protocol may be used to move the files from thepacking machine to the destination machine where rwflowappend isrunning, though we have written the rwsender(8) andrwreceiver(8) to perform this task.

Separate rwflowpack and rwflowappend processes are alsorecommended if you want another process (such as the Analysis Pipeline<http://tools.netsa.cert.org/analysis-pipeline/>) to process the SiLKFlow records as they are generated. 

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is anexact match for an option. A parameter to an option may be specifiedas --arg=param or --arg param, though the first form isrequired for options that take optional parameters. 

General Configuration

The following switches are required:
--incoming-directory=DIR_PATH
Periodically scan the directory DIR_PATH for incremental files toappend to the hourly files. As rwflowappend scans DIR_PATH, itignores a file if its name begins with a dot (".") or if its size is0 bytes. When a file is first detected, its size is recorded, and thefile must have the same size for two consecutive scans beforerwflowappend will append it to the appropriate hourly file. Theinterval between scans is set by --polling-interval. DIR_PATHmust be a complete directory path.
--root-directory=DIR_PATH
Append to existing hourly files and create new hourly files in thedirectory tree rooted at this location. The directory tree has thesame subdirectory structure as that created by rwflowpack.DIR_PATH must be a complete directory path.
--error-directory=DIR_PATH
Store in this directory incremental files that were NOT successfullyappended to an hourly file. DIR_PATH must be a complete directorypath.

The following switches are optional:

--archive-directory=DIR_PATH
Move each incremental file to DIR_PATH or a subdirectory of itafter rwflowappend has successfully appended the incremental fileto an hourly file. If this switch is not provided, the incrementalfiles are deleted once they are successfully appended to an hourlyfile. When the --flat-archive switch is also provided, incrementalfiles are moved into the top of DIR_PATH; when --flat-archive isnot given, each incremental file is moved to a subdirectory ofDIR_PATH that mirrors the path of the hourly file to which theincremental file was appended. Removing files from thearchive-directory is not the job of rwflowappend; the systemadministrator should implement a separate process to clean thisdirectory. This switch is required when the --post-command switchis present.
--flat-archive
When archiving incremental files via --archive-directory, move thefiles into the top of the archive-directory, not into subdirectoriesof it. This switch has no effect if --archive-directory is notalso specified. This switch may be used to allow another process towatch for new files appearing in the archive-directory.
--post-command=COMMAND
Run COMMAND on each incremental file after rwflowappend hassuccessfully appended it to an hourly file and moved it into thearchive-directory. Each occurrence of the string %s in COMMANDis replaced with the full path to the incremental file in thearchive-directory, and each occurrence of "%%" is replaced with "%".If any other character follows "%", rwflowappend exits with anerror. When using this feature, the --archive-directory must bespecified. The exit status of COMMAND is ignored. See also therwpollexec(8) daemon.
--hour-file-command=COMMAND
Run COMMAND upon creation of a new hourly file. The string %sin COMMAND is replaced with the full path to the hourly file, andthe string "%%" is replaced with "%". If any other characterfollows "%", rwflowappend exits with an error. The exit status ofCOMMAND is ignored.
--threads=N
Invoke rwflowappend with N threads reading the incremental filesand writing to the repository. When this switch is not provided,rwflowappend runs with a single thread. Since SiLK 3.8.2.
--reject-hours-past=NUM
Reject incremental files containing records whose starting hour occursmore than this number of hours in the past relative to the currenthour. Incremental files that violate this value are moved into theerror directory. Times are compared using the starting hour of theflow record and the current hour. For example, flow records thatstart at 18:02:56 and 18:58:04 are considered 1 hour in the pastwhether the current time is 19:01:47 or 19:59:33. When performinglive data collection, it is not uncommon to get flows one to two hoursin the past due to the flow generator's active timeout (often 30minutes) and the time to transfer the flow records through thecollection system. The default is to accept all incremental files.
--reject-hours-future=NUM
Similar to --reject-hours-past, but reject incremental filescontaining records whose starting hour occurs more than this number ofhours in the future relative to the current hour. Future dated flowrecords are rare, but can occur due to time drift at the sensor. Thedefault is to accept all incremental files.
--no-file-locking
Do not use advisory write locks. Normally, rwflowappend obtains awrite lock on an hourly file prior to writing records to it. Thewrite lock prevents two instances of rwflowappend from writing tothe same hourly file simultaneously. However, attempting to use awrite lock on some file systems causes rwflowappend to exit with anerror, and this switch can be use when writing data to these filesystems.
--polling-interval=NUM
Check the incoming directory for new incremental files every NUMseconds. The default polling interval is 15 seconds.
--byte-order=ENDIAN
Set the byte order for newly created SiLK Flow files. When appendingrecords to an existing file, the byte order of the file is maintained.The argument is one of the following:
as-is
Maintain the byte order of the incremental files (i.e., the byte orderspecified to rwflowpack). This is the default.
native
Use the byte order of the machine where rwflowappend is running.
big
Use network byte order (big endian) for the flow files.
little
Write the flow files in little endian format.
--compression-method=COMP_METHOD
Specify the compression library to use when creating new hourly files.When this switchis not given, newly created hourly files maintain the compressionmethod used by the incremental file (i.e., the compression methodspecified to rwflowpack). When appending to an existing hourlyfile, the compression method of the file is maintained. The validvalues for COMP_METHOD are determined by which external librarieswere found when SiLK was compiled. To see the available compressionmethods and the default method. use the --help or --versionswitch. SiLK can support the following COMP_METHOD values when therequired libraries are available.
none
Do not compress the output using an external library.
zlib
Use the zlib(3) library for compressing the output. Using zlibproduces the smallest output files at the cost of speed.
lzo1x
Use the lzo1x algorithm from the LZO real time compression libraryfor compression. This compression provides good compression with lessmemory and CPU overhead.
snappy
Use the snappy library for compression, and always compress theoutput regardless of the destination. This compression provides goodcompression with less memory and CPU overhead. Since SiLK 3.13.0.
best
Use lzo1x if available, otherwise use snappy if available, otherwiseuse zlib if available.
--site-config-file=FILENAME
Read the SiLK site configuration from the named file FILENAME.When this switch is not provided, rwflowappend searches for thesite configuration file in the locations specified in the ``FILES''section.
 

Logging and Daemon Configuration

One of the following mutually-exclusive switches is required:
--log-destination=DESTINATION
Specify the destination where logging messages are written. WhenDESTINATION begins with a slash "/", it is treated as a filesystem path and all log messages are written to that file; there is nolog rotation. When DESTINATION does not begin with "/", it mustbe one of the following strings:
none
Messages are not written anywhere.
stdout
Messages are written to the standard output.
stderr
Messages are written to the standard error.
syslog
Messages are written using the syslog(3) facility.
both
Messages are written to the syslog facility and to the standard error(this option is not available on all platforms).
--log-directory=DIR_PATH
Use DIR_PATH as the directory where the log files are written.DIR_PATH must be a complete directory path. The log files have theform

 DIR_PATH/LOG_BASENAME-YYYYMMDD.log

where YYYYMMDD is the current date and LOG_BASENAME is theapplication name or the value passed to the --log-basename switchwhen provided. The log files are rotated: At midnight local time, anew log is opened, the previous file is closed, and the commandspecified by --log-post-rotate is invoked on the previous day's logfile. (Old log files are not removed by rwflowappend; theadministrator should use another tool to remove them.) When thisswitch is provided, a process-ID file (PID) is also written in thisdirectory unless the --pidfile switch is provided.

--log-pathname=FILE_PATH
Use FILE_PATH as the complete path to the log file. The log fileis not rotated.

The following set of switches is optional:

--log-level=LEVEL
Set the severity of messages that are logged. The levels from mostsevere to least are: "emerg", "alert", "crit", "err", "warning","notice", "info", "debug". The default is "info".
--log-sysfacility=NUMBER
Set the facility that syslog(3) uses for logging messages. Thisswitch takes a number as an argument. The default is a value thatcorresponds to "LOG_USER" on the system where rwflowappend isrunning. This switch produces an error unless--log-destination=syslog is specified.
--log-basename=LOG_BASENAME
Use LOG_BASENAME in place of the application name in the name oflog files in the log directory. See the description of the--log-directory switch. This switch does not affect the name ofthe process-ID file.
--log-post-rotate=COMMAND
Run COMMAND on the previous day's log file after log rotation.When this switch is not specified, the previous day's log file iscompressed with gzip(1). When the switch is specified andCOMMAND is the empty string, no action is taken on the log file.Each occurrence of the string %s in COMMAND is replaced with thefull path to the log file, and each occurrence of "%%" is replacedwith "%". If any other character follows "%", rwflowappend exitswith an error. Specifying this switch without also using--log-directory is an error.
--pidfile=FILE_PATH
Set the complete path to the file in which rwflowappend writes itsprocess ID (PID) when it is running as a daemon. No PID file iswritten when --no-daemon is given. When this switch is notpresent, no PID file is written unless the --log-directory switchis specified, in which case the PID is written toLOGPATH/rwflowappend.pid.
--no-chdir
Do not change directory to the root directory. When rwflowappendbecomes a daemon process, it changes its current directory to the rootdirectory so as to avoid potentially running on a mounted file system.Specifying --no-chdir prevents this behavior, which may be usefulduring debugging. The application does not change its directory when--no-daemon is given.
--no-daemon
Force rwflowappend to run in the foreground---it does not become adaemon process. This may be useful during debugging.
--help
Print the available options and exit.
--version
Print the version number and information about how SiLK wasconfigured, then exit the application.
 

ENVIRONMENT

SILK_CONFIG_FILE
This environment variable is used as the value for the--site-config-file when that switch is not provided.
SILK_PATH
This environment variable gives the root of the install tree. Whensearching for configuration files, rwflowappend may use thisenvironment variable. See the ``FILES'' section for details.
 

FILES

${SILK_CONFIG_FILE}
ROOT_DIRECTORY/silk.conf
${SILK_PATH}/share/silk/silk.conf
${SILK_PATH}/share/silk.conf
/usr/share/silk/silk.conf
/usr/share/silk.conf
Possible locations for the SiLK site configuration file which arechecked when the --site-config-file switch is not provided, whereROOT_DIRECTORY/ is the directory specified to the--root-directory switch.
 

SEE ALSO

rwflowpack(8), rwreceiver(8), rwsender(8), rwpollexec(8),rwfilter(1), silk(7), gzip(1), syslog(3), zlib(3),The SiLK Installation Handbook 

NOTES

rwflowappend does not check the integrity of an hourly file beforeappending records to it.

Prior to SiLK 3.6.0 when a write error occurred, rwflowappend couldleave a partially written record or compressed block in the hourlyfile. If a partially written compressed block remained and additionalcompressed blocks were appended, these compressed blocks could not beread by other SiLK tools. If a partially written record remained andadditional records were appended, SiLK tools would read the unaligneddata as if it were aligned and produce garbage records. AlthoughSiLK 3.6.0 works around the issue on write errors, similar issues canoccur if rwflowappend is suddenly killed (e.g., by "kill -9").

When a write error occurs, rwflowappend may leave a zero byte filein the data repository. Such files do affect the exit status ofrwfilter(1), though rwfilter warns about being unable to readthe header from the file.

As of SiLK 3.1.0, rwflowappend obtains an advisory write lock onthe hourly file it is writing, allowing multiple rwflowappendprocesses to write to the same hourly file. File locking may bedisabled by using the --no-file-locking switch. If this switch isenabled, the administrator must ensure that multiple rwflowappendprocesses do not attempt to write to the same hourly filesimultaneously.


 

Index

NAME
SYNOPSIS
DESCRIPTION
OPTIONS
General Configuration
Logging and Daemon Configuration
ENVIRONMENT
FILES
SEE ALSO
NOTES

This document was created byman2html,using the manual pages.