SEARCH
NEW RPMS
DIRECTORIES
ABOUT
FAQ
VARIOUS
BLOG
DONATE


YUM REPOSITORY

 
 

MAN page from CentOS Other silk-common-3.19.1-3.el8.x86_64.rpm

sensor.conf

Section: SiLK Tool Suite (5)
Updated: 2021-01-04
Index 

NAME

sensor.conf - Sensor Configuration file for rwflowpack and flowcap 

DESCRIPTION

As part of collecting flow data, the rwflowpack(8) andflowcap(8) daemons need to know what type of data they arecollecting and how to collect it (e.g., listen on 10000/udp forNetFlow v5; listen on 4740/tcp for IPFIX). In addition, therwflowpack daemon needs information on how to categorize the flow:for example, to label the flows collected at a border router asincoming or outgoing. The Sensor Configuration file, sensor.conf,contains this information, and this manual page describes the syntaxof the file (see ``SYNTAX'' below) and provides some exampleconfigurations (see ``EXAMPLES'').

The sensor.conf file may have any name, and it may reside in anylocation. The name and location of the file is specified by the--sensor-configuration switch to rwflowpack and flowcap.

The Sensor Configuration File defines the following concepts:

probe
A probe specifies a source for flow data. The source could be a porton which flowcap or rwflowpack collects NetFlow or IPFIX datafrom a flow generator such as a router or the yaf software(<http://tools.netsa.cert.org/yaf/>). In rwflowpack, the sourcecan be a directory to periodically poll for files containing NetFlowv5 PDUs, IPFIX records, or SiLK Flow records. When defining a probe,you must specify a unique name for the probe and the probe's type.
group
A group is a named list that contains one of the following: CIDRblocks, the names of IPset files, or integers representing SNMPinterfaces or VLAN identifiers. The use of groups is optional; theprimary purpose of a group is to allow the administrator to specify acommonly used list (such as the IP space of the network beingmonitored) in a single location.
sensor
A sensor represents a logical collection point for the purposes ofanalysis. The sensor contains configuration values that rwflowpackuses to categorize each flow record depending on how the record movesbetween networks at the collection point. Since the sensors and thecategories (known as flowtypes or as class/type pairs) arealso used for analysis, they are defined in the Site Configurationfile, described in silk.conf(5). The Sensor Configuration filemaps sensors to probes and specifies the rules required to categorizethe data. Usually one sensor corresponds to one probe; however, asensor may be comprised of multiple probes, or the flow data collectedat a single probe may be handled by multiple sensors.

The next section of this manual page describes the syntax of thesensor.conf file.

Using the syntax to configure a sensor requires knowledge of thepacking logic that rwflowpack is using. The packing logic isthe set of rules that rwflowpack uses to assign a flowtype to eachrecord it processes. The default packing logic is for the twowaysite, which is described in the packlogic-twoway(3) manual page.Additional packing logic rules are available (e.g.,packlogic-generic(3)).

The last major section of this document is ``EXAMPLES'' where severalcommon configurations are shown. These examples assume rwflowpackis using the packing logic from the twoway site. 

SYNTAX

When parsing the Sensor Configuration file, blank lines are ignored.At any location in a line, the character "#" indicates the beginningof a comment, which continues to the end of the line. These commentsare ignored.

All other lines begin with optional leading whitespace, a commandname, and one or more arguments to the command. Command names are asequence of non-whitespace characters, not including the character"#". Arguments are textual atoms: any sequence of non-whitespace,non-"#" characters, including numerals and punctuation.

There are four contexts for commands: top-level, probe block, groupblock, and sensor block. The probe block, group block, and sensorblock contexts are used to describe individual features of probes,groups, and sensors, respectively.

The valid commands for each context are described below. 

Top-Level Commands

In addition to the commands to begin a probe, group, or sensor block,the top-level context supports the following command:
include path
The include command is used to include the contents of another filewhose location is path. This may be used to separate largeconfigurations into logical units. The argument to include must bea double-quoted string.
 

Probe Block

With the exception of the probe command, the commands listed beloware accepted within the probe context. Within a probe block, one andonly one of the following must be specified: listen-on-port tolisten on a network socket, poll-directory to poll a directory forfiles, read-from-file to read a single file, orlisten-on-unix-socket to listen on a UNIX domain socket. Thesecommands are described below.
probe probe-name probe-type
The probe command is used in the top-level context to begin a newprobe block which continues to the end probe command. Thearguments to the probe command are the name of the probe beingdefined and the probe type. The probe-name must be unique amongall probes. It must begin with a letter, and it may not containwhitespace characters or the slash character ("/"). When a probe isassociated with a single sensor, it is good practice to give the probethe same name as the sensor. The probe-type must be one of thefollowing:
netflow-v5
This probe processes NetFlow v5 protocol data units (PDU) that thedaemon reads from a UDP port or from a file. NetFlow may be generatedby a router or by software that reads packet capture (pcap(3)) dataand generates NetFlow v5 records.
netflow
This is an alias for netflow-v5 for backwards compatibility. Thisalias is deprecated, and it may be removed in a future release.
ipfix
An IPFIX probe processes Internal Protocol Flow Information eXchangerecords that the daemon reads over the network from an IPFIX sourcesuch as yaf(1). An IPFIX probe can also poll a directory for filesgenerated by the yaf program. To support IPFIX probes, SiLK mustbe built with support for the external library libfixbuf, version1.7.0 or later. Both yafand libfixbuf are available from <http://tools.netsa.cert.org/>.
netflow-v9
This probe processes NetFlow v9 protocol data units (PDU) that thedaemon reads from a UDP port from a router. To support NetFlow v9probes, SiLK must be built with support for the external librarylibfixbuf, version 1.7.0 or later.
sflow
This probe processes sFlow v5 records that the daemon reads from a UDPport. To support sFlow probes, SiLK must be built with support forthe external library libfixbuf, version 1.7.0 or later. Since SiLK3.9.0.
silk
A SiLK probe processes the records contained in SiLK Flow filescreated by previous invocations of rwflowpack. The flows will becompletely re-packed, as if they were just received over the network.The sensor and flowtype values in each flow will be ignored. Notethat SiLK usually removes the SNMP interfaces from its flow records,and it is likely that you will be unable to use the SNMP interfaces topack the flows.
end probe
The end probe command ends the definition of a probe. Following anend probe command, top-level commands are again accepted.
listen-on-port port
This command configures the probe to accept flow records over thenetwork, and port specifies the network port number where the probeshould listen for flow data. The protocol command is required whenlisten-on-port is specified, and the listen-as-host andaccept-from-host commands are optional. Multiple probes may usethe same value for port as long as the probes are the same type andthe accept-from-host command is specified in each probe block.Probes of different types may not bind to the same port, meaning thecombination of the following three values must be different:listen-on-port, protocol, and listen-as-host. When listeningto IPFIX data from yaf, this is the value specified to yaf's--ipfix-port switch. When listening to NetFlow from a Ciscorouter, this is the port that was specified to the Cisco IOScommand

 ip flow-export [ip-address] [port]
protocol { tcp | udp }
This command, required when listen-on-port is given, specifieswhether the port is a "tcp" or "udp" port. IPFIX probes supportboth types; the only permitted value for all other probe types is"udp". When listening to IPFIX data from yaf, this is the valuespecified to yaf's --ipfix switch.
accept-from-host host-name [host-name...]
This optional command specifies the hosts that are allowed to connectto the port where the probe is listening. The argument is a list ofIP addresses and/or hostnames separated by whitespace and/or a comma.When this command is not present, any host may connect. The commandmay only be specified when the listen-on-port command is alsopresent. When multiple probes use the same listen-on-port,protocol, and listen-as-host values, the accept-from-hostswitch must be used so that rwflowpack may assign incoming recordsto a specified probe. When listening for NetFlow, this attributewould be the IP address of the router as seen from the machine runningrwflowpack or flowcap. (Prior to SiLK 3.10.1, theaccept-from-host command accepted only a single argument.)
listen-as-host host-name
This optional command is used on a multi-homed machine to specify theaddress the probe should listen on (bind(2) to). Its value is thename of the host or its IP address. If not present, the program willlisten on all the machine's addresses. The command may only bespecified when the listen-on-port command is also present. Forlistening to NetFlow, the value would be the ip-address that wasspecified to the Cisco IOS command

 ip flow-export [ip-address] [port]
listen-on-unix-socket path-to-unix-socket
The value contains the path name to a UNIX domain socket where theflow generator writes its data. The parent directory ofpath-to-unix-socket must exist. Multiple probes may not use thesame path-to-unix-socket.
poll-directory directory-path
When this command is given, rwflowpack will periodically poll thedirectory-path to look for files to process. flowcap will exitwith an error if you attempt to use probes that contain this commandsince flowcap does not support reading data from files. Multipleprobes may not use the same directory-path. When polling thedirectory, zero length files and files whose name begin with a dot(".") are ignored. This command may be used with the following probetypes:
*
For SiLK probes, each file must be a valid SiLK Flow file.
*
IPFIX probes can process files created by the yaf program.
*
A NetFlow v5 probe will process files containing NetFlow v5 PDUs. Theformat of these files is specified in the description of theread-from-file command.
read-from-file dummy-value
When this command is given, rwflowpack will read NetFlow v5 recordsfrom the file specified by the --netflow-file command line switch.The value to the read-from-file command is completely ignored, andwe recommend you use "/dev/null" as the value. flowcap will exitwith an error if you attempt to use probes that contain this commandsince flowcap does not support reading data from files. The formatof a NetFlow v5 file is that the file's length should be an integermultiple of 1464 bytes, where 1464 is the maximum length of theNetFlow v5 PDU. Each 1464 block should contain the 24-byte NetFlow v5header and space for thirty 48-byte flow records, even if fewerNetFlow records are valid. rwflowpack will accept NetFlow v5 filesthat have been compressed with the gzip(1) program.
log-flags { none | { all | bad | default | firewall-event | missing | record-timestamps | sampling | show-templates | ... } }
This optional command accepts a comma- and/or space-separated list ofnames that specify which messages to log for this probe. If notspecified, the default is default, which is equivalent to bad,missing, sampling. The possible values are:
all
Log everything.
bad
Write messages about an individual NetFlow v5 record where the packetor octet count is zero, the packet count is larger than the octetcount, or the duration of the flow is larger than 45 days.
default
Enable the following values: bad, missing, sampling. This isthe default value. Since SiLK 3.10.0. (Prior to SiLK 3.10.0,all was the default.)
firewall-event
When the "firewall-event" quirks flag is set and the probe isprocessing NetFlow v9 or IPFIX, write messages about records that areignored because the firewall event information element on the recordis something other than flow deleted or flow denied. Since SiLK3.8.1.
missing
Examine the sequence numbers of NetFlow v5 packets and write messagesabout missing and out-of-sequence packets. (You may suppress messagesregarding out-of-sequence NetFlow v9 or IPFIX packets for allprobes by setting the SILK_LIBFIXBUF_SUPPRESS_WARNINGS environmentvariable.)
none
Log nothing. It is an error to combine this value with any other.
record-timestamps
Log the timestamps that appear on each record. This produces a lot ofoutput, and it is primarily used for debugging. Since SiLK 3.10.0.
sampling
Write messages constructed by parsing the NetFlow v9 Options Templatesthat specify the sampling algorithm (when samplingAlgorithm andsamplingInterval IEs are present) or flow sampler mode (whenflowSamplerMode and flowSamplerRandomInterval IEs are present).Since SiLK 3.8.0.
show-templates
Write messages to the log file describing each IPFIX template that isread by this file-base or TCP probe. (UDP probes must still rely onthe SILK_IPFIX_PRINT_TEMPLATES environment variable.) The messagecontains embedded new lines, with the template ID and domain on thefirst line, and each of the template's elements on the followinglines. Each element is described by its name, its IE number with theprivate enterprise number if any, and its length in the template.Scope elements in options templates are marked. The format is thatdescribed in Section 10.2 ofRFC7013 <https://tools.ietf.org/html/rfc7013>. Since SiLK 3.19.0.
interface-values { snmp | vlan }
This optional command specifies the values that should be stored inthe "input" and "output" fields of the SiLK Flow records that areread from the probe. If this command is not given, the default issnmp. Note that NetFlow v5 probes only support snmp.
snmp
Store the index of the network interface card (ifIndex) where theflows entered and left the router, respectively.
vlan
Store the VLAN identifier for the source and destination networks,respectively. If only one VLAN ID is available, "input" is set tothat value and "output" is set to 0.

This setting does not affect whether rwflowpack(8) stores the"input" and "output" fields to its output files. Storage of thosefields is controlled by rwflowpack's --pack-interfaces switch.

quirks { none | { firewall-event | missing-ips | nf9-out-is-reverse | nf9-sysuptime-seconds | zero-packets ... } }
This optional command is used to indicate that special (or quirky)handling of the incoming data is desired. The value none disablesall quirks, and that is the default setting. If the value is notnone, it may be a list of one or more of the values specifiedbelow separated by commas and/or spaces. Since SiLK 3.8.0.
firewall-event
Enable checking for firewall event information elements (IEs) whenprocessing IPFIX or NetFlow v9 flow records. This quirk must beenabled when collecting data from a Cisco ASA. The IPFIX firewallEventIE is 233. The Cisco elements are NF_F_FW_EVENT (IE 40005) andNF_F_FW_EXT_EVENT (IE 33002). When this quirk is active, firewallevents that match the value 2 (flow deleted) are categorized as normalflows, firewall events that match the value 3 (flow denied) areusually put into one of non-routed types (e.g., innull, outnull,see packlogic-twoway(3) and packlogic-generic(3) for details),and all other firewall events values are dropped. (Note that a logmessage is generated for these dropped records; to suppress thesemessages, use the log-flags command.) When this quirk is notprovided, SiLK handles these records normally, which may result induplicate flow records. (Prior to SiLK 3.8, SiLK dropped all flowrecords that contained a firewall event IE.) Since SiLK 3.8.0.
missing-ips
Store a flow record even when the record's NetFlow v9/IPFIX templatedoes not contain IP addresses. One change in SiLK 3.8.0 was to ignoreflow records that do not have a source and/or destination IP address;this quirk allows one to undo the effect of that change. Since SiLK3.8.1.
nf9-out-is-reverse
Change handling of the OUT_BYTES and OUT_PKTS information elements tomatch that in libfixbuf prior to 1.8.0. Specifically, treatinformation elements 23 and 24 (OUT_BYTES and OUT_PKTS in RFC3954) asreverseOctetDeltaCount and reversePacketDeltaCount, respectively.Starting with libfixbuf-1.8.0, those NetFlow v9 elements are mapped topostOctetDeltaCount and postPacketDeltaCount, respectively. SinceSiLK 3.17.2.
nf9-sysuptime-seconds
Work around an issue with NetFlow v9 records created by somemiddleboxes (e.g., SonicWall) where the sysUpTime field in the packetheader is reported in seconds instead of in milliseconds. Theincorrect units cause the time stamps on flow records to be futuredated. In addition, adjust the time fields on single packet flowrecords. Since SiLK 3.14.0.
none
Do not enable any quirks.
zero-packets
Enable support for flow records either that do not contain a validpackets field, such as those from the Cisco ASA series of routers, orthat have an unusually large bytes-per-packet ratio. When this quirkis active, SiLK sets the packet count to 1 when the incoming IPFIX orNetFlow v9 flow record has a the packet count if 0. This quirk maymodify the file format used by rwflowpack for IPv4 records in orderto support large byte-per-packet ratios. Since SiLK 3.8.0.
priority value
This optional command is deprecated. It exists for backwardscompatibility and will be removed in the next major release.

To summarize the probe types and the input they can accept:

 Probe Type   Berkeley    Directory  UnixDomain    Single               Socket      Polling     Socket       File ==========  ==========  ==========  ==========  ========== ipfix        tcp/udp        yes netflow-v5     udp          yes                    yes netflow-v9     udp sflow          udp silk                        yes
 

Lists of CIDR Blocks, IPsets, or Integers

This subsection describes the syntax of a list of CIDR blocks, a listof IPset file names, and a list of integers. These lists are used inthe sensor block and group block commands described below.

A group block (see ``Group Block'') allows you to assign names tothese lists. Once the name is defined, it may be referenced in otherlists of the same type by prepending the ``at'' character ("@") to thegroup's name.

The lists are:

cidr-block-list
A cidr-block-list (or ipblock-list) contains one or more CIDR blocksor group references that represent an address space. Adjacent valuesin the list may be separated by multiple whitespace (space or tab)characters and/or a single comma. When IPv4 and IPv6 addressescombined, IPv4 addresses are mapped into the ::ffff:0:0/96 netblock.For lists containing more than a few CIDR blocks, consider using anIPset list instead.
ipset-list
An ipset-list contains the path names of one or more binary IPsetfiles or group references. To create an IPset file, use therwsetbuild(1) tool. Each path name may be a double-quoted string("example"); the quote characters are not necessary if the pathname does not contain whitespace or any special characters(single-quote "'", double-quote """, comma ",", or pound "#").Adjacent values in the list may be separated by multiple whitespace(space or tab) characters and/or a single comma. When multiple IPsetfiles are specified, a new IPset is created in memory and the contentsof the files are merged into it. rwflowpack(8) exits with an errorif the IPset file does not exist or does not contain any IP addresses.Since SiLK 3.10.0.
interface-list
An interface-list contains one or more integers between 0 and 65535,inclusive, or group references or that represent SNMP interfaceindexes or VLAN identifiers. Adjacent values in the list may beseparated by multiple whitespace (space or tab) characters and/or asingle comma.
 

Group Block

The use of group blocks is optional. They are a convenience to defineand give a name to a list of commonly used CIDR blocks, IPset files,or integer values that are treated as SNMP interfaces or VLANidentifiers. Groups may be used in sensor blocks (``Sensor Block'')as described in the descriptions for the discard-when,discard-unless, network-name-ipblocks,network-name-ipsets and network-name-interfaces commands,below.

The commands in a group definition must all be of the same type.For example, you cannot mix ipblocks and ipsets commands in asingle group definition, even though both contain IP addresses.

The contents of an existing group may be added to the current groupblock by using a group reference after the appropriate keyword as longas both groups are the same type. A group reference is the name ofthe group prefixed by the ``at'' character ("@"). When a groupreference is used, the contents of the existing groups are copied intothe current group.

For examples of group blocks, see ``Group definitions'' below.

The group command is used at top-level to begin a group definitionblock, and the remaining commands are accepted within the group block.

group group-name
The group command begins a new group definition block whichcontinues to the end group command. The argument to thegroup command is the name of the group being defined. Thegroup-name must be unique among all groups. It must begin with aletter, and it may not contain whitespace characters or the slashcharacter ("/").
end group
The end group command ends the definition of a group. Following anend group command, top-level commands are again accepted.
interfaces interface-list
The interfaces command adds integer values to a group, where eachinteger is treated as an SNMP interface number or VLAN identifier. Aninterface-list is a list of integers or group references as definedabove (``Lists of CIDR Blocks, IPsets, or Integers''). Theinterfaces command may appear multiple times in a group block.
ipblocks cidr-block-list
The ipblocks command adds CIDR block values to a group. Thecidr-block-list is described above. The ipblocks command mayappear multiple times in a group block.
ipsets ipset-list
The ipset command adds the IP addresses specified in an IPset fileto a group. The ipsets command may appear multiple times in agroup block. Since SiLK 3.10.0.
 

Sensor Block

The information from the sensor block is used by rwflowpack todetermine how to categorize a flow; that is, in which file the flowrecord is stored. The packlogic-twoway(3) manual page describeshow rwflowpack may use the sensor blocks to determine a record'scategory.

When the Sensor Configuration file is used with flowcap, no sensorsneed to be defined. In fact, flowcap completely ignores all textinside each sensor block.

The sensor block works with the packing logic to determine whererwflowpack stores flow records. The packing logic plug-inspecifies a list of network names, and you will refer to thesenetworks when you configure the sensor block. Most plug-ins providethe "external", "internal", and "null" names, where internal refersto network being monitored, null are flows that were blocked by therouter's access control list, and external is everything else.

Several of the commands described below that categorize flow recordsrequire as an argument a list of CIDR blocks, a list of IPset files,or a list of integers. The syntax of these lists is described in the``Lists of CIDR Blocks, IPsets, or Integers'' section above.

As part of determining how to process a flow record, rwflowpackmay check a record's source or destination IP address against acidr-block-list or an ipset-set. Note the following:

*
for a cidr-block-list, the IP address is sequentially compared to eachelement of the list, stopping once a match is made
*
when comparing an IPv4 address to an IPv6 list, the IPv4 address isconverted to IPv6 by mapping it into the ::ffff:0:0/96 prefix forpurposes of the comparison
*
when comparing an IPv6 address to an IPv4 list, an IPv6 address in the::ffff:0:0/96 prefix is converted to IPv4 for purposes of thecomparison and any other IPv6 address fails the comparison

As part of determining how to process a flow record, rwflowpack maycheck whether the record's "input" or "output" fields are aninterface-list. Whether the "input" and "output" fields containSNMP interfaces or VLAN identifiers is determined by theinterface-values command in the probe block (c.f. ``Probe Block'').

The sensor command is used in the top-level context to begin asensor configuration block, and the remaining commands are acceptedwithin the sensor block.

sensor sensor-name
The sensor command begins a new sensor configuration block. Ittakes as an argument the name of the sensor being configured, and thatsensor must be defined in the Site Configuration file (seesilk.conf(5)). A sensor block is closed with the end sensorcommand. You may have multiple sensor blocks that have the samesensor-name.
end sensor
The end sensor command ends the configuration of a sensor.Following an end sensor command, top-level commands are againaccepted.
probe-type-probes probe-name [probe-name ...]
This command associates the listed probe names of the given probe typewith the sensor. The probes do not have to be defined before they areused. (Note this also means that a mistyped probe-name will not bedetected.) For example, netflow-v5-probes S1 says that S1 is anetflow-v5 probe; whenever flow data arrives on the S1 probe, thesensor associated with the probe notices that data is available andprocesses it. Adjacent probe names in the argument list may beseparated by space or tab characters and/or a single comma.
source-network network-name
This command causes the sensor to assume that all flows originatedfrom the network named network-name. For example, if a sensor isassociated a probe that only monitors incoming traffic, you could use"source-network external" to specify that all traffic originated fromthe external network.
destination-network network-name
This command causes the sensor to assume that all flows were sent tothe network named network-name.
network-name-ipblocks {cidr-block-list | remainder}
This command specifies the IP-space that is assigned to the networknamed network-name. The value of the command can be the keywordremainder or a cidr-block-list as defined above. When the value isthe keyword remainder, the IP-space for network-name isconceptually all IPs not assigned to other networks on this sensor.The remainder keyword may only appear one time within a sensorblock.
network-name-ipsets {ipset-list | remainder}
This command specifies the IP-space that is assigned to the networknamed network-name. The value of the command can be the keywordremainder or an ipset-list as defined above. When the value is thekeyword remainder, the IP-space for network-name is conceptuallyall IPs not assigned to other networks on this sensor. Theremainder keyword may only appear one time within a sensor block.
network-name-interfaces {interface-list | remainder}
This command specifies the SNMP interface index(es) or VLANidentifiers that are assigned to the network named network-name.The value of the command may be the keyword remainder or aninterface-list as defined above. When the value is the keywordremainder, the interface list is computed by finding all interfacevalues not assigned to other networks on this sensor. Theremainder keyword may only appear one time within a sensor block.
isp-ip ip-address [ip-address ...]
This optional command may be used for a sensor that processes NetFlowdata. The value to the command is a list of IP addresses indotted-decimal notation, where the IPs are the addresses of the NICson the router. For traffic that doesn't leave the router (and thuswas sent to the router's null-interface), some packing-logic plug-insuse these IPs to distinguish legitimate traffic for the router (e.g.,routing protocol traffic, whose destination address would be in thislist) from traffic that violated the router's access control list(ACL).

The following optional sensor block commands provide a way to filterthe flow records that rwflowpack packs for a sensor. Each filterbegins with either discard-when or discard-unless, mentions aflow record field, and ends with a list of values.

The discard-when command causes the sensor to ignore the flowrecord if the property matches any of the elements in the list. Whena match is found, rwflowpack immediately stops processing therecord for the current sensor and the flow is not packed for thissensor.

The discard-unless command causes the sensor to ignore the flowrecord unless the property matches one of the elements in the list.That is, the flow record is packed only if its property matches one ofthe values specified in the list, and, when multiple discard-unlesscommands are present, if the flow record matches the valuesspecified in each.

For each individual property, only one of discard-when ordiscard-unless may be specified.

discard-when source-interfaces interface-list
Instructs rwflowpack to discard a flow record for this sensor ifthe value in the flow's "input" field is listed ininterface-list. When paired with VLAN tagging (see theinterface-values command in the probe block), this allows theadministrator to discard flows that have a specific VLAN tag. Thecommands discard-when source-interfaces and discard-unlesssource-interfaces may not be specified on the same sensor, but otherdiscard- commands are permitted.
discard-unless source-interfaces interface-list
Instructs rwflowpack to discard the flow record for this sensorunless the flow's "input" field is in interface-list. That is,the flow record may be packed only if its "input" field matches oneof the values specified in interface-list. When paired with VLANtagging, this allows one to discard flows that do not have a specificVLAN tag. The commands discard-when source-interfaces anddiscard-unless source-interfaces may not be specified on the samesensor, but other discard- commands are permitted.
discard-when destination-interfaces interface-list
Discards a flow for this sensor when the flow's "output" fieldmatches a value in interface-list. May not appear in the samesensor block with discard-unless destination-interfaces.
discard-unless destination-interfaces interface-list
Discards a flow for this sensor unless the flow's "output" fieldmatches a value in interface-list. May not appear in the samesensor block with discard-when destination-interfaces.
discard-when any-interfaces interface-list
Discards a flow for this sensor when either the flow's "input" or its"output" field matches a value in interface-list. May not appearin the same sensor block with discard-unless any-interfaces.
discard-unless any-interfaces interface-list
Discards a flow for this sensor unless either the flow's "input" orits "output" field matches a value in interface-list. May notappear in the same sensor block with discard-unless any-interfaces.
discard-when source-ipblocks cidr-block-list
Discards a flow for this sensor when the flow's source IP address,"sIP", matches one of the CIDR blocks in cidr-block-list. May notappear in the same sensor block with discard-unlesssource-ipblocks.
discard-unless source-ipblocks cidr-block-list
Discards a flow for this sensor unless the flow's source IP address,"sIP", matches one of the CIDR blocks in cidr-block-list. May notappear in the same sensor block with discard-when source-ipblocks.
discard-when destination-ipblocks cidr-block-list
Discards a flow for this sensor when the flow's destination IPaddress, "dIP", matches one of the CIDR blocks in cidr-block-list.May not appear in the same sensor block with discard-unlessdestination-ipblocks.
discard-unless destination-ipblocks cidr-block-list
Discards a flow for this sensor unless the flow's destination IPaddress, "dIP", matches one of the CIDR blocks in cidr-block-list.May not appear in the same sensor block with discard-whendestination-ipblocks.
discard-when any-ipblocks cidr-block-list
Discards a flow for this sensor when either the flow's source IP orits destination IP address matches one of the CIDR blocks incidr-block-list. May not appear in the same sensor block withdiscard-unless any-ipblocks.
discard-unless any-ipblocks cidr-block-list
Discards a flow for this sensor unless either the flow's source IP orits destination IP address matches one of the CIDR blocks incidr-block-list. May not appear in the same sensor block withdiscard-when any-ipblocks.
discard-when source-ipsets ipset-list
Discards a flow for this sensor when the flow's source IP address,"sIP", is in one of IPset files in ipset-list. May notappear in the same sensor block with discard-unless source-ipsets.Since SiLK 3.10.0.
discard-unless source-ipsets ipset-list
Discards a flow for this sensor unless the flow's source IP address,"sIP", is in one of IPset files in ipset-list. May not appear inthe same sensor block with discard-when source-ipsets. SinceSiLK 3.10.0.
discard-when destination-ipsets ipset-list
Discards a flow for this sensor when the flow's destination IPaddress, "dIP", is in one of the IPset files in ipset-list. Maynot appear in the same sensor block with discard-unlessdestination-ipsets. Since SiLK 3.10.0.
discard-unless destination-ipsets ipset-list
Discards a flow for this sensor unless the flow's destination IPaddress, "dIP", is in one of the IPset files in ipset-list. Maynot appear in the same sensor block with discard-whendestination-ipsets. Since SiLK 3.10.0.
discard-when any-ipsets ipset-list
Discards a flow for this sensor when either the flow's source IP orits destination IP address is in one of the IPset files inipset-list. May not appear in the same sensor block withdiscard-unless any-ipsets. Since SiLK 3.10.0.
discard-unless any-ipsets ipset-list
Discards a flow for this sensor unless either the flow's source IP orits destination IP address is in one of the IPset files inipset-list. May not appear in the same sensor block withdiscard-when any-ipsets. Since SiLK 3.10.0.
 

EXAMPLES

All these examples assume you are using the packlogic-twoway(3)packing logic plug-in to rwflowpack(8). 

Group definitions

The following shows how to create groups that can be used in othergroup blocks or in certain commands within a sensor block.

 group G01     interfaces 1 2, 3     interfaces 4 end group group G02     interfaces 5 @G01 end group group G03     interfaces @G02     interfaces 6 end group group G11     ipblocks 192.0.2.0/27  192.0.2.32/27,  192.0.2.64/26     ipblocks 192.0.2.128/25 end group group G12     ipblocks 198.51.100.0/24  @G11 end group group G13     ipblocks @G12     ipblocks 203.0.113.0/24 end group group G21     ipsets /var/sets/ip1.set  /var/sets/ip2.set,  /var/sets/ip3.set     ipsets /var/sets/ip4.set end group group G22     ipsets /var/sets/ip5.set  @G21 end group group G23     ipsets @G22     ipsets /var/sets/ip6.set end group
 

NetFlow v5 Categorized by SNMP Interface

The following two blocks define a probe that listens on 9900/udp forNetFlow v5 from a router. The probe only accepts traffic originatingfrom 172.16.22.22 or 172.16.33.33. The associated sensor uses theSNMP interfaces to categorize the flows, where traffic that enters therouter on interface 1 and leaves on interface 8 is in, trafficentering on 8 and leaving on 1 is out, traffic from 1 to 0 isinnull, traffic from 8 to 8 is int2int, etc.

  probe S1 netflow-v5      listen-on-port 9901      protocol udp      accept-from-host 172.16.22.22 172.16.33.33  end probe  sensor S1      netflow-v5-probes S1      external-interfaces 1      internal-interfaces 8      null-interfaces 0  end sensor
 

NetFlow v5 Categorized by IP Address

The probe in this example is the same as above, except theadministrator has chosen to log only messages about bad packets(messages about missing packets will be ignored). The sensor iscategorizing flows by the source and destination IP address in theflow record. The internal network is defined as 128.2.0.0/16, and allother IPs are defined as external. For example, HTTP traffic whosesource is 128.2.0.1 and destination is google.com will be categorizedas outweb; the reply (source of google.com and destination128.2.0.1) will be inweb.

  probe S2 netflow-v5      listen-on-port 9902      protocol udp      accept-from-host 172.16.22.22 172.16.33.33      log-flags bad                     # ignore missing pkts  end probe  sensor S2      netflow-v5-probes S2      internal-ipblocks 128.2.0.0/16      external-ipblocks remainder  end sensor
 

IPFIX Categorized by IP Address

This example uses an IPFIX probe to collect the flows on port9903/tcp, where the probe binds to address 192.168.1.92. The sensorconfiguration is the same as in the previous example, but a groupdefinition is used to define the internal network.

  probe S3 ipfix      listen-on-port 9903      protocol tcp      listen-as-host 192.168.1.92  end probe  group my-network      ipblocks 128.2.0.0/16  end group  sensor S3      ipfix-probes S3      internal-ipblocks @my-network      external-ipblocks remainder  end sensor
 

IPFIX Read from Files

This example uses the same sensor configuration as above. The probeprocesses files that have been created by yaf(1) and stored in thedirectory /tmp/var/yaf/.

  probe S4 ipfix      poll-directory /tmp/var/yaf  end probe  sensor S4      ipfix-probes S4      internal-ipblock 128.2.0.0/16      external-ipblock remainder  end sensor
 

NetFlow v9 Categorized by IP Address

This example uses a NetFlow v9 probe to collect the flows on port9905/udp, where the probe binds to address 192.168.1.92. The sensorconfiguration uses an IPset file to define the internal network.

  probe S5 netflow-v9      listen-on-port 9905      protocol udp      listen-as-host 192.168.1.92  end probe  sensor S5      netflow-v9-probes S5      internal-ipsets /var/sets/my-network.set      external-ipsets remainder  end sensor
 

sFlow v5 Categorized by IP Address

This example uses an sFlow probe to collect the flows on port9906/udp, where the probe binds to the IPv6 address ::1. The sensorconfiguration uses an IPset file to define the internal network.

  probe S19 sflow      listen-on-port 9906      protocol udp      listen-as-host ::1  end probe  sensor S19      sflow-probes S19      internal-ipsets /var/sets/my-network.set      external-ipsets remainder  end sensor
 

NetFlow v9 from a Cisco ASA Router

When collecting NetFlow v9 data from a Cisco ASA (Adaptive SecurityAppliance), specify the quirks statement as shown in this exampleto enable special handling of the NetFlow data.

  probe S20 netflow-v9      listen-on-port 9988      protocol udp      quirks firewall-event zero-packets  end probe  sensor S20      netflow-v9-probes S20      internal-ipsets /var/sets/my-network.set      external-ipsets remainder  end sensor
 

Multiple Sources Becoming One Sensor (One Port)

Consider a scenario where there are multiple input streams that needto be treated as a single sensor. For example, you use multiplerouters for load-balancing but you want them treated as a singlelogical sensor. In this configuration, you send all the input streamsto a single port, and you define a single probe listening on thatport. As long as the streams have a unique source IP, the streamswill be treated distinctly.

The following sensor and probe blocks accept any number of TCP-basedIPFIX connections to port 9907 and any number of NetFlow v5connections to 9908. This configuration works for all types of inputas SiLK 3.4.0 when using libfixbuf-1.2.0. See the configuration inthe following example for a alternate approach.

  probe S7 ipfix      listen-on-port 9907      protocol tcp  end probe  sensor S7      ipfix-probes S7      internal-ipblocks 128.2.0.0/16      external-ipblocks remainder  end sensor  probe S8 netflow-v5      listen-on-port 9908      protocol udp      log-flags bad  end probe  sensor S8      netflow-v5-probes S8      internal-ipblocks 128.2.0.0/16      external-ipblocks remainder  end sensor
 

Multiple Sources Becoming One Sensor (Multiple Ports)

Like the previous example, this example configuration causes multipleinput streams to be treated as a single sensor. In this solution,each stream arrives on a separate port where it is collected by aseparate probe. The sensor block combines the probes into one sensor.This type of approach works with all types of input for all releasesof SiLK.

  probe S6-p1 netflow-v9      listen-on-port 9961      protocol udp  end probe  probe S6-p2 netflow-v9      listen-on-port 9962      protocol udp  end probe  probe S6-p3 netflow-v9      listen-on-port 9963      protocol udp  end probe  sensor S6      netflow-v9-probes S6-p1, S6-p2, S6-p3      internal-ipblocks 128.2.0.0/16      external-ipblocks remainder  end sensor
 

Multiple Sources Becoming One Sensor (Specific Directions)

Consider the case of using yaf on a monitor at the border of anetwork where all traffic entering the network arrives at the monitoron one network interface card (NIC) and all traffic leaving thenetwork arrives at the monitor on a different NIC. Since yaf doesnot support multiple interfaces yet, you must run two yafprocesses, one for each NIC. The sensor configuration for thismonitor would list two probes, each listening on a different port, andtwo sensor blocks both packing to the same sensor. Each sensor blockpacks the traffic as incoming or outgoing depending on which probereceived the traffic.

  probe S9-in ipfix      listen-on-port 9991      protocol tcp  end probe  probe S9-out ipfix      listen-on-port 9992      protocol tcp  end probe  sensor S9      ipfix-probes S9-in      source-network external      destination-network internal  end sensor  sensor S9      ipfix-probes S9-out      source-network internal      destination-network external  end sensor
 

Multiple Sources to Multiple Sensors (Same Port)

Suppose your network has multiple flow generators that you wish totreat as separate sensors, but you would like to minimize the numberof open ports on your firewall. To support this configuration,configure the probes to distinguish the traffic based on the sourceaddress. Specifically, create a separate probe for each sensor wherethe probes of the same type use the same listen-on-port value butdifferent accept-from-host values. (Different probe types may notbind the same port; the combination of listen-on-port, protocol,and listen-as-host must be unique for different probe types.) Thefollowing configuration uses a NetFlow v5 probe, which works for allversions of SiLK. A similar configuration works for any type of inputas of SiLK 3.4.0 and libfixbuf-1.2.0.

  probe S10 netflow-v5      listen-on-port 9910      accept-from-host 172.16.22.10      protocol udp  end probe  probe S11 netflow-v5      listen-on-port 9910      accept-from-host 172.16.22.11      protocol udp  end probe  group my-network2      ipblocks 128.2.0.0/16  end group  sensor S10      netflow-v5-probes S10      internal-ipblocks @my-network2      external-ipblocks remainder  end sensor  sensor S11      netflow-v5-probes S11      internal-ipblocks @my-network2      external-ipblocks remainder  end sensor
 

Single Source Becoming Multiple Sensors

Suppose you have instrumented a single router but you wish to splitthe traffic into two sensors, where one part of the network (monitoredby sensor S12) is defined as 128.2.0.0/17, and the other (sensor S13)as 128.2.128.0/17. Traffic between 128.2.0.1 and google.com will beassigned to sensor S12, but it will so appear as ext2ext trafficfor sensor S13 unless you explicitly discard that traffic using thediscard-unless command.

  probe S12-S13 ipfix      listen-on-port 9912      protocol tcp  end probe  group S12-space      ipblocks 128.2.0.0/17  end group  group S13-space      ipblocks 128.2.128.0/17  end group  sensor S12      ipfix-probes S12-S13      discard-unless any-ipblock @S12-space      internal-ipblocks @S12-space      external-ipblocks remainder  end sensor  sensor S13      ipfix-probes S12-S13      discard-unless any-ipblock @S13-space      internal-ipblocks @S13-space      external-ipblocks remainder  end sensor
 

Discarding Flows Using VLAN Tags

You can configure rwflowpack to discard flows that do not have aparticular VLAN tag. First, specify the interface-values commandto instruct the probe to put the VLAN id into the fields thattypically store the SNMP interfaces. On the sensor, use thediscard-unless command to discard flows that do not have thedesired VLAN tag (114 in this example). Often you will not use theVLAN tags to determine a flow's direction (category) since there is asingle VLAN tag on each flow; instead, you specify the IP space of themonitored network in the sensor block. (However, see the nextexample.)

  probe S14 ipfix      listen-on-port 9914      protocol tcp      interface-values vlan  end probe  sensor S14      ipfix-probes S14      discard-unless any-interface 114      internal-ipblocks 128.2.0.0/16      external-ipblocks remainder  end sensor
 

Categorizing Flows Using VLAN Tags

By repeating a sensor block and using different discard-unlesscommands in each block, you may configure rwflowpack tocategorize flow records based on VLAN tags. Suppose yaf ismonitoring a connection where incoming flows are marked with VLAN tag151 and outgoing flows are marked with 152. You simply discard anytraffic that does not have the wanted VLAN tag, and use thesource-network and destination-network commands to assign thedirection to the flow. In this example, any flow record that does nothave one of the expected VLAN tags has its source-network set to"null", but since rwflowpack does not expect a flow record tooriginate from the null network, it stores the record in the othercategory for later analysis/debugging. (This example requires SiLK3.1 or later.)

  probe S15 ipfix      listen-on-port 9915      protocol tcp      interface-values vlan  end probe  sensor S15      # vlan ID 151 is incoming      ipfix-probes S15      discard-unless source-interface 151      source-network       external      destination-network  internal  end sensor  sensor S15      # vlan ID 152 is outgoing      ipfix-probes S15      discard-unless source-interface 152      source-network       internal      destination-network  external  end sensor  sensor S15      # discard flows that have known IDs      # force unknown IDs into the "other" category      ipfix-probes S15      discard-when source-interface 151,152      source-network       null      destination-network  internal  end sensor
 

IPFIX Collected by a DAG Card

When yaf generates flow records from a multi-port Endace DAG card,it is possible to use the port where the traffic was seen tocategorize the traffic in rwflowpack.

To do this, include the --dag-interface switch on the yafcommand line. This switch causes yaf to store the DAG port wherethe packet was collected into the equivalent of the SNMP input field,and yaf sets the SNMP output field to an offset of the port,specifically the port plus 256 (0x100|port).

Assume DAG port 0 is connected to the external side of the network (soit sees incoming traffic), and assume DAG port 1 is on the internalside. For incoming traffic, yaf sets the input and output valuesto 0 and 256, respectively. For outgoing traffic, the values are 1and 257.

The sensor.conf configuration file for rwflowpack would be:

  probe S16-dag ipfix      listen-on-port 9916      protocol tcp  end probe  sensor S16      ipfix-probes S16-dag      external-interface 0,257      internal-interface 1,256  end sensor

When rwflowpack processes the IPFIX flow records, it treats flowrecords having an input of 0 and an output of 256 as traffic movingfrom an external interface to an internal interface, and rwflowpackpacks those records as incoming. Similarly for the outgoing flowrecords. 

Repacking of SiLK Flows by IP Address

A probe whose type is "silk" must get its flows by polling a directory ofSiLK Flowfiles. The flows can be re-categorized based on the IP addresses orbased on the SNMP interfaces (beware: often the SNMP interface valuesare 0 in SiLK Flow data). In this example, the files in the directory/var/tmp/old-data/ are processed. The internal network is definedas 128.2.0.0/16, and all other IPs are defined as external.

  probe S17 silk      poll-directory /var/tmp/old-data  end probe  sensor S17      silk-probes S17      internal-ipblock 128.2.0.0/16      external-ipblock remainder  end sensor
 

NetFlow From a File Categorized by SNMP Interfaces

Instead of listening on a UDP port for NetFlow traffic, you canconfigure the probe to process a single file containing NetFlow v5PDUs. This example assumes you are running rwflowpack with theswitches --input-mode=file --netflow-file=FILENAME. The--netflow-file switch overrides the read-from-file command onthe probe. rwflowpack will exit once it processes that singlefile.

  probe S18 netflow-v5      log-flags bad                     # ignore missing pkts      read-from-file /dev/null          # use --netflow-file=<file>  end probe  sensor S18      netflow-v5-probes S18      external-interface 182      internal-interface 189      null-interface 0  end sensor
 

SEE ALSO

rwflowpack(8), flowcap(8), packlogic-twoway(3),packlogic-generic(3), rwsetbuild(1), silk.conf(5),silk(7), SiLK Installation Handbook, pcap(3), yaf(1),gzip(1) 

NOTES

Support for using double-quoted strings for IPset path names was addedin SiLK 3.17.2.

The accept-from-host command began to accept a list of arguments inSiLK 3.10.1.

SiLK