Section: Corosync Cluster Engine Programmer's Manual (5)
Updated: 2012-01-12


corosync.conf - corosync executive configuration file






The corosync.conf instructs the corosync executive about various parametersneeded to control the corosync executive. Empty lines and lines starting with# character are ignored. The configuration file consists of bracketed top leveldirectives. The possible directive choices are:

totem { }
This top level directive contains configuration options for the totem protocol.
logging { }
This top level directive contains configuration options for logging.
quorum { }
This top level directive contains configuration options for quorum.
nodelist { }
This top level directive contains configuration options for nodes in cluster.

It is also possible to specify the top level parametercompatibility.This directive indicates the level of compatibility requested by the user. Theoption whitetank can be specified to remain backward compatable withopenais-0.80.z. The option none can be specified to only be compatablewith corosync-1.Y.Z. Extra processing during configuration changes isrequired to remain backward compatable.

The default is whitetank. (backwards compatibility)

Within thetotemdirective, an interface directive is required. There is also one configurationoption which is required:

Within theinterfacesub-directive of totem there are four parameters which are required. There isone parameter which is optional.

This specifies the ring number for the interface. When using the redundantring protocol, each interface should specify separate ring numbers to uniquelyidentify to the membership protocol which interface to use for which redundantring. The ringnumber must start at 0.

This specifies the network address the corosync executive should bindto. For example, if the local interface is with netmask255.255.255.0, set bindnetaddr to If the local interfaceis with netmask, set bindnetaddr to192.168.5.64, and so forth.

This may also be an IPV6 address, in which case IPV6 networking will be used.In this case, the full address must be specified and there is no automaticselection of the network interface within a specific subnet as with IPv4.

If IPv6 networking is used, the nodeid field in nodelist must be specified.

This is optional and can be set to yes. If it is set to yes, the broadcastaddress will be used for communication. If this option is set, mcastaddrshould not be set.

This is the multicast address used by corosync executive. The defaultshould work for most networks, but the network administrator should be queriedabout a multicast address to use. Avoid 224.x.x.x because this is a "config"multicast address.

This may also be an IPV6 multicast address, in which case IPV6 networkingwill be used. If IPv6 networking is used, the nodeid field in nodelist mustbe specified.

It's not needed to use this option if cluster_name option is used. If both optionsare used, mcastaddr has higher priority.

This specifies the UDP port number. It is possible to use the same multicastaddress on a network with the corosync services configured for differentUDP ports.Please note corosync uses two UDP ports mcastport (for mcast receives) and mcastport - 1 (for mcast sends).If you have multiple clusters on the same network using the same mcastaddr please configure the mcastports with a gap.

This specifies the Time To Live (TTL). If you run your cluster on a routednetwork then the default of "1" will be too small. This option providesa way to increase this up to 255. The valid range is 0..255.Note that this is only valid on multicast transport types.

Within thetotemdirective, there are seven configuration options of which one is required,five are optional, and one is required when IPV6 is configured in the interfacesubdirective. The required directive controls the version of the totemconfiguration. The optional option unless using IPV6 directive controlsidentification of the processor. The optional options control secrecy andauthentication, the redundant ring mode of operation and maximum network MTUfield.

This specifies the version of the configuration file. Currently the onlyvalid version for this directive is 2.

This configuration option is optional and is only relevant when no nodeid isspecified. Some corosync clients require a signed 32 bit nodeid that is greaterthan zero however by default corosync uses all 32 bits of the IPv4 address spacewhen generating a nodeid. Set this option to yes to force the high bit to bezero and therefor ensure the nodeid is a positive signed 32 bit integer.

WARNING: The clusters behavior is undefined if this option is enabled on onlya subset of the cluster (for example during a rolling upgrade).

This specifies that HMAC/SHA1 authentication should be used to authenticateall messages. It further specifies that all data should be encrypted with thesober128 encryption algorithm to protect data from eavesdropping.

Enabling this option adds a 36 byte header to every message sent by totem whichreduces total throughput. Encryption and authentication consume 75% of CPUcycles in aisexec as measured with gprof when enabled.

For 100mbit networks with 1500 MTU frame transmissions:A throughput of 9mb/sec is possible with 100% cpu utilization when thisoption is enabled on 3ghz cpus.A throughput of 10mb/sec is possible wth 20% cpu utilization when thisoptin is disabled on 3ghz cpus.

For gig-e networks with large frame transmissions:A throughput of 20mb/sec is possible when this option is enabled on3ghz cpus.A throughput of 60mb/sec is possible when this option is disabled on3ghz cpus.

The default is on.

This specifies the mode of redundant ring, which may be none, active, orpassive. Active replication offers slightly lower latency from transmitto delivery in faulty network environments but with less performance.Passive replication may nearly double the speed of the totem protocolif the protocol doesn't become cpu bound. The final option is none, inwhich case only one network interface will be used to operate the totemprotocol.

If only one interface directive is specified, none is automatically chosen.If multiple interface directives are specified, only active or passive maybe chosen.

This specifies the network maximum transmit unit. To set this value beyond1500, the regular frame MTU, requires ethernet devices that support large, oralso called jumbo, frames. If any device in the network doesn't support largeframes, the protocol will not operate properly. The hosts must also have theirmtu size set from 1500 to whatever frame size is specified here.

Please note while some NICs or switches claim large frame support, they support9000 MTU as the maximum frame size including the IP header. Setting the netmtuand host MTUs to 9000 will cause totem to use the full 9000 bytes of the frame.Then Linux will add a 18 byte header moving the full frame size to 9018. As aresult some hardware will not operate properly with this size of data. A netmtuof 8982 seems to work for the few large frame devices that have been tested.Some manufacturers claim large frame support when in fact they support framesizes of 4500 bytes.

Increasing the MTU from 1500 to 8982 doubles throughput performance from 30MB/secto 60MB/sec as measured with evsbench with 175000 byte messages with the secauthdirective set to off.

When sending multicast traffic, if the network frequently reconfigures, chances arethat some device in the network doesn't support large frames.

Choose hardware carefully if intending to use large frame support.

The default is 1500.

This directive controls the virtual synchrony filter type used to identifya primary component. The preferred choice is YKD dynamic linear voting,however, for clusters larger then 32 nodes YKD consumes alot of memory. Forlarge scale clusters that are created by changing the MAX_PROCESSORS_COUNT#define in the C code totem.h file, the virtual synchrony filter "none" isrecommended but then AMF and DLCK services (which are currently experimental)are not safe for use.

The default is ykd. The vsftype can also be set to none.

This directive controls the transport mechanism used. If the interface towhich corosync is binding is an RDMA interface such as RoCEE or Infiniband, the"iba" parameter may be specified. To avoid the use of multicast entirely, aunicast transport parameter "udpu" can be specified. This requires specifyingthe list of members in nodelist directive, that could potentially make upthe membership before deployment.

The default is udp. The transport type can also be set to udpu or iba.

This specifies the name of cluster and it's used for automatic generatingof multicast address.

Within thetotemdirective, there are several configuration options which are used to controlthe operation of the protocol. It is generally not recommended to change anyof these values without proper guidance and sufficient testing. Some networksmay require larger values if suffering from frequent reconfigurations. Someapplications may require faster failure detection times which can be achievedby reducing the token timeout.

This timeout specifies in milliseconds until a token loss is declared after notreceiving a token. This is the time spent detecting a failure of a processorin the current configuration. Reforming a new configuration takes about 50milliseconds in addition to this timeout.

The default is 1000 milliseconds.

This timeout specifies in milliseconds after how long before receiving a tokenthe token is retransmitted. This will be automatically calculated if tokenis modified. It is not recommended to alter this value without guidance fromthe corosync community.

The default is 238 milliseconds.

This timeout specifies in milliseconds how long the token should be held bythe representative when the protocol is under low utilization. It is notrecommended to alter this value without guidance from the corosync community.

The default is 180 milliseconds.

This value identifies how many token retransmits should be attempted beforeforming a new configuration. If this value is set, retransmit and hold willbe automatically calculated from retransmits_before_loss and token.

The default is 4 retransmissions.

This timeout specifies in milliseconds how long to wait for join messages inthe membership protocol.

The default is 50 milliseconds.

This timeout specifies in milliseconds an upper range between 0 and send_jointo wait before sending a join message. For configurations with less then32 nodes, this parameter is not necessary. For larger rings, this parameteris necessary to ensure the NIC is not overflowed with join messages onformation of a new ring. A reasonable value for large rings (128 nodes) wouldbe 80msec. Other timer values must also change if this value is changed. Seekadvice from the corosync mailing list if trying to run larger configurations.

The default is 0 milliseconds.

This timeout specifies in milliseconds how long to wait for consensus to beachieved before starting a new round of membership configuration. The minimumvalue for consensus must be 1.2 * token. This value will be automaticallycalculated at 1.2 * token if the user doesn't specify a consensus value.

For two node clusters, a consensus larger then the join timeout but less thentoken is safe. For three node or larger clusters, consensus should be largerthen token. There is an increasing risk of odd membership changes, which stilguarantee virtual synchrony, as node count grows if consensus is less thantoken.

The default is 1200 milliseconds.

This timeout specifies in milliseconds how long to wait before checking fora partition when no multicast traffic is being sent. If multicast trafficis being sent, the merge detection happens automatically as a function ofthe protocol.

The default is 200 milliseconds.

This timeout specifies in milliseconds how long to wait before checkingthat a network interface is back up after it has been downed.

The default is 1000 millseconds.

This constant specifies how many rotations of the token without receiving anyof the messages when messages should be received may occur before a newconfiguration is formed.

The default is 2500 failures to receive a message.

This constant specifies how many rotations of the token without any multicasttraffic should occur before the merge detection timeout is started.

The default is 30 rotations.

[HeartBeating mechanism]Configures the optional HeartBeating mechanism for faster failure detection. Keep inmind that engaging this mechanism in lossy networks could cause faulty loss declarationas the mechanism relies on the network for heartbeating.

So as a rule of thumb use this mechanism if you require improved failure in low tomedium utilized networks.

This constant specifies the number of heartbeat failures the system should toleratebefore declaring heartbeat failure e.g 3. Also if this value is not set or is 0 then theheartbeat mechanism is not engaged in the system and token rotation is the methodof failure detection

The default is 0 (disabled).

[HeartBeating mechanism]This constant specifies in milliseconds the approximate delay that your network takesto transport one packet from one machine to another. This value is to be set by systemengineers and please dont change if not sure as this effects the failure detectionmechanism using heartbeat.

The default is 50 milliseconds.

This constant specifies the maximum number of messages that may be sent on onetoken rotation. If all processors perform equally well, this value could belarge (300), which would introduce higher latency from origination to deliveryfor very large rings. To reduce latency in large rings(16+), the defaults area safe compromise. If 1 or more slow processor(s) are present among fastprocessors, window_size should be no larger then 256000 / netmtu to avoidoverflow of the kernel receive buffers. The user is notified of this bythe display of a retransmit list in the notification logs. There is no lossof data, but performance is reduced when these errors occur.

The default is 50 messages.

This constant specifies the maximum number of messages that may be sent by oneprocessor on receipt of the token. The max_messages parameter is limited to256000 / netmtu to prevent overflow of the kernel transmit buffers.

The default is 17 messages.

This constant defines the maximum number of times on receipt of a tokena message is checked for retransmission before a retransmission occurs. Thisparameter is useful to modify for switches that delay multicast packetscompared to unicast packets. The default setting works well for nearly allmodern switches.

The default is 5 messages.

This specifies the time in milliseconds to wait before decrementing theproblem count by 1 for a particular ring to ensure a link is not markedfaulty for transient network failures.

The default is 2000 milliseconds.

This specifies the number of times a problem is detected with a link beforesetting the link faulty. Once a link is set faulty, no more data istransmitted upon it. Also, the problem counter is no longer decremented whenthe problem count timeout expires.

A problem is detected whenever all tokens from the proceeding processor havenot been received within the rrp_token_expired_timeout. Therrp_problem_count_threshold * rrp_token_expired_timeout should be atleast 50milliseconds less then the token timeout, or a complete reconfigurationmay occur.

The default is 10 problem counts.

This specifies the number of times a problem is detected with multicast beforesetting the link faulty for passive rrp mode. This variable is unused in activerrp mode.

The default is 10 times rrp_problem_count_threshold.

This specifies the time in milliseconds to increment the problem counter forthe redundant ring protocol after not having received a token from all ringsfor a particular processor.

This value will automatically be calculated from the token timeout andproblem_count_threshold but may be overridden. It is not recommended tooverride this value without guidance from the corosync community.

The default is 47 milliseconds.

This specifies the time in milliseconds to check if the failed ring can beauto-recovered.

The default is 1000 milliseconds.

Within theloggingdirective, there are several configuration options which are all optional.

The following 3 options are valid only for the top level logging directive:

This specifies that a timestamp is placed on all log messages.

The default is off.

This specifies that file and line should be printed.

The default is off.

This specifies that the code function name should be printed.

The default is off.

The following options are valid both for top level logging directiveand they can be overriden in logger_subsys entries.

These specify the destination of logging output. Any combination ofthese options may be specified. Valid options areyesandno.

The default is syslog and stderr.

Please note, if you are using to_logfile and want to rotate the file, use logrotate(8)with the option

/var/log/corosync.log {    missingok    compress    notifempty    daily    rotate 7    copytruncate}

If theto_logfiledirective is set toyes, this option specifies the pathname of the log file.

No default.

This specifies the logfile priority for this particular subsystem. Ignored if debug is on.Possible values are: alert, crit, debug (same as debug = on), emerg, err, info, notice, warning.

The default is: info.

This specifies the syslog facility type that will be used for any messagessent to syslog. options are daemon, local0, local1, local2, local3, local4,local5, local6 & local7.

The default is daemon.

This specifies the syslog level for this particular subsystem. Ignored if debug is on.Possible values are: alert, crit, debug (same as debug = on), emerg, err, info, notice, warning.

The default is: info.

This specifies whether debug output is logged for this particular logger.

The default is off.

Within theloggingdirective, logger_subsys directives are optional.

Within thelogger_subsyssub-directive, all of the above logging configuration options are valid andcan be used to override the default settings.The subsys entry, described below, is mandatory to identify the subsystem.

This specifies the subsystem identity (name) for which logging is specified. This is thename used by a service in the log_init () call. E.g. 'CPG'. This directive isrequired.

Within thequorumdirective it is possible to specify the quorum algorithm to use with the

directive. At the time of writing only corosync_votequorum is supported. Please referto quorum modules man pages (8) for specific config options.

Within thenodelistdirective it is possible to specify specific informations about nodes in cluster. Directivecan contain onlynodesub-directive, which specifies every node that should be a member of the membership, and wherenon-default options are needed. Every node must have at least ring0_addr field filled.

For UDPU, every node that should be a member of the membership must be specified.

Possible options are:

This specifies ip address of one of the nodes. X is ring number.

This configuration option is optional when using IPv4 and required when usingIPv6. This is a 32 bit value specifying the node identifier delivered to thecluster membership service. If this is not specified with IPv4, the node idwill be determined from the 32 bit IP address the system to which the systemis bound with ring identifier of 0. The node identifier value of zero isreserved and should not be used.




The corosync executive configuration file.







This document was created byman2html,using the manual pages.