TCPCollector

The TCPCollector class collects metrics on TCP stats

Dependencies

  • /proc/net/netstat
  • /proc/net/snmp

Allowed Metric Names

NameDescription
SyncookiesSentAn application wasn't able to accept a connection fast enough, so the kernel couldn't store an entry in the queue for this connection. Instead of dropping it, it sent a cookie to the client.
SyncookiesRecvAfter sending a cookie, it came back to us and passed the check.
SyncookiesFailedAfter sending a cookie, it came back to us but looked invalid.
EmbryonicRsts
PruneCalled
RcvPrunedIf the kernel is really really desperate and cannot give more memory to this socket even after dropping the ofo queue, it will simply discard the packet it received. This is Really Bad.
OfoPrunedWhen a socket is using too much memory (rmem), the kernel will first discard any out-of-order packet that has been queued (with SACK).
OutOfWindowIcmps
LockDroppedIcmps
ArpFilter
TW
TWRecycled
TWKilled
PAWSPassive
PAWSActive
PAWSEstab
DelayedACKsWe waited for another packet to send an ACK, but didn't see any, so a timer ended up sending a delayed ACK.
DelayedACKLockedWe wanted to send a delayed ACK but failed because the socket was locked. So the timer was reset.
DelayedACKLostWe sent a delayed and duplicated ACK because the remote peer retransmitted a packet, thinking that it didn't get to us.
ListenOverflowsWe completed a 3WHS but couldn't put the socket on the accept queue, so we had to discard the connection.
ListenDropsWe couldn't accept a connection because one of: we had no route to the destination, we failed to allocate a socket, we failed to allocate a new local port bind bucket. Note: this counter also include all the increments made to ListenOverflows
TCPPrequeued
TCPDirectCopyFromBacklog
TCPDirectCopyFromPrequeue
TCPPrequeueDropped
TCPHPHits
TCPHPHitsToUser
TCPPureAcks
TCPHPAcks
TCPRenoRecoveryA packet was lost and we recovered after a fast retransmit.
TCPSackRecoveryA packet was lost and we recovered by using selective acknowledgements.
TCPSACKReneging
TCPFACKReorderWe detected re-ordering using FACK (Forward ACK -- the highest sequence number known to have been received by the peer when using SACK -- FACK is used during congestion control).
TCPSACKReorderWe detected re-ordering using SACK.
TCPRenoReorderWe detected re-ordering using fast retransmit.
TCPTSReorderWe detected re-ordering using the timestamp option.
TCPFullUndoWe detected some erroneous retransmits and undid our CWND reduction.
TCPPartialUndoWe detected some erroneous retransmits, a partial ACK arrived while we were fast retransmitting, so we were able to partially undo some of our CWND reduction.
TCPDSACKUndoWe detected some erroneous retransmits, a D-SACK arrived and ACK'ed all the retransmitted data, so we undid our CWND reduction.
TCPLossUndoWe detected some erroneous retransmits, a partial ACK arrived, so we undid our CWND reduction.
TCPLoss
TCPLostRetransmit
TCPRenoFailures
TCPSackFailures
TCPLossFailures
TCPFastRetrans
TCPForwardRetrans
TCPSlowStartRetrans
TCPTimeouts
TCPRenoRecoveryFail
TCPSackRecoveryFail
TCPSchedulerFailed
TCPRcvCollapsed
TCPDSACKOldSent
TCPDSACKOfoSent
TCPDSACKRecv
TCPDSACKOfoRecv
TCPSACKDiscardWe got a completely invalid SACK block and discarded it.
TCPDSACKIgnoredOldWe got a duplicate SACK while retransmitting so we discarded it.
TCPDSACKIgnoredNoUndoWe got a duplicate SACK and discarded it.
TCPAbortOnSynWe received an unexpected SYN so we sent a RST to the peer.
TCPAbortOnDataWe were in FIN_WAIT1 yet we received a data packet with a sequence number that's beyond the last one for this connection, so we RST'ed.
TCPAbortOnCloseWe received data but the user has closed the socket, so we have no wait of handing it to them, so we RST'ed.
TCPAbortOnMemoryThis is Really Bad. It happens when there are too many orphaned sockets (not attached a FD) and the kernel has to drop a connection. Sometimes it will send a reset to the peer, sometimes it wont.
TCPAbortOnTimeoutThe connection timed out really hard.
TCPAbortOnLingerWe killed a socket that was closed by the application and lingered around for long enough.
TCPAbortFailedWe tried to send a reset, probably during one of teh TCPABort* situations above, but we failed e.g. because we couldn't allocate enough memory (very bad).
TCPMemoryPressuresNumber of times a socket was put in "memory pressure" due to a non fatal memory allocation failure (reduces the send buffer size etc).
TCPBacklogDropWe received something but had to drop it because the socket's receive queue was full.
RtoAlgorithmThe algorithm used to determine the timeout value used for retransmitting unacknowledged octets.
RtoMinThe minimum value permitted by a TCP implementation for the retransmission timeout, measured in milliseconds. More refined semantics for objects of this type depend upon the algorithm used to determine the retransmission timeout. In particular, when the timeout algorithm is ``rsre '' (3), an object of this type has the semantics of the LBOUND quantity described in RFC 793.
RtoMaxThe maximum value permitted by a TCP implementation for the retransmission timeout, measured in milliseconds. More refined semantics for objects of this type depend upon the algorithm used to determine the retransmission timeout. In particular, when the timeout algorithm is ``rsre'' (3), an object of this type has the semantics of the UBOUND quantity described in RFC 793.
MaxConnThe limit on the total number of TCP connections the entity can support. In entities where the maximum number of connections is dynamic, this object should contain the value -1.
ActiveOpensThe number of times TCP connections have made a direct transition to the SYN-SENT state from the CLOSED state.
PassiveOpensThe number of times TCP connections have made a direct transition to the SYN-RCVD state from the LISTEN state.
AttemptFailsThe number of times TCP connections have made a direct transition to the CLOSED state from either the SYN-SENT state or the SYN-RCVD state, plus the number of times TCP connections have made a direct transition to the LISTEN state from the SYN-RCVD state.
EstabResetsThe number of times TCP connections have made a direct transition to the CLOSED state from either the ESTABLISHED state or the CLOSE-WAIT state.
CurrEstabThe number of TCP connections for which the current state is either ESTABLISHED or CLOSE- WAIT.
InSegsThe total number of segments received, including those received in error. This count includes segments received on currently established connections.
OutSegsThe total number of segments sent, including those on current connections but excluding those containing only retransmitted octets.
RetransSegsThe total number of segments retransmitted - that is, the number of TCP segments transmitted containing one or more previously transmitted octets.
InErrsThe total number of segments received in error (for example, bad TCP checksums).
OutRstsThe number of TCP segments sent containing the RST flag.

Options

Setting Default Description Type
allowed_names ListenOverflows, ListenDrops, TCPLoss, TCPTimeouts, TCPFastRetrans, TCPLostRetransmit, TCPForwardRetrans, TCPSlowStartRetrans, CurrEstab, TCPAbortOnMemory, TCPBacklogDrop, AttemptFails, EstabResets, InErrs, ActiveOpens, PassiveOpens list of entries to collect, empty to collect all str
byte_unit byte Default numeric output(s) str
enabled False Enable collecting these metrics bool
measure_collector_time False Collect the collector run time in ms bool
metrics_blacklist None Regex to match metrics to block. Mutually exclusive with metrics_whitelist NoneType
metrics_whitelist None Regex to match metrics to transmit. Mutually exclusive with metrics_blacklist NoneType

Example Output

servers.hostname.tcp.ActiveOpens 0.0
servers.hostname.tcp.ArpFilter 0.0
servers.hostname.tcp.AttemptFails 0.0
servers.hostname.tcp.CurrEstab 3.0
servers.hostname.tcp.DelayedACKLocked 144.0
servers.hostname.tcp.DelayedACKLost 10118.0
servers.hostname.tcp.DelayedACKs 125491.0
servers.hostname.tcp.EmbryonicRsts 0.0
servers.hostname.tcp.EstabResets 0.0
servers.hostname.tcp.IPReversePathFilter 0.0
servers.hostname.tcp.InErrs 0.0
servers.hostname.tcp.InSegs 1.0
servers.hostname.tcp.ListenDrops 0.0
servers.hostname.tcp.ListenOverflows 0.0
servers.hostname.tcp.LockDroppedIcmps 4.0
servers.hostname.tcp.MaxConn -1.0
servers.hostname.tcp.OfoPruned 0.0
servers.hostname.tcp.OutOfWindowIcmps 10.0
servers.hostname.tcp.OutRsts 0.0
servers.hostname.tcp.OutSegs 0.0
servers.hostname.tcp.PAWSActive 0.0
servers.hostname.tcp.PAWSEstab 0.0
servers.hostname.tcp.PAWSPassive 0.0
servers.hostname.tcp.PassiveOpens 0.0
servers.hostname.tcp.PruneCalled 0.0
servers.hostname.tcp.RcvPruned 0.0
servers.hostname.tcp.RetransSegs 0.0
servers.hostname.tcp.RtoAlgorithm 0.0
servers.hostname.tcp.RtoMax 0.0
servers.hostname.tcp.RtoMin 0.0
servers.hostname.tcp.SyncookiesFailed 9.0
servers.hostname.tcp.SyncookiesRecv 0.0
servers.hostname.tcp.SyncookiesSent 0.0
servers.hostname.tcp.TCPAbortFailed 0.0
servers.hostname.tcp.TCPAbortOnClose 38916.0
servers.hostname.tcp.TCPAbortOnData 143.0
servers.hostname.tcp.TCPAbortOnLinger 0.0
servers.hostname.tcp.TCPAbortOnMemory 0.0
servers.hostname.tcp.TCPAbortOnSyn 0.0
servers.hostname.tcp.TCPAbortOnTimeout 68.0
servers.hostname.tcp.TCPBacklogDrop 0.0
servers.hostname.tcp.TCPDSACKIgnoredNoUndo 1025.0
servers.hostname.tcp.TCPDSACKIgnoredOld 292.0
servers.hostname.tcp.TCPDSACKOfoRecv 0.0
servers.hostname.tcp.TCPDSACKOfoSent 0.0
servers.hostname.tcp.TCPDSACKOldSent 10175.0
servers.hostname.tcp.TCPDSACKRecv 1580.0
servers.hostname.tcp.TCPDSACKUndo 2448.0
servers.hostname.tcp.TCPDeferAcceptDrop 0.0
servers.hostname.tcp.TCPDirectCopyFromBacklog 35660.0
servers.hostname.tcp.TCPDirectCopyFromPrequeue 19340531.0
servers.hostname.tcp.TCPFACKReorder 0.0
servers.hostname.tcp.TCPFastRetrans 1184.0
servers.hostname.tcp.TCPForwardRetrans 41.0
servers.hostname.tcp.TCPFullUndo 0.0
servers.hostname.tcp.TCPHPAcks 12673896.0
servers.hostname.tcp.TCPHPHits 10361792.0
servers.hostname.tcp.TCPHPHitsToUser 5667.0
servers.hostname.tcp.TCPLoss 188.0
servers.hostname.tcp.TCPLossFailures 7.0
servers.hostname.tcp.TCPLossUndo 6538.0
servers.hostname.tcp.TCPLostRetransmit 7.0
servers.hostname.tcp.TCPMD5NotFound 0.0
servers.hostname.tcp.TCPMD5Unexpected 0.0
servers.hostname.tcp.TCPMemoryPressures 0.0
servers.hostname.tcp.TCPMinTTLDrop 0.0
servers.hostname.tcp.TCPPartialUndo 0.0
servers.hostname.tcp.TCPPrequeueDropped 0.0
servers.hostname.tcp.TCPPrequeued 114232.0
servers.hostname.tcp.TCPPureAcks 1003528.0
servers.hostname.tcp.TCPRcvCollapsed 0.0
servers.hostname.tcp.TCPRenoFailures 0.0
servers.hostname.tcp.TCPRenoRecovery 0.0
servers.hostname.tcp.TCPRenoRecoveryFail 0.0
servers.hostname.tcp.TCPRenoReorder 0.0
servers.hostname.tcp.TCPSACKDiscard 0.0
servers.hostname.tcp.TCPSACKReneging 0.0
servers.hostname.tcp.TCPSACKReorder 0.0
servers.hostname.tcp.TCPSackFailures 502.0
servers.hostname.tcp.TCPSackMerged 1121.0
servers.hostname.tcp.TCPSackRecovery 364.0
servers.hostname.tcp.TCPSackRecoveryFail 13.0
servers.hostname.tcp.TCPSackShiftFallback 3091.0
servers.hostname.tcp.TCPSackShifted 2356.0
servers.hostname.tcp.TCPSchedulerFailed 0.0
servers.hostname.tcp.TCPSlowStartRetrans 2540.0
servers.hostname.tcp.TCPSpuriousRTOs 9.0
servers.hostname.tcp.TCPTSReorder 0.0
servers.hostname.tcp.TCPTimeWaitOverflow 0.0
servers.hostname.tcp.TCPTimeouts 15265.0
servers.hostname.tcp.TW 89479.0
servers.hostname.tcp.TWKilled 0.0
servers.hostname.tcp.TWRecycled 0.0