[[common-counters]]
== Osmocom Counters

The following gives an overview of all the types of counters available:

=== Osmo Counters (deprecated)

Osmo counters are the oldest type of counters added to Osmocom projects.
They are not grouped.

* Printed as part of VTY show stats
* Increment, Decrement
* Accessible through the control interface: counter.<counter_name>

=== Rate Counters

Rate counters count rates of events.

* Printed as part of VTY show stats
* Intervals: per second, minute, hour, day or absolute value
* Increment only
* Accessible through the control interface
* Rate counters are grouped and different instances per group can exist

The control interface command to get a counter (group) is:

rate_ctr.per_{sec,min,hour,day,abs}.<group_name>.<idx>.[counter_name]

It is possible to get all counters in a group by omitting the counter name

=== Stat Item

Stat items are a grouped replacement for osmo counters.

* Printed as part of VTY show stats
* Replacement for osmo counters
* Not yet available through the control interface
* Grouped and indexed like rate counters
* Items have a unit
* Keeps a list of the last values measured, so could return an average, min,
  max, std. deviation. So far this is not implemented in any of the reporting
  options.

=== Statistic Levels

There are three levels on which a statistic can be aggregated in Osmocom
projects: globally, per-peer and per-subscriber.

==== Global

These are global statistics.

==== Peer

These statistics relate to a peer the program connects to such as the NSVC in
an SGSN.

This level also includes reporting global statistics.

==== Subscriber

These statistics are related to an individual mobile subscriber. An example
would be bytes transferred in an SGSN PDP context.

This level also includes global and peer-based statistics.

=== Stats Reporter

The stats reporter periodically collects osmo counter, rate counter and
stat item values and sends them to a backend. Currently implemented are
outputting to the configured log targets and a statsd connector.

==== Configuring a stats reporter

Periodically printing the statistics to the log can be done in the following
way:

.Log statistics
====

----
OsmoBSC> enable
OsmoBSC# configure terminal
OsmoBSC(config)# stats interval 60 <1>
OsmoBSC(config)# stats reporter log <2>
OsmoBSC(config-stats)# level global <3>
OsmoBSC(config-stats)# enable <4>
----
====

<1> The interval determines how often the statistics are reported.
<2> Write the statistic information to any configured log target.
<3> Report only `global` statistics (can be `global`, `peer`, or
    `subscriber`).
<4> Enable the reporter, `disable` will disable it again.

The counter values can also be sent to any aggregation/visualization tool that
understands the statsd format, for example a statsd server with graphite or
prometheus using the statsd_exporter together with grafana.

The statsd format is specified in https://github.com/b/statsd_spec

.Report statistics to statsd
====

----
OsmoBSC> enable
OsmoBSC# configure terminal
OsmoBSC(config)# stats interval 10
OsmoBSC(config)# stats reporter statsd <1>
OsmoBSC(config-stats)# prefix BSC1 <2>
OsmoBSC(config-stats)# level subscriber <3>
OsmoBSC(config-stats)# remote-ip 1.2.3.4 <4>
OsmoBSC(config-stats)# remote-port 8125 <5>
OsmoBSC(config-stats)# enable
----
====

<1> Configure the statsd reporter.
<2> Prefix the reported statistics. This is useful to distinguish statistics
    from multiple instances of the same service.
<3> Report only `global` statistics or include `peer` or `subscriber`
    statistics as well.
<4> IP address of the statsd server.
<5> UDP port of the statsd server. Statsd by default listens to port 8125.

You can use Netdata (https://learn.netdata.cloud/) as a statsd server which does not require special configuration
to show rate counters.
By default all the rate counters will be exposed to the StatsD plugin (listening on 127.0.0.1:8125) and displayed on the
Netdata dashboard available via: http://localhost:19999
The list of available charts which includes all the rate counters reported via statsD is available through:
http://localhost:19999/api/v1/charts

=== Socket stats

libosmocore provides features to monitor the status of TCP connections. This
can be a helpful source of information when the links between network
components are unreliable (e.g. satellite link between BTS and BSC).

NOTE: This feature is only available for certain types of TCP connections. At
the moment only RSL/OML connections between OsmoBSC and the connected BTSs can
be monitored.

==== Configuration

The gathering of the TCP connection statistics is done via syscalls. This has
to be taken into account for the configuration. Since syscalls are rather
expensive and time consuming the overall performance of the application may
suffer when many TCP connections are present. This may be the case for BSCs
with a large number of BTSs connected to it.

The statistics are gathered in batches per interval. A batch size of 5 would
mean that only 5 TCP connections per interval are evaluated and the next 5
connections in the next interval and so on.

It is recommended to choose a large reporting interval and a reasonable small
batch size to distribute the syscall load as even as possible.

.Report statistics to statsd
====
----
OsmoBSC> enable
OsmoBSC# configure terminal
stats-tcp interval 10 <1>
stats-tcp batch-size 5 <2>
----
====

<1> Set the gathering interval (sec.)
<2> Set how many TCP sockets statistics to gather per interval.

==== Generated stats items

[options="header"]
|===
| Name | Description
| tcp:unacked | unacknowledged packets.
| tcp:lost | unacknowledged packets.
| tcp:retrans | lost packets.
| tcp:rtt | retransmitted packets.
| tcp:rcv_rtt | roundtrip-time (receive).
| tcp:notsent_bytes | bytes not yet sent.
| tcp:rwnd_limited | time (usec) limited by receive window.
| tcp:sndbuf_limited | Time (usec) limited by send buffer.
| tcp:reord_seen | reordering events seen.
|===

The item group index is the file descriptor number. The item group name
consists of a static prefix (e.g. "ipa-rsl"), followed by the IP addresses
and ports of both peers.

.VTY output of a stats item group of a TCP connection
====
----
stats tcp (15)('ipa-rsl,r=10.9.1.143:38455<->l=10.9.1.162:3003'):
 unacknowledged packets:        0
 lost packets:        0
 retransmitted packets:        0
 roundtrip-time:      583
 roundtrip-time (receive):        0
 bytes not yet sent:        0
 time (usec) limited by receive window:        0
 Time (usec) limited by send buffer:        0
 reordering events seen:        0
----
====
