[[mgw_pooling]]
== MGW Pooling

{program-name} is able to use a pool of media gateway (MGW) instances.
The aim of MGW pooling is to evenly distribute the RTP voice stream load across
multiple MGW instances. This can help to scale out over multiple VMs or physical
machines. Until osmo-mgw includes multithreading support, it may also be used to
scale-out to multiple cores on a single host.

The load distribution is managed in such a way that when a new call is placed,
the pool will automatically assign the call to the available MGW with the lowest load.

MGW pooling is recommended for larger RAN or CN installations. For small networks
and lab installations the classic method with one MGW per {program-name} offers
sufficient performance.

=== MGW pool VTY configuration

In {program-name} the MGW is controlled via an MGCP-Client. The VTY commands to
configure the MGCP-Client are part of the several 'mgw' nodes, one per
MGCP-Client to set up.

To setup an MGW pool, the user must first install multiple OsmoMGW instances, so
that they won’t interfere with each other. This can be done using different
local host IP addresses or different ports. When OsmoMGW is installed from
packages, the systemd configuration may also require adjustment.

By default, MGCP-Client will use whatever source IP address is resolved by the
kernel routing subsystem, and will also ask the kernel to pick a free UDP port.

Example configuration with two MGCP-Client instances in a pool:
----
 mgw 0
  description media-gw-0 <2>
  remote-ip 127.0.0.1
  remote-port 2432
  local-ip 127.0.0.1
  local-port 2431
  endpoint-domain mgw0 <1>
 mgw 1
  description media-gw-1 <2>
  remote-ip 127.0.0.1
  remote-port 2430
  local-ip 127.0.0.1
  local-port 2429
  endpoint-domain mgw1 <1>
----

<1> When working with multiple MGW / MGCP-Client instances, the domain name for
each MGW should be different. Otherwise it won't be possible to distinguish the
endpoint names in the log. It should also be noted that the domain name must
match the configuration of the related OsmoMGW instance.

<2> It is also possible to assign a descriptive name to each MGW instance. The
MGCP client specific log lines will then use this name as logging context. If
no description is set, the domain name will be used.

=== MGW pool management

The MGW pool is fully runtime-manageable. The only limitation
is that an MGCP-Client can not be restarted or removed as long as it is serving
calls (see also: <<mgw_pooling_blocking>>).

==== MGW pool status

The VTY implements a 'show mgw-pool' command that lists the currently configured
MGW pool members, their status and call utilization. The following snippet shows
an output example run on OsmoBSC, but it should be also available on other
applications supporting the MGW pooling VTY features:

----
OsmoBSC> show mgw-pool
% MGW-Pool:
%  MGW 0:media-gw-0
%   MGCP link:     connected,UP
%   service:       unblocked
%   ongoing calls: 1
%  MGW 1:media-gw-1
%   MGCP link:     connected,UP
%   service:       unblocked
%   ongoing calls: 0
----

==== Adding an MGW / MGCP-Client to the MGW pool

To add a new MGCP-Client to the pool, the 'mgw' node is used. Like with the
'bts' or the 'msc' node a reference number is used that usually starts at 0.
However it is still possible to assign any number from 0-255. The enumeration
also may contain gaps. The following snippet shows an output example run on
OsmoBSC, but it should be also available on other applications supporting the
MGW pooling VTY features:

----
OsmoBSC> enable
OsmoBSC# configure terminal
OsmoBSC(config)# network
OsmoBSC(config-net)# mgw 2
OsmoBSC(config-mgw)# ?
  keepalive        Monitor if the MGCP link against MGW is still usable
  local-ip         local bind to connect to MGW from
  local-port       local port to connect to MGW from
  remote-ip        remote IP address to reach the MGW at
  remote-port      remote port to reach the MGW at
  endpoint-domain  Set the domain name to send in MGCP messages, e.g. the part 'foo' in 'rtpbridge/*@foo'.
  reset-endpoint   Add an endpoint name that should be reset (DLCX) on connect to the reset-endpoint list,e.g. 'rtpbridge/*'
----

The newly added MGW will immediately appear in the mgw-pool list but it won't
be used until its configuration finished by reconnecting it.

----
% MGW-Pool:
%  MGW 0:media-gw-0
%   MGCP link:     connected,UP
%   service:       unblocked
%   ongoing calls: 2
%  MGW 1:media-gw-1
%   MGCP link:     connected,UP
%   service:       unblocked
%   ongoing calls: 3
%  MGW 2:mgw <1>
%   MGCP link:     disconnected,DOWN
%   service:       unblocked
%   ongoing calls: 0
----

<1> In this example a description is not set yet, so the domain name ("mgw")
is displayed.

==== Reconnecting an MGW / MGCP-Client

It may become necessary to reconnect an MGCP-Client. This is the case when the
VTY configuration was changed at runtime. In order to make the changes effective
the MGW configuration must be reloaded by reconnecting the MGW connection. Also
newly created MGW instances require a reconnect once their configuration is
done.

To reconnect an MGCP-Client use the 'reconnect' VTY command:
----
OsmoBSC# mgw 2 reconnect
----

The mgcp-client status should immediately change to 'connected'. The MGW is now
ready to be used for new calls.

----
OsmoBSC# show mgw-pool
% MGW-Pool:
%  MGW 0:media-gw-0
%   MGCP link:     connected,UP
%   service:       unblocked
%   ongoing calls: 2
%  MGW 1:media-gw-1
%   MGCP link:     connected,UP
%   service:       unblocked
%   ongoing calls: 3
%  MGW 2:mgw
%   MGCP link:     connected,UP
%   service:       unblocked
%   ongoing calls: 0
----

It should be noted that MGCP a protocol is used via UDP, the connect only
happens locally to forward the UDP datagrams properly (state printed in
`mgcp-client: (dis)connected` above). Also (unless a reset endpoint is
configured like in the example config above) there will be no immediate
interaction with the MGW. However, the log should at least confirm the connect
worked and the MGCP client has been created successfully.

----
Mon Aug  2 17:15:00 2021 DLMGCP mgcp_client.c:788 MGCP client: using endpoint domain '@mgw'
Mon Aug  2 17:15:00 2021 DLMGCP mgcp_client.c:908 MGCP GW connection: r=127.0.0.1:2427<->l=127.0.0.1:2727
----

For that reason, it is strongly advised to enable the `keepalive` feature in
{program-name} to schedule periodical MGCP queries against the MGW, in order to
make sure it is reachable (the state `MGCP link: UP|DOWN` printed above). See
section <<mgw_pooling_keepalive>> below for more information.

[[mgw_pooling_keepalive]]
==== Monitor MGCP link (keepalive) / MGCP-Client

The `keepalive` feature in {program-name} allows scheduling periodical queries
on the MGCP layer in order to make sure it is reachable and hence obtain
information on the state of the MGCP link. This is in turn used by the MGW Pool
when picking an MGW from the pool: MGWs whose link is considered to be DOWN are
skipped.

The feature consists of:
- A `keepalive request-interval` which will trigger a transmission of an MGCP
AuditEndpoint command targeting endpoint with name `keepalive request-endpoint`.
This interval is updated every time any message is transmitted in the MGCP link,
meaning the MGCP AuditEndpoint message is only triggered if no message has been
transmitted since `keepalive request-interval` seconds ago.
- A `keepalive timeout` which upon triggering (because no message was received
over that amount of time) will then consider the MGW to be non-reachable (link
DOWN).

The `keepalive` parameters are to be preferrably configured so that
`"keepalive request-interval" * 2 < "keepalive timeout"`.

Example VTY configuration of `keepalive` feature in {program-name}:
----
 mgw 0
  ...
  keepalive request-interval 20 <1>
  keepalive request-endpoint null <2>
  keepalive timeout 50 <3>
----

<1> Transmit an MGCP AuditEndpoint message to the MGW if no message has been
sent to it over last 20 seconds
<2> The MGCP AuditEndpoint targets the `null` endpoint. This is a special
endpoint available at OsmoMGW for those purposes, but any available endpoint can
be configured and used instead.
<3> Consider the MGCP link to be DOWN if no message is received from the MGW
over the last 50 seconds

NOTE: The `keepalive` feature is disabled by default, and must be explicitly
configured in order to enable it.

[[mgw_pooling_blocking]]
==== Blocking an MGW / MGCP-Client

If it becomes apparent that an MGCP-Client must be restarted or removed from
the config (maintenance) the operator can put that MGCP-Client into a blocked
mode. A blocked MGCP-Client will still serve the ongoing calls but it will not
be picked for the assignment of new calls.

To block an MGCP-Client use the 'block' VTY command:
----
OsmoBSC# mgw 2 block
OsmoBSC# show mgw-pool
% MGW-Pool:
%  MGW 0:media-gw-0
%   MGCP link:     connected,UP
%   service:       unblocked
%   ongoing calls: 11
%  MGW 1:media-gw-1
%   MGCP link:     connected,UP
%   service:       unblocked
%   ongoing calls: 12
%  MGW 2:mgw
%   MGCP link:     connected,UP
%   service:       blocked
%   ongoing calls: 10
----

When the number of ongoing calls has tapered off, the MGW / MGCP-Client can be
restarted or removed if necessary.

----
OsmoBSC# show mgw-pool
% MGW-Pool:
%  MGW 0:media-gw-0
%   MGCP link:     connected,UP
%   service:       unblocked
%   ongoing calls: 15
%  MGW 1:media-gw-1
%   MGCP link:     connected,UP
%   service:       unblocked
%   ongoing calls: 14
%  MGW 2:mgw
%   MGCP link:     connected,UP
%   service:       blocked
%   ongoing calls: 0
----

If the blockade should be reverted, the 'unblock' VTY command can be used in
the same way to remove the blockade. (Reconnecting will not remove the
blockade.)

==== Removing an MGW / MGCP-Client

An MGCP-Client is removed from the pool using the 'no mgw' command from the
configure terminal. The MGCP-Client instance will automatically be terminated
and the related resources are freed. The only requirement is that there are no
ongoing calls on the selected instance.

----
OsmoBSC# configure terminal
OsmoBSC(config)# network
OsmoBSC(config-net)# no mgw 2
----
