openwisp-monitoring 0.1

UNKNOWN

Whenever a new device is created it will have UNKNOWN as it’s default Heath Status.

It implies that the system doesn’t know whether the device is reachable yet.

OK

Everything is working normally.

PROBLEM

One of the metrics has a value which is not in the expected range (the threshold value set in the alert settings has been crossed).

Example: CPU usage should be less than 90% but current value is at 95%.

CRITICAL

One of the metrics defined in OPENWISP_MONITORING_CRITICAL_DEVICE_METRICS has a value which is not in the expected range (the threshold value set in the alert settings has been crossed).

Example: ping is by default a critical metric which is expected to be always 1 (reachable).

Device Status

This metric stores the status of the device for viewing purposes.

Ping

Uptime:

Packet loss:

Round Trip Time:

Traffic

WiFi Clients

Memory Usage

CPU Load

Disk Usage

Ping

This check returns information on device uptime and RTT (Round trip time). The Charts uptime, packet loss and rtt are created. The fping command is used to collect these metrics. You may choose to disable auto creation of this check by setting OPENWISP_MONITORING_AUTO_PING to False.

Configuration applied

This check ensures that the openwisp-config agent is running and applying configuration changes in a timely manner. You may choose to disable auto creation of this check by using the setting OPENWISP_MONITORING_AUTO_DEVICE_CONFIG_CHECK.

OPENWISP_MONITORING_SHORT_RETENTION_POLICY

The default retention policy used to store raw device data.

This data is only used to assess the recent status of devices, keeping it for a long time would not add much benefit and would cost a lot more in terms of disk space.

OPENWISP_MONITORING_AUTO_PING

Whether ping checks are created automatically for devices.

OPENWISP_MONITORING_AUTO_DEVICE_CONFIG_CHECK

This setting allows you to choose whether config_applied checks should be created automatically for newly registered devices. It’s enabled by default.

OPENWISP_MONITORING_AUTO_CHARTS

Automatically created charts.

OPENWISP_MONITORING_CRITICAL_DEVICE_METRICS

Device metrics that are considered critical:

when a value crosses the boundary defined in the “threshold value” field of the alert settings related to one of these metric types, the health status of the device related to the metric moves into CRITICAL.

By default, if devices are not reachable by pings they are flagged as CRITICAL.

OPENWISP_MONITORING_HEALTH_STATUS_LABELS

This setting allows to change the health status labels, for example, if we want to use online instead of ok and offline instead of critical, you can use the following configuration:

OPENWISP_MONITORING_HEALTH_STATUS_LABELS = {
    'ok': 'online',
    'problem': 'problem',
    'critical': 'offline'
}

OPENWISP_MONITORING_MANAGEMENT_IP_ONLY

By default, only the management IP will be used to perform active checks to the devices.

If the devices are connecting to your OpenWISP instance using a shared layer2 network, hence the OpenWSP server can reach the devices using the last_ip field, you can set this to False.

OPENWISP_MONITORING_DEVICE_RECOVERY_DETECTION

When device recovery detection is enabled, recoveries are discovered as soon as a device contacts the openwisp system again (eg: to get the configuration checksum or to send monitoring metrics).

This feature is enabled by default.

If you use OpenVPN as the management VPN, you may want to check out a similar integration built in openwisp-network-topology: when the status of an OpenVPN link changes (detected by monitoring the status information of OpenVPN), the network topology module will trigger the monitoring checks. For more information see: Network Topology Device Integration

OPENWISP_MONITORING_MAC_VENDOR_DETECTION

Indicates whether mac addresses will be complemented with hardware vendor information by performing lookups on the OUI (Organization Unique Identifier) table.

This feature is enabled by default.

OPENWISP_MONITORING_WRITE_RETRY_OPTIONS

# default value of OPENWISP_MONITORING_RETRY_OPTIONS:

dict(
    max_retries=None,
    retry_backoff=True,
    retry_backoff_max=600,
    retry_jitter=True,
)

Retry settings for recoverable failures during metric writes.

By default if a metric write fails (eg: due to excessive load on timeseries database at that moment) then the operation will be retried indefinitely with an exponential random backoff and a maximum delay of 10 minutes.

This feature makes the monitoring system resilient to temporary outages and helps to prevent data loss.

For more information regarding these settings, consult the celery documentation regarding automatic retries for known errors.

OPENWISP_MONITORING_METRICS

This setting allows to define additional metric configuration or to override the default metric configuration defined in openwisp_monitoring.monitoring.configuration.DEFAULT_METRICS.

For example, if you want to change only the field_name of clients metric to wifi_clients you can use:

from django.utils.translation import gettext_lazy as _

OPENWISP_MONITORING_METRICS = {
    'clients': {
        'label': _('WiFi clients'),
        'field_name': 'wifi_clients',
    },
}

For example, if you want to change only the default alert settings of memory metric you can use:

OPENWISP_MONITORING_METRICS = {
    'memory': {
        'alert_settings': {'threshold': 75, 'tolerance': 5}
    },
}

For example, if you want to change only the notification of config_applied metric you can use:

from django.utils.translation import gettext_lazy as _

OPENWISP_MONITORING_METRICS = {
    'config_applied': {
        'notification': {
            'problem': {
                'verbose_name': 'Configuration PROBLEM',
                'verb': _('has not been applied'),
                'email_subject': _(
                    '[{site.name}] PROBLEM: {notification.target} configuration '
                    'status issue'
                ),
                'message': _(
                    'The configuration for device [{notification.target}]'
                    '({notification.target_link}) {notification.verb} in a timely manner.'
                ),
            },
            'recovery': {
                'verbose_name': 'Configuration RECOVERY',
                'verb': _('configuration has been applied again'),
                'email_subject': _(
                    '[{site.name}] RECOVERY: {notification.target} {notification.verb} '
                    'successfully'
                ),
                'message': _(
                    'The device [{notification.target}]({notification.target_link}) '
                    '{notification.verb} successfully.'
                ),
            },
        },
    },
}

Or if you want to define a new metric configuration, which you can then call in your custom code (eg: a custom check class), you can do so as follows:

from django.utils.translation import gettext_lazy as _

OPENWISP_MONITORING_METRICS = {
    'top_fields_mean': {
        'name': 'Top Fields Mean',
        'key': '{key}',
        'field_name': '{field_name}',
        'label': '_(Top fields mean)',
        'related_fields': ['field1', 'field2', 'field3'],
    },
}

OPENWISP_MONITORING_CHARTS

This setting allows to define additional charts or to override the default chart configuration defined in openwisp_monitoring.monitoring.configuration.DEFAULT_CHARTS.

For example, if you want to change the traffic chart to show MB (megabytes) instead of GB (Gigabytes) you can use:

OPENWISP_MONITORING_CHARTS = {
    'traffic': {
        'unit': ' MB',
        'description': (
            'Network traffic, download and upload, measured on '
            'the interface "{metric.key}", measured in MB.'
        ),
        'query': {
            'influxdb': (
                "SELECT SUM(tx_bytes) / 1000000 AS upload, "
                "SUM(rx_bytes) / 1000000 AS download FROM {key} "
                "WHERE time >= '{time}' AND content_type = '{content_type}' "
                "AND object_id = '{object_id}' GROUP BY time(1d)"
            )
        },
    }
}

Or if you want to define a new chart configuration, which you can then call in your custom code (eg: a custom check class), you can do so as follows:

from django.utils.translation import gettext_lazy as _

OPENWISP_MONITORING_CHARTS = {
    'ram': {
        'type': 'line',
        'title': 'RAM usage',
        'description': 'RAM usage',
        'unit': 'bytes',
        'order': 100,
        'query': {
            'influxdb': (
                "SELECT MEAN(total) AS total, MEAN(free) AS free, "
                "MEAN(buffered) AS buffered FROM {key} WHERE time >= '{time}' AND "
                "content_type = '{content_type}' AND object_id = '{object_id}' "
                "GROUP BY time(1d)"
            )
        },
    }
}

In case you just want to change the colors used in a chart here’s how to do it:

OPENWISP_MONITORING_CHARTS = {
    'traffic': {
        'colors': ['#000000', '#cccccc']
    }
}

register_metric

This function is used to register a new metric configuration from anywhere in your code.

An example usage has been shown below.

from django.utils.translation import gettext_lazy as _
from openwisp_monitoring.monitoring.configuration import register_metric

# Define configuration of your metric
metric_config = {
    'label': _('Ping'),
    'name': 'Ping',
    'key': 'ping',
    'field_name': 'reachable',
    'related_fields': ['loss', 'rtt_min', 'rtt_max', 'rtt_avg'],
    'charts': {
        'uptime': {
            'type': 'bar',
            'title': _('Uptime'),
            'description': _(
                'A value of 100% means reachable, 0% means unreachable, values in '
                'between 0% and 100% indicate the average reachability in the '
                'period observed. Obtained with the fping linux program.'
            ),
            'summary_labels': [_('Average uptime')],
            'unit': '%',
            'order': 200,
            'colorscale': {
                'max': 100,
                'min': 0,
                'label': _('Reachable'),
                'scale': [
                    [0, '#c13000'],
                    [0.5, '#deed0e'],
                    [1, '#7db201'],
                ],
                'map': [
                    [100, '#7db201', _('Reachable')],
                    [33, '#deed0e', _('Partly reachable')],
                    [None, '#c13000', _('Unreachable')],
                ],
                'fixed_value': 100,
            },
            'query': chart_query['uptime'],
        },
        'packet_loss': {
            'type': 'bar',
            'title': _('Packet loss'),
            'description': _(
                'Indicates the percentage of lost packets observed in ICMP probes. '
                'Obtained with the fping linux program.'
            ),
            'summary_labels': [_('Average packet loss')],
            'unit': '%',
            'colors': '#d62728',
            'order': 210,
            'query': chart_query['packet_loss'],
        },
        'rtt': {
            'type': 'scatter',
            'title': _('Round Trip Time'),
            'description': _(
                'Round trip time observed in ICMP probes, measuered in milliseconds.'
            ),
            'summary_labels': [
                _('Average RTT'),
                _('Average Max RTT'),
                _('Average Min RTT'),
            ],
            'unit': _(' ms'),
            'order': 220,
            'query': chart_query['rtt'],
        },
    },
    'alert_settings': {'operator': '<', 'threshold': 1, 'tolerance': 0},
    'notification': {
        'problem': {
            'verbose_name': 'Ping PROBLEM',
            'verb': 'cannot be reached anymore',
            'level': 'warning',
            'email_subject': _(
                '[{site.name}] {notification.target} is not reachable'
            ),
            'message': _(
                'The device [{notification.target}] {notification.verb} anymore by our ping '
                'messages.'
            ),
        },
        'recovery': {
            'verbose_name': 'Ping RECOVERY',
            'verb': 'has become reachable',
            'level': 'info',
            'email_subject': _(
                '[{site.name}] {notification.target} is reachable again'
            ),
            'message': _(
                'The device [{notification.target}] {notification.verb} again by our ping '
                'messages.'
            ),
        },
    },
}

# Register your custom metric configuration
register_metric('ping', metric_config)

The above example will register one metric configuration (named ping), three chart configurations (named rtt, packet_loss, uptime) as defined in the charts key, two notification types (named ping_recovery, ping_problem) as defined in notification key.

The AlertSettings of ping metric will by default use threshold and tolerance defined in the alert_settings key. You can always override them and define your own custom values via the admin.

Note: It will raise ImproperlyConfigured exception if a metric configuration is already registered with same name (not to be confused with verbose_name).

If you don’t need to register a new metric but need to change a specific key of an existing metric configuration, you can use OPENWISP_MONITORING_METRICS.

unregister_metric

This function is used to unregister a metric configuration from anywhere in your code.

An example usage is shown below.

from openwisp_monitoring.monitoring.configuration import unregister_metric

# Unregister previously registered metric configuration
unregister_metric('metric_name')

Note: It will raise ImproperlyConfigured exception if the concerned metric configuration is not registered.

register_chart

This function is used to register a new chart configuration from anywhere in your code.

An example usage has been shown below.

from openwisp_monitoring.monitoring.configuration import register_chart

# Define configuration of your chart
chart_config = {
    'type': 'histogram',
    'title': 'Histogram',
    'description': 'Histogram',
    'top_fields': 2,
    'order': 999,
    'query': {
        'influxdb': (
            "SELECT {fields|SUM|/ 1} FROM {key} "
            "WHERE time >= '{time}' AND content_type = "
            "'{content_type}' AND object_id = '{object_id}'"
        )
    },
}

# Register your custom chart configuration
register_chart('chart_name', chart_config)

Note: It will raise ImproperlyConfigured exception if a chart configuration is already registered with same name (not to be confused with verbose_name).

If you don’t need to register a new chart but need to change a specific key of an existing chart configuration, you can use OPENWISP_MONITORING_CHARTS.

unregister_chart

This function is used to unregister a chart configuration from anywhere in your code.

An example usage is shown below.

from openwisp_monitoring.monitoring.configuration import unregister_chart

# Unregister previously registered chart configuration
unregister_chart('chart_name')

Note: It will raise ImproperlyConfigured exception if the concerned chart configuration is not registered.

TimeseriesWriteException

Path: openwisp_monitoring.db.exceptions.TimeseriesWriteException

If there is any failure due while writing data in timeseries database, this exception shall be raised with a helpful error message explaining the cause of the failure. This exception will normally be caught and the failed write task will be retried in the background so that there is no loss of data if failures occur due to overload of Timeseries server. You can read more about this retry mechanism at OPENWISP_MONITORING_WRITE_RETRY_OPTIONS.

InvalidMetricConfigException

Path: openwisp_monitoring.monitoring.exceptions.InvalidMetricConfigException

This exception shall be raised if the metric configuration is broken.

InvalidChartConfigException

Path: openwisp_monitoring.monitoring.exceptions.InvalidChartConfigException

This exception shall be raised if the chart configuration is broken.

Live documentation

A general live API documentation (following the OpenAPI specification) at /api/v1/docs/.

Browsable web interface

Additionally, opening any of the endpoints listed below directly in the browser will show the browsable API interface of Django-REST-Framework, which makes it even easier to find out the details of each endpoint.

List of endpoints

Since the detailed explanation is contained in the Live documentation and in the Browsable web page of each point, here we’ll provide just a list of the available endpoints, for further information please open the URL of the endpoint in your browser.

GET /v1/monitoring/device/{pk}/?key={key}&status=true

POST /v1/monitoring/device/{pk}/?key={key}

device_metrics_received

Path: openwisp_monitoring.device.signals.device_metrics_received

Arguments:

• instance: instance of Device whose metrics have been received

• request: the HTTP request object

This signal is emitted when device metrics are received to the DeviceMetric view (only when using HTTP POST).

The signal is emitted just before a successful response is returned, it is not sent if the response was not successful.

health_status_changed

Path: openwisp_monitoring.device.signals.health_status_changed

Arguments:

• instance: instance of DeviceMonitoring whose status has been changed

• status: the status by which DeviceMonitoring’s existing status has been updated with

This signal is emitted only if the health status of DeviceMonitoring object gets updated.

threshold_crossed

Path: openwisp_monitoring.monitoring.signals.threshold_crossed

Arguments:

• metric: Metric object whose threshold defined in related alert settings was crossed

• alert_settings: AlertSettings related to the Metric

• target: related Device object

• first_time: it will be set to true when the metric is written for the first time. It shall be set to false afterwards.

first_time parameter can be used to avoid initiating unneeded actions. For example, sending recovery notifications.

This signal is emitted when the threshold value of a Metric defined in alert settings is crossed.

pre_metric_write

Path: openwisp_monitoring.monitoring.signals.pre_metric_write

Arguments:

• metric: Metric object whose data shall be stored in timeseries database

• values: metric data that shall be stored in the timeseries database

This signal is emitted for every metric before the write operation is sent to the timeseries database.

post_metric_write

Path: openwisp_monitoring.monitoring.signals.post_metric_write

Arguments:

• metric: Metric object whose data is being stored in timeseries database

• values: metric data that is being stored in the timeseries database

This signal is emitted for every metric after the write operation is successfully executed in the background.

1. Initialize your custom module

The first thing you need to do in order to extend any openwisp-monitoring app is create a new django app which will contain your custom version of that openwisp-monitoring app.

A django app is nothing more than a python package (a directory of python scripts), in the following examples we’ll call these django apps as mycheck, mydevicemonitoring, mymonitoring but you can name it how you want:

django-admin startapp mycheck
django-admin startapp mydevicemonitoring
django-admin startapp mymonitoring

Keep in mind that the command mentioned above must be called from a directory which is available in your PYTHON_PATH so that you can then import the result into your project.

Now you need to add mycheck to INSTALLED_APPS in your settings.py, ensuring also that openwisp_monitoring.check has been removed:

INSTALLED_APPS = [
    # ... other apps ...
    # 'openwisp_monitoring.check',        <-- comment out or delete this line
    # 'openwisp_monitoring.device',       <-- comment out or delete this line
    # 'openwisp_monitoring.monitoring'    <-- comment out or delete this line
    'mycheck',
    'mydevicemonitoring',
    'mymonitoring',
]

For more information about how to work with django projects and django apps, please refer to the “Tutorial: Writing your first Django app” in the django docunmentation.

2. Install openwisp-monitoring

Install (and add to the requirement of your project) openwisp-monitoring:

pip install --U https://github.com/openwisp/openwisp-monitoring/tarball/master

3. Add EXTENDED_APPS

Add the following to your settings.py:

EXTENDED_APPS = ['device_monitoring', 'monitoring', 'check']

4. Add openwisp_utils.staticfiles.DependencyFinder

Add openwisp_utils.staticfiles.DependencyFinder to STATICFILES_FINDERS in your settings.py:

STATICFILES_FINDERS = [
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
    'openwisp_utils.staticfiles.DependencyFinder',
]

5. Add openwisp_utils.loaders.DependencyLoader

Add openwisp_utils.loaders.DependencyLoader to TEMPLATES in your settings.py:

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'OPTIONS': {
            'loaders': [
                'django.template.loaders.filesystem.Loader',
                'django.template.loaders.app_directories.Loader',
                'openwisp_utils.loaders.DependencyLoader',
            ],
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    }
]

6. Inherit the AppConfig class

Please refer to the following files in the sample app of the test project:

• sample_check/__init__.py.

• sample_check/apps.py.

• sample_monitoring/__init__.py.

• sample_monitoring/apps.py.

• sample_device_monitoring/__init__.py.

• sample_device_monitoring/apps.py.

For more information regarding the concept of AppConfig please refer to the “Applications” section in the django documentation.

7. Create your custom models

To extend check app, refer to sample_check models.py file.

To extend monitoring app, refer to sample_monitoring models.py file.

To extend device_monitoring app, refer to sample_device_monitoring models.py file.

Note:

• For doubts regarding how to use, extend or develop models please refer to the “Models” section in the django documentation.

• For doubts regarding proxy models please refer to proxy models.

8. Add swapper configurations

Add the following to your settings.py:

# Setting models for swapper module
# For extending check app
CHECK_CHECK_MODEL = 'YOUR_MODULE_NAME.Check'
# For extending monitoring app
MONITORING_CHART_MODEL = 'YOUR_MODULE_NAME.Chart'
MONITORING_METRIC_MODEL = 'YOUR_MODULE_NAME.Metric'
MONITORING_ALERTSETTINGS_MODEL = 'YOUR_MODULE_NAME.AlertSettings'
# For extending device_monitoring app
DEVICE_MONITORING_DEVICEDATA_MODEL = 'YOUR_MODULE_NAME.DeviceData'
DEVICE_MONITORING_DEVICEMONITORING_MODEL = 'YOUR_MODULE_NAME.DeviceMonitoring'

Substitute <YOUR_MODULE_NAME> with your actual django app name (also known as app_label).

9. Create database migrations

Create and apply database migrations:

./manage.py makemigrations
./manage.py migrate

For more information, refer to the “Migrations” section in the django documentation.

10. Create your custom admin

To extend check app, refer to sample_check admin.py file.

To extend monitoring app, refer to sample_monitoring admin.py file.

To extend device_monitoring app, refer to sample_device_monitoring admin.py file.

To introduce changes to the admin, you can do it in the two ways described below.

Note: for doubts regarding how the django admin works, or how it can be customized, please refer to “The django admin site” section in the django documentation.

from openwisp_monitoring.check.admin import CheckAdmin

CheckAdmin.list_display.insert(1, 'my_custom_field')
CheckAdmin.ordering = ['-my_custom_field']

from openwisp_monitoring.device.admin import DeviceAdmin

DeviceAdmin.list_display.insert(1, 'my_custom_field')
DeviceAdmin.ordering = ['-my_custom_field']

from openwisp_monitoring.monitoring.admin import MetricAdmin, AlertSettingsAdmin

MetricAdmin.list_display.insert(1, 'my_custom_field')
MetricAdmin.ordering = ['-my_custom_field']
AlertSettingsAdmin.list_display.insert(1, 'my_custom_field')
AlertSettingsAdmin.ordering = ['-my_custom_field']

from django.contrib import admin

from openwisp_monitoring.check.admin import CheckAdmin as BaseCheckAdmin
from swapper import load_model

Check = load_model('check', 'Check')

admin.site.unregister(Check)

@admin.register(Check)
class CheckAdmin(BaseCheckAdmin):
    # add your changes here

from django.contrib import admin

from openwisp_monitoring.device_monitoring.admin import DeviceAdmin as BaseDeviceAdmin
from swapper import load_model

Device = load_model('config', 'Device')

admin.site.unregister(Device)

@admin.register(Device)
class DeviceAdmin(BaseDeviceAdmin):
    # add your changes here

from django.contrib import admin

from openwisp_monitoring.monitoring.admin import (
    AlertSettingsAdmin as BaseAlertSettingsAdmin,
    MetricAdmin as BaseMetricAdmin
)
from swapper import load_model

Metric = load_model('Metric')
AlertSettings = load_model('AlertSettings')

admin.site.unregister(Metric)
admin.site.unregister(AlertSettings)

@admin.register(Metric)
class MetricAdmin(BaseMetricAdmin):
    # add your changes here

@admin.register(AlertSettings)
class AlertSettingsAdmin(BaseAlertSettingsAdmin):
    # add your changes here

11. Create root URL configuration

Please refer to the urls.py file in the test project.

For more information about URL configuration in django, please refer to the “URL dispatcher” section in the django documentation.

12. Create celery.py

Please refer to the celery.py file in the test project.

For more information about the usage of celery in django, please refer to the “First steps with Django” section in the celery documentation.

13. Import Celery Tasks

Add the following in your settings.py to import celery tasks from device_monitoring app.

CELERY_IMPORTS = ('openwisp_monitoring.device.tasks',)

14. Create the custom command run_checks

Please refer to the run_checks.py file in the test project.

For more information about the usage of custom management commands in django, please refer to the “Writing custom django-admin commands” section in the django documentation.

15. Import the automated tests

When developing a custom application based on this module, it’s a good idea to import and run the base tests too, so that you can be sure the changes you’re introducing are not breaking some of the existing features of openwisp-monitoring.

In case you need to add breaking changes, you can overwrite the tests defined in the base classes to test your own behavior.

For, extending check app see the tests of sample_check app to find out how to do this.

For, extending device_monitoring app see the tests of sample_device_monitoring app to find out how to do this.

For, extending monitoring app see the tests of sample_monitoring app to find out how to do this.

Other base classes that can be inherited and extended

The following steps are not required and are intended for more advanced customization.

# mydevice/api/views.py
from openwisp_monitoring.device.api.views import (
    DeviceMetricView as BaseDeviceMetricView
)

class DeviceMetricView(BaseDeviceMetricView):
    # add your customizations here ...
    pass

url(
    r'^api/v1/monitoring/device/(?P<pk>[^/]+)/$',
    views.device_metric,
    name='api_device_metric',
),

# urls.py
from mydevice.api.views import DeviceMetricView

urlpatterns = [
    # ... other URLs
    url(r'^(?P<path>.*)$', DeviceMetricView.as_view(), name='api_device_metric',),
]