openwisp-network-topology 1.0

Deploy it in production

An automated installer is provided by the OpenWISP project: ansible-openwisp2.

Ensure to follow the instructions explained in the following section: Enabling the network topology module.

Install stable version from pypi

Install from pypi:

pip install openwisp-network-topology

Install development version

Install tarball:

pip install https://github.com/openwisp/openwisp-network-topology/tarball/master

Alternatively you can install via pip using git:

pip install -e git+git://github.com/openwisp/openwisp-network-topology#egg=openwisp-network-topology

If you want to contribute, install your cloned fork:

git clone git@github.com:<your_fork>/openwisp-network-topology.git
cd openwisp-network-topology
python setup.py develop

Installing for development

Install sqlite:

sudo apt-get install sqlite3 libsqlite3-dev

Install your forked repo:

git clone git://github.com/<your_fork>/openwisp-network-topology
cd openwisp-network-topology/
python setup.py develop

Install test requirements:

pip install -r requirements-test.txt

Create database:

cd tests/
./manage.py migrate
./manage.py createsuperuser

You can access the admin interface at http://127.0.0.1:8000/admin/.

Run tests with:

./runtests.py

Run qa tests:

./run-qa-checks

Setup (integrate in an existing django project)

Add openwisp_network_topology and its dependencies to INSTALLED_APPS:

INSTALLED_APPS = [
    # other apps
    'openwisp_network_topology',
    'openwisp_users.accounts',
    'allauth',
    'allauth.account',
    'openwisp_users',
    'rest_framework',
]

Add the URLs to your main urls.py:

from django.contrib import admin

urlpatterns = [
    # ... other urls in your project ...
    path('', include('openwisp_network_topology.urls')),
    path('admin/', admin.site.urls),
]

Then run:

./manage.py migrate

Creating a topology

1. Create a topology object by going to Network Topology > Topologies > Add topology.

2. Give an appropriate label to the topology.

3. Select the topology format from the dropdown menu. The topology format determines which parser should be used to process topology data.

4. Select the Strategy for updating this topology.

   • If you are using FETCH strategy, then enter the URL for fetching topology data in the Url field.

   • If you are using RECEIVE strategy, you will get the URL for sending topology data. The RECEIVE strategy provides an additional field expiration time. This can be used to add delay in marking missing links as down.

Sending data for topology with RECEIVE strategy

1. Copy the URL generated by OpenWISP for sending the topology data.

   E.g., in our case the URL is http://127.0.0.1:8000/api/v1/network-topology/topology/d17e539a-1793-4be2-80a4-c305eca64fd8/receive/?key=cMGsvio8q0L0BGLd5twiFHQOqIEKI423.

2. Create a script (eg: /opt/send-topology.sh) which sends the topology data using POST, in the example script below we are sending the status log data of OpenVPN but the same code can be applied to other formats by replacing cat /var/log/openvpn/tun0.stats with the actual command which returns the network topology output:

#!/bin/bash

# Get OpenVPN topology data from OpenVPN management interface
cat /var/log/openvpn/tun0.stats |
    # Upload the topology data to OpenWISP
    curl -s -X POST \
        --data-binary @- \
        --header "Content-Type: text/plain" \
        http://127.0.0.1:8000/api/v1/network-topology/topology/d17e539a-1793-4be2-80a4-c305eca64fd8/receive/?key=cMGsvio8q0L0BGLd5twiFHQOqIEKI423

3. Add the /opt/send-topology.sh script created in the previous step to the crontab, here’s an example which sends the topology data every 5 minutes:

# flag script as executable
chmod +x /opt/send-topology.sh
# open crontab
crontab -e

## Add the following line and save

echo */5 * * * * /opt/send-topology.sh

4. Once the steps above are completed, you should see nodes and links being created automatically, you can see the network topology graph from the admin page of the topology change page (you have to click on the View topology graph button in the upper right part of the page) or, alternatively, a non-admin visualizer page is also available at the URL /topology/topology/<TOPOLOGY-UUID>/.

update_topology

After topology URLs (URLs exposing the files that the topology of the network) have been added in the admin, the update_topology management command can be used to collect data and start playing with the network graph:

./manage.py update_topology

The management command accepts a --label argument that will be used to search in topology labels, eg:

./manage.py update_topology --label mytopology

save_snapshot

The save_snapshot management command can be used to save the topology graph data which could be used to view the network topology graph sometime in future:

./manage.py save_snapshot

The management command accepts a --label argument that will be used to search in topology labels, eg:

./manage.py save_snapshot --label mytopology

upgrade_from_django_netjsongraph

If you are upgrading from django-netjsongraph to openwisp-network-topology, there is an easy migration script that will import your topologies, users & groups to openwisp-network-topology instance:

./manage.py upgrade_from_django_netjsongraph

The management command accepts an argument --backup, that you can pass to give the location of the backup files, by default it looks in the tests/ directory, eg:

./manage.py upgrade_from_django_netjsongraph --backup /home/user/django_netjsongraph/

The management command accepts another argument --organization, if you want to import data to a specific organization, you can give its UUID for the same, by default the data is added to the first found organization, eg:

./manage.py upgrade_from_django_netjsongraph --organization 900856da-c89a-412d-8fee-45a9c763ca0b

Note: you can follow the tutorial to migrate database from django-netjsongraph.

create_device_nodes

This management command can be used to create the initial DeviceNode relationships when the integration with OpenWISP Controller is enabled in a pre-existing system which already has some devices and topology objects in its database.

./manage.py create_device_nodes

FETCH strategy

Topology data will be fetched from a URL.

When some links are not detected anymore they will be flagged as “down” straightaway.

RECEIVE strategy

Topology data is sent directly from one or more nodes of the network.

The collector waits to receive data in the payload of a POST HTTP request; when such a request is received, a key parameter it’s first checked against the Topology key.

If the request is authorized the collector proceeds to update the topology.

If the data is sent from one node only, it’s highly advised to set the expiration_time of the Topology instance to 0 (seconds), this way the system works just like in the FETCH strategy, with the only difference that the data is sent by one node instead of fetched by the collector.

If the data is sent from multiple nodes, you SHOULD set the expiration_time of the Topology instance to a value slightly higher than the interval used by nodes to send the topology, this way links will be flagged as “down” only if they haven’t been detected for a while. This mechanism allows to visualize the topology even if the network has been split in several parts, the disadvantage is that it will take a bit more time to detect links that go offline.

OPENWISP_NETWORK_TOPOLOGY_PARSERS

Additional custom netdiff parsers.

OPENWISP_NETWORK_TOPOLOGY_SIGNALS

String representing python module to import on initialization.

Useful for loading django signals or to define custom behaviour.

OPENWISP_NETWORK_TOPOLOGY_TIMEOUT

Timeout when fetching topology URLs.

OPENWISP_NETWORK_TOPOLOGY_LINK_EXPIRATION

If a link is down for more days than this number, it will be deleted by the update_topology management command.

Setting this to False will disable this feature.

OPENWISP_NETWORK_TOPOLOGY_VISUALIZER_CSS

Path of the visualizer css file. Allows customization of css according to user’s preferences.

OPENWISP_NETWORK_TOPOLOGY_NODE_EXPIRATION

If a node has not been modified since the days specified and if it has no links, it will be deleted by the update_topology management command. This depends on OPENWISP_NETWORK_TOPOLOGY_LINK_EXPIRATION being enabled. Replace False with an integer to enable the feature.

OPENWISP_NETWORK_TOPOLOGY_API_URLCONF

Use the urlconf option to change receive api url to point to another module, example, myapp.urls.

OPENWISP_NETWORK_TOPOLOGY_API_BASEURL

If you have a seperate instance of openwisp-network-topology on a different domain, you can use this option to change the base of the url, this will enable you to point all the API urls to your openwisp-network-topology API server’s domain, example value: https://mytopology.myapp.com.

OPENWISP_NETWORK_TOPOLOGY_API_AUTH_REQUIRED

When enabled, the API endpoints will only allow authenticated users who have the necessary permissions to access the objects which belong to the organizations the user manages.

Live documentation

A general live API documentation (following the OpenAPI specification) at /api/v1/docs/.

Browsable web interface

Additionally, opening any of the endpoints listed below directly in the browser will show the browsable API interface of Django-REST-Framework, which makes it even easier to find out the details of each endpoint.

List of endpoints

Since the detailed explanation is contained in the Live documentation and in the Browsable web page of each point, here we’ll provide just a list of the available endpoints, for further information please open the URL of the endpoint in your browser.

GET /api/v1/network-topology/topology/

/api/v1/network-topology/topology/?organization=371791ec-e3fe-4c9a-8972-3e8b882416f6&strategy=fetch

GET /api/v1/network-topology/topology/?include_unpublished=true

POST /api/v1/network-topology/topology/

GET /api/v1/network-topology/topology/{id}/

GET /api/v1/network-topology/topology/{id}/?include_unpublished=true

PUT /api/v1/network-topology/topology/{id}/

PATCH /api/v1/network-topology/topology/{id}/

DELETE /api/v1/network-topology/topology/{id}/

GET /api/v1/network-topology/topology/{id}/history/?date=2020-08-08

POST /api/v1/network-topology/topology/{id}/receive/

GET /api/v1/network-topology/link/

/api/v1/network-topology/link/?status=down&topology=7fce01bd-29c0-48b1-8fce-0508f2d75d36

POST /api/v1/network-topology/link/

GET /api/v1/network-topology/link/{id}/

PUT /api/v1/network-topology/link/{id}/

PATCH /api/v1/network-topology/link/{id}/

DELETE /api/v1/network-topology/link/{id}/

GET /api/v1/network-topology/node/

/api/v1/network-topology/node/?organization=371791ec-e3fe-4c9a-8972-3e8b882416f6&topology=7fce01bd-29c0-48b1-8fce-0508f2d75d36

POST /api/v1/network-topology/node/

GET /api/v1/network-topology/node/{id}/

PUT /api/v1/network-topology/node/{id}/

PATCH /api/v1/network-topology/node/{id}/

DELETE /api/v1/network-topology/node/{id}/

Example: overriding the <script> tag

Here’s a step by step guide on how to change the javascript options passed to netjsongraph.js, remember to replace <project_path> with the absolute filesytem path of your project.

Step 1: create a directory in <project_path>/templates/netjsongraph

Step 2: open your settings.py and edit the TEMPLATES['DIRS'] setting so that it looks like the following example:

# settings.py
TEMPLATES = [
    {
        'DIRS': [os.path.join(BASE_DIR, 'templates')],
        # ... all other lines have been omitted for brevity ...
    }
]

Step 3: create a new file named netjsongraph-script.html in the new <project_path>/templates/netjsongraph/ directory, eg:

<!-- <project_path>/templates/netjsongraph/netjsongraph-script.html -->
<script>
    window.__njg_el__ = window.__njg_el__ || "body";
    window.__njg_default_url__ = "{{ graph_url }}";
    window.loadNetJsonGraph = function(graph){
        graph = graph || window.__njg_default_url__;
        d3.select("svg").remove();
        d3.select(".njg-overlay").remove();
        d3.select(".njg-metadata").remove();
        return d3.netJsonGraph(graph, {
            el: window.__njg_el__,
            // customizations of netjsongraph.js
            linkClassProperty: "status",
            defaultStyle: false,
            labelDy: "-1.4em",
            circleRadius: 8,
            charge: -100,
            gravity: 0.3,
            linkDistance: 100,
            linkStrength: 0.2,
        });
    };
    window.graph = window.loadNetJsonGraph();
    window.initTopologyHistory(jQuery);
</script>

1. Initialize your custom module

The first thing you need to do is to create a new django app which will contain your custom version of openwisp-network-topology.

A django app is nothing more than a python package (a directory of python scripts), in the following examples we’ll call this django app sample_network_topology, but you can name it how you want:

django-admin startapp sample_network_topology

If you use the integration with openwisp-controller, you may want to extend also the integration app if you need:

django-admin startapp sample_integration_device

Keep in mind that the command mentioned above must be called from a directory which is available in your PYTHON_PATH so that you can then import the result into your project.

Now you need to add sample_network_topology to INSTALLED_APPS in your settings.py, ensuring also that openwisp_network_topology has been removed:

INSTALLED_APPS = [
    # ... other apps ...
    'openwisp_utils.admin_theme',
    # all-auth
    'django.contrib.sites',
    'openwisp_users.accounts',
    'allauth',
    'allauth.account',
    'allauth.socialaccount',
    # (optional) openwisp_controller - required only if you are using the integration app
    'openwisp_controller.pki',
    'openwisp_controller.config',
    'reversion',
    'sortedm2m',
    # network topology
    # 'sample_network_topology' <-- uncomment and replace with your app-name here
    # (optional) required only if you need to extend the integration app
    # 'sample_integration_device' <-- uncomment and replace with your integration-app-name here
    'openwisp_users',
    # admin
    'django.contrib.admin',
    # rest framework
    'rest_framework',
]

For more information about how to work with django projects and django apps, please refer to the django documentation.

2. Install openwisp-network-topology

Install (and add to the requirement of your project) openwisp-network-topology:

pip install openwisp-network-topology

3. Add EXTENDED_APPS

Add the following to your settings.py:

EXTENDED_APPS = ('openwisp_network_topology',)

4. Add openwisp_utils.staticfiles.DependencyFinder

Add openwisp_utils.staticfiles.DependencyFinder to STATICFILES_FINDERS in your settings.py:

STATICFILES_FINDERS = [
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
    'openwisp_utils.staticfiles.DependencyFinder',
]

5. Add openwisp_utils.loaders.DependencyLoader

Add openwisp_utils.loaders.DependencyLoader to TEMPLATES in your settings.py:

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'OPTIONS': {
            'loaders': [
                'django.template.loaders.filesystem.Loader',
                'django.template.loaders.app_directories.Loader',
                'openwisp_utils.loaders.DependencyLoader',
            ],
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    }
]

6. Inherit the AppConfig class

Please refer to the following files in the sample app of the test project:

• sample_network_topology/__init__.py.

• sample_network_topology/apps.py.

For the integration with openwisp-controller, see:

• sample_integration_device/__init__.py.

• sample_integration_device/apps.py.

You have to replicate and adapt that code in your project.

For more information regarding the concept of AppConfig please refer to the “Applications” section in the django documentation.

7. Create your custom models

Please refer to sample_app models file use in the test project.

You have to replicate and adapt that code in your project.

Note: for doubts regarding how to use, extend or develop models please refer to the “Models” section in the django documentation.

8. Add swapper configurations

Once you have created the models, add the following to your settings.py:

# Setting models for swapper module
TOPOLOGY_LINK_MODEL = 'sample_network_topology.Link'
TOPOLOGY_NODE_MODEL = 'sample_network_topology.Node'
TOPOLOGY_SNAPSHOT_MODEL = 'sample_network_topology.Snapshot'
TOPOLOGY_TOPOLOGY_MODEL = 'sample_network_topology.Topology'
# if you use the integration with OpenWISP Controller and/or OpenWISP Monitoring
TOPOLOGY_DEVICE_DEVICENODE_MODEL = 'sample_integration_device.DeviceNode'

Substitute sample_network_topology with the name you chose in step 1.

9. Create database migrations

Create and apply database migrations:

./manage.py makemigrations
./manage.py migrate

For more information, refer to the “Migrations” section in the django documentation.

10. Create the admin

Refer to the admin.py file of the sample app.

To introduce changes to the admin, you can do it in two main ways which are described below.

Note: for more information regarding how the django admin works, or how it can be customized, please refer to “The django admin site” section in the django documentation.

from openwisp_network_topology.admin import TopologyAdmin, LinkAdmin, NodeAdmin

# TopologyAdmin.list_display.insert(1, 'my_custom_field') <-- your custom change example
# LinkAdmin.list_display.insert(1, 'my_custom_field') <-- your custom change example
# NodeAdmin.list_display.insert(1, 'my_custom_field') <-- your custom change example

from django.contrib import admin
from swapper import load_model

from openwisp_network_topology.admin import (
    TopologyAdmin as BaseTopologyAdmin,
    LinkAdmin as BaseLinkAdmin,
    NodeAdmin as BaseNodeAdmin
)

Node = load_model('topology', 'Node')
Link = load_model('topology', 'Link')
Topology = load_model('topology', 'Topology')

admin.site.unregister(Topology)
admin.site.unregister(Link)
admin.site.unregister(Node)

@admin.register(Topology, TopologyAdmin)
class TopologyAdmin(BaseTopologyAdmin):
    # add your changes here

@admin.register(Link, LinkAdmin)
class LinkAdmin(BaseLinkAdmin):
    # add your changes here

@admin.register(Node, NodeAdmin)
class NodeAdmin(BaseNodeAdmin):
    # add your changes here

11. Create root URL configuration

Please read and replicate according to your project needs:

The following can be used to register all the urls in your
``urls.py``.

# If you've extended visualizer views (discussed below).
# Import visualizer views & function to add it.
# from openwisp_network_topology.utils import get_visualizer_urls
# from .sample_network_topology.visualizer import views

urlpatterns = [
    # If you've extended visualizer views (discussed below).
    # Add visualizer views in urls.py
    # path('topology/', include(get_visualizer_urls(views))),
    path('', include('openwisp_network_topology.urls')),
    path('admin/', admin.site.urls),
]

For more information about URL configuration in django, please refer to the “URL dispatcher” section in the django documentation.

12. Setup API urls

You need to create a file api/urls.py (the name & path of the file must match) inside your app, which contains the following:

from openwisp_network_topology.api import views
# When you want to modify views, please change views location
# from . import views
from openwisp_network_topology.utils import get_api_urls

urlpatterns = get_api_urls(views)

13. Extending management commands

To extend the management commands, create sample_network_topology/management/commands directory and two files in it:

• save_snapshot.py

• update_topology.py

14. Import the automated tests

When developing a custom application based on this module, it’s a good idea to import and run the base tests too, so that you can be sure the changes you’re introducing are not breaking some of the existing features of openwisp-network-topology.

Refer to the tests.py file of the sample app.

In case you need to add breaking changes, you can overwrite the tests defined in the base classes to test your own behavior.

For testing you also need to extend the fixtures, you can copy the file openwisp_network_topology/fixtures/test_users.json in your sample app’s fixtures/ directory.

Now, you can then run tests with:

# the --parallel flag is optional
./manage.py test --parallel sample_network_topology

Substitute sample_network_topology with the name you chose in step 1.

For more information about automated tests in django, please refer to “Testing in Django”.

Other base classes that can be inherited and extended

The following steps are not required and are intended for more advanced customization.