logreduce 0.5.0

Local file usage

• Compare two files or directories without building a model:

$ logreduce diff testr-nodepool-01/output.good testr-nodepool-01/output.fail
0.232 | testr-nodepool-01/output.fail:0677:  File "voluptuous/schema_builder.py", line 370, in validate_mapping
0.462 | testr-nodepool-01/output.fail:0678:    raise er.MultipleInvalid(errors)
0.650 | testr-nodepool-01/output.fail:0679:  voluptuous.error.MultipleInvalid: required key not provided @ data['providers'][2]['cloud']

• Compare two files or directories:

$ logreduce dir preprod-logs/ /var/log/

• Or build a model first and run it separately:

$ logreduce dir-train sosreport.clf old-sosreport/ good-sosreport/
$ logreduce dir-run sosreport.clf new-sosreport/

Zuul job usage

Logreduce can query Zuul build database to train a model.

• Extract novelty from a job logs:

$ logreduce job http://logs.openstack.org/...

# Reduce comparaison to a single project (e.g. for tox jobs)
$ logreduce job --project openstack/nova http://logs.openstack.org/...

# Compare using many baselines
$ logreduce job --count 10 http://logs.openstack.org/...

# Include job artifacts
$ logreduce job --include-path logs/ http:/logs.openstack.org/...

• Or build a model first and run it separately:

$ logreduce job-train --job job_name job_name.clf
$ logreduce job-run job_name.clf http://logs.openstack.org/.../

Journald usage

Logreduce can look for anomaly in journald, comparing the last day/week/month to the previous one:

• Extract novelty from last day journal:

$ logreduce journal --range day

• Build a model using journal of last month and look for novelty in last week:

$ logreduce journal-train --range month good-journal.clf
$ logreduce journal-run --range week good-journal.clf

Filters configuration

Some content yields false positives that can be ignored through filters. Using the –config command line attribute, filters can be set for exclude_files, exclude_paths and exclude_lines. Here is an example filters configuration file:

filters:
  exclude_files:
    - "deployment-hieradata.j2.yaml"
    - "tempest.html"
  exclude_paths:
    - "group_vars/Compute"
    - "group_vars/Controller"
    - "group_vars/Undercloud"
  exclude_lines:
    # neutron dhcp interface
    - "^tap[^ ]*$"
    # IPA cookies
    - "^.*[Cc]ookie.*ipa_session="

API

• PUT /anomaly/new: receive user report request from os_loganalyze

• PUT /anomaly: import an anomaly report (json file generated by standalone cli)

• GET /anomaly/{anomaly_id}: return an anomaly details

• POST /anomaly/{anomaly_id}: update anomaly status

• POST /anomaly/{anomaly_id}/logfile/{logfile_id}: update scores

• DELETE /anomaly/{anomaly_id}/logfile/{logfile_id}: remove a file

• GET /anomalies: return the list of anomalies

• GET /status: return the list of worker jobs

Installation

Here is a brief documentation to setup the server components:

# Setup service
useradd -m -d /var/lib/logreduce -c "Logreduce Daemon" logreduce
mkdir /etc/logreduce /var/log/logreduce
chown -R logreduce /var/log/logreduce

# Install extra requirements
sudo -u logreduce pip3 install --user "logreduce[server]"

# Copy configuration
cp etc/logreduce/config.yaml /etc/logreduce/
cp etc/httpd/log-classify.conf /etc/httpd/conf.d/
cp etc/systemd/*.service /lib/systemd/system
chown -R logreduce /var/log/logreduce

# Update configuration
# Edit public_url with the http server public address

# Start the service
systemctl enable logreduce-*
systemctl start logreduce-*
systemctl status logreduce-*
# -> status should display 'INFO logreduce.ServerWorker: Connected'
# -> /var/log/logreduce/worker.log should say 'Starting RPC listener'

# Build and install the web interface
cd web
yarn install
yarn build
rsync -a build/ /usr/share/log-classify/
# -> curl localhost/log-classify should return the index.html

Test format

Each tests case is composed of:

• A .good file (or directory) that holds the baseline

• A .fail file (or directory)

• A info.yaml file that describe expected output:

threshold: float # set the distance threshold for the test
anomalies:
  - optional: bool  # to define minor anomalies not considered false positive
    lines: |        # the expected lines to be highlighted
      Traceback...
      RuntimeError...

Evaluate

To run the evaluation, first install logreduce-tests:

git clone https://softwarefactory-project.io/r/logreduce-tests
pushd logreduce-tests
python3 setup.py develop --user

logreduce-tests expect tests directories as argument:

$ logreduce-tests tests/testr-zuul-[0-9]*
[testr-zuul-01]: 100.00% accuracy,  5.00% false-positive
[testr-zuul-02]:  80.00% accuracy,  0.00% false-positive
...
Summary:  90.00% accuracy,  2.50% false-positive

Add –debug to display false positive and missing chunks.