gocept.logging 0.6

Testing support

To help inspecting the extra values, gocept.logging comes with a specialized handler for testing:

>>> import gocept.logging
>>> import logging

>>> log = logging.getLogger('testing')
>>> handler = gocept.logging.TestingHandler()
>>> log.addHandler(handler)
>>> log.warning('Hello, world!', extra={'foo': 'bar'})
>>> handler.messages[0].extra['foo']
'bar'

The TestingHandler records each log message as a namedtuple of type gocept.logging.testing.LogMessage so you an easily access all parts of the message.

Setup with ini file

If you have a paste.ini for your application, you might use something like this:

[loggers]
keys = root

[handlers]
keys = console, syslog

[formatters]
keys = generic, keyvalue

[logger_root]
level = INFO
handlers = console, syslog

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[formatter_generic]
format = %(asctime)s %(levelname)-5.5s %(name)s: %(message)s

[handler_syslog]
class = logging.handlers.SysLogHandler
args = ()
formatter = keyvalue

[formatter_keyvalue]
class = gocept.logging.SyslogKeyValueFormatter

Setup with ZConfig

If you have a Zope application, you might use something like this:

<eventlog>
  <logfile>
    formatter zope.exceptions.log.Formatter
    format %(asctime)s %(levelname)-5.5s %(name)s: %(message)s
    path STDOUT
  </logfile>
  <syslog>
    formatter gocept.logging.SyslogKeyValueFormatter
  </syslog>
</eventlog>

syslogd configuration

rsyslog:

$EscapeControlCharactersOnReceive off
$MaxMessageSize 64k
user.* @localhost:5140

The first two lines are to support tracebacks, which are multiline and might take up some space. The last line tells rsyslogd to forward all messages of the user facility (which is what stdlib logging uses by default) via syslog UDP protocol to localhost port 5140 (where logstash might be listening).

logstash configuration

input {
        tcp {
                host => "localhost"
                port => 5140
                type => syslog
        }
        udp {
                host => "localhost"
                port => 5140
                type => syslog
        }
}

filter {
        grok {
                type => "syslog"
                pattern => [ "(?m)<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" ]
        }
        syslog_pri {
                type => "syslog"
        }
        date {
                type => "syslog"
                match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
        }
        mutate {
                type => "syslog"
                exclude_tags => "_grokparsefailure"
                replace => [ "@source_host", "%{syslog_hostname}" ]
                replace => [ "@message", "%{syslog_program}: %{syslog_message}" ]
        }
        mutate {
                type => "syslog"
                remove => [ "syslog_hostname", "syslog_timestamp" ]
        }
        kv {
                exclude_tags => "_grokparsefailure"
                type => "syslog"
        }
}

output {
        elasticsearch { embedded => true }
}