Structured Logging with Python and CEE Syslog Handler
cee_syslog_handler is an extension to the python syslog logging handler with support for structured json messages.
Today we have released cee_syslog_handler version 0.3.1. The
cee_syslog_handler is an extension to the python syslog logging handler with
support for structured JSON messages. The message formatting is the same as in graypy.
While the mitre common event expression
project is discontinued, the @cee: cookie lives on as a way to define JSON messages in the rsyslog message normalization module.
As a simple example, we will log a test message with an extra field some_value.
from cee_syslog_handler import CeeSysLogHandler
import logging
logger = logging.getLogger('log')
logger.setLevel(logging.DEBUG)
ch = CeeSysLogHandler(address=("localhost", 514))
ch.setLevel(logging.DEBUG)
logger.addHandler(ch)
logger.info("test", extra={"some_value": 55})
The result is a single line in the syslog with a JSON representation of the log message (including the _some_value field):
2015-03-09T20:16:59.360858+00:00 localhost : @cee: {"_process_name": "MainProcess", "_some_value": 55,
"level": 6, "timestamp": 1425932219.360644, "_pid": 13295, "facility": "log", "_function": "<module>",
"source_facility": "log", "_thread_name": "MainThread", "host": "precise64", "version": "1.0",
"file": "testlog.py", "message": "test", "line": 12, "short_message": "test"}
The real power of having a structured representation of your logs is revealed when you combine it with tools like jq the lightweight command-line json processor.
For example, to get back the extra value you logged in the example, run:
sed -r 's/.*@cee: //;tx;d;:x' < syslog | jq '._some_value'
Using Syslog with CEE is a lighter-weight solution for centralized structured logging than I proposed in my EuroPython 2014 talk Log everything with Logstash and Elasticsearch, but it can be easily combined with Elasticsearch as shown in parsing cee logs with rsyslog. Or it can be used as an intermediate step to feed logs into Logstash/Graylog if you prefer having a graphical search interface and also want to keep the benefit of rsyslog's stability, and to have your log messages as text files too, just in case some Java software has issues.