Django logs to ELK using Filebeat

I’ve written a post about how to send Django logs to ELK stack. You can read it here. In that post I’ve used logstash client with a sidecar docker container. Logstash client works but it needs too much resources. Nowadays it’s better to use Filebeat as data shipper instead of Logstash client. Filebeat it’s also a part of ELK stack. It’s a golang binary much lightweight than logstash client.

The idea is almost the same than the other post. Here we’ll also build a sidecar container with our django application logs mounted.

version: '3'
  # Application
    image: elk:latest
    command: /bin/bash ./
      context: .
      dockerfile: Dockerfile
      DEBUG: 'True'
      - logs_volume:/src/logs
      - static_volume:/src/staticfiles
    image: elk-nginx:latest
      context: .docker/nginx
      dockerfile: Dockerfile
      - static_volume:/src/staticfiles
      - 8000:8000
      - api
    image: filebeat:latest
      context: .docker/filebeat
      dockerfile: Dockerfile
      - logs_volume:/app/logs
    command: filebeat -c /etc/filebeat/filebeat.yml -e -d "*" -strict.perms=false
      - api


With filebeat we can perform actions to prepare our logs to be ready to be stored within elasticsearch. But, at least here, it’s much more easy to prepare the logs in the django application:

class CustomisedJSONFormatter(json_log_formatter.JSONFormatter):
    def json_record(self, message: str, extra: dict, record: logging.LogRecord):
        context = extra
        django = {
            'app': settings.APP_ID,
            'filename': record.filename,
            'funcName': record.funcName,
            'msecs': record.msecs,
        if record.exc_info:
            django['exc_info'] = self.formatException(record.exc_info)

        return {
            'message': message,
            'timestamp': now(),
            'level': record.levelname,
            'context': context,
            'django': django

And in we use our CustomisedJSONFormatter

    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'simple': {
            'format': '[%(asctime)s] %(levelname)s|%(name)s|%(message)s',
            'datefmt': '%Y-%m-%d %H:%M:%S',
        "json": {
            '()': CustomisedJSONFormatter,
    'handlers': {
        'applogfile': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': Path(BASE_DIR).resolve().joinpath('logs', 'app.log'),
            'maxBytes': 1024 * 1024 * 15,  # 15MB
            'backupCount': 10,
            'formatter': 'json',
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'formatter': 'simple'
    'root': {
        'handlers': ['applogfile', 'console'],
        'level': 'DEBUG',

And that’s all. Our Application logs centralized in ELK and ready to consume with Kibana

Source code available here

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.