NeuroAgent

How to Configure Log Duplication in Django Projects

Complete guide to avoiding log duplication in Django and saving important logs to the database only for specific views.

#django-logging#python-logging#database-logging#duplicate-logs#view-logging#django-optimization#log-handlers
Question

How to configure log duplication in Python logging for a Django project?

In my Django project, I have basic logging configured in views.py:

python
from django.shortcuts import render
from models import ORM_Model
logging.basicConfig(
    format="%(time)s - %(name)s %(lineno)d - %(levelname)s - %(message)s", level=logging.INFO,
    #datefmt='%H:%M:%S',
    fmt='%(asctime)s.%(msecs)03d',
)
logger = logging.getLogger(__name__)

#### BEGIN VIEWS 
def view_info_app(request,*args, **kwargs):
    """
    everything output here is very important!
    """
    logger.info("APP: view_info_app")
    info_objects = ORM_Model.objects.get(id=123)
    logger.info("Very important information - %s",info_objects)
    return render("templates/index.html",{})

def view_index_app(request,*args, **kwargs):
    """
    this is not important
    """
    logger.info("APP: view_index_app")
    info_objects = ORM_Model.objects.filter()
    logger.info("Unnecessary information - %s",info_objects)
    return render("templates/index.html",{})

I now need to save some logs to a database in addition to console output. How can I implement this only for specific views?

NeuroAgent

Main Problem of Log Duplication in Django

The main problem with log duplication in Django arises from a conflict between basicConfig() and Django’s built-in logging system. To avoid duplication and save logs to the database only for specific views, you should use configuration through LOGGING in settings.py.

Contents

Main Causes of Log Duplication

Log duplication in Django projects is usually caused by the following factors:

  1. Conflict between basicConfig() and Django LOGGING
    Your current basicConfig() setup conflicts with Django’s logging system, which is already configured by default.

  2. Logger inheritance (propagate=True)
    When using multiple loggers with propagate=True, logs may be duplicated as they are processed by multiple handlers.

  3. Reconfiguration of the root logger
    If there’s a root logger configuration in Django settings, this can lead to duplication when using basicConfig().

Preventing Duplication Configuration

To avoid duplication, completely remove basicConfig() from your views.py and configure logging through settings.py:

python
# DELETE THESE LINES from views.py:
# logging.basicConfig(...)
# logger = logging.getLogger(__name__)

Instead, use in views.py:

python
import logging
logger = logging.getLogger(__name__)  # Only this line is needed

Database Logging

Option 1: Using django-logdb

Add to requirements.txt:

django-logdb

In settings.py:

python
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '{levelname} {asctime} {module} {process:d} {thread:d} {message}',
            'style': '{',
        },
    },
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'formatter': 'verbose',
            'level': 'INFO',
        },
        'database': {
            'class': 'django_logdb.handlers.DatabaseHandler',
            'formatter': 'verbose',
            'level': 'DEBUG',
        },
    },
    'loggers': {
        'my_app': {
            'handlers': ['console', 'database'],
            'level': 'DEBUG',
            'propagate': False,
        },
        'django': {
            'handlers': ['console'],
            'level': 'INFO',
            'propagate': True,
        },
    },
}

Option 2: Custom Handler

Create a file logger/handlers.py:

python
import logging
from django.db import models

class DBHandler(logging.Handler):
    def emit(self, record):
        logger_entry = LogEntry(
            level=record.levelname,
            message=record.getMessage(),
            module=record.module,
            function_name=record.funcName,
            line_number=record.lineno,
            timestamp=record.created
        )
        logger_entry.save()

class LogEntry(models.Model):
    level = models.CharField(max_length=20)
    message = models.TextField()
    module = models.CharField(max_length=100)
    function_name = models.CharField(max_length=100)
    line_number = models.IntegerField()
    timestamp = models.DateTimeField(auto_now_add=True)
    
    class Meta:
        app_label = 'logger'

Selective Logging for Specific Views

Method 1: Different Loggers for Different Views

python
# views.py
import logging

logger_important = logging.getLogger('important_logs')
logger_regular = logging.getLogger('regular_logs')

def view_info_app(request, *args, **kwargs):
    logger_important.info("APP: view_info_app")
    info_objects = ORM_Model.objects.get(id=123)
    logger_important.info("Very important information - %s", info_objects)
    return render(request, "templates/index.html", {})

def view_index_app(request, *args, **kwargs):
    logger_regular.info("APP: view_index_app")
    info_objects = ORM_Model.objects.filter()
    logger_regular.info("Unnecessary information - %s", info_objects)
    return render(request, "templates/index.html", {})

Method 2: Using Filters

python
# settings.py
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'simple': {
            'format': '[%(asctime)s] %(levelname)s|%(name)s|%(message)s',
            'datefmt': '%Y-%m-%d %H:%M:%S',
        },
    },
    'filters': {
        'important_only': {
            '()': 'django_filters.logging.MessageFilter',
            'message': 'Very important information|APP: view_info_app',
        },
    },
    'handlers': {
        'applogfile': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': os.path.join(BASE_DIR, 'django_blend.log'),
            'backupCount': 10,
            'formatter': 'simple',
        },
        'database_important': {
            'level': 'INFO',
            'class': 'logger.handlers.DBHandler',
            'formatter': 'simple',
            'filters': ['important_only'],
        },
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'formatter': 'simple'
        },
    },
    'loggers': {
        'important_logs': {
            'handlers': ['database_important'],
            'level': 'INFO',
            'propagate': False,
        },
        'regular_logs': {
            'handlers': ['applogfile', 'console'],
            'level': 'DEBUG',
            'propagate': False,
        },
        'django': {
            'handlers': ['console'],
            'level': 'INFO',
            'propagate': True,
        },
    },
}

Method 3: Decorators for Selective Logging

python
# decorators.py
import logging
import functools

def log_to_database(view_func):
    @functools.wraps(view_func)
    def wrapper(request, *args, **kwargs):
        db_logger = logging.getLogger('database_logs')
        db_logger.info(f"VIEW: {view_func.__name__} started")
        result = view_func(request, *args, **kwargs)
        db_logger.info(f"VIEW: {view_func.__name__} completed")
        return result
    return wrapper

# views.py
@log_to_database
def view_info_app(request, *args, **kwargs):
    logger = logging.getLogger(__name__)
    logger.info("APP: view_info_app")
    info_objects = ORM_Model.objects.get(id=123)
    logger.info("Very important information - %s", info_objects)
    return render(request, "templates/index.html", {})

def view_index_app(request, *args, **kwargs):
    logger = logging.getLogger(__name__)
    logger.info("APP: view_index_app")
    info_objects = ORM_Model.objects.filter()
    logger.info("Unnecessary information - %s", info_objects)
    return render(request, "templates/index.html", {})

Complete Configuration Example

python
# settings.py
import os
from pathlib import Path

BASE_DIR = Path(__file__).resolve().parent.parent

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'simple': {
            'format': '[%(asctime)s] %(levelname)s|%(name)s|%(message)s',
            'datefmt': '%Y-%m-%d %H:%M:%S',
        },
        'verbose': {
            'format': '{levelname} {asctime} {module} {process:d} {thread:d} {message}',
            'style': '{',
        },
    },
    'handlers': {
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'formatter': 'simple',
        },
        'file': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': os.path.join(BASE_DIR, 'django.log'),
            'maxBytes': 10485760,  # 10MB
            'backupCount': 5,
            'formatter': 'simple',
        },
        'database': {
            'level': 'INFO',
            'class': 'django_logdb.handlers.DatabaseHandler',
            'formatter': 'verbose',
        },
    },
    'loggers': {
        'myapp.views.important': {
            'handlers': ['database'],
            'level': 'INFO',
            'propagate': False,
        },
        'myapp.views': {
            'handlers': ['file', 'console'],
            'level': 'DEBUG',
            'propagate': False,
        },
        'django': {
            'handlers': ['console'],
            'level': 'WARNING',
            'propagate': True,
        },
    },
}
python
# views.py
import logging
from django.shortcuts import render
from .models import ORM_Model

logger = logging.getLogger(__name__)
logger_important = logging.getLogger('myapp.views.important')

def view_info_app(request, *args, **kwargs):
    """
    Everything logged here is very important!
    """
    logger_important.info("APP: view_info_app")
    info_objects = ORM_Model.objects.get(id=123)
    logger_important.info("Very important information - %s", info_objects)
    logger.info("Regular log for view_info_app")
    return render(request, "templates/index.html", {})

def view_index_app(request, *args, **kwargs):
    """
    This is not important
    """
    logger.info("APP: view_index_app")
    info_objects = ORM_Model.objects.filter()
    logger.info("Unnecessary information - %s", info_objects)
    return render(request, "templates/index.html", {})

Optimization Recommendations

  1. Use different logger names for different parts of the application

    python
    logger = logging.getLogger(__name__)  # Automatically creates hierarchy

  2. Set propagate=False to avoid duplication

    python
    'myapp': {
    'handlers': ['database'],
    'level': 'INFO',
    'propagate': False,  # Important!
},

  3. Use logging levels for filtering

    python
    'handlers': {
    'important_only': {
        'level': 'INFO',  # Only INFO and above
        'class': 'django_logdb.handlers.DatabaseHandler',
    },
},

  4. For large projects, consider using specialized packages:

  5. Test configuration in different environments

    python
    # Example environment
LOGGING['loggers']['django']['level'] = os.getenv('DJANGO_LOG_LEVEL', 'INFO')

Sources

  1. Django Documentation - Logging
  2. Stack Overflow - Django duplicate log output
  3. django-logdb Package
  4. SigNoz - Django Logging Guide
  5. GitHub - django-db-logger
  6. Medium - Comprehensive Django Logging Guide
  7. James Lin - Custom Django Log Handler

Conclusion

  1. Remove basicConfig() from your code and use only LOGGING in settings.py
  2. Configure different loggers with different handlers to avoid duplication
  3. Use propagate=False to prevent log inheritance
  4. For database logging, use specialized handlers or packages
  5. For selective logging, apply different logger names, filters, or decorators

These solutions will allow you to effectively manage logging in your Django project, avoid duplication, and save important logs only for specific views to the database.

How to create a custom log handler for Django with saving to PostgreSQL?How to configure log rotation in Django projects with different importance levels?How to integrate Django logging with monitoring systems like ELK or Graylog?How to optimize performance when logging large volumes of data in Django?How to configure asynchronous logging in Django for improved performance?How to use middleware for automatic logging of all requests in Django?
Ask NeuroAgent