NeuroAgent

Configuring Auto-Deletion of Logs in Grafana Loki with File System in Docker

Complete guide to properly configuring automatic deletion of old logs in Grafana Loki with file system storage in Docker. Solutions for retention_period and compactor issues.

#docker-configuration#file-system-storage#grafana-loki#log-management#log-retention#monitoring-tools
Question

How to configure automatic log deletion in Grafana Loki with filesystem storage in Docker?

I’ve tried various options from the official Grafana Loki documentation (retention., delete.) but they don’t work. Please advise on how to properly enable automatic deletion of old logs in my configuration:

loki-config.yaml:
auth_enabled: false

server:
http_listen_port: 3100
grpc_listen_port: 9096
log_level: debug
grpc_server_max_concurrent_streams: 1000

common:
instance_addr: 127.0.0.1
path_prefix: /tmp/loki
storage:
filesystem:
chunks_directory: /tmp/loki/chunks
rules_directory: /tmp/loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory

query_range:
parallelise_shardable_queries: true
align_queries_with_step: true
results_cache:
cache:
embedded_cache:
enabled: true
max_size_mb: 100

limits_config:
metric_aggregation_enabled: true
reject_old_samples: true
reject_old_samples_max_age: 10s
retention_period: 10s

compactor:
working_directory: /loki/compactor
compaction_interval: 10s
retention_enabled: true
retention_delete_delay: 0s
delete_request_store: filesystem
delete_request_cancel_period: 10s
delete_max_interval: 10s

table_manager:
retention_deletes_enabled: true

schema_config:
configs:
- from: 2020-10-24
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h

pattern_ingester:
enabled: true
metric_aggregation:
loki_address: localhost:3100
flush_check_period: 1m

ruler:
alertmanager_url: http://localhost:9093

frontend:
encoding: protobuf

docker-compose.yaml:
services:
loki:
image: grafana/loki
container_name: loki
command: -config.file=/etc/loki/config/loki-config.yaml
volumes:
- ./loki-config.yaml:/etc/loki/config/loki-config.yaml
- ./loki-data:/tmp/loki
ports:
- “3100:3100”

grafana:
image: grafana/grafana
container_name: grafana
ports:
- “3000:3000”
volumes:
- ./grafana-data:/var/lib/grafana
depends_on:
- loki

How to properly configure retention period and automatic log deletion in Grafana Loki with filesystem storage in Docker?

NeuroAgent

The problem with automatic log deletion in Grafana Loki with filesystem storage in Docker arises due to configuration mismatch. Here’s how to properly configure automatic deletion of old logs:

For proper automatic log deletion in Grafana Loki with filesystem storage in Docker, you need to perform the following settings:

yaml
limits_config:
  retention_period: 720h  # Minimum 24 hours (e.g., 30 days = 720h)
  # ... other settings

compactor:
  working_directory: /tmp/loki/retention  # Outside chunks_directory
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h  # Delay before deletion
  retention_delete_worker_count: 150
  delete_request_store: filesystem  # Important for filesystem
  # ... other settings

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        period: 24h  # Must be a multiple of retention_period

Main issues in your configuration:

  1. Incorrect retention period:

    • retention_period: 10s is less than the minimum possible 24 hours
    • Must be a multiple of the indexing period (24h)

  2. Missing required parameters:

    • Missing retention_delete_worker_count
    • retention_delete_delay is set to 0s (recommended minimum 1-2 hours)

  3. Incorrect structure:

    • delete_request_store is specified but not used in limits_config
    • working_directory should be a separate directory, not coinciding with chunks_directory

Corrected configuration:

yaml
loki-config.yaml:
limits_config:
  retention_period: 720h  # 30 days
  # ... other settings

compactor:
  working_directory: /tmp/loki/retention
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
  delete_request_store: filesystem
  # ... other settings

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        period: 24h

Docker configuration for retention:

yaml
docker-compose.yaml:
services:
  loki:
    image: grafana/loki:latest
    volumes:
      - ./loki-config.yaml:/etc/loki/config/loki-config.yaml
      - loki-data:/tmp/loki
      - loki-retention:/tmp/loki/retention  # Separate volume for compactor
    # ... other settings

volumes:
  loki-data:
  loki-retention:  # Required for retention

Verification:

  1. Ensure the retention directory exists and is writable:
bash
mkdir -p ./loki-data/retention
chmod 755 ./loki-data/retention
  1. Restart the container:
bash
docker-compose down && docker-compose up -d
  1. Check logs for errors:
bash
docker-compose logs loki
  1. After 24 hours, observe deletion of old logs through Grafana or Loki CLI

Important notes:

  • Minimum retention period: 24 hours
  • Period multiples: retention_period must be a multiple of index.period
  • Loki version: Use recent versions (v2.8+ or v3.x)
  • Deletion granularity: Logs are deleted in batches after compaction

Common errors:

  1. Missing separate volume for retention
  2. Incorrect directory permissions
  3. Too small retention_delete_delay
  4. Insufficient space for compactor operation

After proper configuration, old logs will be automatically deleted after the specified retention period. For monitoring, use Grafana with Loki metrics dashboards (log_entries_total).

How to monitor the log deletion process and verify retention functionality in Grafana Loki?What are the best practices for log retention policies in distributed systems?What alternative data storage options can be used for Loki instead of file system in Docker?How to optimize Loki performance when working with large volumes of log data?How to configure automatic log backup before deletion in Loki?
Ask NeuroAgent