Reduce Docker Overlay2 Space Usage: Nextcloud Fix

Docker overlay2 filesystem can consume disk space even when /var/lib/docker appears to have sufficient free space due to how the overlay2 storage driver manages container layers and writeable layers. This issue commonly occurs with Nextcloud containers because they accumulate data writes and cache files that aren’t automatically cleaned by standard Docker commands. To reclaim space from your Docker overlay2 filesystem and prevent future Nextcloud crashes, you’ll need targeted cleanup strategies that address the specific ways overlay2 accumulates unused data.

Contents

• Understanding Docker Overlay2 Storage
• Why Standard Docker Cleanup Commands Fail
• Advanced Diagnostics for Overlay2 Space Usage
• Immediate Cleanup Solutions
• Nextcloud-Specific Overlay2 Issues
• Preventive Measures for Long-Term Management
• Moving Docker to a Larger Storage Volume
• Manual Overlay2 Cleanup (Last Resort)

Understanding Docker Overlay2 Storage

The Docker overlay2 storage driver is the default filesystem driver for Docker containers on modern Linux systems. It uses a layered approach that combines multiple read-only layers with a single writeable layer. When you run docker system df, you might see that Docker reports sufficient free space, but the overlay2 directory continues to grow. This happens because overlay2 stores container data differently than traditional filesystems.

In the overlay2 architecture, each container has its own directory structure within /var/lib/docker/overlay2. These directories contain the merged folder (which combines all layers) and the diff folder (containing changes made by the container). The official Docker documentation explains that the overlay driver can consume a large portion of the disk because all container writes and image layers are stored in the same filesystem.

When containers write data, especially frequently accessed files like Nextcloud’s database and application files, these operations create additional layers that aren’t visible in standard Docker disk usage reports. The overlay2 filesystem expands mainly due to unused or cached data that Docker does not automatically clean, which is why your Nextcloud server crashed despite apparent free space.

Why Standard Docker Cleanup Commands Fail

Standard Docker cleanup commands like docker system prune or docker image prune often don’t address the root cause of overlay2 filesystem bloat because they focus on removing unused images, containers, and build cache, not the specific filesystem structures that consume space within the overlay2 directory.

The VirtualizationHowTo guide explains that the overlay2 filesystem can grow large when unused images, containers, or build cache accumulate, but standard prune commands may not catch everything. Specifically, the overlay2 directory can retain orphaned layers, unmerged changes, and temporary build artifacts that aren’t properly tracked by Docker’s cleanup routines.

When you run docker system prune, it removes:

• All stopped containers
• All networks not used by at least one container
• All dangling images
• All build cache

However, it doesn’t address:

• Unmerged container layers
• Orphaned overlay2 directories
• Nextcloud-specific cache and temp files
• Docker’s own internal metadata and log files

This is why your Nextcloud server crashed despite running cleanup commands - the overlay2 filesystem was still consuming disk space through mechanisms that standard Docker cleanup doesn’t address. The FossGuides resource specifically notes that after running standard prune commands, the server’s free space might not increase dramatically until you address the underlying overlay2 directory issues.

Advanced Diagnostics for Overlay2 Space Usage

Before attempting cleanup, you need to understand exactly what’s consuming space in your overlay2 directory. Here are the diagnostic tools that will help identify the root cause of your space issues:

Start by examining the disk usage of your Docker directories:

du -a /var/lib/docker | sort -n -r | head -n 10

This command will show you the top 10 space consumers in your Docker directory structure. As shown in the Nextcloud community post, you might see something like:

37636092 /var
37466452 /var/lib
37351564 /var/lib/docker
37239188 /var/lib/docker/overlay2
35790720 /var/lib/docker/overlay2/04b75732e23f872f18c73e8444df90e0be3b7fc34612693a62f5973ec0136e3f
18145112 /var/lib/docker/overlay2/04b75732e23f872f18c73e8444df90e0be3b7fc34612693a62f5973ec0136e3f/merged

Next, use Docker’s built-in diagnostic tools:

docker system df -v

This command provides a more detailed breakdown of how Docker is using space, showing exactly which images, containers, and volumes are consuming the most space.

For Nextcloud-specific issues, check the application’s cache and temporary files:

docker exec <your-nextcloud-container> df -h
docker exec <your-nextcloud-container> du -sh /var/www/html/data

The BitDoze guide recommends using these diagnostic commands to identify specific areas where Nextcloud might be accumulating data that isn’t properly managed by Docker’s overlay2 filesystem.

Immediate Cleanup Solutions

With a clear understanding of what’s consuming space, you can now implement targeted cleanup solutions. Start with these immediate actions that address the most common causes of overlay2 filesystem bloat:

1. Aggressive Docker System Prune

Run a comprehensive system prune that removes all unused objects, including volumes that might be retaining data:

docker system prune -a --volumes

As noted in the Stack Overflow discussion, this specific command combination was able to reclaim around 50GB of disk space in one case. The -a flag removes all unused images, not just dangling ones, and --volumes removes all unused volumes.

2. Nextcloud-Specific Cleanup

Nextcloud accumulates various types of cache and temporary files that can consume significant overlay2 space. Clean these up:

docker exec <your-nextcloud-container> php occ maintenance:mode --on
docker exec <your-nextcloud-container> php occ files:scan --all
docker exec <your-nextcloud-container> php occ files:cleanup
docker exec <your-nextcloud-container> php occ files:scan-app-data
docker exec <your-nextcloud-container> php occ config:system:set 'memcache.local' --value='\OC\Memcache\APCu' --type=string

These commands put Nextcloud in maintenance mode, scan and clean up the file system, optimize the database, and ensure proper caching configuration.

3. Overlay2 Layer Cleanup

Remove specific overlay2 layers that are no longer referenced by any container:

docker system prune -f
docker image prune -a -f
docker container prune -f
docker volume prune -f
docker builder prune -a -f

Run these commands in sequence to ensure all types of Docker objects are cleaned. The VirtualizationHowTo guide recommends this approach as it systematically removes unused components that contribute to overlay2 growth.

Nextcloud-Specific Overlay2 Issues

Nextcloud has unique characteristics that make it particularly prone to overlay2 filesystem space consumption. Understanding these issues will help you implement more effective long-term solutions.

Database and Cache Growth

Nextcloud databases and caches grow continuously as users upload files, make changes, and the system performs background operations. These write-intensive operations create multiple overlay2 layers that aren’t automatically cleaned up. The FossGuides resource mentions that after Nextcloud-specific cleanup, the server’s free space increased dramatically, preventing a crash caused by overlay filesystem exhaustion.

Temporary File Accumulation

Nextcloud creates numerous temporary files during file operations, previews generation, and maintenance tasks. These files often remain in the overlay2 filesystem even after they should have been deleted. The Help.NextCloud forum post shows real-world examples of how these files can consume gigabytes of space in overlay2 directories.

Preview and Thumbnail Generation

Nextcloud generates previews and thumbnails for various file types, storing them in the filesystem. These files accumulate over time and aren’t automatically managed by Docker’s overlay2 cleanup routines. Each preview generation operation creates new layers in the overlay2 directory structure.

App and Plugin Data

Nextcloud apps and plugins store their data in the filesystem, which can grow unpredictably. Some apps may create their own caching mechanisms that don’t respect Docker’s storage management.

To address these Nextcloud-specific issues, implement regular maintenance schedules that include:

# Set up a cron job for regular Nextcloud maintenance
0 3 * * * docker exec <your-nextcloud-container> php occ maintenance:mode --on
0 4 * * * docker exec <your-nextcloud-container> php occ files:scan --all
0 5 * * * docker exec <your-nextcloud-container> php occ files:cleanup
0 6 * * * docker exec <your-nextcloud-container> php occ maintenance:mode --off

Preventive Measures for Long-Term Management

After immediate cleanup, implement preventive measures to ensure your overlay2 filesystem doesn’t become bloated again. These long-term strategies will help you maintain optimal disk usage for your Nextcloud Docker setup.

Regular Maintenance Schedule

Set up automated cleanup tasks to run periodically. Create a shell script containing:

#!/bin/bash
# weekly-overlay2-cleanup.sh

# Stop Nextcloud for maintenance
docker exec nextcloud php occ maintenance:mode --on

# Run Docker cleanup commands
docker system prune -f
docker image prune -a -f
docker container prune -f
docker volume prune -f
docker builder prune -a -f

# Nextcloud specific cleanup
docker exec nextcloud php occ files:scan --all
docker exec nextcloud php occ files:cleanup
docker exec nextcloud php occ config:system:set 'memcache.local' --value='\OC\Memcache\APCu' --type=string

# Restart Nextcloud
docker exec nextcloud php occ maintenance:mode --off

Make this script executable and set up a weekly cron job to run it:

chmod +x weekly-overlay2-cleanup.sh
crontab -e
# Add: 0 3 * * 0 /path/to/weekly-overlay2-cleanup.sh

Docker Configuration Tweaks

Adjust your Docker configuration to better manage overlay2 space. Edit or create /etc/docker/daemon.json:

{
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.size=10G",
    "overlay2.mount_program=/usr/bin/fuse2fs",
    "overlay2.override_kernel_check=true"
  ]
}

The official Docker documentation recommends these settings to optimize overlay2 performance and space usage. The size parameter limits the maximum size of container layers, preventing runaway growth.

Nextcloud Configuration Optimization

Configure Nextcloud to minimize filesystem writes:

1. Enable Redis for caching by adding to config/config.php:

   php

   'memcache.local' => '\\OC\\Memcache\\Redis',
   'memcache.distributed' => '\\OC\\Memcache\\Redis',
   'memcache.locking' => '\\OC\\Memcache\\Redis',
   'redis' => [
     'host' => 'redis',
     'port' => 6379,
   ],
   

2. Set up proper cron jobs instead of web cron:

   bash

   docker exec nextcloud php occ config:system:set 'cron.interval' --value='15' --type=integer
   

3. Configure preview generation settings to limit storage:

   bash

   docker exec nextcloud php occ config:system:set 'enabledPreviewProviders' --value='["OC\\Preview\\Image","OC\\Preview\\MP3","OC\\Preview\\TXT","OC\\Preview\\MarkDown"]' --type=array
   

Monitoring and Alerting

Set up monitoring to alert you when overlay2 space usage approaches critical levels. Create a monitoring script:

#!/bin/bash
# monitor-overlay2.sh

OVERLAY2_USAGE=$(du -s /var/lib/docker/overlay2 | cut -f1)
THRESHOLD=107374182400  # 100GB in bytes

if [ "$OVERLAY2_USAGE" -gt "$THRESHOLD" ]; then
  echo "ALERT: Overlay2 usage is $OVERLAY2_USAGE bytes, exceeding threshold of $THRESHOLD bytes"
  # Send alert via email, Slack, or other notification method
  echo "ALERT: Overlay2 usage is $OVERLAY2_USAGE bytes" | mail -s "Docker Overlay2 Space Alert" admin@example.com
fi

Set up a daily cron job to run this monitoring script.

Moving Docker to a Larger Storage Volume

If your system disk is consistently too small for Docker’s overlay2 filesystem, the best solution is to move Docker to a larger storage volume. This approach prevents space issues entirely while providing better performance for I/O-intensive applications like Nextcloud.

Preparing the New Storage

First, identify a suitable storage location with adequate space. This could be:

1. A separate partition on your system
2. An external USB drive
3. A network-mounted filesystem (NFS)
4. A cloud storage solution (with proper caching)

For this example, we’ll use a separate partition at /mnt/docker-storage:

# Create and format the new partition (adjust as needed)
fdisk /dev/sdb  # Create new partition
mkfs.ext4 /dev/sdb1
mkdir -p /mnt/docker-storage
mount /dev/sdb1 /mnt/docker-storage

Moving Docker Data

Stop Docker and move its data to the new location:

# Stop Docker service
sudo systemctl stop docker
sudo systemctl stop docker.socket

# Move Docker data
mv /var/lib/docker /mnt/docker-storage/

# Create symbolic link
ln -s /mnt/docker-storage/docker /var/lib/docker

# Start Docker service
sudo systemctl start docker

Configuring Mount Persistence

Ensure the new partition mounts automatically on boot by adding it to /etc/fstab:

/dev/sdb1  /mnt/docker-storage  ext4  defaults  0  2

Verifying the Move

After moving Docker to the new storage, verify everything is working correctly:

# Check disk usage
df -h
du -sh /var/lib/docker

# Start your Nextcloud container
docker-compose up -d

# Verify containers are running
docker ps

The BitDoze guide emphasizes that moving Docker’s root to a larger or faster disk is one of the most effective long-term solutions for overlay2 space issues, especially for applications like Nextcloud that generate substantial data.

Manual Overlay2 Cleanup (Last Resort)

If all other methods fail to reclaim sufficient space, you may need to manually clean the overlay2 directory. This is a high-risk operation that should only be performed as a last resort, with proper backups and precautions.

Preparation Steps

Before proceeding with manual cleanup:

1. Back up your data:

   bash

   # Create a complete backup
   docker exec nextcloud tar -czf /tmp/nextcloud-backup.tar.gz /var/www/html
   docker cp nextcloud:/tmp/nextcloud-backup.tar.gz /path/to/backup/location/
   

2. Stop Docker service:

   bash

   sudo systemctl stop docker
   

3. Identify active containers and their overlay2 directories:

   bash

   ls -la /var/lib/docker/overlay2/
   # Note the directory names for active containers
   

Safe Manual Cleanup

1. Remove clearly unused overlay2 directories:

   bash

   # Find directories that don't correspond to active containers
   sudo find /var/lib/docker/overlay2 -name "diff" -type d | xargs -I {} sh -c 'if [ ! -L {} ]; then echo "Potential unused: {}"; fi'
   

2. Remove specific orphaned layers (use with extreme caution):

   bash

   # Remove layers that aren't referenced by any container
   sudo find /var/lib/docker/overlay2 -name "link" -type f | xargs -I {} sh -c 'if ! grep -q "$(basename $(dirname {}))" /var/lib/docker/image/overlay2/repositories-aux.json; then echo "Orphaned layer: $(dirname {})"; sudo rm -rf "$(dirname {})"; fi'
   

3. Clean up Docker’s metadata:

   bash

   # Remove corrupted metadata files
   sudo find /var/lib/docker/image/overlay2/imagedb/content/sha256 -type f -size 0 -delete
   

Recovery Steps

If something goes wrong during manual cleanup:

1. Restore from backup:

   bash

   # Restore your Nextcloud data
   docker cp /path/to/backup/location/nextcloud-backup.tar.gz nextcloud:/tmp/
   docker exec nextcloud tar -xzf /tmp/nextcloud-backup.tar.gz -C /
   

2. Reset Docker completely (as a last resort):

   bash

   # WARNING: This removes all Docker data!
   sudo rm -rf /var/lib/docker
   sudo systemctl start docker
   

The VirtualizationHowTo guide warns that manual deletion inside /var/lib/docker/overlay2 should be avoided whenever possible. If a full reset is needed, stop Docker, back up data, move the directory, then restart. The guide specifically recommends this approach only when all other methods have failed.

Sources

1. VirtualizationHowTo: Docker Overlay2 Cleanup: 5 Ways to Reclaim Disk Space

2. FossGuides: How To Free Up Disk Space From Docker Overlay2

3. BitDoze: Reclaim Disk Space by Cleaning Up /var/lib/docker/overlay2

4. Docker Documentation: OverlayFS storage driver

5. Nextcloud Community: NCP on docker, filling up rootfs and Buster update confusion

6. Stack Overflow: Is it safe to clean docker/overlay2/

Conclusion

Reducing Docker overlay2 filesystem space usage requires a multi-pronged approach that addresses both immediate cleanup and long-term prevention. When your Nextcloud server crashes due to overlay2 filesystem exhaustion despite apparent free space, it’s typically because the overlay2 directory contains accumulated layers, orphaned data, and application-specific caches that standard Docker cleanup commands don’t address.

The most effective solutions involve a combination of aggressive Docker system pruning with volume removal, Nextcloud-specific maintenance operations, and implementing regular automated cleanup schedules. For persistent issues, moving Docker to a larger storage volume provides the most robust long-term solution, eliminating space constraints entirely while improving performance for I/O-intensive applications like Nextcloud.

By following the diagnostic steps, implementing targeted cleanup commands, and establishing preventive measures, you can maintain optimal overlay2 filesystem usage and prevent future Nextcloud crashes due to disk space exhaustion. Remember to always back up your data before performing significant cleanup operations, especially when using manual overlay2 directory manipulation as a last resort.