Why dotMemory Doesn't Save Snapshots in Docker

When dotMemory displays the message “SNAPSHOT #1 READY” and then hangs, this is often related to specific behaviors in Docker containers and the handling of large snapshots. Here are the main causes and solutions to this problem.

Contents

• Common Causes of the Problem
• Immediate Solutions
• Alternative Methods for Saving Snapshots
• Configuration for Docker Containers
• Prevention of Similar Issues
• When to Contact JetBrains Support

Common Causes of the Problem

Snapshot Processing Issue
After the “SNAPSHOT #1 READY” message, dotMemory begins processing the collected data, which can take a significant amount of time. In Docker containers, this process is often slowed down due to:

• Limited resources (CPU, memory)
• Filesystem access issues
• Container isolation conflicts
• Large amounts of data to process (you have ~11.86 M objects)

Important: According to JetBrains support documentation, if a snapshot gets stuck in the processing state, you should wait about 10 minutes and check for the “JetBrains.Common.ExternalStorage.CLR40.exe” process in Task Manager.

Immediate Solutions

1. Wait for Processing to Complete

The simplest solution is to just wait. Processing large snapshots can take anywhere from 10 to 30 minutes.

# Wait 10-15 minutes before checking results
docker exec -it app ls -la /dotMemory/snapshots/

2. Check Access Rights

Ensure that the container has write permissions to the specified directory:

docker exec -it app touch /dotMemory/snapshots/test.txt
# If error occurs - no access rights

3. Use an Alternative Command

Try adding the --no-gc flag and changing the save path:

docker exec -it app /dotMemory/dotmemory get-snapshot 1 --save-to-dir=/tmp/snapshots --no-gc

Alternative Methods for Saving Snapshots

1. Using Service Messages

For remote containers, you can use service messages:

# Create a file with commands
echo '##dotMemory["get-snapshot"]' > /tmp/dotmemory_commands
echo '##dotMemory["disconnect"]' >> /tmp/dotmemory_commands

# Copy to container and execute
docker cp /tmp/dotmemory_commands app:/tmp/commands
docker exec -it app /dotMemory/dotmemory service-messages --file=/tmp/commands

2. Using gcore as an Alternative

If dotMemory can’t handle it, use standard Linux tools:

# Connect to the container
docker exec -it app bash

# Use gcore to create a memory dump
gcore -o /dotMemory/snapshots/memory_dump 1

3. Scheduled Snapshots

Use scheduled snapshots instead of immediate ones:

docker exec -it app /dotMemory/dotmemory attach 1 --trigger-on-activation --trigger-timer=5m --trigger-max-snapshots=1 --save-to-dir=/dotMemory/snapshots

Configuration for Docker Containers

1. Increase Container Resources

If possible, increase the memory and CPU limits for the container:

# docker-compose.yml
services:
  app:
    mem_limit: 2g
    cpus: 2.0

2. Use the Correct Base Image

Ensure you’re using an image with a supported .NET version:

# Use an up-to-date image with full profiling support
FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS base

3. Configure Proper Paths

Avoid using complex paths inside the container:

# Better to use /tmp or /var/tmp for temporary files
docker exec -it app /dotMemory/dotmemory get-snapshot 1 --save-to-dir=/var/tmp/dotmemory_snapshots

Prevention of Similar Issues

1. Regular Profiling

Perform profiling regularly, not only when memory leaks are detected:

# Create a script for regular snapshots
#!/bin/bash
docker exec app /dotMemory/dotmemory attach 1 --trigger-on-activation --trigger-timer=1h --trigger-max-snapshots=24 --save-to-dir=/dotMemory/snapshots

2. Resource Monitoring

Set up monitoring of memory usage in the container:

# Monitor memory usage
docker stats app --no-stream

3. Use dotMemory Unit

For automated memory testing, use dotMemory Unit:

[Test]
public void MemoryLeakTest()
{
    GC.Collect();
    GC.WaitForPendingFinalizers();
    
    var initialSnapshot = TestRunner.GetMemorySnapshot();
    
    // Your code for checking leaks
    
    GC.Collect();
    GC.WaitForPendingFinalizers();
    
    var finalSnapshot = TestRunner.GetMemorySnapshot();
    
    finalSnapshot.CompareWith(initialSnapshot).WhereObjectsAre((_, diff) => diff > 0).CheckShouldBeEmpty();
}

When to Contact JetBrains Support

Contact JetBrains support in the following cases:

1. If waiting more than 30 minutes yields no results
2. If the “JetBrains.Common.ExternalStorage.CLR40.exe” process consumes more than 1GB of memory
3. If snapshots consistently fail to save in different containers
4. When encountering errors with HRESULT 0x8013136A

Note: According to JetBrains documentation, certain HRESULT errors indicate issues with connecting the profiler to the .NET runtime.

Sources

1. JetBrains dotMemory Documentation - Profile Application in Docker Container
2. Taking snapshot with dotMemory is very slow - JetBrains Support
3. dotMemory command line scheduled snapshots - Stack Overflow
4. Can we use dotMemory profiler for .NET application in Azure Kubernetes - JetBrains Support
5. DotMemory error calling ./dotMemory.sh get-snapshot on linux - JetBrains Support

Conclusion

The issue with dotMemory hanging after the “SNAPSHOT #1 READY” message is typically resolved by the following approaches:

1. Patience is key: Most cases are resolved simply by waiting 10-30 minutes
2. Check resources: Ensure the container has sufficient memory and CPU
3. Use alternative methods: gcore and service messages may work better in Docker environments
4. Configure proper paths: Avoid complex paths and check access rights
5. Regular profiling: Prevent issues with regular memory monitoring

For .NET 8 applications with explicit memory leaks, it’s recommended to first try the simple approach of waiting, then move to more complex methods such as using gcore or changing the container configuration.