Docker Storage Drivers: overlay2 vs devicemapper Performance Comparison

Our Docker builds were slow. 15 minutes to build an image. Container I/O was terrible. We were using devicemapper in loopback mode. Worst choice possible.

I switched to overlay2. Build time dropped to 3 minutes. I/O performance improved 5x. Here’s what I learned about Docker storage drivers.

Table of Contents

The Problem

Docker storage drivers:

• overlay2: Modern, fast (recommended)
• devicemapper: Older, slower
• aufs: Legacy
• btrfs: Special use cases
• zfs: Special use cases

Wrong choice = slow performance.

Check Current Driver

docker info | grep "Storage Driver"
# Storage Driver: devicemapper

overlay2 (Recommended)

Modern, performant, stable.

Requirements:

• Linux kernel 4.0+
• XFS or ext4 filesystem

Configure:

/etc/docker/daemon.json:

{
  "storage-driver": "overlay2"
}

Restart Docker:

sudo systemctl restart docker

devicemapper

Older driver. Two modes:

1. loop-lvm (BAD - don’t use):

• Uses loopback devices
• Terrible performance
• Default on some systems

2. direct-lvm (OK):

• Uses dedicated block device
• Better performance
• Complex setup

Configure direct-lvm:

{
  "storage-driver": "devicemapper",
  "storage-opts": [
    "dm.directlvm_device=/dev/xvdf",
    "dm.thinp_percent=95",
    "dm.thinp_metapercent=1",
    "dm.thinp_autoextend_threshold=80",
    "dm.thinp_autoextend_percent=20"
  ]
}

Performance Comparison

Benchmark setup:

# Create test container
docker run -d --name test-io ubuntu:18.04 sleep infinity

# Write test
docker exec test-io dd if=/dev/zero of=/tmp/test bs=1M count=1000

# Read test
docker exec test-io dd if=/tmp/test of=/dev/null bs=1M

Results:

Driver	Write Speed	Read Speed	Build Time
overlay2	500 MB/s	600 MB/s	3 min
devicemapper (direct-lvm)	200 MB/s	250 MB/s	8 min
devicemapper (loop-lvm)	50 MB/s	60 MB/s	15 min

Winner: overlay2 (5-10x faster)

Migration to overlay2

Warning: Requires recreating containers!

Step 1: Backup images

# Save important images
docker save myapp:latest > myapp.tar

Step 2: Stop Docker

sudo systemctl stop docker

Step 3: Backup Docker data

sudo cp -au /var/lib/docker /var/lib/docker.bak

Step 4: Configure overlay2

sudo cat > /etc/docker/daemon.json <<EOF
{
  "storage-driver": "overlay2"
}
EOF

Step 5: Remove old data

sudo rm -rf /var/lib/docker/*

Step 6: Start Docker

sudo systemctl start docker

Step 7: Restore images

docker load < myapp.tar

Step 8: Recreate containers

docker-compose up -d

Filesystem Considerations

XFS (recommended for overlay2):

# Format disk
sudo mkfs.xfs -n ftype=1 /dev/sdb

# Mount
sudo mkdir -p /var/lib/docker
sudo mount /dev/sdb /var/lib/docker

# Add to /etc/fstab
echo "/dev/sdb /var/lib/docker xfs defaults 0 0" | sudo tee -a /etc/fstab

ext4 (also works):

sudo mkfs.ext4 /dev/sdb
sudo mount /dev/sdb /var/lib/docker

Volume Performance

Test volume I/O:

# Named volume
docker volume create test-vol
docker run --rm -v test-vol:/data ubuntu:18.04 dd if=/dev/zero of=/data/test bs=1M count=1000

# Bind mount
docker run --rm -v /tmp/test:/data ubuntu:18.04 dd if=/dev/zero of=/data/test bs=1M count=1000

Results:

Type	Write Speed	Use Case
Named volume	500 MB/s	Production data
Bind mount	600 MB/s	Development
tmpfs	2000 MB/s	Temporary data

Image Layer Caching

overlay2 excels at layer caching:

FROM ubuntu:18.04

# These layers are cached
RUN apt-get update
RUN apt-get install -y python3

# Only this layer rebuilds if code changes
COPY app.py /app/

Build times:

Driver	First Build	Cached Build
overlay2	3 min	10 sec
devicemapper	8 min	2 min

Container Density

overlay2 uses less disk space:

# Check disk usage
docker system df

# With overlay2
Images: 10 (5 GB)
Containers: 50 (2 GB)
Volumes: 20 (10 GB)
Total: 17 GB

# With devicemapper
Images: 10 (8 GB)
Containers: 50 (5 GB)
Volumes: 20 (10 GB)
Total: 23 GB

35% less disk usage!

Monitoring Storage

Prometheus metrics:

# Disk usage
node_filesystem_avail_bytes{mountpoint="/var/lib/docker"}

# I/O operations
rate(node_disk_io_time_seconds_total[5m])

# Read/write bytes
rate(node_disk_read_bytes_total[5m])
rate(node_disk_written_bytes_total[5m])

Cleanup

Remove unused data:

# Remove unused images
docker image prune -a

# Remove unused volumes
docker volume prune

# Remove everything unused
docker system prune -a --volumes

Automated cleanup:

#!/bin/bash
# cleanup.sh

# Remove images older than 7 days
docker image prune -a --filter "until=168h" -f

# Remove stopped containers
docker container prune -f

# Remove unused volumes
docker volume prune -f

Cron job:

0 2 * * * /usr/local/bin/cleanup.sh

Production Recommendations

1. Use overlay2:

{
  "storage-driver": "overlay2"
}

2. Separate disk for Docker:

# Dedicated disk
/dev/sdb -> /var/lib/docker

3. Monitor disk usage:

# Alert when > 80%
df -h /var/lib/docker

4. Regular cleanup:

# Weekly cleanup
docker system prune -a -f

5. Limit log size:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Troubleshooting

Issue: Out of disk space

# Check usage
docker system df

# Clean up
docker system prune -a --volumes

Issue: Slow I/O

# Check driver
docker info | grep "Storage Driver"

# Switch to overlay2 if not already

Issue: Can’t start container

# Check logs
journalctl -u docker

# Check disk space
df -h /var/lib/docker

Benchmark Script

#!/bin/bash

echo "=== Docker Storage Driver Benchmark ==="
echo "Driver: $(docker info | grep 'Storage Driver' | awk '{print $3}')"
echo ""

# Write test
echo "Write test (1GB)..."
time docker run --rm ubuntu:18.04 dd if=/dev/zero of=/tmp/test bs=1M count=1000 2>&1 | grep copied

# Read test
echo "Read test..."
docker run -d --name test ubuntu:18.04 dd if=/dev/zero of=/tmp/test bs=1M count=1000
time docker exec test dd if=/tmp/test of=/dev/null bs=1M 2>&1 | grep copied
docker rm -f test

# Build test
echo "Build test..."
cat > Dockerfile.test <<EOF
FROM ubuntu:18.04
RUN apt-get update && apt-get install -y python3
COPY . /app
EOF

time docker build -f Dockerfile.test -t test:latest .
docker rmi test:latest
rm Dockerfile.test

Results

Before (devicemapper loop-lvm):

• Build time: 15 minutes
• Write speed: 50 MB/s
• Disk usage: 23 GB

After (overlay2):

• Build time: 3 minutes (80% faster)
• Write speed: 500 MB/s (10x faster)
• Disk usage: 17 GB (26% less)

Lessons Learned

1. Always use overlay2 - Unless you have specific requirements
2. Avoid loop-lvm - Terrible performance
3. Dedicated disk - Better performance and isolation
4. Monitor disk usage - Prevent out-of-space issues
5. Regular cleanup - Automate it

Conclusion

Docker storage driver choice significantly impacts performance. overlay2 is the clear winner for most use cases.

Key takeaways:

1. Use overlay2 storage driver
2. XFS or ext4 filesystem
3. Dedicated disk for /var/lib/docker
4. Monitor and clean up regularly
5. Benchmark your workload

Switch to overlay2. Your builds and containers will thank you.