Microservices Configuration Management: Consul vs etcd in Production

We had 30 microservices. Each had config files. Changing a database URL meant updating 30 repos, 30 deployments. It took hours. Mistakes were common.

I implemented centralized config with Consul. Now config changes take 30 seconds. No deployments. Services auto-reload. Game changer.

The Problem

Config in files:

# service1/config.yaml
database:
  host: db.example.com
  port: 5432
  
redis:
  host: redis.example.com
  port: 6379

Issues:

Scattered across repos
Requires deployment to change
No audit trail
Secrets in version control
Environment-specific configs

Consul Overview

Consul provides:

Key-Value store: Configuration storage
Service discovery: Find services
Health checking: Monitor services
Multi-datacenter: Global config

Installing Consul

wget https://releases.hashicorp.com/consul/1.6.1/consul_1.6.1_linux_amd64.zip
unzip consul_1.6.1_linux_amd64.zip
sudo mv consul /usr/local/bin/

Start server:

consul agent -server -bootstrap-expect=3 -data-dir=/tmp/consul -node=server1 -bind=10.0.1.1 -ui

Storing Configuration

Store config in Consul:

# Database config
consul kv put config/database/host db.example.com
consul kv put config/database/port 5432
consul kv put config/database/username dbuser

# Redis config
consul kv put config/redis/host redis.example.com
consul kv put config/redis/port 6379

# Feature flags
consul kv put config/features/new-ui true

Hierarchical structure:

config/
  database/
    host
    port
    username
  redis/
    host
    port
  features/
    new-ui
    beta-feature

Reading Configuration

Python client:

import consul

c = consul.Consul(host='localhost', port=8500)

# Get single value
db_host = c.kv.get('config/database/host')[1]['Value'].decode('utf-8')

# Get all database config
index, data = c.kv.get('config/database/', recurse=True)
db_config = {item['Key'].split('/')[-1]: item['Value'].decode('utf-8') for item in data}

print(db_config)
# {'host': 'db.example.com', 'port': '5432', 'username': 'dbuser'}

Go client:

package main

import (
    "fmt"
    "github.com/hashicorp/consul/api"
)

func main() {
    client, _ := api.NewClient(api.DefaultConfig())
    kv := client.KV()
    
    // Get value
    pair, _, _ := kv.Get("config/database/host", nil)
    fmt.Println(string(pair.Value))
    
    // Get all database config
    pairs, _, _ := kv.List("config/database/", nil)
    for _, p := range pairs {
        fmt.Printf("%s: %s\n", p.Key, string(p.Value))
    }
}

Watch for Changes

Auto-reload on config change:

import consul
import time

c = consul.Consul()

def load_config():
    index, data = c.kv.get('config/database/', recurse=True)
    return {item['Key'].split('/')[-1]: item['Value'].decode('utf-8') for item in data}

# Initial load
config = load_config()
index = c.kv.get('config/database/')[0]

# Watch for changes
while True:
    index, data = c.kv.get('config/database/', index=index, wait='30s')
    if data:
        new_config = load_config()
        if new_config != config:
            print("Config changed!")
            config = new_config
            # Reload application
            reload_app(config)

Environment-Specific Config

Organize by environment:

# Production
consul kv put config/prod/database/host prod-db.example.com

# Staging
consul kv put config/staging/database/host staging-db.example.com

# Development
consul kv put config/dev/database/host localhost

Load based on environment:

import os

env = os.getenv('ENVIRONMENT', 'dev')
db_host = c.kv.get(f'config/{env}/database/host')[1]['Value'].decode('utf-8')

Secrets Management

Use Consul with Vault:

# Store secret in Vault
vault kv put secret/database password=supersecret

# Reference in Consul
consul kv put config/database/password-path secret/database

Application code:

import hvac

# Get Vault path from Consul
vault_path = c.kv.get('config/database/password-path')[1]['Value'].decode('utf-8')

# Get secret from Vault
vault_client = hvac.Client(url='http://localhost:8200')
secret = vault_client.secrets.kv.v2.read_secret_version(path='database')
password = secret['data']['data']['password']

etcd Alternative

Install etcd:

wget https://github.com/etcd-io/etcd/releases/download/v3.4.0/etcd-v3.4.0-linux-amd64.tar.gz
tar xvf etcd-v3.4.0-linux-amd64.tar.gz
sudo mv etcd-v3.4.0-linux-amd64/etcd* /usr/local/bin/

Start:

etcd --name node1 --data-dir /tmp/etcd

Store config:

etcdctl put /config/database/host db.example.com
etcdctl put /config/database/port 5432

Python client:

import etcd3

etcd = etcd3.client(host='localhost', port=2379)

# Set value
etcd.put('/config/database/host', 'db.example.com')

# Get value
value, metadata = etcd.get('/config/database/host')
print(value.decode('utf-8'))

# Watch for changes
watch_id = etcd.add_watch_callback('/config/database/', callback)

Consul vs etcd

Consul:

✅ Built-in service discovery
✅ Health checking
✅ Multi-datacenter support
✅ Web UI
❌ More complex

etcd:

✅ Simpler
✅ Better Kubernetes integration
✅ Faster for pure KV
❌ No built-in service discovery
❌ No health checking

Our choice: Consul (needed service discovery)

Service Discovery

from consul import Consul

c = Consul()

c.agent.service.register(
    name='web-app',
    service_id='web-app-1',
    address='10.0.1.10',
    port=8080,
    check={
        'http': 'http://10.0.1.10:8080/health',
        'interval': '10s'
    }
)

Discover service:

# Find healthy instances
index, services = c.health.service('web-app', passing=True)

for service in services:
    print(f"{service['Service']['Address']}:{service['Service']['Port']}")

Health Checks

HTTP check:

{
  "check": {
    "id": "web-app-health",
    "name": "Web App Health",
    "http": "http://localhost:8080/health",
    "interval": "10s",
    "timeout": "1s"
  }
}

TCP check:

{
  "check": {
    "id": "redis-health",
    "name": "Redis Health",
    "tcp": "localhost:6379",
    "interval": "10s"
  }
}

Script check:

{
  "check": {
    "id": "disk-health",
    "name": "Disk Health",
    "args": ["/usr/local/bin/check_disk.sh"],
    "interval": "30s"
  }
}

Configuration Templates

Use consul-template:

consul-template -template "config.tmpl:config.yaml"

Template (config.tmpl):

database:
  host: {{ key "config/database/host" }}
  port: {{ key "config/database/port" }}
  username: {{ key "config/database/username" }}

redis:
  host: {{ key "config/redis/host" }}
  port: {{ key "config/redis/port" }}

features:
  new_ui: {{ key "config/features/new-ui" }}

Auto-reload on change:

consul-template \
  -template "config.tmpl:config.yaml:systemctl reload myapp" \
  -wait 5s

Kubernetes Integration

ConfigMap from Consul:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  database.host: {{ consul_key "config/database/host" }}
  database.port: {{ consul_key "config/database/port" }}

Or use Consul directly:

apiVersion: v1
kind: Pod
metadata:
  name: web-app
spec:
  containers:
  - name: app
    image: web-app:latest
    env:
    - name: CONSUL_HTTP_ADDR
      value: "consul:8500"

Monitoring

Prometheus metrics:

# Consul cluster health
consul_up

# Key-value operations
rate(consul_kvs_apply[5m])

# Service health
consul_health_service_status

Backup and Restore

Backup:

consul snapshot save backup.snap

Restore:

consul snapshot restore backup.snap

Automated backups:

#!/bin/bash
DATE=$(date +%Y%m%d_%H%M%S)
consul snapshot save /backups/consul_$DATE.snap
find /backups -name "consul_*.snap" -mtime +7 -delete

Results

Before:

Config in 30 repos
Hours to change config
Frequent mistakes
No audit trail

After:

Centralized config
30-second config changes
Auto-reload
Complete audit trail
99.99% uptime

Lessons Learned

Start with Consul - Service discovery is valuable
Use templates - consul-template is powerful
Monitor health - Catch issues early
Backup regularly - Automate it
Organize hierarchically - env/service/key

Conclusion

Centralized configuration management is essential for microservices. Consul provides config, service discovery, and health checking in one tool.

Key takeaways:

Centralize configuration
Watch for changes
Use service discovery
Implement health checks
Automate backups

Stop managing config files. Use Consul for microservices configuration.

Table of Contents