Datadog vs. Grafana / OpenTelemetry: The $30k/mo Observability Tax & TCO Teardown

1. The Custom Metric & Cardinality Multiplier Trap

The primary driver of unexpected five-figure Datadog invoices is not host agent pricing, but the custom metric cardinality multiplier. By default, Datadog plans include 100 custom metrics per host license. Any metric emitted beyond this quota is billed at $5.00 per 100 metric series per month ($0.05 per active series).

In modern microservice architectures, developers frequently append context tags to application metrics. Consider a standard counter metric tracking payment checkouts:

# Example DogStatsD emission with high-cardinality tags
statsd.increment('checkout.payment.processed', tags=[
    'environment:production',
    'payment_method:credit_card',
    'currency:usd',
    f'user_id:{user.id}',        # 10,000 active users
    f'merchant_id:{merchant.id}'  # 500 active merchants
])

Because Datadog defines a "metric series" as each unique combination of tag key-value pairs, adding user_id and merchant_id generates 5,000,000 distinct time-series. At $0.05 per series, this single line of application code generates a $250,000/month billing overage before engineering alerts trigger.

# Monthly Datadog Bill Calculation
Total_Cost = (Hosts * $31) + (MAX(0, Custom_Metrics - (Hosts * 100)) * $0.05) + (Log_GB * $2.10)

2. Log Ingestion vs. Indexing: The $2.50/GB Hidden Markup

Log management represents the second major cost cliff. Datadog utilizes a dual-tier pricing model for logs:

• Ingestion Pipeline: $0.10 per GB for log ingestion, parsing, and live tailing.
• Indexing & Retention: $1.70 per million log events for 15-day retention ($2.50 per million for 30-day retention).

Assuming an average log message size of 750 bytes, 1 GB contains approximately 1.33 million log events. Therefore, indexing 1 GB of logs in Datadog costs:

$$ ext{Effective Log Cost} = \$0.10 + (1.33 imes \$1.70) = \$2.36 ext{ per GB}$$

At an enterprise scale of 100 GB/day (3,000 GB/month), Datadog log indexing costs $7,080 per month ($84,960/year). In contrast, Amazon S3 Standard object storage costs $0.023 per GB/month. Storing the identical 3,000 GB of compressed logs in an S3 bucket with Grafana Loki indexing costs less than $69.00/month.

3. OpenTelemetry Collector Architecture & Vendor Independence

To eliminate lock-in and decouple telemetry ingestion from vendor pricing, engineering teams utilize the OpenTelemetry (OTel) Collector. Deployed as a Kubernetes DaemonSet, the OTel Collector receives traces, metrics, and logs from application runtimes via the vendor-neutral OTLP standard (gRPC/HTTP), applies local batching and sampling, and routes data to multiple backends simultaneously.

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
data:
  otel-collector-config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318

    processors:
      batch:
        timeout: 1s
        send_batch_size: 1024
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 20

    exporters:
      # Export to Self-Hosted Grafana Mimir
      prometheusremotewrite:
        endpoint: "http://mimir-distributor.monitoring.svc:8080/api/v1/push"
      
      # Dual-ship to Datadog for zero-downtime validation
      datadog:
        api:
          key: "${DD_API_KEY}"
          site: "datadoghq.com"

    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [prometheusremotewrite, datadog]

4. The Grafana LGTM Stack: Infrastructure Cost Modeling

The open-source LGTM stack (Loki, Grafana, Tempo, Mimir) provides an enterprise-grade drop-in replacement for Datadog's three pillars of observability:

For a 50-node Kubernetes cluster generating 80 GB/day of logs and 6,500 custom metrics, the empirical monthly TCO comparison is stark:

=======================================================================
DATADOG SAAS (50 Hosts + 6,500 Custom Metrics + 80 GB/day Logs):
- Host Infra + APM: 50 * $31/mo                     = $1,550.00
- Custom Metrics: (6,500 - 5,000) = 1,500 * $0.05   = $75.00
- Log Ingestion & Indexing: 80GB * 30 * $2.10        = $5,040.00
- Synthetic & Process Monitoring Add-ons             = $450.00
-----------------------------------------------------------------------
TOTAL DATADOG SAAS COST:                            = $7,115.00 / month
                                                    ($85,380.00 / year)

=======================================================================
SELF-HOSTED GRAFANA LGTM ON AWS (m6i.xlarge Spot + S3):
- 3x m6i.xlarge EKS Worker Nodes (Spot/Compute)     = $285.00
- Amazon S3 Object Storage (Metrics/Logs/Traces)    = $65.00
- Network Data Transfer & EBS Volumes               = $45.00
- Allocated Maintenance Hours (3 hrs/mo @ $100/hr)  = $300.00
-----------------------------------------------------------------------
TOTAL SELF-HOSTED LGTM COST:                        = $695.00 / month
                                                    ($8,340.00 / year)

NET ANNUAL BOTTOM-LINE SAVINGS:                     = $77,040.00 / year (90.2% TCO Reduction)
=======================================================================

5. Zero-Downtime Migration Playbook: Dual-Emitting Telemetry

Migrating away from Datadog without risking production outage alerts requires a 3-step staged migration:

1. Deploy the OTel Collector DaemonSet: Install the OpenTelemetry Collector across all Kubernetes nodes and configure your application frameworks (OpenTelemetry Java Agent, Go auto-instrumentation, Python OTel SDK) to export to localhost:4317.
2. Dual-Ship Telemetry: Configure the OTel Collector pipeline to fan out telemetry to both Datadog and your new Grafana Mimir instance. Verify dashboard parity, alerting rule equivalence, and latency metrics for 14 calendar days.
3. Decommission Datadog Agents: Once Grafana OnCall or PagerDuty alert integrations are validated, disable the Datadog exporter in the OTel Collector ConfigMap and terminate Datadog enterprise subscription seats.