How to Set Up Container Monitoring in Datadog
Containers are ephemeral by nature — they spin up, scale out, crash, and get replaced in seconds. Traditional server monitoring doesn't work in this environment because there's no stable host to track. Container monitoring requires a different approach: one that tracks metrics at the container, pod, and service level and correlates them with the orchestrator's view of the world.
Datadog's container monitoring provides a unified view of your containerized infrastructure — from the Kubernetes cluster level down to individual container processes. This guide walks you through configuring Datadog to monitor Docker and Kubernetes workloads effectively, set up meaningful alerts, and build dashboards that help your team understand system health at a glance.
Step-by-step guide
Install the Datadog Agent as a DaemonSet
Deploy the Datadog Agent to your Kubernetes cluster as a DaemonSet so it runs on every node. Use the official Datadog Helm chart: 'helm install datadog-agent datadog/datadog' with your API key and cluster name configured in values.yaml. The DaemonSet ensures that as nodes scale up and down, the agent automatically covers the entire cluster without manual intervention.
- Set the DD_API_KEY and DD_CLUSTER_NAME environment variables in your Helm values file
- Enable the Cluster Agent by setting 'clusterAgent.enabled: true' in values.yaml for Kubernetes-native metrics collection
- Configure resource limits for the agent pods to prevent them from consuming excessive cluster resources
Enable Container and Process Monitoring
Configure the Datadog Agent to collect container-level metrics by enabling the Docker or containerd integration. In your values.yaml, set 'datadog.containerExclude' to filter out system containers you don't care about (like kube-proxy or coredns sidecars). Enable process monitoring by setting 'datadog.processAgent.enabled: true' — this gives you visibility into individual processes running inside containers, which is essential for debugging resource consumption issues.
- Set 'datadog.dockerLabelsAsTags' to automatically convert Docker labels into Datadog tags for filtering
- Enable live container view by setting 'datadog.processAgent.processCollection: true' to see real-time container processes in the Datadog UI
Configure Kubernetes Integration
Enable the Kubernetes integration in the Datadog Cluster Agent to collect orchestrator-level metrics: pod status, deployment replica counts, node conditions, and resource requests vs. limits. These metrics tell you not just whether containers are healthy, but whether the orchestrator is managing them correctly. Configure RBAC permissions so the Cluster Agent can read Kubernetes API resources without elevated cluster-admin privileges.
- Verify that the Cluster Agent is collecting Kubernetes state metrics by checking the 'kubernetes_state.*' metric namespace in Datadog
- Enable Kubernetes events collection to surface pod scheduling failures, OOM kills, and node pressure events in your Datadog event stream
Set Up Application Performance Monitoring (APM)
Install Datadog's APM tracing library in your application containers to get distributed tracing across services. For a Node.js application, add 'dd-trace' and initialize it at the top of your entry point. For Python, add 'ddtrace' and use the 'ddtrace-run' wrapper. Configure the trace agent endpoint to point to the Datadog Agent running on the same node via the Kubernetes downward API (DD_AGENT_HOST set to the node IP).
- Set service name, environment, and version tags in your APM configuration so traces are properly attributed
- Enable runtime metrics collection to correlate application-level performance with container resource consumption
Build Container Monitoring Dashboards
Create a dashboard with four sections: Cluster Overview (node count, pod counts by status, resource utilization), Service Health (request rate, error rate, latency per service from APM), Container Resources (CPU and memory usage by deployment with limits overlaid), and Alerts (active monitors and recent triggered alerts). Use template variables for namespace, deployment, and service so the dashboard can be filtered dynamically to drill into specific workloads.
- Add a 'Top Containers by CPU' top list widget to quickly identify resource-hungry containers
- Include a 'Pods by Status' pie chart showing Running, Pending, Failed, and CrashLoopBackOff counts
- Add a 'Container Restarts' time series to spot instability trends
Configure Alerts for Container Health
Set up monitors for the critical container health signals: container restart count (alert if a pod restarts more than 3 times in 10 minutes — indicates CrashLoopBackOff), CPU throttling percentage (alert above 50% — indicates the container needs higher CPU limits), memory usage approaching limit (alert above 85% — indicates OOM kill risk), and pod scheduling failures (alert if pods stay in Pending state for more than 5 minutes). Each alert should include the container name, namespace, and relevant dashboard link in the notification.
- Use Datadog's anomaly detection for CPU and memory metrics to catch unusual patterns without setting static thresholds
- Set up a composite monitor that triggers only when both high error rate AND high resource consumption occur together, reducing false positives
Common mistakes
Not Setting Resource Requests and Limits
Monitoring containers without resource requests and limits is like monitoring a car without a speedometer — you see the metrics but have no reference point for whether they're normal. Always set CPU and memory requests and limits on your workloads so Datadog can show utilization as a percentage of the allocation.
Alerting on Container-Level Metrics Without Context
A single container using 95% CPU might be perfectly fine if it's a batch job, but alarming if it's a web server. Context matters — use tags to differentiate between workload types and set different thresholds for different services. A one-size-fits-all alert policy generates noise.
Collecting Every Metric Without Filtering
The Datadog Agent can collect hundreds of metrics per container. Without filtering, you'll generate massive data volumes and costs. Configure 'datadog.containerExclude' to skip system containers and limit custom metric collection to the metrics you actually alert on or dashboard.
Ignoring Kubernetes Events
Kubernetes events contain critical information about scheduling failures, image pull errors, liveness probe failures, and resource pressure that raw metrics don't capture. Enable event collection and create monitors for high-severity events like FailedScheduling and OOMKilled.
Tips
Use Datadog's Containers Map view for a visual overview of your entire cluster — it shows container density, resource usage, and health status on a treemap that makes large deployments comprehensible.
Tag all containers with 'service,' 'env,' and 'version' using Datadog's Unified Service Tagging for consistent filtering across metrics, traces, and logs.
Set up log collection alongside container monitoring so you can pivot from a CPU spike on the dashboard directly to the application logs that explain it.
Use Datadog's Notebooks feature to document incident investigations that reference specific container metrics, creating a knowledge base for future troubleshooting.
How Vantage helps
Vantage helps product teams incorporate monitoring and observability requirements into their product specifications from the start. When writing PRDs for new services or features, Vantage ensures that non-functional requirements like container resource limits, alerting thresholds, and dashboard needs are captured alongside the functional requirements.