How to Set Up Log Management in Datadog
Logs are the last line of defense when something goes wrong in production. Metrics tell you that error rates spiked. Traces tell you which service is slow. But logs tell you exactly what happened — the malformed request payload, the database connection timeout, the null pointer that crashed the worker. Without centralized log management, debugging production issues means SSH-ing into servers, grep-ing through files, and hoping the relevant log has not been rotated out.
Datadog's log management platform collects logs from every service, parses them into structured data, and provides a single search interface across your entire stack. Combined with Datadog's metrics and APM, you can jump from an alert to the relevant logs in seconds. This guide covers setting up Datadog log management from agent installation through building log-based monitors that catch issues before users report them.
Step-by-step guide
Plan your log collection architecture
Before installing anything, decide what to collect and how. Identify every log source: application logs (API, workers, frontend errors), infrastructure logs (nginx, PostgreSQL, Redis), and cloud service logs (AWS CloudWatch, S3 access logs). For each source, determine the transport method: Datadog Agent for servers and containers, direct API submission for serverless functions, and cloud integration for managed services. Set a retention policy based on compliance needs and budget — 15 days is default, extend to 30-90 days for production logs.
- List every log source in your infrastructure with its format and volume estimate
- Decide retention periods: 15 days for debug-level, 30 days for info, 90 days for error and above
- Estimate monthly log volume in GB to predict costs (Datadog charges per ingested GB)
Install and configure the Datadog Agent
Install the Datadog Agent on every host that generates logs. For Docker environments, run the Agent as a container with log collection enabled. For Kubernetes, deploy the Agent as a DaemonSet. Configure the Agent's datadog.yaml to enable log collection: set logs_enabled to true. For each log source, create a configuration file in conf.d/ that specifies the log file path, service name, and source tag. The service and source tags are critical for linking logs to APM traces and for applying correct parsing pipelines.
- Install the Datadog Agent using the one-line install script for your OS or container runtime
- Set logs_enabled: true in datadog.yaml
- Create a conf.d/your-service.d/conf.yaml for each application with type, path, service, and source
- For Docker, set log labels or use autodiscovery with container annotations
Implement structured logging in your applications
Structured logs (JSON format with consistent fields) are dramatically easier to search and alert on than unstructured text. Configure your application loggers to output JSON with these standard fields: timestamp, level (info/warn/error), service, message, and a trace_id for APM correlation. Add contextual fields relevant to your application: user_id, request_id, endpoint, duration_ms. Avoid logging sensitive data — mask or exclude PII, tokens, and passwords from log output.
- Configure your logger to output JSON format with consistent field names
- Include trace_id and span_id for APM correlation in every log line
- Add a logger middleware that automatically attaches request_id and user_id to all logs in a request
- Audit log output for PII and add exclusion rules for sensitive fields
Set up log processing pipelines
In the Datadog web UI, navigate to Logs > Configuration > Pipelines. Create a pipeline for each service that transforms raw logs into structured, searchable data. Add processors for: Grok Parser (to extract fields from unstructured log lines), Attribute Remapper (to standardize field names across services), Status Remapper (to map your log levels to Datadog's severity levels), and Trace ID Remapper (to link logs to APM traces). Test each pipeline with sample logs before applying to production.
- Create one pipeline per service (API, workers, database, etc.)
- Add a Grok Parser for any log sources that output non-JSON format
- Add a Status Remapper to map your level field to Datadog's standard severity
- Add a Trace ID Remapper using your trace_id field for APM correlation
Build log-based monitors and alerts
Create monitors that alert on log patterns indicating problems. Essential monitors: error rate spike (count of error-level logs exceeds 2x the rolling average), specific error patterns (new error messages that have never appeared before), and absence-based monitors (a critical service stops logging entirely, indicating a crash). Route alerts to appropriate channels: critical alerts to PagerDuty, warnings to Slack, and informational to email digest.
- Create a log monitor for error rate: alert when error count exceeds 2x the 1-hour rolling average
- Create a log monitor for specific fatal patterns: 'database connection refused', 'out of memory'
- Create an absence monitor that fires if no logs are received from a critical service for 5 minutes
- Configure notification routing: critical to PagerDuty, warn to Slack, info to email
Create saved views and dashboards
Build saved views in the Log Explorer for common debugging scenarios: all errors for a specific service, logs for a specific user (filtered by user_id), and logs matching a specific request (filtered by request_id). Create a log dashboard with widgets showing: error count by service over time, top 10 error messages, log volume by level, and a live tail of the most recent error logs. Pin this dashboard for on-call engineers who need instant visibility during incidents.
- Create saved views for 'Production Errors', 'Slow Queries', and 'Authentication Failures'
- Build a dashboard with error count time series, top error messages table, and volume by level
- Add a dashboard widget for live tail of error-level logs
- Share the dashboard URL in your on-call runbook
Optimize costs with log filtering and exclusion
Log volume directly impacts Datadog costs. After running for a week, review which logs provide value and which are noise. Set up exclusion filters to drop verbose debug logs from production, health check logs from load balancers, and repetitive info logs that do not aid debugging. Use Datadog's metrics from logs feature to extract counts and aggregates from high-volume logs, then exclude the raw logs. This gives you the statistical insight without the storage cost.
Common mistakes
Logging everything at debug level in production
Debug-level logging in production generates massive volume, increases costs, and buries important signals in noise. Use info level as your production default, warn for concerning conditions, and error for failures. Enable debug-level logging only temporarily during active investigations using a feature flag or environment variable.
Not correlating logs with traces
Logs and APM traces are exponentially more valuable when linked. Without trace_id in your logs, debugging a slow request requires manually correlating timestamps across services. Adding trace_id to every log line lets you click from an APM trace directly to all logs for that request across all services.
Inconsistent log formats across services
When your API logs JSON, your worker logs plain text, and your database logs syslog format, you need three different parsing pipelines and cannot search across services with consistent field names. Standardize on JSON logging with consistent field names across all services.
Tips
Use Datadog's Log Patterns feature to automatically identify recurring log patterns — this surfaces the most common log messages and helps you decide which to keep, alert on, or exclude.
Add custom facets for your most-queried fields (user_id, endpoint, status_code) so they appear in the Log Explorer sidebar for quick filtering.
Set up daily log volume reports sent to your engineering channel — sudden volume increases often indicate a new bug causing excessive error logging.
Use Datadog's Sensitive Data Scanner to automatically detect and mask credit card numbers, email addresses, and other PII that may accidentally appear in logs.
How Vantage helps
Vantage integrates with Datadog to bring observability context into product planning. When writing PRDs and generating tickets, Vantage can surface relevant operational patterns and error trends, helping teams plan features with production reliability in mind from the requirements phase.