Datadog Agent

​

Coverage Matrix

​

Datadog Agent Overview

Datadog Agent collects metrics, logs, and traces from Kubernetes clusters, running as a DaemonSet on each node. It integrates with the Kubernetes API and workloads to provide observability, forwarding data to Datadog for monitoring. The Cluster Agent coordinates metadata collection and reduces API load, improving efficiency in large clusters. Datadog Agent supports auto-discovery, integrations, and security monitoring, making it a flexible solution for infrastructure visibility. Chkk ensures its seamless deployment, monitoring, and upgrade safety in Kubernetes environments.

​

Chkk Coverage

​

Curated Release Notes

Chkk curates Datadog Agent release notes, highlighting new features, breaking changes, and operational impacts. Instead of parsing every upstream detail, engineers receive targeted insights on critical changes, deprecated configurations, and feature adjustments. If an update requires RBAC changes or introduces new API dependencies, Chkk flags it beforehand. Release summaries also assess potential resource impact and performance shifts. This approach streamlines decision-making and prevents misconfigurations.

​

Preflight & Postflight Checks

Before upgrades, Chkk runs preflight checks to verify Kubernetes version compatibility, deprecated fields, and required permissions. It ensures configuration changes (e.g., log collection toggles, new API calls) won’t disrupt monitoring. Post-upgrade, Chkk confirms Agent pods are healthy, data is flowing, and no unexpected failures have occurred. It detects memory spikes, missing logs, or CRD issues early, preventing monitoring blind spots. These validations reduce risk and improve upgrade confidence.

​

Version Recommendations

Chkk monitors Datadog Agent’s lifecycle, notifying teams of security risks and upcoming EOL versions. It suggests stable upgrade paths based on support status and community-reported issues, ensuring compatibility and reliability. If a version has known bugs, Chkk recommends skipping or waiting for a fix. It also tracks Kubernetes API changes that might impact older Agent versions. This proactive approach minimizes unplanned outages and ensures ongoing support.

​

Upgrade Templates

Chkk provides Upgrade Templates for in-place rolling updates and blue-green deployments. In-place upgrades roll out Agents node by node while monitoring resource impact. Blue-green strategies deploy a parallel Agent set for validation before cluster-wide adoption. Chkk includes rollback steps, ensuring quick recovery if issues arise. These templates integrate with GitOps workflows and reduce manual intervention risks. Teams gain predictable, controlled upgrades with minimal downtime.

​

Preverification

Preverification simulates Datadog Agent upgrades in an isolated environment before live deployment. This dry-run detects configuration conflicts, RBAC gaps, or API incompatibilities without affecting production. If a new Agent version crashes due to missing dependencies, Chkk flags it early. Engineers can iterate on fixes before rolling out the upgrade. This process enhances stability and prevents production incidents.

​

Supported Packages

Chkk supports Helm, Datadog Operator, Kubernetes YAML, and Kustomize-based deployments. It tracks Helm chart versions, validates Operator CRs, and ensures static manifests stay in sync. Custom Agent images and private registries are fully supported, ensuring consistency across environments. If integrations like kube-state-metrics are used, Chkk verifies their compatibility with Agent versions. This flexibility allows seamless adoption across different Kubernetes setups.

​

Common Operational Considerations

• Configuration Pitfalls: Features like log collection and APM tracing must be explicitly enabled; default settings might leave critical data unmonitored. Always review datadog.yaml settings to ensure required integrations and logs are correctly configured.
• Resource Utilization & Tuning: The Agent can consume significant CPU/memory in large clusters, often due to frequent metric polling. Tune min_collection_interval and filter unnecessary logs to optimize resource usage.
• Integration & Autodiscovery: Properly annotate pods for Autodiscovery to automatically enable monitoring for services like Redis or NGINX. Running kube-state-metrics alongside the Agent ensures full cluster observability.
• Cluster Agent Usage: The Datadog Cluster Agent offloads metadata collection, reducing API server strain. Large clusters should deploy it for improved efficiency and scalability.
• Kubernetes Compatibility: Kubernetes API changes (e.g., EndpointSlices replacing Endpoints) may require Agent updates and RBAC adjustments. Always verify Kubernetes version compatibility before upgrading.
• Staying Up-to-Date: Monthly Agent updates are recommended for security and stability. Avoid deploying versions with known issues, and test in a canary environment before full rollout.

​

Additional Resources

• Datadog Agent Documentation
• Datadog Agent Releases