Connectivity issue for agents 3.6 to 5.32.6
Incident Report for Datadog EU
Resolved
The incident has been resolved. For additional information on upgrading agents 3.6 to 5.32.6, see https://docs.datadoghq.com/agent/faq/certificate_verify_failed-error/.
Posted May 30, 2020 - 16:22 EDT
Update
We've released Datadog agent 5.32.7 which fixes the certificate problem and restores connectivity to Datadog.

To upgrade from a previous version of agent 5, use the install command for your platform:

Centos/Red Hat: sudo yum check-update && sudo yum install datadog-agent
Debian/Ubuntu: sudo apt-get update && sudo apt-get install datadog-agent
Windows: Download the Datadog Agent installer (https://s3.amazonaws.com/ddagent-windows-stable/ddagent-cli-latest.msi). start /wait msiexec /qn /i ddagent-cli-latest.msi
More platforms and configuration management options detailed at https://app.datadoghq.com/account/settings?agent_version=5#agent
Posted May 30, 2020 - 15:54 EDT
Update
What happened?

On Saturday May 30th, 2020, at 10:48 UTC, an SSL root certificate belonging to a certificate authority and used to sign some of the Datadog certificates expired, and caused some agents to lose connectivity with Datadog endpoints. Because this root certificate is embedded in agents from version 3.6 to version 5.32.6, some users will need to take action to restore connectivity.

What versions of the agent are affected?

Agent versions spanning 3.6.x to 5.32.6 embed the expired root certificate and are affected. Agent versions 6.x and 7.x are unaffected; if you are using these agents, no action is required on your part.

Fixing without updating the agent

We’re actively working on a new version of agent 5 but if you’d like to address this without an update, the following is the quickest path to resolution.

On Linux:

sudo rm /opt/datadog-agent/agent/datadog-cert.pem && sudo service datadog-agent restart

On Windows:

rm "C:\Program Files (x86)\Datadog\Datadog Agent\files\datadog-cert.pem"
net stop /y datadogagent ; net start /y datadogagent
Posted May 30, 2020 - 12:47 EDT
Monitoring
A fix has been implemented and we are monitoring the results. We are contacting customers directly who are running versions of the agent which may still have connectivity issues.
Posted May 30, 2020 - 11:05 EDT
This incident affected: API.