How to deliver your logs/metrics to Datadog using Amazon Kinesis Data Firehose

AWS Data and Analytics services have evolved continuously. Recently, for example, I have seen how Amazon Kinesis Data Firehose allows third-party destinations such as Dynatrace, Datadog, and NewRelic, among others. These new integrations will allow us to take our log and metric flows easily to these providers.

Tres logos Dynatrace Datadog New Relic
Some of the third party integrations

When I saw this new release, I began to work on an architecture that I had previously implemented (you can see the architecture in this link). Since we had an on-premise ingest of many logs to AWS through a VPN, I decided then to test the new functionalities and implement the integration with Datadog to see how it made the logs reach Datadog without taking advantage of what was previously implemented.

Chart shows logs to AWS through a VPN

I made the integration with Datadog to test this new functionality and then be able to take the advantages that this tool provides:

  • Log parsing.
  • Log enrichment.
  • Generation of metrics.
  • Filtering and prioritization.

How does the integration work?

The integrations with these providers are all done through an API KEY; you have to select the correct endpoint and use the API KEY generated from the service.

Chart shows integrations with providers done through an API KEY

Datadog Integration

1- Go to the Datadog Configuration page.

2- Create an API KEY.

Chart create API KEY

3- Create Amazon Kinesis Data Firehose and select the endpoint for Datadog.

Create Amazon Kinesis Data Firehose

4- Select the HTTP endpoint URL.

Select the HTTP endpoint URL.

5- Put the API KEY from step 2.

Put the API KEY from step 2.

Check the integration

Check in the Amazon Kinesis Data Firose by monitoring the Records read from Amazon Kinesis Data Stream (Sum).

Chart to Check the integration

Check on Datadog logs. The logs in the image are generated from the sample KPL.

Logs are generated from the sample KPL.

Once you have all the logs in Datadog, you can take advantage of  the functionalities it provides.

Final architecture and conclusions

Chart shows final architecture

In the architecture above, Amazon Kinesis Data Streams receive the information from an on-premise application running the KPL. This information is transmitted over the VPN and using a VPC endpoint; it reaches the Kinesis data streams. It feeds the entire Amazon Kinesis Data Firehose flow to reach the Datadog.

Here are some conclusions and highlights:

1- Integrations for managing logs and metrics are easier through Amazon Kinesis Data Firehose.

2- All integrations must use an API KEY; you must consult third-party provider documentation to do this.

3- These integrations will allow you to take advantage of the functionalities provided by these third-party providers in the management of logs and monitoring.

4- Data that could not be sent to the third-party provider can be stored in a local bucket.

5- It will be possible to centralize N amount of log sources in an Amazon Kinesis Data Stream and use Amazon Kinesis Data Firehose to be sent to Datadog. This use case is widespread since many companies use these tools from other providers as their monitoring and logging core.

6- In this post, I talk about Dynatrace, Datadog, and NewRelic. Still, you have to consider other third-party integrations such as LogicMonitor, MongoDB Cloud, Splunk, and Sumo Logic.

7- Many people might wonder why we do not directly install the Datadog agent on the instance and avoid the use of Kinesis. In this scenario, using Amazon Kinesis Data Stream and Amazon Kinesis Data Firehose allows more flows within AWS that can use the logs, such as a flow that feeds S3 with the logs using Amazon Quicksight and visualizes the information.

Trending Topics
Data & AI
Globant Experience
Healthcare & Life Sciences
Media & Entertainment

Subscribe to our newsletter

Receive the latests news, curated posts and highlights from us. We’ll never spam, we promise.

More From

The CloudOps Studio combines the best cloud technologies, continuous integration, and continuous delivery practices along with cloud operations management and unique capabilities to facilitate new and more efficient ways of doing business.