BDWatchdogFaaS

BDWatchdogFaaS is a real-time monitoring tool for Funcions-as-a-Service in both AWS and Azure.

BDWatchdogFaaS logo

BDWatchdogFaaS processs the logs from Lambda and Azure Functions in order to build and store a common model to both providers, allowing to consult, analyze and monitor function logs from AWS and Azure indistinctly. The transformation of logs into the common model is done by a FaaS in the corresponding provider, which in near real-time ingests, processes and sends the data to a common datalake. In addition, the data is forwarded to a Power BI dashboard so that the serverless functions can be monitored easily.

Deployment

BDWatchdogFaaS is available in the AWS Serverless Application Repository (SAR), so you can deploy it easily in your environment.

Click here to start using BDWatchdogFaaS.

Architecture

BDWatchdogFaaS is composed of 4 different modules:

• Azure monitoring module
• AWS monitoring module
• Stats module
• Visualization module

BDWatchdogFaaS architecture

Azure

This module deals with the monitoring of Azure Functions and it has been implemented completely using Azure services. When an Azure Function is executed, Azure automatically generate a series of logs which are stored in Application Insights, the APM tool provided by Azure.

Once in Application Insights, logs are immediately sent to a queue of EvenHub, which work as trigger to a Azure Function, called AzureLogsForwarder.

This FaaS handles the task of processing logs to extract the relevant information, and then forwards it to the statistics and visualization modules.

Azure monitoring module

AWS

This module deals with the monitoring of AWS Lambda. Analogous to the Azure monitoring module, this module has been built on top of AWS services.

First, when a Lambda is executed, AWS registers the logs in a log group of Cloudwatch, the monitoring service of AWS. Using a subscription filter logs are automatically sent to a Lambda function, called AWSLogsForwarder, which is trigger as response to the incoming data.

AWS monitoring module

Stats

This module has been developed to store the logs processed by the previous modules in an AWS DynamoDB Serverless table. DynamoDB is a fully managed, key-value NoSQL database designed to run high-performance applications provided by AWS.

The table keeps all processed logs from both Azure Functions and AWS Lambda in a single point, so that another Lambda function, LogStatsForwarder, can read the data and obtain overall statistics about the FaaS being monitored in both cloud providers. These statistics are forwarded to the visualization module as well.

Stats module

Visualization

Finally, the visualization module is built making use of the capabilities for real-time visualization of streaming datasets in Power BI Service. This application allows the developers to define streaming datasets that can be feeded through an API, sending the data using HTTP requests.

Visualization module

In order to work with BDWatchdog, you have to create these 4 streaming datasets in Power BI Service:

1. Azure logs

   [
       {
       "id" :"AAAAA555555",
       "source" :"AAAAA555555",
       "timestamp" :"2023-09-04T19:14:13.406Z",
       "funcName" :"AAAAA555555",
       "duration" :98.6,
       "result" :"AAAAA555555",
       "operationId" :"AAAAA555555"
       }
   ]
   

2. Lambda logs

   [
       {
       "id" :"AAAAA555555",
       "source" :"AAAAA555555",
       "timestamp" :"2023-09-04T19:12:33.210Z",
       "duration" :98.6,
       "billedDuration" :98.6,
       "memorySize" :98.6,
       "memoryUsed" :98.6,
       "operationId" :"AAAAA555555",
       "result" :"AAAAA555555",
       "funcName" :"AAAAA555555"
       }
   ]
   

3. Common logs

   [
       {
       "id" :"AAAAA555555",
       "source" :"AAAAA555555",
       "timestamp" :"2023-09-04T19:15:44.764Z",
       "funcName" :"AAAAA555555",
       "duration" :98.6,
       "result" :"AAAAA555555",
       "operationId" :"AAAAA555555"
       }
   ]
   

4. Stats

   [
       {
       "totalInvocations" :98.6,
       "totalSuccesses" :98.6,
       "totalFailures" :98.6,
       "percentageSuccess" :98.6,
       "avgDuration" :98.6,
       "avgSuccessDuration" :98.6,
       "avgFailureDuration" :98.6,
       "avgMemory" :98.6,
       "avgSuccessMemory" :98.6,
       "avgFailureMemory" :98.6,
       "timestamp" :"2023-09-04T19:17:00.077Z",
       "avgMemorySize" :98.6,
       "avgSuccessMemorySize" :98.6,
       "avgFailureMemorySize" :98.6
       }
   ]