Why Terraform?

In this post I described how to display AWS Billing metrics in Grafana Cloud. Therefore it was necessary to create manually the data source and the dashboard. With Terraform, you can describe the setup as code and benefit from the full advantages of IaC.

Terraform

Terraform is a tool for infrastructure as code and works with many different provider. Terraform comes with a CLI for the deployments.

Grafana Provider and Dashboard declaration

For this use case, you need a Grafana data source and a Grafana dashboard. These configurations have to defined in a .tf file like this one

At first the provider.

provider "grafana" {
  url  = var.grafana_url
  auth = var.grafana_api_key
}

Then the data source and dashboard. The dashboard section links to the file dashboards/aws-billing.json.

resource "grafana_data_source" "cloudwatch" {
  type = "cloudwatch"
  name = "CloudWatch"

  json_data {
    default_region = var.region
    auth_type      = "keys"
  }

  secure_json_data {
    access_key = var.access_key_grafana
    secret_key = var.secret_key_grafana
  }
}

resource "grafana_dashboard" "metrics" {
  config_json = file("dashboards/aws-billing.json")
}

For security reasons and flexible sharing of the template, the parameters for secrets and variables like region are in a .env file. This is the template for that.

# The URL of the Grafana instance
export TF_VAR_grafana_url=
# The API key of the Grafana instance
export TF_VAR_grafana_api_key=
# IAM user like described here: https://grafana.com/docs/grafana/latest/datasources/cloudwatch/#iam-policies
export TF_VAR_access_key_grafana=
export TF_VAR_secret_key_grafana=
# Default region of the data source
export TF_VAR_region=

The declaration of Terraform variables looks like that.

variable "grafana_url" {}
variable "grafana_api_key" {}
variable "access_key_grafana" {}
variable "secret_key_grafana" {}
variable "region" {}

In this case it’s in the file variable.tf like described here.

Grafana API Key

Terraform can “communicate” with Grafana via an API key. Navigate to this URL “https://<>/org/apikeys” and create on with the role “Admin”.

Put the API key into the .env file.

Usage of the env file

Before the creation of the S3 Backend and the deployment run the command source .env.

Terraform AWS S3 backend

This setup so far works for the first deployment. Changes and a redeployment lead to an error because the resource already exists. Therefore it’s necessary to extend the setup with a Terraform backend. In this example, it’s a S3 backend.

Unfortunately, it’s not possible to use variables here. This is discussed in this issue with some approaches for workarounds. I use this one, more or less.

Concrete I put a script around the command terraform init. This script can use the environment variables and create a terraform file for the backend.

#!/bin/sh
cat > ./backend.tf << EOF
terraform {
  backend "s3" {
    bucket = "${TF_VAR_s3_bucket_name}"
    key    = "${TF_VAR_backend_key}"
    region = "${TF_VAR_region}"
  }
}
EOF
 
terraform init -input=false

This script creates the bucket.

#!/bin/sh
 
aws s3api create-bucket --bucket $TF_VAR_s3_bucket_name --region $TF_VAR_region

For the backend, it needs an IAM user. This script creates the user and return access and secret key. Put that into the .env file.

# Name of the Terrafrom S3 backend for state handling
export TF_VAR_s3_bucket_name=
# Name of the state file
export TF_VAR_backend_key=terraform.tfstate
# IAM user credentials to access S3 and write the state file
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=

#!/bin/sh
 
aws iam create-user --user-name terraform_state
 
aws iam create-access-key --user-name terraform_state

This script creates and attach the missing policy.

#!/bin/bash
 
cat > ./scripts/policy << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::${TF_VAR_s3_bucket_name}"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::${TF_VAR_s3_bucket_name}/*"
    }
  ]
}
EOF
 
aws iam create-policy --policy-name terraform_state --policy-document file://scripts/policy
 
ARN=$(aws iam list-policies --query 'Policies[?PolicyName==`terraform_state`].Arn' --output text)
 
aws iam attach-user-policy --policy-arn $ARN --user-name terraform_state

Deployment commands

Once the S3 backend is created, you’re a few commands away from the deployment.

At first, the initialization of Terraform, which is wrapped in a script.

sh scripts/terraformInit.sh

For the next commands, the Terraform CLI is sufficient.

validate: terraform validate

plan: terraform plan

apply: terraform apply

Grafana Dasboard changes

The dashboard can now be changed directly via the JSON file in the folder dashboards. The easier way is to do that manually in Grafana and copy the changed JSON via the share functionality.

Overwrite the file aws-billing.json with the JSON from Grafana and redeploy.

CI/CD pipeline

The local deployment is also possible with a CI/CD pipeline. In this example it’s with GitHub actions. Instead of the .env file, the variables and credentials coming from GitHub secrets.

Code

https://github.com/JohannesKonings/aws-grafana-billing-dashboard