Managing Grafana Dashboards With Terraform

We’ve all done it — deleted a graph from a dashboard, realised we still need it but have forgotten the query. Use Terraform to go back in time and save yourself the headache

Grafana logo

Grafana is a great visualisation tool. It is powerful, flexible, and generally easy to use.

However, manually administering the dashboards has some cons, for example, accidental deletions of graphs, people “testing” changes and forgetting to clean them up, and general DR capabilities.

Join the Config as Code revolution and store your dashboards as code for ultimate peace of mind.

Terraform

Terraform is probably the most used solution for As Code, and the Grafana project has a Terraform provider to simplify managing Grafana. Here’s a list of those providers.

This won't be a full Terraform tutorial and will assume some knowledge of Terraform.

Let’s Get Started

Authentication

In Terraform, we need to add the Grafana provider to give us access to the Grafana resources. Here’s how to do that:

// Update versions as appropriate for time of reading
terraform {
  required_version = ">= 1.3.0"
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = "1.34.0"
    }
  }
}

provider "grafana" {
  url  = "http://grafana.example.com/"
  auth = var.grafana_auth
}

In this example, the url is the URL of your Grafana instance and auth is an acceptable way to authenticate with Grafana.

As you can see, we need some Terraform credentials to authenticate Grafana.

You could use your username and password in a basic auth combination, e.g., username:password, but a better practice is to use an API key.

Recent versions of Grafana API keys have been replaced with Service Accounts, so we need to set one up to use Terraform.

There is a bit of a chicken and egg dilemma here. Ideally, you would use Terraform to manage your Grafana Service Accounts. but we need to manually create one to use Terraform. Once Terraform is up and running, you could replace the manually created account with one managed by Terraform.

In Grafana, click on the settings (cog) icon from the LHM and choose Service Accounts from the menu ribbon.

Choose Add service account, and call it whatever you want, e.g., Terraform and give it Editor permissions.

Creating Service Account in Grafana UI

Now, choose Add service account token and set the expiry date as appropriate for your security policies.

Copy the token and store it inline with your security policies too. This can now be added to the Terraform provider auth section (or in a var) we started above.

The token can be added to an Environment Variable called GRAFANA_AUTH and the auth argument be omitted from the provider for a more secure, conscious, and flexible approach

Now you can set up the Terraform state as required and run the terraform init

The Dashboard

Dashboards in Grafana are stored internally as JSON, and it is this JSON that Terraform uses to represent the dashboards as code.

For me, the best way to start looking after a dashboard in Terraform is to create it in Grafana manually, export the JSON, and import it into Terraform.

Here, I have manually created a basic dashboard with a couple of panels in Grafana:

Basic Dashboard

To export the JSON, click on the dashboard settings (cog) icon near the top right and then choose JSON Model from the menu.

From here, copy all the JSON into a file in your Terraform directory and name it as you choose.

Terraform Resource

In the Terraform code, we can now add our dashboard resource:

resource "grafana_dashboard" "terraform_maintained" {
  config_json = file("tf_dashboard.json")
}

Now my folder structure looks like the following:

.
├── dashboard.tf
├── providers.tf
├── terraform.tfstate
└── tf_dashboard.json

If we try and do a Terraform apply here, an error will be thrown with a version mismatch. It is an odd error as it says it cannot create the dashboard because it already exists.

Error: status: 412, body: {"message":"The dashboard has been changed by someone else","status":"version-mismatch"}

As stated earlier, we need to import the dashboard that’s been manually created into Terraform (see the docs on how to import if using Grafana organizations).

terraform import grafana_dashboard.terraform_maintained GC797v0Vz

Once imported, the Terraform will apply.

Drift Detection and Correction

Making a manual change to the dashboard will result in Terraform drift (where Terraform shows a change in the configuration).

Let's say someone was testing a change on the dashboard and forgot to remove it:

Change to dashboard

Running a Terraform plan will show the delta between what is stored in the JSON file and what is live (I won't show this as it is quite long).

Running Terraform apply again will set the Dashboard back to how it is defined in the stored JSON before the changes.

We can even delete the whole dashboard. Running Terraform apply will restore it for us.

This same method can be used if you mess something up, want to restore it, and it can also be used to move from one Grafana instance to another (by updating where the provider points to).

Updating the JSON

Of course, there will be times when changes do need to be made to the dashboard. These can either be done in the JSON file and rolled out with Terraform or made manually. After that, the JSON can be re-exported and updated in the Terraform location.

Conclusion

Maintaining dashboards in Terraform is a simple task. One day, it will dig you out of a deep hole, so there is no reason not to do it :).

I would look at storing as much of your Grafana config in Terraform for the same drift and DR reasons, so check out the full provider docs to see what can be done.