Monitoring Geth with InfluxDB and Grafana

This tutorial will help you set up monitoring for your Geth node so you can better understand its performance and identify potential problems.

Prerequisites

• You should already be running an instance of Geth.
• Most of the steps and examples are for linux environment, basic terminal knowledge will be helpful.
• Check out this video overview of Geth's suite of metrics: Monitoring an Ethereum infrastructure by Péter Szilágyi(opens in a new tab).

Monitoring stack

An Ethereum client collects lots of data which can be read in the form of a chronological database. To make monitoring easier, you can feed this into data visualisation software. There are multiple options available:

• Prometheus(opens in a new tab) (pull model)
• InfluxDB(opens in a new tab) (push model)
• Telegraf(opens in a new tab)
• Grafana(opens in a new tab)
• Datadog(opens in a new tab)
• Chronograf(opens in a new tab)

There's also Geth Prometheus Exporter(opens in a new tab), an option preconfigured with InfluxDB and Grafana. You can set it up easily using docker and Ethbian OS(opens in a new tab) for RPi 4.

In this tutorial, we'll set up your Geth client to push data to InfluxDB to create a database and Grafana to create a graph visualisation of the data. Doing it manually will help you understand the process better, alter it, and deploy in different environments.

Setting up InfluxDB

First, let's download and install InfluxDB. Various download options can be found at Influxdata release page(opens in a new tab). Pick the one that suits your environment. You can also install it from a repository(opens in a new tab). For example in Debian based distribution:

1curl -tlsv1.3 --proto =https -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add
2source /etc/lsb-release
3echo "deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
4sudo apt update
5sudo apt install influxdb -y
6sudo systemctl enable influxdb
7sudo systemctl start influxdb
8sudo apt install influxdb-client

After successfully installing InfluxDB, make sure it's running on background. By default, it is reachable at localhost:8086. Before using influx client, you have to create new user with admin privileges. This user will serve for high level management, creating databases and users.

1curl -XPOST "http://localhost:8086/query" --data-urlencode "q=CREATE USER username WITH PASSWORD 'password' WITH ALL PRIVILEGES"

Now you can use influx client to enter InfluxDB shell(opens in a new tab) with this user.

1influx -username 'username' -password 'password'

Directly communicating with InfluxDB in its shell, you can create database and user for geth metrics.

1create database geth
2create user geth with password choosepassword

Verify created entries with:

1show databases
2show users

Leave InfluxDB shell.

1exit

InfluxDB is running and configured to store metrics from Geth.

Preparing Geth

After setting up database, we need to enable metrics collection in Geth. Pay attention to METRICS AND STATS OPTIONS in geth --help. Multiple options can be found there, in this case we want Geth to push data into InfluxDB. Basic setup specifies endpoint where InfluxDB is reachable and authentication for the database.

1geth --metrics --metrics.influxdb --metrics.influxdb.endpoint "http://0.0.0.0:8086" --metrics.influxdb.username "geth" --metrics.influxdb.password "chosenpassword"

This flags can be appended to a command starting the client or saved to the configuration file.

You can verify that Geth is successfully pushing data, for instance by listing metrics in database. In InfluxDB shell:

1use geth
2show measurements

Setting up Grafana

Next step is installing Grafana which will interpret data graphically. Follow installation process for your environment in Grafana documentation. Make sure to install OSS version if you don't want otherwise. Example installation steps for Debian distributions using repository:

1curl -tlsv1.3 --proto =https -sL https://packages.grafana.com/gpg.key | sudo apt-key add -
2echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
3sudo apt update
4sudo apt install grafana
5sudo systemctl enable grafana-server
6sudo systemctl start grafana-server

When you've got Grafana running, it should be reachable at localhost:3000. Use your preferred browser to access this path, then login with the default credentials (user: admin and password: admin). When prompted, change the default password and save.

You will be redirected to the Grafana home page. First, set up your source data. Click on the configuration icon in the left bar and select "Data sources".

There aren't any data sources created yet, click on "Add data source" to define one.

For this setup, select "InfluxDB" and proceed.

Data source configuration is pretty straight forward if you are running tools on the same machine. You need to set the InfluxDB address and details for accessing the database. Refer to the picture below.

If everything is complete and InfluxDB is reachable, click on "Save and test" and wait for the confirmation to pop up.

Grafana is now set up to read data from InfluxDB. Now you need to create a dashboard which will interpret and display it. Dashboards properties are encoded in JSON files which can be created by anybody and easily imported. On the left bar, click on "Create and Import".

For a Geth monitoring dashboard, copy the ID of this dashboard(opens in a new tab) and paste it in the "Import page" in Grafana. After saving the dashboard, it should look like this:

You can modify your dashboards. Each panel can be edited, moved, removed or added. You can change your configurations. It's up to you! To learn more about how dashboards work, refer to Grafana's documentation(opens in a new tab). You might also be interested in Alerting(opens in a new tab). This lets you set up alert notifications for when metrics reach certain values. Various communication channels are supported.