Integrating Grafana into Icinga

Integrating Grafana graph inside Icinga2 WebUI

The following guide will show how to integrate Grafana graphs into an existing Icinga2 installation.

Icinga2

Hopefully, it’s already installed!

Installing Grafana + InfluxDB

Add repos

sudo bash -c 'cat <<EOF > /etc/apt/sources.list.d/influxdata.list
# InfluxDB
deb https://repos.influxdata.com/debian buster stable
# Grafana
deb https://packages.grafana.com/oss/deb stable main
EOF
'

Add keys

wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
wget -q -O - https://repos.influxdata.com/influxdb.key | sudo apt-key add -

Update package list

sudo apt update

Install packages

sudo apt install grafana influxdb

Launches services

sudo systemctl daemon-reload
sudo systemctl enable --now grafana-server
sudo systemctl enable --now influxdb

NB : Grafana should now run on port 3000, and InfluxDB listen for data on port 8086.
Secure those ports as needed, they run on 0.0.0.0 by default.
Default Grafana credentials is admin/admin, change it ASAP. :slight_smile:

Configure Grafana

Enable Grafana image renderer plugin

sudo grafana-cli plugins install grafana-image-renderer

The following command will install a LOT of stuff (X11 related…) in order for Grafana to generate graphs previews…
See : https://grafana.com/docs/grafana/latest/administration/image_rendering/#troubleshooting

sudo apt install libxdamage1 libxext6 libxi6 libxtst6 libnss3 libnss3 libcups2 libxss1 libxrandr2 libasound2 libatk1.0-0 libatk-bridge2.0-0 libpangocairo-1.0-0 libpango-1.0-0 libcairo2 libatspi2.0-0 libgtk3.0-cil libgdk3.0-cil libx11-xcb-dev
sudo systemctl restart grafana

Install Grafana Module for Icinga

sudo git clone https://github.com/Mikesch-mp/icingaweb2-module-grafana /usr/share/icingaweb2/modules/grafana

NB: module name must be “grafana”.

Imports dashboards into Grafana

Download “incinga2-default” dashboard from here : https://github.com/Mikesch-mp/icingaweb2-module-grafana/tree/master/dashboards/influxdb and import it into Grafana.

In order to automate the process, simply create the following file :

sudo bash -c 'cat <<EOF > /etc/grafana/provisioning/dashboards/sample.yaml
# # config file version
apiVersion: 1

providers:
 - name: 'default'
   orgId: 1
   folder: ''
   folderUid: ''
   type: file
   options:
     path: /path/to/icinga2-default.json
EOF
'

NB: the dashboard ( icinga2-default.json) must contains an uid, title and datasource matching the config from the icinga2web module (see later).

“title”: “icinga2-default”,
“uid”: “hosLL2oMz”,

See example in attachment. icinga2-default.json (5.3 KB)
Modify uid, title, and datasource according to your needs.

Then,

sudo systemctl restart grafana-server

Adjust Grafana config to use

Depending on your needs, one might need to adjust Grafana config at : /etc/grafana/grafana.ini.

Generate Grafana viewer only token

Browse to Grafana WebUI > Configuration > API Key and generate a “viewer” API key.

Or via CLI:

curl -X POST -H "Content-Type: application/json" -d '{"name":"apikeycurl", "role": "Viewer", "name": "icinga"}' http://admin:password@localhost:3000/api/auth/keys

Optional : import Grafana Dashboard for Icinga

Go to your Grafana WebUI and import dashboard number 1572. This is an extra dashboard to view Icinga metrics directly into Grafana.

Configure influxDB

Create DB, user and grants.

Run influxdb client

influx

Then create db, user, grants.

Connected to http://localhost:8086 version 1.8.3
#InfluxDB shell version: 1.8.3
#> create database icinga2
#> create user icinga2 with password 'XXXXX'
#> grant all on "icinga2" to "icinga2"

yep… “;” are not required :slight_smile:

Retention

By default, Influxdb will keep an infinite amount of data, which can become problematic over time.
One should define a custom retention period with the following commands :

Connected to http://localhost:8086 version 1.8.3
#InfluxDB shell version: 1.8.3
#> CREATE RETENTION POLICY one_year ON inciga2 DURATION 365d REPLICATION 1 DEFAULT
#> DROP RETENTION POLICY autogen on icinga2

Security

Depending on the network topology, influxdb should use SSL.

Known issues

There is an inconsistency in the data sent by Icinga2 to InfluxDB and the data requested by Icinga2 WebUI in order to render graph.
The column "service"is missing from InfluxDB, so in order to have all graphs displayed correctly, we have to create this column manually.

Configure Icinga

Enable module on Icinga WebUI

sudo icingacli module enable grafana

Configure influxdb module

This module allows Icinga to send data in InfluxDB (in order to let Grafana generate graphs, for example).

sudo icinga2 feature enable influxdb

sudo bash -c 'cat <<EOF > /etc/icinga2/features-enabled/influxdb.conf
object InfluxdbWriter "influxdb" {

host = "127.0.0.1"
port = 8086
database = "icinga2"
username = "icinga2"
password = "XXXXX"

enable_send_thresholds = true
enable_send_metadata = true

host_template = {
measurement = "$host.check_command$"
tags = {
hostname = "$host.name$"
}
}
service_template = {
measurement = "$service.check_command$"
tags = {
hostname = "$host.name$"
service = "$service.name$"
}
}
}

EOF
'

NB: change credentials, dband host info.

sudo systemctl restart icinga2

Configure icinga module

This module allows Grafana graphs to be displayed inside Icinga WebUI.

sudo bash -c 'cat <<EOF > /etc/icingaweb2/modules/grafana/config.ini
[grafana]
version = "1"
host = "127.0.0.1:3000"
authentication = "token"
apitoken = "YYYYYYY"
protocol = "http"
timerangeAll = "1w/w"
defaultdashboard = "icinga2-default"
defaultdashboarduid = "hosLL2oMz"
defaultdashboardpanelid = "1"
defaultorgid = "1"
shadows = "0"
theme = "light"
datasource = "icinga"
accessmode = "proxy"
debug = "0"
directrefresh = "no"
height = "280"
width = "640"
enableLink = "yes"
EOF
'

NB: change credentials and host info!

Other

Probably found a bug in Icinga InfluxDB module:

Fresh install :

The “base” graph which calls the command hostalive, indeed shows no data. Other graphs works as expected.

After digging a little, it seems that the URL sent by Icinga2 to Grafana looks like this :

http://127.0.0.1:3000/render/d-solo/YYYY/icinga2-default?var-hostname=XXXXX&var-service=hostalive&var-command=hostalive&panelId=1&orgId=1&width=640&height=280&theme=light&from=now-6h&to=now

The one, for apt (for example), looks the same however “hostalive” is replaced twice by "apt, like so :

http://127.0.0.1:3000/render/d-solo/YYYY/icinga2-default?var-hostname=XXXXX&var-service=apt&var-command=apt&panelId=1&orgId=1&width=640&height=280&theme=light&from=now-6h&to=now

If we inspect the influxDB table for “apt”, there is a column called “service”. However, there is none for the table “hostalive”.

Now if we call the following URL (leaving var-service blank), the graph is works as expected :

http://127.0.0.1:3000/render/d-solo/YYYY/icinga2-default?var-hostname=XXXXX&var-service=&var-command=hostalive&panelId=1&orgId=1&width=640&height=280&theme=light&from=now-6h&to=now

Sources & references

Other than the various link cited during the guide,

https://agir.april.org/issues/4439#change-17826


1 Like

Amazing work :tada: I’ll start working on playbooks this saturday. And begin with a grafana playbook.

Any advice on how to do that programmatically ? Or maybe preparing a configuration file before launching ?

There are a few other work items for which I’d be most grateful if you could explain how to do that from a CLI or a REST API or anything not manual really :wink:

Downloading is ok, it’s the import it into Grafana. that I’ll need to do without manual intervention.

For the sake of simplicity influxdb + grafana + icinga can be forced to be on the same host so SSL should not be necessary. It does not scale but will it hold a maximum of 20 services over 20 hosts on a 2GB RAM and 1 core machine? Organization self-hosting Enough typically rely on less than 20 hosts and 20 services.

Same question: how can this be done from CLI/REST/other.

Blocking the port can be done at the firewall level (if it runs in Docker, then just don’t expose the ports), this is what I am doing on my remote machines : only open 22, 80 and 443. Then I have an NGiNX reverse proxy (in Docker), dispatching the requests to other Docker containers. They all belong to the same Docker network bridge.

For the credentials, you can use Docker variables, such as :

GF_SECURITY_ADMIN_PASSWORD=MyS3curP$$

Of course, you can use an Ansible variable and store it in Ansible vault.

This is what I would do and have been doing for my own setups.

Looks like it can be done : https://grafana.com/docs/grafana/latest/administration/provisioning/#dashboards but I never did it, it will require additional testing on my part.

It can be done like this, thank you Burp Suite :

curl -X POST -H "Content-Type: application/json" -d '{"name":"apikeycurl", "role": "Viewer", "name": "icinga"}' http://admin:password@localhost:3000/api/auth/keys

1 Like

I’m confused because the HOWTO suggests installing grafana from packages and not using docker. I’m happy with both: which one do you recommend ?

I would be grateful if you could do that. A few years ago I automated a grafana install and I vaguely remember it required more than adding a file somewhere. Not much more but … the devil is in the details :wink:

Done, check the original doc. :slight_smile:

1 Like

Excellent! I’m all set to start working on the playbooks :slight_smile:

Is this still an issue? If so what inconsistency does it refer to exactly?

Yes, it is!

It is the issue an Icinga side where the data sent to InfluxDB is missing a column for the “hostalive” service, as described in the “Other” part of the guide.

I’ll write a bug report on their Github and hope for the best.

1 Like

Could you please let me know which versions I should pin? Whenever possible we pin the package versions so that there are no unexpected versions change before they get a chance of being tested.

grafana                              7.3.2 
influxdb                             1.8.3-1
1 Like

The master branch is a few months ahead of the latest release. Since it has not been updated in over six months, I’m tempted to pin the commit hash of the current master head instead of relying on the latest release tag. Do you have an opinion? We would not want the install to mysteriously fail if a bugous commit is added to master :wink:

I think it’s a good idea to pin it like you said. Right now, it’s doing what we need.

If the bug I found is later fixed, then we’ll do another serie of tests and maybe pin that new version in our playbook.

1 Like

I would very much welcome some guidance as to what my needs are regarding the datasource :stuck_out_tongue:

Although I could invent a way to create uid, maybe there are recommendations on how to pick a good one?

It really can be anything you want. Datasource title / name, should be something relevant like “icinga”. Since it is the name used by Grafana for the “connector” that links to the correct InfluxDB.

About uid, it seems to be 9 characters, alphanumerical and case sensitive.

1 Like

Grafana configuration is not very complicated but it requires a little work. I’m considering using https://github.com/cloudalchemy/ansible-grafana instead of doing it manually. I’ll give it a try now.

I have no clue what a datasource is :slight_smile:

Will it work if I use this “icinga2-default.json” file as is ? I’m sorry to be so ignorant and be entirely dependent on you to spell out in every detail how things must be done. I do not need to understand why, just how :wink:

yes it will, this is why I provided it.

Just make sure to have matching uid in /etc/icingaweb2/modules/grafana/config.ini.

Sam 21 nov 2020, à 16:53, Loïc Dachary via Enough a écrit :

1 Like

should be:

i.e. only one "name"

I have completed the HOWTO (I think :wink: but I’m not sure where to look to see that it works? A screenshot would help as well as directions such as “go in the service and…”.

Also I’m pretty sure I did something wrong (would be surprised to get it right the first time around) and it would be helpful to have some indication as to where error logs can be found for grafana & influxdb.

Very excited to see that complete :slight_smile: