Prometheus & Grafana

Prerequisites

Golang

Make sure you have Golang installed. For instructions on how to install here click here. To make sure it is installed run correctly:
1
go version
Copied!
Output example:
1
go version go1.16.3 linux/amd64
Copied!

Check Metrics

Make sure metrics are being exposed. For this check you can run:
1
curl localhost:9900/metrics | grep ^hmy
Copied!
Output example:
1
% Total % Received % Xferd Average Speed Time Time Time Current
2
Dload Upload Total Spent Left Speed
3
100 6087 0 6087 0 0 849k 0 --:--:-- --:--:-- -hmy_consensus_blskeys{index="0",pubkey="892f005b61d2c4f482a87a1da7226c210dbb6175b52ba391a941bcdab577f90767d0bbb5282f0d073b73eeb706c14810"} 0
4
-:--hmy_consensus_finality_bucket{le="800"} 1959
5
:-hmy_consensus_finality_bucket{le="1000"} 1960
6
- hmy_consensus_finality_bucket{le="1250"} 1960
7
8hmy_consensus_finality_bucket{le="1562.5"} 1960
8
49hmy_consensus_finality_bucket{le="1953.125"} 1960
9
khmy_consensus_finality_bucket{le="2441.40625"} 1960 (...continues)
Copied!

Setup Prometheus

Create User

1
sudo useradd -M -r -s /bin/false prometheus
Copied!

Create Directories

1
sudo mkdir /etc/prometheus /var/lib/prometheus
Copied!

Extract and Install

Download the latest version of Prometheus here.
The example show below is for Prometheus 2.26.0. Change it accordingly to the version you have downloaded.
1
tar xzf prometheus-2.26.0.linux-amd64.tar.gz
2
sudo cp prometheus-2.26.0.linux-amd64/{prometheus,promtool} /usr/local/bin/
3
sudo chown prometheus:prometheus /usr/local/bin/{prometheus,promtool}
4
sudo cp -r prometheus-2.26.0.linux-amd64/{consoles,console_libraries} /etc/prometheus/
5
sudo cp prometheus-2.26.0.linux-amd64/prometheus.yml /etc/prometheus/prometheus.yml
Copied!

Configure Targets

Open prometheus.yml configuration file:
1
sudo vi /etc/prometheus/prometheus.yml
Copied!
Change the default port to 9900 and save the file:
1
static_configs:
2
- targets: ['localhost:9900']
Copied!

Setup Permissions

1
sudo chown -R prometheus:prometheus /etc/prometheus
2
sudo chown prometheus:prometheus /var/lib/prometheus
Copied!

Setup Systemd

Create prometheus systemd file:
1
sudo vi /etc/systemd/system/prometheus.service
Copied!
Add the following to prometheus.service file:
1
[Unit]
2
Description=Prometheus Time Series Collection and Processing Server
3
Wants=network-online.target
4
After=network-online.target
5
6
[Service]
7
User=prometheus
8
Group=prometheus
9
Type=simple
10
ExecStart=/usr/local/bin/prometheus \
11
--config.file /etc/prometheus/prometheus.yml \
12
--storage.tsdb.path /var/lib/prometheus/ \
13
--web.console.templates=/etc/prometheus/consoles \
14
--web.console.libraries=/etc/prometheus/console_libraries
15
16
[Install]
17
WantedBy=multi-user.target
Copied!

Enable and Start Prometheus

1
sudo chmod 755 /etc/systemd/system/prometheus.service
2
sudo systemctl daemon-reload
3
sudo systemctl enable prometheus
4
sudo systemctl start prometheus
5
sudo systemctl status prometheus
Copied!

Check Metrics

You can now load Prometheus on your browser: http://Your_IP_Address:9090. You can run up or any other query to see if everthing is working as expected:
Querying Prometheus

Setup Grafana

Install

Follow instructions here on how to download and install Grafana for your OS.
Make sure to enable and start Grafana server.
Once started, you should be able to load Grafana on your browser: http://Your_IP_Address:3000. Default user and password is admin.

Add Prometheus Datasource

Go to http://Your_IP_Address:3000/datasourcesand add Prometheu as datasource:
Prometheus Datasource

Import Dashboard

Go to http://Your_IP_Address:3000/dashboard/import and load the json file below:
Load json file
Json file:
1
{
2
"annotations": {
3
"list": [
4
{
5
"builtIn": 1,
6
"datasource": "-- Grafana --",
7
"enable": true,
8
"hide": true,
9
"iconColor": "rgba(0, 211, 255, 1)",
10
"name": "Annotations & Alerts",
11
"type": "dashboard"
12
}
13
]
14
},
15
"description": "Monitoring Dashboard",
16
"editable": true,
17
"gnetId": null,
18
"graphTooltip": 0,
19
"id": 1,
20
"links": [],
21
"panels": [
22
{
23
"collapsed": true,
24
"datasource": null,
25
"gridPos": {
26
"h": 1,
27
"w": 24,
28
"x": 0,
29
"y": 0
30
},
31
"id": 2,
32
"panels": [],
33
"title": "Status",
34
"type": "row"
35
},
36
{
37
"datasource": null,
38
"fieldConfig": {
39
"defaults": {
40
"color": {
41
"mode": "thresholds"
42
},
43
"mappings": [],
44
"thresholds": {
45
"mode": "absolute",
46
"steps": [
47
{
48
"color": "green",
49
"value": null
50
},
51
{
52
"color": "red",
53
"value": 80
54
}
55
]
56
}
57
},
58
"overrides": []
59
},
60
"gridPos": {
61
"h": 5,
62
"w": 6,
63
"x": 0,
64
"y": 1
65
},
66
"id": 4,
67
"options": {
68
"orientation": "auto",
69
"reduceOptions": {
70
"calcs": [
71
"lastNotNull"
72
],
73
"fields": "",
74
"values": false
75
},
76
"showThresholdLabels": false,
77
"showThresholdMarkers": true,
78
"text": {}
79
},
80
"pluginVersion": "7.5.3",
81
"targets": [
82
{
83
"exemplar": true,
84
"expr": "up",
85
"interval": "",
86
"legendFormat": "",
87
"refId": "A"
88
}
89
],
90
"timeFrom": null,
91
"timeShift": null,
92
"title": "Harmony Online",
93
"type": "gauge"
94
},
95
{
96
"datasource": null,
97
"description": "",
98
"fieldConfig": {
99
"defaults": {
100
"color": {
101
"mode": "thresholds"
102
},
103
"mappings": [],
104
"thresholds": {
105
"mode": "absolute",
106
"steps": [
107
{
108
"color": "green",
109
"value": null
110
},
111
{
112
"color": "red",
113
"value": 80
114
}
115
]
116
},
117
"unit": "short"
118
},
119
"overrides": []
120
},
121
"gridPos": {
122
"h": 5,
123
"w": 10,
124
"x": 6,
125
"y": 1
126
},
127
"id": 6,
128
"options": {
129
"orientation": "auto",
130
"reduceOptions": {
131
"calcs": [
132
"lastNotNull"
133
],
134
"fields": "",
135
"values": false
136
},
137
"showThresholdLabels": false,
138
"showThresholdMarkers": true,
139
"text": {}
140
},
141
"pluginVersion": "7.5.3",
142
"targets": [
143
{
144
"exemplar": true,
145
"expr": "promhttp_metric_handler_requests_in_flight",
146
"hide": true,
147
"instant": false,
148
"interval": "",
149
"legendFormat": "",
150
"refId": "Pending"
151
},
152
{
153
"exemplar": true,
154
"expr": "rate(promhttp_metric_handler_requests_total[5m])",
155
"hide": false,
156
"instant": false,
157
"interval": "",
158
"legendFormat": "",
159
"refId": "Total"
160
}
161
],
162
"timeFrom": null,
163
"timeShift": null,
164
"title": "Metric Handler",
165
"type": "gauge"
166
}
167
],
168
"refresh": false,
169
"schemaVersion": 27,
170
"style": "dark",
171
"tags": [],
172
"templating": {
173
"list": []
174
},
175
"time": {
176
"from": "now-6h",
177
"to": "now"
178
},
179
"timepicker": {},
180
"timezone": "",
181
"title": "Harmony Dashboard",
182
"uid": "V6MHr8_Gk",
183
"version": 5
184
}
Copied!

Load Dashboard

Once you save it you should see the dashboard loaded on Grafana:
Last modified 7mo ago