This section describes how to use these dashboards for a cluster.
Access Hub Grafana Dashboards#
The Grafana for each cluster can be accessed at
For example, the Grafana for community hubs running on our GCP project is accessible at
To access the Grafana dashboards you’ll need a username and password.
These can be accessed using
sops (see sops overview for how to set up
sops on your machine).
See Log in to the cluster-spcific Grafana dashboard for how to find the credentials information.
The Central Grafana#
The Grafana deployment in the
2i2c cluster ingests data from all the 2i2c clusters and will soon be able to be used as “the central Grafana”.
TODO: should add more info once this is ready to use.
Set up Grafana Dashboards for a cluster#
This guide will walk through the steps required to setup a suite of Grafana dashboards for a cluster.
support chart is a helm chart maintained by the 2i2c Engineers that consists of common tools used to support JupyterHub deployments in the cloud.
These tools are
ingress-nginx, for controlling ingresses and load balancing;
cert-manager, for automatically provisioning TLS certificates from Let’s Encrypt; Prometheus, for scraping and storing metrics from the cluster and hub; and Grafana, for visualising the metrics retreived by Prometheus.
support.values.yaml file in your chosen cluster folder#
infrastructure repo, the full filepath should be:
Add the following helm chart values to your
<grafana-domain> should follow the pattern
<prometheus-domain> should follow the pattern
prometheusIngressAuthSecret: enabled: true grafana: ingress: hosts: - <grafana-domain> tls: - secretName: grafana-tls hosts: - <grafana-domain> prometheus: server: ingress: enabled: true hosts: - <prometheus-domain> tls: - secretName: prometheus-tls hosts: - <prometheus-domain>
Only 2i2c staff + our centralized grafana should be able to access the
prometheus data on a cluster from outside the cluster. The basic auth
feature of nginx-ingress is used to restrict this. A
file is used to provide these secret credentials.
prometheusIngressAuthSecret: username: <output of pwgen -s 64 1> password: <output of pwgen -s 64 1>
We use the pwgen program, commonly installed by default in many operating systems, to generate the password.
Once you create the file, encrypt it in-place with
sops --in-place --encrypt <file-name>.
Add the following config as a top-level key to your
Note this filepath is relative to the location of your
support: helm_chart_values_files: - support.values.yaml - enc-support.secret.values.yaml
support chart via the
deployer tool to deploy the support chart to the cluster.
See Manually deploy a config change for details on how to setup the tool locally.
python3 deployer deploy-support CLUSTER_NAME
Setting DNS records#
support chart has been successfully deployed, retrieve the external IP address for the
ingress-nginx load balancer.
kubectl --namespace support get svc support-ingress-nginx-controller
Add the following DNS records via Namecheap.com:
<cluster-name>.2i2c.cloud, used for the primary hub (if it exists).
*.<cluster-name>.2i2c.cloud, for all other hubs, grafana and prometheus instances.
The DNS records should be
A records if using GCP or Azure (where external IP is an
IPv4 address), or
CNAME records if using AWS (where external IP is a domain name).
Wait a while for the DNS to propagate!
Log in to the cluster-spcific Grafana dashboard#
GRAFANA_URL will present you with a login page.
Here are the credentials for logging in:
password: located in
Register the cluster’s Prometheus Server with the central Grafana#
Once you have deployed the support chart, you must also register this cluster as a datasource for the central Grafana dashboard. This will allow you to visualize cluster statistics not only from the cluster-specific Grafana deployement but also from the central dashboard, that aggregates data from all the clusters.
update_central_grafana_datasources.py script in the deployer to let the central Grafana know about this new prometheus server:
$ python3 deployer/update_central_grafana_datasources.py <grafana-cluster-name>
is the name of the cluster where the central Grafana lives. Right now, this defaults to “2i2c”.
Setting up Grafana Dashboards#
Once you have logged into grafana as the admin user, create a new API key. You can do this by selecting the gear icon from the left-hand menu, and then selecting API keys. The key you create needs admin permissions.
Keep this key safe as you won’t be able to retrieve it!
Create the file
config/clusters/<cluster>/grafana-token.secret.yaml with the following content.
grafana_token: PASTE_YOUR_API KEY HERE
Then encrypt this file using
sops like so:
sops --output config/clusters/<cluster>/enc-grafana-token.secret.yaml --encrypt config/clusters/<cluster>/grafana-token.secret.yaml
The encrypted file can now be committed to the repository.
Once you’ve pushed the encrypted
grafana_token to the GitHub repository, manually trigger the
deploy-grafana-dashboards workflow using the “Run workflow” button from here to deploy the dashboards.
The workflow only runs when manually triggered.
Any re-triggering of the workflow after the initial deployment will overwrite any dashboard created from the Grafana UI and not stored in the