Monitoring with Prometheus and Grafana

Data Migrator provides detailed metrics about recent and ongoing events, activity, and performance through the UI or the CLI. Use Prometheus and Grafana to capture and visualize this data:

• Prometheus is an open-source monitoring system that records real-time metrics in a time series database.

• Grafana is an open-source interactive visualization web application that provides charts, graphs, and alerting functionality when connected to supported data sources.

Prometheus endpoints

Collect Prometheus metrics for Data Migrator or the UI using each component's specific endpoint.

Data Migrator Prometheus endpoint

Data Migrator Prometheus endpoint located at:

 http(s)://<hostname>:18080/actuator/prometheus

info

If you are not doing so already, you may need to enable access to the REST API on your network. Learn more here.

UI Prometheus endpoint

To access the UI endpoint, enable it in the UI configuration, and restart the service.

1. Edit the UI configuration file.

   vi /etc/wandisco/ui/application-prod.properties

2. Add the following configuration line to enable the endpoint.

   management.metrics.export.prometheus.enabled=true 

3. Restart the UI service.

   systemctl restart livedata-ui

The UI Prometheus endpoint when enabled is located at:

http://<hostname>:8081/management/prometheus

tip

If you have https enabled on the UI, the endpoint will redirect to https on port 8443 automatically

Set up monitoring with Prometheus and Grafana

Set up a host machine

You will need a machine with network access to the Data Migrator node on port 18080.

note

We do not recommend installing additional applications on the Data Migrator node.

For more information on the minimum requirements to run Grafana, see the official documentation.

Install Prometheus

See First steps with Prometheus for information on installing Prometheus on your new node.

Install Grafana

See Install Grafana for information on installing Grafana on your new node.

In these examples, we use the .tar file installation steps.

Configure Prometheus

See the official documentation for information on configuring Prometheus.

To let Prometheus know which source to collect data from, create or update the Prometheus configuration file to point to the correct endpoint on the Data Migrator host. Use the default configuration file (prometheus.yml) or create your own and point to it when running Prometheus.

note

The Data Migrator Prometheus endpoint used here is http(s)://hostname:18080/actuator/prometheus so we can collect Data Migrator specific metrics.

The following example configuration shows Prometheus pointing to a Data Migrator instance with IP 172.31.44.14 and a metrics_path value of /actuator/prometheus:

scrape_interval: 1s
evaluation_interval: 1s

alerting:
alertmanagers:
- static_configs:
    - targets:

scrape_configs:
- job_name: "Data Migrator Prometheus"
metrics_path: '/actuator/prometheus'
static_configs:
  - targets: ["172.31.44.14:18080"]

Collect data

Start Prometheus

See the official documentation for your version for information on how to run Prometheus.

Example

Starting Prometheus pointing to a specified config file from the.tar file installation method:

/var/tmp/builds/prometheus/prometheus-2.33.5.linux-amd64/prometheus --config.file="prometheus.yml

Prometheus will start collecting data from the endpoint specified in its configuration and store it in its own time-series database, ready to be analyzed. When you stop the process, it will stop collecting data.

Visualize data

Start Grafana

See the official documentation for your version for information on how to run Grafana.

Starting Grafana - .tar file installation method:

/var/tmp/grafana/grafana-8.4.3/bin/grafana-server

While Grafana is running, go to the hostname or IP on port 3000 to sign in. Enter the following default credentials and update them when prompted:

**Username**: admin
**Password**: admin

Add the data source

When signed in to Grafana for the first time, you won't see any data until you've set up a data source. Create a Prometheus data source to get started.

Visualize the data

See the Grafana documentation for information on using panels, visualizations, and dashboards to visualize your data.

View Prometheus metrics

See the endpoint below for the latest list of available metrics for Prometheus:

URL for Data Migrator Prometheus metrics documentation on your server:

http://localhost:18080/actuator/prometheus

URL for Hive Migrator Prometheus metrics documentation on your server:

http://localhost:6780/prometheus

info

For more information on accessing the Data Migrator REST API documentation, see API Reference.