Skip to content

Configure Alerts#

Introduction#

Alerts are sent from Hopsworks using Prometheus' Alert manager. In order to send alerts we first need to configure the Alert manager.

Prerequisites#

Administrator account on a Hopsworks cluster.

Step 1: Go to alerts configuration#

To configure the Alert manager click on your name in the top right corner of the navigation bar and choose Cluster Settings from the dropdown menu. In the Cluster Settings' Alerts tab you can configure the alert manager to send alerts via email, slack or pagerduty.

Configure alerts
Configure alerts

Step 2: Configure Email Alerts#

To send alerts via email you need to configure an SMTP server. Click on the Configure button on the left side of the email row and fill out the form that pops up.

Configure Email Alerts
Configure Email Alerts
  • Default from: the address used as sender in the alert email.
  • SMTP smarthost: the Simple Mail Transfer Protocol (SMTP) host through which emails are sent.
  • Default hostname (optional): hostname to identify to the SMTP server.
  • Authentication method: how to authenticate to the SMTP server. CRAM-MD5, LOGIN or PLAIN.

Optionally cluster wide Email alert receivers can be added in Default receiver emails. These receivers will be available to all users when they create event triggered alerts.

Step 3: Configure Slack Alerts#

Alerts can also be sent via Slack messages. To be able to send Slack messages you first need to configure a Slack webhook. Click on the Configure button on the left side of the slack row and past in your Slack webhook in Webhook.

Configure slack Alerts
Configure slack Alerts

Optionally cluster wide Slack alert receivers can be added in Slack channel/user. These receivers will be available to all users when they create event triggered alerts.

Step 4: Configure Pagerduty Alerts#

Pagerduty is another way you can send alerts from Hopsworks. Click on the Configure button on the left side of the pagerduty row and fill out the form that pops up.

Configure Pagerduty Alerts
Configure Pagerduty Alerts

Fill in Pagerduty URL: the URL to send API requests to.

Optionally cluster wide Pagerduty alert receivers can be added in Service key/Routing key. By first choosing the PagerDuty integration type:

  • global event routing (routing_key): when using PagerDuty integration type Events API v2.
  • service (service_key): when using PagerDuty integration type Prometheus.

Then adding the Service key/Routing key of the receiver(s). PagerDuty provides documentation on how to integrate with Prometheus' Alert manager.

Step 5: Advanced configuration#

If you are familiar with Prometheus' Alert manager you can also configure alerts by editing the yaml/json file directly.

Advanced configuration
Advanced configuration

Example: Adding the yaml snippet shown below in the global section of the alert manager configuration will have the same effect as creating the SMTP configuration as shown in section 1 above.

global:
    smtp_smarthost: smtp.gmail.com:587
    smtp_from: hopsworks@gmail.com
    smtp_auth_username: hopsworks@gmail.com
    smtp_auth_password: XXXXXXXXX
    smtp_auth_identity: hopsworks@gmail.com
 ...

To test the alerts by creating triggers from Jobs and Feature group validations see Alerts.

The yaml syntax in the UI is slightly different in that it does not allow double quotes (it will ignore the values but give no error). Below is an example configuration, that can be used in the UI, with both email and slack receivers configured for system alerts.

global:
    smtp_smarthost: smtp.gmail.com:587
    smtp_from: hopsworks@gmail.com
    smtp_auth_username: hopsworks@gmail.com
    smtp_auth_password: XXXXXXXXX
    smtp_auth_identity: hopsworks@gmail.com
    resolveTimeout: 5m
templates:
  - /srv/hops/alertmanager/alertmanager-0.17.0.linux-amd64/template/*.tmpl
route:
  receiver: default
  routes:
    - receiver: email
      continue: true
      match:
        type: system-alert
    - receiver: slack
      continue: true
      match:
        type: system-alert
  groupBy:
    - alertname
  groupWait: 10s
  groupInterval: 10s
receivers:
  - name: default
  - name: email
    emailConfigs:
      - to: someone@logicalclocks.com
        from: hopsworks@logicalclocks.com
        smarthost: mail.hello.com
        text: >-
          summary: {{ .CommonAnnotations.summary }} description: {{
          .CommonAnnotations.description }}
  - name: slack
    slackConfigs:
      - apiUrl: >-
          https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
        channel: '#general'
        text: >-
          <!channel> summary: {{ .Annotations.summary }} description: {{
          .Annotations.description }}