Endpoint monitoring with Blackbox Exporter

Overview
﻿Blackbox Exporter is used for endpoint monitoring and can help generate meaningful uptime and availability metrics. Blackbox Exporter allows blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP, ICMP and gRPC.
While most of the exporters use static configurations and then expose metrics accordingly, Blackbox Exporter works a little differently.
To setup Blackbox Exporter, we define modules which contain a set of targets/urls to monitor. Prometheus agent queries the blackbox exporter with these targets/urls which generates endpoint metrics like uptime, latency etc.
A sample configuration is shown below
Configuration
Add the following YAML snippets to the OpsVerse agent's values.yaml file.
YAML
victoria-metrics-agent:
  extraScrapeConfigs:
    - job_name: 'blackbox-exporter'
      scrape_timeout: 30s
      scrape_interval: 30s
      metrics_path: /probe
      params:
        module: [http_2xx]

      static_configs:
        - targets:
          - <Add target endpoint(s) here>

      relabel_configs:
        - source_labels: [__address__]
          target_label: __param_target
        - source_labels: [__param_target]
          target_label: instance
        - target_label: __address__
          replacement: devopsnow-agent-blackbox-exporter.devopsnow:9115
victoria-metrics-agent:
  extraScrapeConfigs:
    - job_name: 'blackbox-exporter'
      scrape_timeout: 30s
      scrape_interval: 30s
      metrics_path: /probe
      params:
        module: [http_2xx]

      static_configs:
        - targets:
          - <Add target endpoint(s) here>

      relabel_configs:
        - source_labels: [__address__]
          target_label: __param_target
        - source_labels: [__param_target]
          target_label: instance
        - target_label: __address__
          replacement: devopsnow-agent-blackbox-exporter.devopsnow:9115
﻿
YAML
blackbox-exporter:
  enabled: true
blackbox-exporter:
  enabled: true
﻿
Run the below command:
Shell
helm upgrade --install devopsnow-agent -n devopsnow \
    --repo https://registry.devopsnow.io/chartrepo/public \
    -f <path to opsverse-values.yaml file>
helm upgrade --install devopsnow-agent -n devopsnow \
    --repo https://registry.devopsnow.io/chartrepo/public \
    -f <path to opsverse-values.yaml file>
﻿
Single Machine
Alternatively, you can install the Prometheus Blackbox Exporter on a single machine.
Prerequisite(s):
You have the OpsVerse Agent running on the machine﻿
Run this command to install the Blackbox Exporter as a systemd unit:
Shell
# for amd64
curl https://raw.githubusercontent.com/OpsVerseIO/installers/main/prometheus-exporters/install-exporter-amd64.sh | sudo bash -s -- -e blackbox  

# for arm64
curl https://raw.githubusercontent.com/OpsVerseIO/installers/main/prometheus-exporters/install-exporter-arm64.sh | sudo bash -s -- -e blackbox  
# for amd64
curl https://raw.githubusercontent.com/OpsVerseIO/installers/main/prometheus-exporters/install-exporter-amd64.sh | sudo bash -s -- -e blackbox  

# for arm64
curl https://raw.githubusercontent.com/OpsVerseIO/installers/main/prometheus-exporters/install-exporter-arm64.sh | sudo bash -s -- -e blackbox  
﻿
Edit the OpsVerse agent config file by using the following command:
Shell
sudo vi /etc/opsverse/agent-config.yaml
sudo vi /etc/opsverse/agent-config.yaml
﻿
Add the Blackbox exporter job under scrape_configs and edit the fields as required.
YAML
- job_name: 'blackbox-exporter'
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
      - <Add target endpoints here>
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: localhost:9115
- job_name: 'blackbox-exporter'
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
      - <Add target endpoints here>
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: localhost:9115
﻿
Edit the Blackbox exporter's config file by using the following command:
Shell
sudo vi /etc/opsverse/exporters/blackbox/config.yaml
sudo vi /etc/opsverse/exporters/blackbox/config.yaml
﻿
If the below content is alerady present in the config file, you can ignore the below step.
Add the below module to Blackbox exporter's config file.
YAML
modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      method: GET
      preferred_ip_protocol: "ip4"
      ip_protocol_fallback: false
      no_follow_redirects: false
      fail_if_ssl: false
      fail_if_not_ssl: false
      tls_config:
        insecure_skip_verify: true
modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      method: GET
      preferred_ip_protocol: "ip4"
      ip_protocol_fallback: false
      no_follow_redirects: false
      fail_if_ssl: false
      fail_if_not_ssl: false
      tls_config:
        insecure_skip_verify: true
﻿
Restart the prom-blackbox-exporter service and also the OpsVerse Agent by using the following commands:
Shell
sudo systemctl restart prom-blackbox-exporter

sudo systemctl restart opsverse-agent
sudo systemctl restart prom-blackbox-exporter

sudo systemctl restart opsverse-agent
﻿
Check status to confirm everything is working fine.
Shell
sudo systemctl status prom-blackbox-exporter

sudo systemctl status opsverse-agent
sudo systemctl status prom-blackbox-exporter

sudo systemctl status opsverse-agent
﻿
Custom Modules for Blackbox Exporter
You can configure custom modules in Blackbox exporter by defining the module in the blackbox config file and then updating scrape_configs section of agent-config.yaml to use the custom module. Some examples are provided below:
Appending Custom Headers with requests
Add the following module to the modules section in /etc/opsverse/exporters/blackbox/config.yaml
YAML
http_2xx_with_header:
  prober: http
  timeout: 5s
  http:
    valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
    method: GET
    headers: # These are your custom headers
      Authentication: Bearer <your-token>
      Accept-Language: en-US
    preferred_ip_protocol: "ip4"
    ip_protocol_fallback: false
    no_follow_redirects: false
    fail_if_ssl: false
    fail_if_not_ssl: false
    tls_config:
      insecure_skip_verify: true
http_2xx_with_header:
  prober: http
  timeout: 5s
  http:
    valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
    method: GET
    headers: # These are your custom headers
      Authentication: Bearer <your-token>
      Accept-Language: en-US
    preferred_ip_protocol: "ip4"
    ip_protocol_fallback: false
    no_follow_redirects: false
    fail_if_ssl: false
    fail_if_not_ssl: false
    tls_config:
      insecure_skip_verify: true
﻿
Then restart the blackbox-exporter as described above and update scrape_configs in /etc/opsverse/agent-config.yaml by adding the custom module to the params block:
YAML
module: [http_2xx_with_header]  # Use the custom module
module: [http_2xx_with_header]  # Use the custom module
﻿
The exporter will now send the defined headers to every target it is monitoring
Sending POST body to target
Add the following module to the modules section in /etc/opsverse/exporters/blackbox/config.yaml
YAML
http_post_2xx:
  prober: http
  timeout: 5s
  http:
    valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
    method: POST
    headers:
      Content-Type: application/json
    body: '{"foo": "bar"}' # This can be a custom body you want to send to the target
    preferred_ip_protocol: "ip4"
    ip_protocol_fallback: false
    no_follow_redirects: false
    fail_if_ssl: false
    fail_if_not_ssl: false
    tls_config:
      insecure_skip_verify: true
http_post_2xx:
  prober: http
  timeout: 5s
  http:
    valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
    method: POST
    headers:
      Content-Type: application/json
    body: '{"foo": "bar"}' # This can be a custom body you want to send to the target
    preferred_ip_protocol: "ip4"
    ip_protocol_fallback: false
    no_follow_redirects: false
    fail_if_ssl: false
    fail_if_not_ssl: false
    tls_config:
      insecure_skip_verify: true
﻿
Then restart the blackbox-exporter as described above and update scrape_configs in /etc/opsverse/agent-config.yaml by adding the custom module to the params block:
YAML
module: [http_post_2xx]  # Use the custom module
module: [http_post_2xx]  # Use the custom module
﻿
The exporter will now make a POST request with predefined body to every target it is monitoring
For more examples of custom blackbox-exporter modules, you can see the blackbox-exporter examples page.
Visualisation
OpsVerse ObserveNow provides an out of the box Blackbox Exporter/Endpoint Monitoring dashboard that can be used to provide a visual overview of your monitored endpoints. 
﻿
﻿
﻿
Did this page help you?
Language Specific Profiling
Overview