Endpoint monitoring with Blackbox Exporter

overview blackbox exporter https //github com/prometheus/blackbox exporter is used for endpoint monitoring and can help generate meaningful uptime and availability metrics blackbox exporter allows blackbox probing of endpoints over http, https, dns, tcp, icmp and grpc while most of the exporters use static configurations and then expose metrics accordingly, blackbox exporter works a little differently to setup blackbox exporter , we define modules which contain a set of targets/urls to monitor prometheus agent queries the blackbox exporter with these targets/urls which generates endpoint metrics like uptime, latency etc a sample configuration is shown below configuration add the following yaml snippets to the opsverse agent's values yaml file victoria metrics agent extrascrapeconfigs \ job name 'blackbox exporter' scrape timeout 30s scrape interval 30s metrics path /probe params module \[http 2xx] static configs \ targets \ \<add target endpoint(s) here> relabel configs \ source labels \[ address ] target label param target \ source labels \[ param target] target label instance \ target label address replacement devopsnow agent blackbox exporter devopsnow 9115 blackbox exporter enabled true run the below command helm upgrade install devopsnow agent n devopsnow \\ \ repo https //registry devopsnow\ io/chartrepo/public \\ f \<path to opsverse values yaml file> single machine alternatively, you can install the prometheus blackbox exporter https //github com/prometheus/blackbox exporter on a single machine prerequisite(s) you have the opsverse agent running on the machine https //docs opsverse io/install the opsverse agent#dszlx run this command to install the blackbox exporter as a systemd unit \# for amd64 curl https //raw\ githubusercontent com/opsverseio/installers/main/prometheus exporters/install exporter amd64 sh | sudo bash s e blackbox \# for arm64 curl https //raw\ githubusercontent com/opsverseio/installers/main/prometheus exporters/install exporter arm64 sh | sudo bash s e blackbox edit the opsverse agent config file by using the following command sudo vi /etc/opsverse/agent config yaml add the blackbox exporter job under scrape configs and edit the fields as required \ job name 'blackbox exporter' metrics path /probe params module \[http 2xx] static configs \ targets \ \<add target endpoints here> relabel configs \ source labels \[ address ] target label param target \ source labels \[ param target] target label instance \ target label address replacement localhost 9115 edit the blackbox exporter's config file by using the following command sudo vi /etc/opsverse/exporters/blackbox/config yaml if the below content is alerady present in the config file, you can ignore the below step add the below module to blackbox exporter's config file modules http 2xx prober http timeout 5s http valid http versions \["http/1 1", "http/2 0"] method get preferred ip protocol "ip4" ip protocol fallback false no follow redirects false fail if ssl false fail if not ssl false tls config insecure skip verify true restart the prom blackbox exporter service and also the opsverse agent by using the following commands sudo systemctl restart prom blackbox exporter sudo systemctl restart opsverse agent check status to confirm everything is working fine sudo systemctl status prom blackbox exporter sudo systemctl status opsverse agent custom modules for blackbox exporter you can configure custom modules in blackbox exporter by defining the module in the blackbox config file and then updating scrape configs section of agent config yaml to use the custom module some examples are provided below appending custom headers with requests add the following module to the modules section in /etc/opsverse/exporters/blackbox/config yaml http 2xx with header prober http timeout 5s http valid http versions \["http/1 1", "http/2 0"] method get headers # these are your custom headers authentication bearer \<your token> accept language en us preferred ip protocol "ip4" ip protocol fallback false no follow redirects false fail if ssl false fail if not ssl false tls config insecure skip verify true then restart the blackbox exporter as described above and update scrape configs in /etc/opsverse/agent config yaml by adding the custom module to the params block module \[http 2xx with header] # use the custom module the exporter will now send the defined headers to every target it is monitoring sending post body to target add the following module to the modules section in /etc/opsverse/exporters/blackbox/config yaml http post 2xx prober http timeout 5s http valid http versions \["http/1 1", "http/2 0"] method post headers content type application/json body '{"foo" "bar"}' # this can be a custom body you want to send to the target preferred ip protocol "ip4" ip protocol fallback false no follow redirects false fail if ssl false fail if not ssl false tls config insecure skip verify true then restart the blackbox exporter as described above and update scrape configs in /etc/opsverse/agent config yaml by adding the custom module to the params block module \[http post 2xx] # use the custom module the exporter will now make a post request with predefined body to every target it is monitoring for more examples of custom blackbox exporter modules, you can see the blackbox exporter examples page https //github com/prometheus/blackbox exporter/blob/master/example yml visualisation opsverse observenow provides an out of the box blackbox exporter/endpoint monitoring dashboard that can be used to provide a visual overview of your monitored endpoints