Apache Kafka

opsverse's kubernetes agent https //docs opsverse io/install the opsverse agent#veqxm can be used to collect metrics from kafka add the following yaml snippet to the agent's values yaml and run the agent based on these instructions https //docs opsverse io/install the opsverse agent#veqxm \# enable this section to pull metrics from kafka kafka exporter enabled true image repository danielqsj/kafka exporter \# overrides the image tag whose default is the chart appversion tag latest annotations prometheus io/scrape "true" kafkaserver \ \<kafka server service name> \<namespace> svc cluster local \<port> refer values yaml https //github com/prometheus community/helm charts/blob/main/charts/prometheus kafka exporter/values yaml file from prometheus community for additional value configurations and default values collecting metrics from strimzi based kafka deployment opsverse's kubernetes agent https //docs opsverse io/install the opsverse agent#veqxm can also be used to collect metrics from a strimzi based kafka deployment to instrument metrics ( jmx and kafka exporter ) from a strimzi based kafka deployment follow the given steps deploy the following config map in the same namespace as your kafka deployment kind configmap apiversion v1 metadata name kafka metrics labels app strimzi data kafka metrics config yml | \# see https //github com/prometheus/jmx exporter for more info about jmx prometheus exporter metrics lowercaseoutputname true rules \# special cases and very specific rules \ pattern kafka server\<type=( +), name=( +), clientid=( +), topic=( +), partition=( )><>value name kafka server $1 $2 type gauge labels clientid "$3" topic "$4" partition "$5" \ pattern kafka server\<type=( +), name=( +), clientid=( +), brokerhost=( +), brokerport=( +)><>value name kafka server $1 $2 type gauge labels clientid "$3" broker "$4 $5" \ pattern kafka server\<type=( +), cipher=( +), protocol=( +), listener=( +), networkprocessor=( +)><>connections name kafka server $1 connections tls info type gauge labels cipher "$2" protocol "$3" listener "$4" networkprocessor "$5" \ pattern kafka server\<type=( +), clientsoftwarename=( +), clientsoftwareversion=( +), listener=( +), networkprocessor=( +)><>connections name kafka server $1 connections software type gauge labels clientsoftwarename "$2" clientsoftwareversion "$3" listener "$4" networkprocessor "$5" \ pattern "kafka server\<type=( +), listener=( +), networkprocessor=( +)><>( +) " name kafka server $1 $4 type gauge labels listener "$2" networkprocessor "$3" \ pattern kafka server\<type=( +), listener=( +), networkprocessor=( +)><>( +) name kafka server $1 $4 type gauge labels listener "$2" networkprocessor "$3" \# some percent metrics use meanrate attribute \# ex) kafka server\<type=(kafkarequesthandlerpool), name=(requesthandleravgidlepercent)><>meanrate \ pattern kafka (\w+)\<type=( +), name=( +)percent\w ><>meanrate name kafka $1 $2 $3 percent type gauge \# generic gauges for percents \ pattern kafka (\w+)\<type=( +), name=( +)percent\w ><>value name kafka $1 $2 $3 percent type gauge \ pattern kafka (\w+)\<type=( +), name=( +)percent\w , ( +)=( +)><>value name kafka $1 $2 $3 percent type gauge labels "$4" "$5" \# generic per second counters with 0 2 key/value pairs \ pattern kafka (\w+)\<type=( +), name=( +)persec\w , ( +)=( +), ( +)=( +)><>count name kafka $1 $2 $3 total type counter labels "$4" "$5" "$6" "$7" \ pattern kafka (\w+)\<type=( +), name=( +)persec\w , ( +)=( +)><>count name kafka $1 $2 $3 total type counter labels "$4" "$5" \ pattern kafka (\w+)\<type=( +), name=( +)persec\w ><>count name kafka $1 $2 $3 total type counter \# generic gauges with 0 2 key/value pairs \ pattern kafka (\w+)\<type=( +), name=( +), ( +)=( +), ( +)=( +)><>value name kafka $1 $2 $3 type gauge labels "$4" "$5" "$6" "$7" \ pattern kafka (\w+)\<type=( +), name=( +), ( +)=( +)><>value name kafka $1 $2 $3 type gauge labels "$4" "$5" \ pattern kafka (\w+)\<type=( +), name=( +)><>value name kafka $1 $2 $3 type gauge \# emulate prometheus 'summary' metrics for the exported 'histogram's \# note that these are missing the ' sum' metric! \ pattern kafka (\w+)\<type=( +), name=( +), ( +)=( +), ( +)=( +)><>count name kafka $1 $2 $3 count type counter labels "$4" "$5" "$6" "$7" \ pattern kafka (\w+)\<type=( +), name=( +), ( +)=( ), ( +)=( +)><>(\d+)thpercentile name kafka $1 $2 $3 type gauge labels "$4" "$5" "$6" "$7" quantile "0 $8" \ pattern kafka (\w+)\<type=( +), name=( +), ( +)=( +)><>count name kafka $1 $2 $3 count type counter labels "$4" "$5" \ pattern kafka (\w+)\<type=( +), name=( +), ( +)=( )><>(\d+)thpercentile name kafka $1 $2 $3 type gauge labels "$4" "$5" quantile "0 $6" \ pattern kafka (\w+)\<type=( +), name=( +)><>count name kafka $1 $2 $3 count type counter \ pattern kafka (\w+)\<type=( +), name=( +)><>(\d+)thpercentile name kafka $1 $2 $3 type gauge labels quantile "0 $4" zookeeper metrics config yml | \# see https //github com/prometheus/jmx exporter for more info about jmx prometheus exporter metrics lowercaseoutputname true rules \# replicated zookeeper \ pattern "org apache zookeeperservice\<name0=replicatedserver id(\\\d+)><>(\\\w+)" name "zookeeper $2" type gauge \ pattern "org apache zookeeperservice\<name0=replicatedserver id(\\\d+), name1=replica (\\\d+)><>(\\\w+)" name "zookeeper $3" type gauge labels replicaid "$2" \ pattern "org apache zookeeperservice\<name0=replicatedserver id(\\\d+), name1=replica (\\\d+), name2=(\\\w+)><>(packets\\\w+)" name "zookeeper $4" type counter labels replicaid "$2" membertype "$3" \ pattern "org apache zookeeperservice\<name0=replicatedserver id(\\\d+), name1=replica (\\\d+), name2=(\\\w+)><>(\\\w+)" name "zookeeper $4" type gauge labels replicaid "$2" membertype "$3" \ pattern "org apache zookeeperservice\<name0=replicatedserver id(\\\d+), name1=replica (\\\d+), name2=(\\\w+), name3=(\\\w+)><>(\\\w+)" name "zookeeper $4 $5" type gauge labels replicaid "$2" membertype "$3" add the following snippet under spec kafka metricsconfig in your kafka deployment file metricsconfig type jmxprometheusexporter valuefrom configmapkeyref name kafka metrics key kafka metrics config yml add the following snippet under spec zookeeper metricsconfig in your kafka deployment file metricsconfig type jmxprometheusexporter valuefrom configmapkeyref name kafka metrics key zookeeper metrics config yml add the following snippet under spec in your kafka deployment file kafkaexporter topicregex " " groupregex " " a sample strimzi based kafka deployment with `metrics instrumentaion enabled can be found here once the above steps are completed add the following snippet under victoria metrics agent extrascrapeconfigs of the agent's values yaml and run the agent based on these instructions https //docs opsverse io/install the opsverse agent#veqxm extrascrapeconfigs \ job name kubernetes strimzi kafka kubernetes sd configs \ role pod metric relabel configs \ action drop regex go source labels \ name \ action drop regex zookeeper source labels \ name relabel configs \ action keep regex kafka exporter source labels \ meta kubernetes pod label app kubernetes io name \ action keep regex kafka|kafkaconnect|kafkamirrormaker|kafkamirrormaker2 source labels \ meta kubernetes pod label strimzi io kind \ action labelmap regex meta kubernetes pod label (strimzi io +) replacement $1 separator ; \ action replace regex ( ) replacement $1 separator ; source labels \ meta kubernetes namespace target label namespace \ action replace regex ( ) replacement $1 separator ; source labels \ meta kubernetes pod name target label kubernetes pod name \ action replace regex ( ) replacement $1 separator ; source labels \ meta kubernetes pod node name target label node name \ action replace regex ( ) replacement $1 separator ; source labels \ meta kubernetes pod host ip target label node ip \ job name kubernetes strimzi kafka additional kubernetes sd configs \ role pod metric relabel configs \ action drop regex go source labels \ name \ action drop regex zookeeper source labels \ name relabel configs \ action keep regex kafka|kafkaconnect|kafkamirrormaker|kafkamirrormaker2 source labels \ meta kubernetes pod label strimzi io kind \ action labelmap regex meta kubernetes pod label (strimzi io +) replacement $1 separator ; \ action replace regex ( ) replacement $1 separator ; source labels \ meta kubernetes namespace target label namespace \ action replace regex ( ) replacement $1 separator ; source labels \ meta kubernetes pod name target label kubernetes pod name \ action replace regex ( ) replacement $1 separator ; source labels \ meta kubernetes pod node name target label node name \ action replace regex ( ) replacement $1 separator ; source labels \ meta kubernetes pod host ip target label node ip to view in a convenient dashboard, you may search for the grafana dashboard kafka jmx dashboard in grafana or import the dashboard from the strimzi repository https //github com/strimzi/strimzi kafka operator/blob/main/examples/metrics/grafana dashboards/strimzi kafka json (ensure you select the metricsserver datasource when prompted) collecting metrics from amazon managed streaming for apache kafka (msk) opsverse's kubernetes agent https //docs opsverse io/install the opsverse agent#veqxm can also be used to collect metrics from amazon msk to instrument metrics from a amazon msk deployment follow the given steps enable open monitoring for the msk cluster by following this document https //docs aws amazon com/msk/latest/developerguide/open monitoring html#enable open monitoring after creation add the following yaml snippet to the agent's values yaml and run the agent based on these instructions https //docs opsverse io/install the opsverse agent#veqxm observe agent victoria metrics agent extrascrapeconfigs \ job name kafka msk jmx static configs \ targets \["\<broker 1 endpoint> 11001","\<broker 2 endpoint> 11001", "\<broker n endpoint> 11001"] \ job name kafka msk node static configs \ targets \["\<broker 1 endpoint> 11002","\<broker 2 endpoint> 11002", "\<broker n endpoint> 11002"] make sure the msk broker endpoints are reachable over by the cluster in which the opsverse agent is being deployed