ObserveNow
...
Integrations
Infrastructure

Apache Kafka

OpsVerse's Kubernetes agent can be used to collect metrics from Kafka.

Add the following YAML snippet to the agent's values.yaml and run the agent based on these instructions.

YAML


Refer values.yaml file from Prometheus community for additional value configurations and default values.

Collecting Metrics from Strimzi Based Kafka Deployment

OpsVerse's Kubernetes agent can also be used to collect metrics from a Strimzi based kafka deployment.

To instrument metrics (JMX and kafka-exporter) from a Strimzi based kafka deployment follow the given steps:

  • Deploy the following config-map in the same namespace as your kafka deployment:
Text

  • Add the following snippet under spec.kafka.metricsConfig in your kafka deployment file:
YAML

  • Add the following snippet under spec.zookeeper.metricsConfig in your kafka deployment file:
YAML

  • Add the following snippet under spec in your kafka deployment file:
YAML


A sample Strimzi based kafka deployment with `metrics instrumentaion enabled can be found here

  • Once the above steps are completed add the following snippet under victoria-metrics-agent.extraScrapeConfigs of the agent's values.yaml and run the agent based on these instructions.
YAML


To view in a convenient dashboard, you may search for the Grafana dashboard : Kafka- JMX Dashboard in Grafana or import the dashboard from the strimzi repository (ensure you select the MetricsServer datasource when prompted).

Collecting Metrics from Amazon Managed Streaming for Apache Kafka (MSK)

OpsVerse's Kubernetes agent can also be used to collect metrics from Amazon MSK. To instrument metrics from a Amazon MSK deployment follow the given steps:

  • Enable open monitoring for the MSK cluster by following this document.
  • Add the following YAML snippet to the agent's values.yaml and run the agent based on these instructions.
YAML


Make sure the MSK broker endpoints are reachable over by the cluster in which the opsverse-agent is being deployed.