Java

springboot metrics instrumentation instrumenting your app please follow the installation steps outlined in this https //docs micrometer io/micrometer/reference/implementations/prometheus html make sure add the management endpoints web exposure include=prometheus property in you application properties file setting the pod annotations once your application is emitting metrics, your pod in k8s will need to be annotated to enable the agent to scrape the metrics following are the relevant annotations prometheus io/scrape 'true' prometheus io/path '/actuator/prometheus' prometheus io/port '8080' java tracing instrumentation opentelemetry is the recommended approach to instrument for tracing please follow the instructions in the following document to instrument your application https //github com/open telemetry/opentelemetry java instrumentation this is achieved by including opentelemetry java agent as shown in this example java javaagent\ path/to/opentelemetry javaagent all jar \\ dotel resource attributes=service name=your service name \\ dotel traces exporter=zipkin \\ dotel metrics exporter=none \\ dotel exporter zipkin endpoint=http //otel collector devopsnow\ svc cluster local 9411 \\ jar myapp jar following args will need to be set otel resource attributes=service name=your service name otel traces exporter=zipkin otel metrics exporter=none otel exporter zipkin endpoint=http //otel collector devopsnow\ svc cluster local 9411 this will make the java application to emit spans which will be collected by the opentelemetry collector installed as part of the opsverse agent and forwarded to the tracing backend trace context (traceid, spanid and traceflags) injection into logs trace id, span id and trace flags can be printed to logs using mdc the mapped diagnostic context (mdc) is an instrument for distinguishing interleaved log output from different sources log output is typically interleaved when a server handles multiple clients near simultaneously the mdc is managed on a per thread basis it contains thread local contextual information which is later copied to each logging event captured by a logging library log4j instrumentation step 1 add instrumented log4j2 and opentelemetry api libraries into the project dependencies maven projects \<dependency> \<groupid>io opentelemetry instrumentation\</groupid> \<artifactid>opentelemetry log4j 2 13 2\</artifactid> \<version>1 9 2 alpha\</version> \<scope>runtime\</scope> \</dependency> \<dependency> \<groupid>io opentelemetry\</groupid> \<artifactid>opentelemetry api\</artifactid> \<version>1 26 0\</version> \</dependency> gradle projects dependencies { runtimeonly("io opentelemetry instrumentation\ opentelemetry log4j 2 13 2 1 9 2 alpha") implementation("io opentelemetry\ opentelemetry api 1 26 0") } any recent versions can be picked for both dependencies step 2 update the log4j2 xml configuration file, typically stored in resources directory, with traceid and spanid the following is an example configuration \<?xml version="1 0" encoding="utf 8"?> \<configuration status="warn"> \<appenders> \<console name="console" target="system out"> \<patternlayout pattern="%d{hh\ mm\ ss sss} \[%t] % 5level %logger{36} traceid %x{trace id} spanid %x{span id} %msg%n" /> \</console> \</appenders> \<loggers> \<root level="all" > \<appenderref ref="console"/> \</root> \</loggers> \</configuration> if you are using springboot and you don't have an explicit log4j2 xml file, please add the following block in application properties file logging pattern level=%d{hh\ mm\ ss sss} \[%t] % 5level %logger{36} traceid %x{trace id} spanid %x{span id} %msg%n logback instrumentation step 1 add instrumented logback library into the project dependencies maven projects \<dependency> \<groupid>io opentelemetry instrumentation\</groupid> \<artifactid>opentelemetry logback 1 0\</artifactid> \<version>1 9 2 alpha\</version> \<scope>runtime\</scope> \</dependency> gradle projects dependencies { runtimeonly("io opentelemetry instrumentation\ opentelemetry logback 1 0 1 9 2 alpha") } any recent versions can be picked for the dependency step 2 update the logback xml configuration file, typically stored in the resources directory, with traceid, spanid, and sampled keys the following is an example configuration \<?xml version="1 0" encoding="utf 8" ?> \<configuration> \<appender name="stdout" class="ch qos logback core consoleappender"> \<encoder> \<pattern>\<!\[cdata\[%date{hh\ mm\ ss sss} \[%thread] % 5level %logger{15}#%line %x{req requesturi} traceid %x{trace id} spanid %x{span id} %msg\n]]>\</pattern> \</encoder> \</appender> \<appender name="otel" class="io opentelemetry instrumentation logback v1 0 opentelemetryappender"> \<appender ref ref="stdout" /> \</appender> \<root> \<level value="debug" /> \<appender ref ref="stdout" /> \</root> \</configuration> if you are using springboot and you don't have an explicit logback xml file, please add the following block in application properties file logging pattern level=%d{hh\ mm\ ss sss} \[%t] % 5level %logger{36} traceid %x{trace id} spanid %x{span id} %msg%n frameworks spring boot for spring boot configuration which uses logback, you can add mdc to log lines by overriding the logging pattern level config in application properties file using the following block logging pattern level=trace id=%mdc{trace id} span id=%mdc{span id} trace flags=%mdc{trace flags} %5p springboot uses logback for logging by default logback can also be configured to handle complex logging needs the aforementioned logging config will work both default and custom logback logging configs java access logs instrumentation springboot comes with an embedded tomcat server the access logs track user session activity and page hit counts springboot's access logs are by default written to a file and not to stdout this means that most of the log collection agents do not automatically collect the access logs from containers there are a few strategies to handle this scenario we recommend updating the springboot app's properties file to make the access logs written to stdout this can be done by setting the following properties on the springboot app properties file note that the keys in server tomcat accesslog pattern can be changed according to requirement usage of json logging pattern helps in creating dashboards and alerts using logql server tomcat accesslog enabled=true server tomcat accesslog pattern={\\"ip\\" \\"%h\\", \\"user\\" \\"%l\\", \\"username\\" \\"%u\\", \\"time\\" \\"%t\\", \\"request\\" \\"%r\\", \\"status\\" \\"%s\\", \\"responsesize\\" \\"%b\\"} server tomcat accesslog directory=/dev server tomcat accesslog prefix=stdout server tomcat accesslog buffered=false \# don't use empty double quotes, server tomcat accesslog suffix= server tomcat accesslog file date format= for more application level configurations visit https //docs spring io/spring boot/docs/current/reference/html/application properties html single machine metrics instrumentation for single machines, you can install the https //github com/prometheus/jmx exporter on a single machine prerequisite(s) you have the https //docs opsverse io/install the opsverse agent#dszlx run this command to download the prometheus jmx exporter agent to your machine \# for amd64 curl https //raw\ githubusercontent com/opsverseio/installers/main/prometheus exporters/install exporter amd64 sh | sudo bash s e jmx \# for arm64 curl https //raw\ githubusercontent com/opsverseio/installers/main/prometheus exporters/install exporter arm64 sh | sudo bash s e jmx at this point, you can start your java runtime with the downloaded agent java javaagent /usr/local/bin/jmx prometheus javaagent 0 16 1 jar=9404 /etc/opsverse/exporters/jmx/config yaml jar yourjar jar the agent will begin scraping the metrics automatically (because the script will drop the scrape target into /etc/opsverse/targets/jmx exporter json for the agent to auto discover) if you change the port of the exporter (from 9404 in the java invocation above), please update /etc/opsverse/targets/jmx exporter json with the new port tracing instrumentation follow the java tracing instrumentation described above to emit and send traces to your jaeger collector endpoint the dotel exporter zipkin endpoint argument should be pointing to your jaeger collector example https //your jaegar endpoint 9411/api/v2/spans