Java

SpringBoot Metrics Instrumentation
Instrumenting your app
Please follow the installation steps outlined in this document.
Make sure add the management.endpoints.web.exposure.include=prometheus property in you application.properties file
Setting the POD annotations
Once your application is emitting metrics, your POD in K8s will need to be annotated to enable the agent to scrape the metrics. Following are the relevant annotations:
YAML
prometheus.io/scrape: 'true'
prometheus.io/path: '/actuator/prometheus'
prometheus.io/port: '8080'
prometheus.io/scrape: 'true'
prometheus.io/path: '/actuator/prometheus'
prometheus.io/port: '8080'
﻿
Java Tracing Instrumentation
OpenTelemetry is the recommended approach to instrument for tracing. Please follow the instructions in the following document to instrument your application:
﻿OpenTelemetry Java Agent﻿
This is achieved by including OpenTelemetry Java agent as shown in this example:
Shell
java -javaagent:path/to/opentelemetry-javaagent-all.jar \
     -Dotel.resource.attributes=service.name=your-service-name \
     -Dotel.traces.exporter=zipkin \
     -Dotel.metrics.exporter=none \
     -Dotel.exporter.zipkin.endpoint=http://otel-collector.devopsnow.svc.cluster.local:9411 \
     -jar myapp.jar
java -javaagent:path/to/opentelemetry-javaagent-all.jar \
     -Dotel.resource.attributes=service.name=your-service-name \
     -Dotel.traces.exporter=zipkin \
     -Dotel.metrics.exporter=none \
     -Dotel.exporter.zipkin.endpoint=http://otel-collector.devopsnow.svc.cluster.local:9411 \
     -jar myapp.jar
﻿
Following args will need to be set:
otel.resource.attributes=service.name=your-service-name
otel.traces.exporter=zipkin
otel.metrics.exporter=none
otel.exporter.zipkin.endpoint=http://otel-collector.devopsnow.svc.cluster.local:9411
This will make the Java application to emit spans which will be collected by the OpenTelemetry collector installed as part of the OpsVerse agent and forwarded to the tracing backend.
Trace context (TraceId, SpanId and TraceFlags) injection into logs
Trace ID, Span ID and trace flags can be printed to logs using MDC. The Mapped Diagnostic Context (MDC) is an instrument for distinguishing interleaved log output from different sources. Log output is typically interleaved when a server handles multiple clients near-simultaneously. The MDC is managed on a per thread basis. It contains thread-local contextual information which is later copied to each logging event captured by a logging library.
Log4j instrumentation
Step 1: Add instrumented log4j2 and opentelemetry-api libraries into the project dependencies:
Maven projects:
Java
<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-log4j-2.13.2</artifactId>
    <version>1.9.2-alpha</version>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
    <version>1.26.0</version>
</dependency>
<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-log4j-2.13.2</artifactId>
    <version>1.9.2-alpha</version>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
    <version>1.26.0</version>
</dependency>
﻿
Gradle projects:
Java
dependencies {
     runtimeOnly("io.opentelemetry.instrumentation:opentelemetry-log4j-2.13.2:1.9.2-alpha")
     implementation("io.opentelemetry:opentelemetry-api:1.26.0")
}
dependencies {
     runtimeOnly("io.opentelemetry.instrumentation:opentelemetry-log4j-2.13.2:1.9.2-alpha")
     implementation("io.opentelemetry:opentelemetry-api:1.26.0")
}
﻿
Any recent versions can be picked for both dependencies.
Step 2: Update the log4j2.xml configuration file, typically stored in resources directory, with traceId and spanId. The following is an example configuration.
Shell
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} traceId: %X{trace_id} spanId: %X{span_id} - %msg%n" />
        </Console>
    </Appenders>
    <Loggers>
        <Root level="All" >
            <AppenderRef ref="Console"/>
        </Root>
    </Loggers>
</Configuration>
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} traceId: %X{trace_id} spanId: %X{span_id} - %msg%n" />
        </Console>
    </Appenders>
    <Loggers>
        <Root level="All" >
            <AppenderRef ref="Console"/>
        </Root>
    </Loggers>
</Configuration>
﻿
If you are using SpringBoot and you don't have an explicit log4j2.xml file, please add the following block in application.properties file.
Shell
logging.pattern.level=%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} traceId: %X{trace_id} spanId: %X{span_id} - %msg%n
logging.pattern.level=%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} traceId: %X{trace_id} spanId: %X{span_id} - %msg%n
﻿
Logback instrumentation
Step 1: Add instrumented logback library into the project dependencies:
Maven projects:
Java
<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-logback-1.0</artifactId>
    <version>1.9.2-alpha</version>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-logback-1.0</artifactId>
    <version>1.9.2-alpha</version>
    <scope>runtime</scope>
</dependency>
﻿
Gradle projects:
Java
dependencies {
     runtimeOnly("io.opentelemetry.instrumentation:opentelemetry-logback-1.0:1.9.2-alpha")
}
dependencies {
     runtimeOnly("io.opentelemetry.instrumentation:opentelemetry-logback-1.0:1.9.2-alpha")
}
﻿
Any recent versions can be picked for the dependency.
Step 2: Update the logback.xml configuration file, typically stored in the resources directory, with traceId, spanId, and sampled keys. The following is an example configuration.
Shell
<?xml version="1.0" encoding="UTF-8" ?>
<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern><![CDATA[%date{HH:mm:ss.SSS} [%thread] %-5level %logger{15}#%line %X{req.requestURI} traceId: %X{trace_id} spanId: %X{span_id} %msg\n]]></pattern>
        </encoder>
    </appender>

    <appender name="OTEL" class="io.opentelemetry.instrumentation.logback.v1_0.OpenTelemetryAppender">
        <appender-ref ref="STDOUT" />
    </appender>

    <root>
        <level value="DEBUG" />
        <appender-ref ref="STDOUT" />
    </root>

</configuration>
<?xml version="1.0" encoding="UTF-8" ?>
<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern><![CDATA[%date{HH:mm:ss.SSS} [%thread] %-5level %logger{15}#%line %X{req.requestURI} traceId: %X{trace_id} spanId: %X{span_id} %msg\n]]></pattern>
        </encoder>
    </appender>

    <appender name="OTEL" class="io.opentelemetry.instrumentation.logback.v1_0.OpenTelemetryAppender">
        <appender-ref ref="STDOUT" />
    </appender>

    <root>
        <level value="DEBUG" />
        <appender-ref ref="STDOUT" />
    </root>

</configuration>
﻿
If you are using SpringBoot and you don't have an explicit logback.xml file, please add the following block in application.properties file.
Shell
logging.pattern.level=%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} traceId: %X{trace_id} spanId: %X{span_id} - %msg%n
logging.pattern.level=%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} traceId: %X{trace_id} spanId: %X{span_id} - %msg%n
﻿
Frameworks
Spring Boot
For Spring Boot configuration which uses logback, you can add MDC to log lines by overriding the logging.pattern.level  config in application.properties file using the following block:
Shell
logging.pattern.level=trace_id=%mdc{trace_id} span_id=%mdc{span_id} trace_flags=%mdc{trace_flags} %5p
logging.pattern.level=trace_id=%mdc{trace_id} span_id=%mdc{span_id} trace_flags=%mdc{trace_flags} %5p
﻿
SpringBoot uses Logback for logging by default. Logback can also be configured to handle complex logging needs. The aforementioned logging config will work both default and custom Logback logging configs.
Java Access Logs Instrumentation
Springboot comes with an embedded Tomcat server. The access logs track user session activity and page hit counts. Springboot's access logs are by default written to a file and not to stdout. This means that most of the log collection agents do not automatically collect the access logs from containers. There are a few strategies to handle this scenario. We recommend updating the Springboot app's properties file to make the access logs written to stdout. This can be done by setting the following properties on the Springboot app.properties file. Note that the keys in server.tomcat.accesslog.pattern can be changed according to requirement. Usage of json logging pattern helps in creating dashboards and alerts using LogQL.
Java
server.tomcat.accesslog.enabled=true
server.tomcat.accesslog.pattern={\"ip\": \"%h\", \"user\": \"%l\", \"username\": \"%u\", \"time\": \"%t\", \"request\": \"%r\", \"status\":\"%s\", \"responsesize\": \"%b\"}
server.tomcat.accesslog.directory=/dev
server.tomcat.accesslog.prefix=stdout
server.tomcat.accesslog.buffered=false
# Don't use empty double quotes,
server.tomcat.accesslog.suffix=
server.tomcat.accesslog.file-date-format=
server.tomcat.accesslog.enabled=true
server.tomcat.accesslog.pattern={\"ip\": \"%h\", \"user\": \"%l\", \"username\": \"%u\", \"time\": \"%t\", \"request\": \"%r\", \"status\":\"%s\", \"responsesize\": \"%b\"}
server.tomcat.accesslog.directory=/dev
server.tomcat.accesslog.prefix=stdout
server.tomcat.accesslog.buffered=false
# Don't use empty double quotes,
server.tomcat.accesslog.suffix=
server.tomcat.accesslog.file-date-format=
﻿
For more application-level configurations visit Common Application Properties.
Single Machine
Metrics Instrumentation
For Single Machines, you can install the Prometheus JMX exporter on a single machine.
Prerequisite(s):
you have the OpsVerse Agent running on the machine.
Run this command to download the Prometheus JMX Exporter agent to your machine:
Shell
# For amd64
curl https://raw.githubusercontent.com/OpsVerseIO/installers/main/prometheus-exporters/install-exporter-amd64.sh | sudo bash -s -- -e jmx

# For arm64
curl https://raw.githubusercontent.com/OpsVerseIO/installers/main/prometheus-exporters/install-exporter-arm64.sh | sudo bash -s -- -e jmx
# For amd64
curl https://raw.githubusercontent.com/OpsVerseIO/installers/main/prometheus-exporters/install-exporter-amd64.sh | sudo bash -s -- -e jmx

# For arm64
curl https://raw.githubusercontent.com/OpsVerseIO/installers/main/prometheus-exporters/install-exporter-arm64.sh | sudo bash -s -- -e jmx
﻿
At this point, you can start your Java runtime with the downloaded agent:
Shell
java -javaagent:/usr/local/bin/jmx_prometheus_javaagent-0.16.1.jar=9404:/etc/opsverse/exporters/jmx/config.yaml -jar yourJar.jar
java -javaagent:/usr/local/bin/jmx_prometheus_javaagent-0.16.1.jar=9404:/etc/opsverse/exporters/jmx/config.yaml -jar yourJar.jar
﻿
The agent will begin scraping the metrics automatically (because the script will drop the scrape target into /etc/opsverse/targets/jmx-exporter.json for the agent to auto-discover).
If you change the port of the exporter (from 9404 in the java ... invocation above), please update /etc/opsverse/targets/jmx-exporter.json with the new port.
Tracing Instrumentation
Follow the Java Tracing Instrumentation described above to emit and send traces to your Jaeger collector endpoint.
The -Dotel.exporter.zipkin.endpoint argument should be pointing to your jaeger collector.
Example: https://your-jaegar-endpoint:9411/api/v2/spans
﻿
Did this page help you?
Python
.NET / .NET Framework