--- title: Elastic Container Service (ECS) slug: elastic-container-service-ecs docTags: createdAt: 2024-07-19T14:31:52.855Z --- Telemetry from Elastic Container Service (ECS) clusters of types Fargate, EC2 or External can be brought into OpsVerse ObserveNow using an OpenTelemetry sidecar deployment. The sidecar is a lightweight container that runs in each application container of your ECS cluster and collects observability data and forwards it to ObserveNow. ## Collecting Logs from ECS Logs from ECS services are collected using AWS `firelens` log driver and `AWS for fluentbit` logs forwarder. The log forwarder is run as an additional container within the same service (aka `sidecar` container). ### Step 1: Add the Log Forwarder Task Add the following block to your task definition under the `containerDefinitions` section and redeploy your services. Update the value of `awslogs-region` to the right region. :::CodeblockTabs Task Definition For The Log Router ```json ... ... { "name": "opsverse-log-collector", "image": "906394416424.dkr.ecr.us-west-2.amazonaws.com/aws-for-fluent-bit:stable", "cpu": 250, "memory": 512, "essential": true, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-create-group": "True", "awslogs-group": "/ecs/ecs-opsverse-log-collector", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "ecs" } }, "firelensConfiguration": { "type": "fluentbit", "options": { "enable-ecs-log-metadata": "true" } } } ... ... ``` ::: ### Step 2: Forward App Logs to the Log Forwarder Sidecar Change the app container's `LogConfiguration` section to the following: :::CodeblockTabs App's LogConfoguration ```json "logConfiguration": { "logDriver": "awsfirelens", "options": { "Name": "loki", "Match": "*", "Host": "", "Port": "443", "tls": "on", "http_user": "devopsnow", "http_passwd": "", "tls.verify": "off" } } ``` ::: The `Opsverse logs endpoint` and `logs password` are available on the [OpsVerse admin console](https://console.opsverse.io/). ### Step 3: View Logs in ObserveNow Logs collected from ECS can be viewed under the `Explore` section of the `Grafana` that is part of `ObserveNow`. All logs have the label `job=fluenbit` added to them. This label can be used to easily search for the ECS logs. ![ECS logs](https://api.archbee.com/api/optimize/xpy-ZuNXAextve6S5Tto2/RmgVDr6jECegBbKGF8auM_screen-shot-2024-07-20-at-110338-pm.png) ## Collecting Metrics and Traces From ECS Container metrics and distributed traces can be collected using the Open Telemetry collector deployed as a sidecar container. ### Step 1: Create The OTel Collector Config The `oterl-collector` sidecar configuration is managed using a parameter in AWS Parameter Store. Create a new parameter named `/ecs/opsverse/otelcol.yaml` with the following configuration: :::CodeblockTabs OTel Collector Config ```yaml extensions: health_check: receivers: awsecscontainermetrics: collection_interval: 30s otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: batch: timeout: 30s exporters: otlphttp: endpoint: "https://" tls: insecure: false headers: Authorization: "Basic Base64{devopsnow:}" prometheusremotewrite: endpoint: "https:///api/v1/write" headers: Authorization: "Basic Base64{devopsnow:}" resource_to_telemetry_conversion: enabled: true debug: verbosity: normal service: extensions: [health_check] pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlphttp, debug] metrics: receivers: [otlp] processors: [batch] exporters: [otlphttp, debug] metrics/aws: receivers: [awsecscontainermetrics] exporters: [prometheusremotewrite, debug] logs: receivers: [otlp] processors: [batch] exporters: [otlphttp, debug] ``` ::: ### Step 2: Update the `ecsTaskExecution` IAM Role The sidecar container needs to have access to the AWS Parameter Store and Cloudwatch logs, add the following policies to the `ecsTaskExecution` role - `AmazonSSMReadOnlyAccess` - `CloudWatchLogsFullAccess` ### Step 3: Add the OTel Collector SideCar Container Update the task definition to run a sidecar container to run the OpenTelemetry image ( `otel/opentelemetry-collector-contrib:0.143.1`) with the config created above and redeploy the task definition to start seeing data from your ECS cluster in ObserveNow. Add the following to the `ContainerDefinitions` section of the task definition: ```json ... ... { "name": "opsverse-otel-collector", "image": "otel/opentelemetry-collector-contrib:0.143.1", "essential": true, "command": [ "--config=env:OTEL_COLLECTOR_CONFIG" ], "secrets": [ { "name": "OTEL_COLLECTOR_CONFIG", "valueFrom": "/ecs/opsverse/otelcol.yaml" } ], "cpu": 256, "memory": 512, "portMappings": [ { "protocol": "tcp", "containerPort": 4317 }, { "protocol": "tcp", "containerPort": 4318 } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/opsverse-otelcol-logs", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "ecs", "awslogs-create-group": "True" } } } ... ... ``` Edit `awslogs-region` as required and redeploy the ECS service. ### Step 4: View the metrics in ObserveNow Metrics collected from ECS can be viewed under the `Explore` section of the `Grafana` that is part of `ObserveNow`. ![](https://api.archbee.com/api/optimize/xpy-ZuNXAextve6S5Tto2/ttz54ckYs8zRZ788F5QZR_screen-shot-2024-07-21-at-13034-am.png) ### Instrumenting Apps to Send Traces The above steps install a `otel collector` that can receive distributed traces from task deployed in ECS. Follow the steps mentioned in the `Application` section of the documentation to instrument apps using OpenTelemetry. Here are some language specific doc pages: - [Java](https://docs.opsverse.io/java) - [Javascript (Node JS and Browser)](https://docs.opsverse.io/javascript-nodejs-and-browser) - [.Net](https://docs.opsverse.io/net-net-framework) - [Python](https://docs.opsverse.io/python) - [Go](https://docs.opsverse.io/go) - [Ruby](https://docs.opsverse.io/ruby) Following `env` variables need to be added to the app's container def: :::CodeblockTabs ECS TaskDefinition ```json "environment": [ { "name": "OTEL_EXPORTER_OTLP_ENDPOINT", "value": "http://localhost:4317" }, { "name": "OTEL_RESOURCE_ATTRIBUTES", "value": "service.name=" } ``` ::: For more detailed info about insturmenting using Open Telemetry, please refer to the official Open Telemtry [this documentation](https://opentelemetry.io/docs/concepts/instrumentation/).