Elastic Container Service (ECS)

telemetry from elastic container service (ecs) clusters of types fargate, ec2 or external can be brought into opsverse observenow using an opentelemetry sidecar deployment the sidecar is a lightweight container that runs in each application container of your ecs cluster and collects observability data and forwards it to observenow collecting logs from ecs logs from ecs services are collected using aws firelens log driver and aws for fluentbit logs forwarder the log forwarder is run as an additional container within the same service (aka sidecar container) step 1 add the log forwarder task add the following block to your task definition under the containerdefinitions section and redeploy your services update the value of awslogs region to the right region task definition for the log router { "name" "opsverse log collector", "image" "906394416424 dkr ecr us west 2 amazonaws com/aws for fluent bit\ stable", "cpu" 250, "memory" 512, "essential" true, "logconfiguration" { "logdriver" "awslogs", "options" { "awslogs create group" "true", "awslogs group" "/ecs/ecs opsverse log collector", "awslogs region" "us east 1", "awslogs stream prefix" "ecs" } }, "firelensconfiguration" { "type" "fluentbit", "options" { "enable ecs log metadata" "true" } } } step 2 forward app logs to the log forwarder sidecar change the app container's logconfiguration section to the following app's logconfoguration "logconfiguration" { "logdriver" "awsfirelens", "options" { "name" "loki", "match" " ", "host" "\<opsverse logs endpoint>", "port" "443", "tls" "on", "http user" "devopsnow", "http passwd" "\<logs password>", "tls verify" "off" } } the opsverse logs endpoint and logs password are available on the opsverse admin console step 3 view logs in observenow logs collected from ecs can be viewed under the explore section of the grafana that is part of observenow all logs have the label job=fluenbit added to them this label can be used to easily search for the ecs logs ecs logs collecting metrics and traces from ecs container metrics and distributed traces can be collected using the open telemetry collector deployed as a sidecar container step 1 create the otel collector config the oterl collector sidecar configuration is managed using a parameter in aws parameter store create a new parameter named /ecs/opsverse/otelcol yaml with the following configuration otel collector config extensions health check receivers awsecscontainermetrics collection interval 30s otlp protocols grpc endpoint 0 0 0 0 4317 http endpoint 0 0 0 0 4318 processors batch timeout 30s exporters otlp endpoint "\<opsverse otel endpoint> 443" tls insecure false headers authorization "basic base64{devopsnow \<password>}" prometheusremotewrite endpoint "https //\<opsverse metrics endpoint>/api/v1/write" headers authorization "basic base64{devopsnow \<password>}" resource to telemetry conversion enabled true logging verbosity normal service extensions \[health check] pipelines traces receivers \[otlp] processors \[batch] exporters \[otlp, logging] metrics receivers \[otlp] processors \[batch] exporters \[otlp, logging] metrics/aws receivers \[awsecscontainermetrics] exporters \[prometheusremotewrite, logging] logs receivers \[otlp] processors \[batch] exporters \[otlp, logging] step 2 update the ecstaskexecution iam role the sidecar container needs to have access to the aws parameter store and cloudwatch logs, add the following policies to the ecstaskexecution role amazonssmreadonlyaccess cloudwatchlogsfullaccess step 3 add the otel collector sidecar container update the task definition to run a sidecar container to run the opentelemetry image ( otel/opentelemetry collector contrib 0 105 0 ) with the config created above and redeploy the task definition to start seeing data from your ecs cluster in observenow add the following to the containerdefinitions section of the task definition { "name" "opsverse otel collector", "image" "otel/opentelemetry collector contrib 0 105 0", "essential" true, "command" \[ " config=env\ otel collector config" ], "secrets" \[ { "name" "otel collector config", "valuefrom" "/ecs/opsverse/otelcol yaml" } ], "cpu" 256, "memory" 512, "portmappings" \[ { "protocol" "tcp", "containerport" 4317 }, { "protocol" "tcp", "containerport" 4318 } ], "logconfiguration" { "logdriver" "awslogs", "options" { "awslogs group" "/ecs/opsverse otelcol logs", "awslogs region" "us east 1", "awslogs stream prefix" "ecs", "awslogs create group" "true" } } } edit awslogs region as required and redeploy the ecs service step 4 view the metrics in observenow metrics collected from ecs can be viewed under the explore section of the grafana that is part of observenow instrumenting apps to send traces the above steps install a otel collector that can receive distributed traces from task deployed in ecs follow the steps mentioned in the application section of the documentation to instrument apps using opentelemetry here are some language specific doc pages java javascript (node js and browser) net python go ruby following env variables need to be added to the app's container def ecs taskdefinition "environment" \[ { "name" "otel exporter otlp endpoint", "value" "http //opsverse otel collector 4317" }, { "name" "otel resource attributes", "value" "service name=\<service name>" } for more detailed info about insturmenting using open telemetry, please refer to the official open telemtry this documentation