AWS RDS Logs Ingester

overview this helm chart deploys the rds logs ingester, a tool designed to fetch logs from amazon rds instances and optionally forward them to a loki instance for log management the deployment is configurable, allowing users to specify rds instances, loki api credentials, and resource limits key features fetch logs from specified rds instances optional integration with loki for log forwarding support for custom labels on top of ingesting the rds instance tags configurable via helm values for easy customization setting up authentication two types of authentication can be set up for accessing aws rds 1\ using service account based auth 2\ creating a secret using aws credentials any one of the above authentication can be set up service account based authentication is used by default in the chart setting up service account based auth before configuring the service account in the chart values, an iam role arn with the appropriate permissions/policies attached needs to be created in the aws iam prerequisites 1\ check if your aws eks cluster already has an associated openid connect provider url to do this navigate to the overview section of you aws eks cluster in the aws console and check for openid connect provider url if one is not associated with the cluster, use the following aws doc to do so https //docs aws amazon com/eks/latest/userguide/authenticate oidc identity provider html 2\ navigate to access management > identity providers in aws iam check for an entry corresponding to you oidc provider url if not present, add an identity provider for the correspoding oidc provider url using the below aws doc https //docs aws amazon com/eks/latest/userguide/enable iam roles for service accounts html 1\ check if your aws eks cluster already has an associated openid connect provider url to do this navigate to the overview section of you aws eks cluster in the aws console and check for openid connect provider url if one is not associated with the cluster, use the following aws doc to do so https //docs aws amazon com/eks/latest/userguide/authenticate oidc identity provider html 2\ navigate to access management > identity providers in aws iam check for an entry corresponding to you oidc provider url if not present, add an identity provider for the correspoding oidc provider url using the below aws doc https //docs aws amazon com/eks/latest/userguide/enable iam roles for service accounts html step 1 creating a new iam role navigate to access management > roles in aws iam create a new role of type web identity using the identity provider corresponding to your oidc provider url set the audience as sts amazonaws com skip the add permissions step as we will be adding necessary inline permission post role creation give an appropriate name for the role and then create the role step 2 attaching custom policy to the role navigate to the permissions section of the newly create role click on add permissions > create inline policy and use the below json template to add necessary policies to the role give an appropriate name and create the policy { "version" "2012 10 17", "statement" \[ { "effect" "allow", "action" \[ "rds\ downloaddblogfileportion", "rds\ describedblogfiles", "rds\ describedbinstances" ], "resource" "arn\ aws\ rds <12 digit aws account id>\ db " } ] } step 3 verify trust relationships navigate to the trust relationships section inside your role verify and update the trust relationships using the below json { "version" "2012 10 17", "statement" \[ { "effect" "allow", "principal" { "federated" "arn\ aws\ iam \<aws account id>\ oidc provider/\<openid connect url>" }, "action" "sts\ assumerolewithwebidentity", "condition" { "stringlike" { "\<openid connect url>\ aud" "sts amazonaws com", "\<openid connect url>\ sub" "system\ serviceaccount rds ingester sa" } } } ] } do not include the https prefix when using the oidc url from the eks console setting up auth via secret using aws credentials use the following command to create a generic secret to store the aws credentials kubectl create secret generic aws credentials rds ingester from literal=aws access key id=\<redacted> from literal=aws secret access key=\<redacted> n \<namespace> if using an aws credentials secret for authentication, it should be enabled in the values file that will be applied while installing the chart the steps for it are mentioned in following step configuring values yaml and installing the chart copy the template in a rds ingester values yaml named file and configure the values appropriately enable/disable the awscredentials and serviceaccount under auth based on the type of authentication you are using auth via service account is set to true by default if you are using aws creditials secret as auth, then set the awscreditials to true and serviceaccount to false you can define multiple rds instances under dbinstances with specific identifiers, regions, and log file filters customize the labels for each database to categorize your logs effectively the chart is configured to push logs to loki credentials for loki should be specified under the loki credentials field configure the iam role arn value of your role under the aws rolearn field configmap basic name rds config auth awscredentials enabled false serviceaccount enabled true dbinstances \ identifier \<rds database 1 identifier> labels \<key1> \<value1> \<key2> \<value2> logfilefilter \<provide a log file filter> region \<rds database region> \ identifier \<rds database 2 identifier> labels \<key1> \<value1> \<key2> \<value2> logfilefilter \<provide a log file filter> region \<rds database 2 region> output loki credentials username \<loki username> password \<loki password> secretname loki remote write credentials host \<loki host> writetolokiapi true writetostdout true image pull secret devopsnow private docker reg crd rds ingester serviceaccount aws rolearn arn\ aws\ iam \<aws account id>\ role/\<iam role name> name rds ingester sa tokenname rds ingester token do not use hyphen ( ) in your label keys use the follow helm commands to deploy the chart // adding opsverse's helm repository helm repo add opsverse helm charts https //registry devopsnow\ io/chartrepo/public // installing the rds ingester helm upgrade install rds logs ingester opsverse helm charts/rds logs ingester n \<namespace> f rds ingester values yaml once the helm chart is deployed using the rds ingester values yaml values file, the aws rds logs should be pulled and pushed to the loki host provided and should be visible in your observenow grafana in the logs datasource