Amazon RDS Logs Ingester
This Helm chart deploys the RDS Logs Ingester, a tool designed to fetch logs from Amazon RDS instances and optionally forward them to a Loki instance for log management. The deployment is configurable, allowing users to specify RDS instances, Loki API credentials, and resource limits.
- Fetch logs from specified RDS instances.
- Optional integration with Loki for log forwarding.
- Support for custom labels on top of ingesting the RDS instance tags
- Configurable via Helm values for easy customization.
- If using AWS AccessID/SecretKey credentials: A Kubernetes secret named aws-credentials containing AWS access keys and region. Use the following command to create a generic secret to store the AWS Credentials:
- If using Service Account based auth, an IAM role ARN with the appropriate permissions/policies attached to be able to interact with the API
- Note: It is recommended to use Service Account based auth as a better security practice
- If storing the AWS credentials in a secret, consider implementing sealed secrets for much greater security: Sealed Secrets
- An image pull secret for accessing the private Docker registry (included in the helm chart)
This guide outlines the process for creating a role with OIDC (OpenID Connect) web identity in AWS EKS (Elastic Kubernetes Service), attaching policies, and configuring a Kubernetes service account for authentication.
- Open the AWS Management Console.
- Access the IAM (Identity and Access Management) service and follow the steps presented in the following AWS documentation to create a new IAM Role:
- Check if the Kubernetes cluster already has an associated OIDC URL. If one is not associated with the cluster, use the following steps to create a new one:
Configure Trusted Entity and Audience
![Document image Document image](https://images.archbee.com/xpy-ZuNXAextve6S5Tto2/RSxrnJFO0JiU0zUmp1B0Q_1.png?format=webp)
Attach Policies
skip this part as we will be adding custom minimal permissions later on, post role-creation.
![Document image Document image](https://images.archbee.com/xpy-ZuNXAextve6S5Tto2/URQqoUs3nOUB5iDKw1WiN_1.png?format=webp)
Review and Create Role
Click Next: Tags (optional), then Next: Review and Create
![Document image Document image](https://images.archbee.com/xpy-ZuNXAextve6S5Tto2/OLnEXiCWKvNU-f2yVeh3F_1.png?format=webp)
- After following the above step to create a role, navigate to the newly created role to add custom permissions:
click on the Add permissions option and select Create inline policy
![Document image Document image](https://images.archbee.com/xpy-ZuNXAextve6S5Tto2/uHg2UlX0dp045uuxpDbGW_1.png?format=webp)
use the following inline policy to set permissions:
- Go to the Trust relationships tab.
- Confirm the correct OIDC details.
use the following template to edit the Role's Trust Relationship:
Note: do not include the https: prefix when using the OIDC URL from the EKS console:
![Document image Document image](https://images.archbee.com/xpy-ZuNXAextve6S5Tto2/oh8V7-xo_rJAtvJzHwC-R_1.png?format=webp)
- Annotate the service account field in values.yaml with the IAM role ARN
- add the ARN value under the serviceAccount.aws.roleArn field
you can find the roleArn value in the summary section of the role console:
![Document image Document image](https://images.archbee.com/xpy-ZuNXAextve6S5Tto2/vl846qVni37gffkrazBtN_1.png?format=webp)
:
In the values.yaml file, users can configure
- dbInstances RDS instances to be monitored
- replicaCount: Number of pod replicas.
- serviceAccount: Specifies the service account details.
- configMap: Contains configurable parameters such as Loki API information.
- image: Docker image details including the repository, tag, and pull secret.
- awsCredentialsSecret: The name of the secret containing AWS credentials.
- namespace: Kubernetes namespace for deployment.
- resources: Resource requests and limits for the pod.
Database Instances: You can define multiple RDS instances under dbInstances with specific identifiers, regions, and log file filters. Customize the labels for each database to categorize your logs effectively.
Loki Integration: The chart is configured to push logs to Loki. Credentials for Loki are specified under the loki.credentials field. This is crucial for secure and efficient log management.
Service Account and AWS Role: A Kubernetes service account linked to an AWS IAM role (rds-ingester-sa) is used for AWS operations.
Resource Requests and Limits: Set under resources, these parameters ensure efficient utilization of your cluster's resources.
Customization Options: Modify values such as replicaCount, namespace etc. as per your requirements in a custom values.yaml
To deploy the chart, use the following commands:
Using custom values.yaml:
Additional useful helm commands:
The Helm chart creates the following Kubernetes resources:
- Deployment: Manages the lifecycle of the RDS Logs Ingester pods.
- ServiceAccount: Manages identity for processes that run in a Pod.
- ConfigMap: Contains non-confidential data in key-value pairs.
- Secret: Manages sensitive information, such as passwords and tokens.
This Helm chart simplifies the deployment and management of the RDS Logs Ingester in a Kubernetes environment. By adjusting values in the values.yaml file, users can tailor the deployment to their specific needs.
- Logs ingested in Loki, as viewed in Grafana.
![Document image Document image](https://images.archbee.com/xpy-ZuNXAextve6S5Tto2/uDdiwLpMRHaKr3HY8gs0r_1.png?format=webp)