AWS Lambda Lifecycle Policy Management

This document explains how to use the S3 object tagging feature in the Taegis Lambda function and how to configure S3 lifecycle policies to optimize storage costs.

What is Object Tagging?

Object tagging is a feature that allows the Taegis Lambda to mark S3 objects as "processed" after they have been successfully uploaded to Taegis. This enables you to create lifecycle policies that automatically delete or transition objects based on their processing status.

CloudFormation Option: EnableObjectTagging

When deploying the Taegis Lambda using CloudFormation, you'll see a parameter called EnableObjectTagging:

Parameter	Description	Default
EnableObjectTagging	Enable tagging of processed S3 objects to support tag-based lifecycle policies	false

When Enabled (true)

• The Lambda function adds a tag TaegisProcessed=true to each S3 object after successful processing.
• The CloudFormation template creates the necessary IAM permissions for the Lambda to perform tagging operations.
• You can use tag-based lifecycle policies to manage objects based on their processing status.

When Disabled (false)

• No tags are added to S3 objects.
• No additional IAM permissions are created.
• You need to rely on time-based lifecycle policies only.

How Tagging Works

When enabled, the Taegis Lambda function:

1. Receives notification of a new object in your S3 bucket.
2. Reads the object data.
3. Uploads the data to Taegis.
4. Upon successful upload, adds a TaegisProcessed=true tag to the original S3 object.

The tagging operation is performed asynchronously to ensure it doesn't impact the primary data processing flow.

Configuring S3 Lifecycle Policies

You can use the TaegisProcessed tag to create efficient lifecycle policies that:

• Delete objects quickly after they've been processed.
• Keep unprocessed objects longer for troubleshooting or other requirements.
• Transition objects to different storage classes based on their status.

Example Lifecycle Policy

Here's an example S3 lifecycle policy that uses the TaegisProcessed tag:

{
  "Rules": [
    {
      "ID": "Delete-Processed-Objects",
      "Status": "Enabled",
      "Filter": {
        "Tag": {
          "Key": "TaegisProcessed",
          "Value": "true"
        }
      },
      "Expiration": {
        "Days": 7
      }
    },
    {
      "ID": "Delete-All-Objects",
      "Status": "Enabled",
      "Filter": {},
      "Expiration": {
        "Days": 30
      }
    }
  ]
}

This policy:

1. Deletes processed objects (tagged with TaegisProcessed=true) after 7 days.
2. Deletes all objects (regardless of tags) after 30 days as a safety measure.

Setting Up the Lifecycle Policy

To configure this policy in the AWS Console:

1. Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.
2. In the Buckets list, choose the bucket that contains your log files.
3. Choose the Management tab, then select Create lifecycle rule.
4. Enter a name for your rule (e.g., "Delete-Processed-Objects").
5. Under Rule scope, select Limit the scope using filters.
6. Select Tag and enter TaegisProcessed for the key and true for the value.
7. Choose Next.
8. Under Lifecycle rule actions, select Expire current versions of objects.
9. Enter the number of days after which to expire objects (e.g., 7).
10. Choose Next and then Create rule.
11. Repeat steps 3-10 to create the safety rule without tag filters and with a longer expiration period (e.g., 30 days).

Enabling Object Tagging for Existing Deployments

To enable object tagging for an existing Taegis Lambda deployment:

1. Download the latest CloudFormation template from the Taegis portal.
2. Follow the Amazon AWS Lambda Update procedure.
3. When updating the CloudFormation stack, set EnableObjectTagging to true.
4. Complete the stack update.
5. Configure your S3 lifecycle policy as described above.

Benefits of Using Object Tagging

• Cost Optimization: Delete processed data quickly to reduce storage costs.
• Troubleshooting: Keep unprocessed data longer for investigation.
• Visibility: Easily identify which objects have been successfully processed.
• Automation: Automate cleanup based on actual processing status rather than just time.

Recommended Configuration

For most environments, we recommend:

• Enabling object tagging (EnableObjectTagging = true)
• Setting up a dual-rule lifecycle policy:
  • 7-day expiration for processed objects
  • 30-day safety expiration for all objects

This configuration balances cost optimization with operational safety, ensuring that processed data is removed promptly while maintaining a safety net for any objects that might not be processed successfully.

Considerations

• The Lambda function must have the necessary IAM permissions to add tags to objects (automatically configured when EnableObjectTagging = true)
• If you're using server-side encryption with customer-managed keys (SSE-KMS), ensure the Lambda's IAM role has permission to use those keys
• Tagging operations are performed asynchronously and won't impact the primary data processing flow
• If tagging fails (e.g., due to permission issues), the Lambda will log the error but continue processing other objects

For more information about AWS S3 lifecycle policies, see the AWS documentation.