Using S3 replication for DevOps

S3 replication and object ownership transfer can be used nicely for DevOps promotions to higher environments.

We have used this feature instead of messy S3 bucket policies etc.

In the above scenario we have used the DevOps account with S3 buckets matching the environment. We created replica S3 buckets in the relevant environment and associated policies. This is very easy to set up.

Every new object that gets copied into the main S3 is automatically replicated into the replica bucket. Also with the new ownership change we can assign ownership to the account who owns the bucket. This was a problem earlier as the object even if copied into a bucket owned by someone else did not transfer ownership automatically.

In the dev, test and other accounts you create a trigger to deploy new versions. This is very easy to control as the CodePipeline or any other mechanism you may choose for deployment can also be CloudFormed or Terraformed into the specific environments.

A backward S3 replica consolidating everything into the devops account for provenance can also be created to keep track of which versions of the artifacts are deployed in each environment. For example, the CodePipeline could update the deployed versions into a manifest file which gets copied back into the DevOps account, also via replication.

When choosing the type of replication for such an scenario you could use the RTC option which allows copies to be done quickly.

From AWS :-

Replicate objects within 15 minutes — You can use S3 Replication Time Control (S3 RTC) to replicate your data in the same AWS Region or across different Regions in a predictable time frame. S3 RTC replicates 99.99 percent of new objects stored in Amazon S3 within 15 minutes (backed by a service level agreement). For more information, see Meeting compliance requirements using S3 Replication Time Control (S3 RTC).

There are also SRR (Same Region Replication) and CRR (Cross Region Replication) Options that can be used effectively in various scenarios. Several other options such as same storage class, retain meta data and ownership as described earlier are also available giving lot of flexibility to address your workflows.

I hope you found this useful. If you have other scenarios where you have used this feature please comment below.