Move S3 Files Using Python Boto3

Using python boto3 module, we can move the s3 files from one bucket to another bucket or with in the same bucket. In this article we will explain how to achieve with an example.

Let’s say, the bucket name is sde-whiletrue-live and the s3 object key be old-location/data.parquet. We would like to move the object to the bucket dev-whiletrue-live and new key location be new-location/data.parquet.

Example

import boto3

#Creating Session With Boto3.
session = boto3.Session(
aws_access_key_id='Your Access Key ID',
aws_secret_access_key='You Secret access key'
)

#Creating S3 Resource From the Session.
s3 = session.resource('s3')

#Create a Source Dictionary That Specifies Bucket Name and Key Name of the Object to Be Copied
source = {
    'Bucket': 'sde-whiletrue-live',
    'Key': 'old-location/data.parquet'
}

# create dest bucket object
bucket = s3.Bucket('sde-whiletrue-live')

# use s3 bucket object method `copy` to copy the object 
bucket.copy(copy_source, 'new-location/data.parquet')

In the above example we have created source dict with Bucket and Key. Also, we have created bucket object with destination bucket. We are using bucket object method copy and passing source dict as first parameter and destination key as second parameter.