How to migrate data from a bucket to another cross region
This guide will help you to migrate data from a bucket in one region to a bucket in another region using “obsutil” Flexible Engine command line tool.
This tool can used from a Linux ECS hosted in any Flexible Engine or from a Windows workstation connected to internet.
Installation on Linux
Prerequisites:
- A Linux ECS is instantiated in Flexible Engine in eu-west-0 or eu-west-1 with outgoing internet connectivity (EIP or NAT Gateway)
- Flexible Engine user has OBS access rights (user belong to default “admin’ group” or belong to a group with OBS service IAM policy allowing read/write access
Steps:
- Connect to ECS using SSH or console “Remote Login”
- Download “obsutil” tools:
wget https://oss.eu-west-0.prod-cloud-ocb.orange-business.com/obsutil/obsutil_ocb_linux_amd64_5.3.4.tar.gz
- Extract archive:
tar -xzvf obsutil_ocb_linux_amd64_5.3.4.tar.gz
- Make the tool executable:
chmod 755 obsutil_linux_amd64_5.3.4/obsutil
- Check the tool is working:
./obsutil_linux_amd64_5.3.4/obsutil version
- Configure obsutil for the bucket to be migrated:
./obsutil_linux_amd64_5.3.4/obsutil config -interactive
Please input your ak:
xxxxxxxxxxxxxxxxxxxxxxxxx
Please input your sk:
xxxxxxxxxxxxxxxxxxxxxxxxx
Please input your endpoint:
oss.eu-west-1.prod-cloud-ocb.orange-business.com
Please input your token:
Config file url:
C:\Users\tools\.obsutilconfig
Update config file successfully!
- Check the access to the bucket to be migrated:
./obsutil_linux_amd64_5.3.4/obsutil ls -limit=5
Bucket CreationDate Location BucketType
obs://bucket001 2018-09-03T01:53:02Z example OBJECT
obs://bucket002 2018-11-01T01:40:01Z example OBJECT
obs://bucket003 2018-10-25T11:45:45Z example OBJECT
obs://bucket004 2018-10-26T02:33:09Z example OBJECT
obs://bucket005 2018-10-26T02:34:50Z example OBJECT
Bucket number is: 5
Installation on Windows workstation
Prerequisites:
- A Windows workstation with outgoing internet connectivity
Steps:
- Download “obsutil” tool using the following link: https://oss.eu-west-0.prod-cloud-ocb.orange-business.com/obsutil/obsutil_ocb_windows_amd64_5.3.4.zip
- Extract the archive into a directory
- Execute the program “obsutil.exe” from the directory
- Configure “obsutil” for the bucket to be migrated:
Enter "exit" or "quit" to logout
Enter "help" or "help command" to show help docs
Input your command:
-->config -interactive
Please input your ak:
xxxxxxxxxxxxxxxxxxxxxxxxxxx
Please input your sk:
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Please input your endpoint:
xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Please input your token:
Config file url:
C:\Users\uvez8210\.obsutilconfig
Update config file successfully!
- If needed, configure your proxy by editing the “.obsutilconfig” file located in your user directory:
proxyUrl=http://username:password@your-proxy:8080
Migration
Steps:
- Create the new bucket:
mb obs://bucket-destination -location=eu-west-0
- Synchronize objects from source bucket with destination bucket:
sync obs://bucket-source obs://bucket-destination
- Delete objects inside source bucket:
rm obs://bucket-source -r -f
- Delete source bucket:
rm obs://bucket-source -f
Usefull link:
https://docs.prod-cloud-ocb.orange-business.com/utiltg/obs/obs_11_0001.html