Rsync to AWS S3 mount as a local drive for free

Mount AMAZON S3 as local drive on both Windows and Linux and rsync to backup data to S3 for free without paying any client software.

s3fs on docker is excellent option to build your own backup server and SMB share to Windows boxes.

s3fs to mount Amazon S3 storage as a local disk on Ubuntu docker or VM. NFS share for Linux client and S3 as SMB share for Windows clients.

S3fs now part of Ubuntu so with the apt-get install you can install like any other inbuilt program.

sudo apt-get install f3fs sudo chmod 640 /etc/passwd-s3fs above above text file place your Amazon key:seceret format above steps common for Docker also.It allows d3fs to ready access data safely. Its the important step to begin with s3cmd + rsync to Amazon S3. You ue S3fs to store data on Amazon Glacier also. With the help of Lifecycle policy- you can achive it. Lets see later. can you say alternative to Cloudberry?
Code language: JavaScript (javascript)

amazon s3 fuse based filesystem now ready. now let’s automate s3fs mount s3 bucket task. By updating /etc/fstab as mentioned below you can automate it.

s3fs#linuxsme /home/sshbox/s3-bucket/ fuse _netdev,enable_content_md5,allow_other,url=https://s3.amazonaws.com,retries=5 0 0

now reboot and see. mount -a | | grep -i bucket must show details. There are many options available but you can use above for Productions loads.

I tried around 300 GB and worked as expected. 25 GB each I tried.

Now you can rsync to Amazon s3 or AWS s3. In my setup I shared one folder from my external drive to Ubuntu VM where S3fs installed. with help of cron, every 1 hr rsync will send data to S3.

*/59 * * * * /usr/bin/flock -n /tmp/lock_file -c "/usr/bin/rsync -arv --ignore-existing /mnt/hgfs/backuptos3/* /home/sshbox/s3-bucket"
Code language: JavaScript (javascript)

what it says every 1 hr – check the folder of external drive and copy them to Amazon S3. to avoid duplicates and cost option ignore-existing will help.

Now you use SAMBA services to mount share to any windows box. S3fs is Fuse based excellt net file system to enable rsync to Amazon S3 as local drive. Now you can use it without paying any client fee to mount remote S3 to local disk.

hgfs,s3fs combination enables a production-grade backup system using Amazon S3. You can use normal ls command on mounted Amazon S3.

But please be remember ever call is API call to Amazon S3 so some cost involved. for retrival of 20 GB data from S3 it may cost you around $1.76.

Leave a comment

Your email address will not be published. Required fields are marked *