Backing up your files from your EC2 machine to amazon S3 bucket

Posted By : Arvind Das | 19-Jun-2012

Amazon S3 bucket are containers for objects (the files you store on s3). I would explain how to make back up of your files from EC2 machine to amazon s3 bucket.

  • Create a S3 account. If you already have an account ,move to step two. Else click here
  • Download your aws access key and aws secret access key.
    • You can download them from your security credentials menu.
  • Now download and install s3fs. You can Download
  • Install these prerequisites before installing.
  • yum install gcc 
    yum install libstdc++-devel 
    yum install gcc-c++ 
    yum install fuse 
    yum install fuse-devel 
    yum install curl-devel 
    yum install libxml2-devel 
    yum install openssl-devel 
    yum install mailcap 
    
  • Installation Steps
    • Unzip the downloaded s3fs-1.61.tar.gz and follow steps further
    tar xvzf s3fs-1.61.tar.gz 
    cd s3fs-1.61/ 
    ./configure --prefix=/usr 
    make 
    make install (as root) 
    
  • Create a file named .passwd-s3fs in your root directory.
  • Vi .passwd-s3fs
    
  • Write your access key id and secret access key in the format as aws_access_key:aws_secret_key
    • Change its mode as “chmod 600 ~/.passwd-s3fs “ so that no other group has access to it.
    • save and exit
  • In this method we will create a mount point for your bucket in your machine, So that everything you put in that directory(mount point) will be automatically moved to S3 bucket.
    • To mount your S3 bucket run
    • s3fs -o allow_other bucket_name /mnt/your_directory_name
      
    • This is the directory where you will put your files to move to S3
    • To copy files to your S3 bucket run rsync. Read the note below before executing this command
    • rsync -av  /you_directory_path_to_files  /mnt/your_directory_name
      
    • [Note] --delete command is to ensure that if you delete files from your bucket it will be deleted from your machine too.
  • If you have issues in building “make” or ./config your s3fs , then you may be running an older version of FUSE. Update that manually as
    • Download Latest Fuse
  •                    tar xvzf fuse-2.8.6.tar.gz
                        cd fuse-2.8.6
                       ./configure –prefix=/usr
                        make 
                        make install
    
  • To check if installation is correct, run
  • modprobe fuse
    

    If it runs clean, OK

    Else it may show an error

    FATAL: Could not load /lib/modules/2.6.16.33-xenU/modules.dep: No such file or directory

  • In such case, download http://s3.amazonaws.com/ec2-downloads/ec2-modules-2.6.16.33-xenU-x86_64.tgz extract
  • tar xvf ec2-modules-2.6.16.33-xenU-x86_64.tgz
    cd lib/modules
    cp -r * /lib/modules/
    

About Author

Author Image
Arvind Das

Technical Product Manager experienced in developing Real Time Planning Solutions, Video CMS with experience in Product Management and technical expertise in Reactive applications , Micro-services . Spending most of his time in planning , new solutions.

Request for Proposal

Name is required

Comment is required

Sending message..