script to sync folder to s3 using cron and s3cmd
so on the command line, i have been looking for a brain dead simple way of syncing my photographs to s3 every x amount of hours.
i wrote a small script to handle that and can be placed in cron to automatically copy new folders to s3. i am working on making this script also sync the files in each folder but that will be a later post.
the script uses s3cmd from http://s3tools.org/s3cmd or in gentoo net-misc/s3cmd.
here is the script :
#!/bin/bash
######
# this script uses s3cmd from http://s3tools.org/s3cmd to sync a folder in linux to s3
#
# make sure that you have configured s3cmd by running s3cmd --configure
#
# this script is usally kept in a cron to be run every x amount of hours
######
S3CMD=`which s3cmd`
S3CMD_OPTIONS="--recursive put"
BKUP_PATH=CHANGE THIS TO PATH WHERE FOLDERS ARE LOCATED
BUCKET=s3://CHANGE THIS TO S3 BUCKET NAME FROM "s3cmd ls"/
LIST=/tmp/S3CMD.$$
DIR=/tmp/DIR.$$
touch $LIST $DIR
#### you should not have to change anything below this line ####
# check if s3cmd exists
if [[ -z ${S3CMD} ]]; then
printf "\ns3cmd not found.\nInstall s3cmd from http://s3tools.org/s3cmd\n"
exit 1
fi
# use s3cmd to print a list of existing folders in bucket
${S3CMD} la > $LIST
# check if folders in $BKUP_PATH exist in s3, if not then copy them there
for DIR_CHECK in `ls -al $BKUP_PATH | awk '{print $9}' | tail -n +4 | tr '/' ' ' ` ; do
if [[ $(grep -c $DIR_CHECK $LIST) -gt 0 ]]; then
printf "\n${DIR_CHECK} exists in s3."
else
printf "\n${DIR_CHECK} does not exist at s3."
printf "\nadding now:\n"
${S3CMD} ${S3CMD_OPTIONS} ${BKUP_PATH}/${DIR_CHECK} ${BUCKET}
fi
done
# delete temporary files
rm -rf $LIST
rm -rf $DIR
script is located here
i realize that this whole script can be replaced with one command, but for simplicity (depending who is using it) i decided to just make it since it was a 5 minute thing.
Leave a Reply