LFTP多线程多(单)文件下载

liuzj

由于mget命令下载速度太慢，就写了个脚本，可以用来进行多线程多(单)文件同步下载，速度比原来得快不少。供大家参考，欢迎转帖。
[code:1]
#!/bin/bash
# PARM1: SOURCE DIRECTORY ON REMOTE HOST
# PARM2: HOST ADDRESS
# PARM3: DESTINATION PATH TO SAVE DOWNLOADED FILES
# PARM4: DYNAMICALLY CREATED SHELL FILE TO RETRIVE REMOTE FILES WITH "PGET" COMMAND

REMOTEPATH=$1
HOSTADDR=$2
DESTPATH=$3
DYNAFILE=$4

# test if REMOTEPATH parameter is specified
if [ "$REMOTEPATH" = "" ]; then
        echo "Parameter: REMOTEPATH was not specified."
        exit 1
fi

# test if HOSTADDR parameter is specified
if [ "$HOSTADDR" = "" ]; then
        HOSTADDR=ftp.abc.com
        echo "Parameter: HOSTADDR was not specified, use default hostname: $HOSTADDR."
fi

# test if REMOTEPATH parameter is specified
if [ "$DESTPATH" = "" ]; then
        echo "Parameter: DESTPATH was not specified."
        exit 1
fi

# make sure $REMOTEPATH $DESTPATH do not end with '/'
#
DESTPATH=`echo $DESTPATH | sed 's/\/$//g'`
REMOTEPATH=`echo $REMOTEPATH | sed 's/\/$//g'`

# test if destination path exists
if [ ! -d $DESTPATH ]; then
# if Destination Path: $DESTPATH does not exist, it will be created automatically.
        mkdir $DESTPATH
        if [ `echo $?` = "1" ]; then
                echo "Cannot create directory: $DESTPATH."
                exit 1
        fi
fi

# test if destination path is writable
if [ ! -w $DESTPATH ]; then
        echo "Destination Path: $DESTPATH is not writable."
        exit 1
fi


tracelog=/tmp/trace.log.$$

USER=`whoami`
THIS_HOST=`hostname -s`
LOGDATE=`date +%d/%m/%Y`

# use traceroute to test connectivity
traceroute $HOSTADDR > $tracelog 2>&1
grep "unkown host" $tracelog >/dev/null 2>&1

if [ `echo $?` = "0" ]; then
        echo "Host: $HOSTADDR is not reachable."
        exit 1
fi

# retrieve file list
lftp $HOSTADDR<<FTPCMD
cd $REMOTEPATH
lcd $DESTPATH
nlist > $DYNAFILE
quit
FTPCMD

# multi file multi connections downloading, uncomment next line
# sed 's/^./pget &/g' $DYNAFILE | sed 's/$/& \&/g' > $DYNAFILE

#single file multi connections downloading, comment next line for multi file multi connections downloading
sed 's/^./pget &/g' $DYNAFILE | cat > $DYNAFILE

echo $DESTPATH | sed 's/^./lcd &/g' > $DYNAFILE.tmp
cat $DYNAFILE.tmp $DYNAFILE | cat > $DYNAFILE
echo $REMOTEPATH | sed 's/^./cd &/g' > $DYNAFILE.tmp
cat $DYNAFILE.tmp $DYNAFILE | cat > $DYNAFILE
echo $HOSTADDR | sed 's/^./lftp &/g' | sed 's/$/&<<FTPCMD/g' > $DYNAFILE.tmp
cat $DYNAFILE.tmp $DYNAFILE | cat > $DYNAFILE
echo '#!/bin/bash' > $DYNAFILE.tmp
cat $DYNAFILE.tmp $DYNAFILE | cat > $DYNAFILE

awk '{print $0} END { print "wait all"} ' $DYNAFILE | cat > $DYNAFILE
awk '{print $0} END { print "quit"} ' $DYNAFILE | cat > $DYNAFILE
awk '{print $0} END { print "FTPCMD"} ' $DYNAFILE | cat > $DYNAFILE
# remove temporary file
rm -f $DYNAFILE.tmp

# change file prvilage and excute the dynamically created shell file
chmod u+x $DYNAFILE
. $DYNAFILE

# rm -f $DYNAFILE[/code:1]

MichaelBibby编辑，以code格式格式化代码
                            ----MichaelBibby 11/07/04

yangyong

矣，lftp不是可以多线程下吗?
pget -n100 http://*****************

MichaelBibby

不错的脚本，加精鼓励

PS：liuzj兄，我编辑了你的帖子，以code格式格式化脚本，这样代码看起来很整洁  

liuzj

主要是用在自动下载目录下的文件，定制化的下载文件。

谢谢加的编辑。

bzimage

嗯，有图形界面就好了