如何将文件从FTP服务器增量复制到Hadoop HDFS

Question

我们有一个FTP服务器，每天都有很多文件上传到FTP服务器，我需要在HDFS中复制所有这些文件。

每次只应下载增量文件，即首次下载10个文件后，FTP上载了5个新文件；在作业的下一次迭代中，它应该只下载HDFS中的新5个文件。

我们没有使用Nifi或Kafka连接。

我们有什么好的解决方案来完成这项任务吗？

Answer 1

您可以在LFTP作业中使用touch文件来实现这一点，下面是我的解释和代码。查看每个步骤的注释。

#!bin/bash
#SomeConfigs
TOUCHFILE='/somepath/inYourLocal/someFilename.touch'
RemoteSFTPserverPath='/Remote/Server/path/toTheFiles'
LocalPath='/Local/Path/toReceiveTheFiles'
FTP_Server_UserName='someUser'
FTP_Server_Password='SomePassword'
ServerIP='//127.12.11.35'

#Transfer files from FTP Server #This is the main command
ftp_command="lftp -e 'mirror --only-missing --newer-than=${TOUCHFILE} --older-than=now-2minutes --parallel=4 --no-recursion --include "SomeFileName*.csv"  ${RemoteSFTPserverPath}/ ${LocalPath}/;exit' -u ${FTP_Server_UserName},${FTP_Server_Password} sftp:${ServerIP}"

#CommandToexecute The Job
eval ${ftp_command}

#After finishing the lftp job You have to update the touch file for the next job
#This will update to current timestamp
touch /somepath/inYourLocal/someFilename.touch

#If you want to update with the last file received time
TchDate=$(stat -c %y "$(ls -1t ${LocalPath} | head -n1)" | date)
touch -d ${TchDate} /somepath/inYourLocal/someFilename.touch

#Stat on latest file in remote server #You can do this way also
TchDate=$(ssh -o StrictHostKeyChecking=no ${FTP_Server_UserName}@${FTP_Server_Password} "stat -c %y \"$(ls -1t ${RemoteSFTPserverPath}/ | head -n1)\" | date")
touch -d ${TchDate} /somepath/inYourLocal/someFilename.touch

#Once you have the files in your local you can copy them to hdfs

hdfs dfs -put -f /Local/Path/toReceiveTheFiles/*.csv /HDFS/PATH

#Remove the files in local so that you can accommodate for the upcoming files
rm -r -f /Local/Path/toReceiveTheFiles/*.csv

在LFTP作业中，您有很多选项man lftp将是您最好的来源