FastDFS集群搭建、Nginx集成FastDFS、FastDHT文件去重、集成FastDHT、SpringBoot集成FastDFS

资源下载安装

1.安装依赖包libfastcommon
https://github.com/happyfish100/libfastcommon/archive/V1.0.35.tar.gz

[root@CentOSX ~]# yum install -y gcc-c++
[root@CentOSX ~]# tar -zxf V1.0.35.tar.gz
[root@CentOSX ~]# cd libfastcommon-1.0.35
[root@CentOSX libfastcommon-1.0.35]# ./make.sh
[root@CentOSX libfastcommon-1.0.35]# ./make.sh install

2.安装FastDFS
https://github.com/happyfish100/fastdfs/archive/V5.11.tar.gz

[root@CentOSX ~]# yum install -y perl-devel
[root@CentOSX ~]# tar -zxf fastdfs-5.11.tar.gz
[root@CentOSX ~]# cd fastdfs-5.11
[root@CentOSX fastdfs-5.11]# ./make.sh
[root@CentOSX fastdfs-5.11]# ./make.sh install

提示:当软件安装结束后，默认FastDFS启动所需的配置⽂文件放置在/etc/fdfs⽬目录下。

[root@CentOSX ~]# yum install -y tree
[root@CentOSX ~]# tree /etc/fdfs/
/etc/fdfs/
!"" client.conf.sample
!"" storage.conf.sample
!"" storage_ids.conf.sample
#"" tracker.conf.sample
0 directories, 4 files
# 可运⾏行行脚本
[root@CentOSX ~]# ls -l /etc/init.d/fdfs_*
-rwxr-xr-x. 1 root root 961 Jun 30 20:22 /etc/init.d/fdfs_storaged
-rwxr-xr-x. 1 root root 963 Jun 30 20:22 /etc/init.d/fdfs_trackerd
# 执⾏行行程序
[root@CentOSX ~]# whereis fdfs_storaged fdfs_trackerd
fdfs_storaged: /usr/bin/fdfs_storaged
fdfs_trackerd: /usr/bin/fdfs_trackerd

配置服务

1.创建fdfs运行所需的数据目录

[root@CentOSX ~]# mkdir -p /data/fdfs/{tracker,storage/store01,storage/store02}
[root@CentOSX ~]# tree /data/
/data/
#"" fdfs
!"" storage
$ #"" store01
| #"" store02
#"" tracker

2.创建启动所需的配置文件

[root@CentOSX ~]# cp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf
[root@CentOSX ~]# cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf
[root@CentOSX ~]# cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf
[root@CentOSX ~]# tree /etc/fdfs/
/etc/fdfs/
!"" client.conf
!"" client.conf.sample
!"" storage.conf
!"" storage.conf.sample
!"" storage_ids.conf.sample
!"" tracker.conf
#"" tracker.conf.sample

3.配置Tracker Server

[root@CentOSX ~]# vi /etc/fdfs/tracker.conf
base_path=/data/fdfs/tracker

4.配置Storage Server

[root@CentOSX ~]# vi /etc/fdfs/storage.conf
group_name=group`[1,2,3]`//分别是group_name=group1，group_name=group2...
base_path=/data/fdfs/storage
store_path_count=2
store_path0=/data/fdfs/storage/store01
store_path1=/data/fdfs/storage/store02
tracker_server=CentOSA:22122
tracker_server=CentOSB:22122
tracker_server=CentOSC:22122

修改Client端

[root@CentOSX ~]# vi /etc/fdfs/client.conf
base_path=/tmp
tracker_server=CentOSA:22122
tracker_server=CentOSB:22122
tracker_server=CentOSC:22122

启动服务器

[root@CentOSX ~]# /etc/init.d/fdfs_trackerd start|stop|restart
Starting FastDFS tracker server:
[root@CentOSX ~]# /etc/init.d/fdfs_storaged start|stop|restart
Starting FastDFS storage server:
[root@CentOSX ~]# ps -aux | grep fdfs
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
root 78950 0.0 0.1 144784 2040 ? Sl 21:06 0:00
/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf
root 79000 13.0 3.2 83520 67144 ? Sl 21:06 0:06
/usr/bin/fdfs_storaged /etc/fdfs/storage.conf
root 79324 0.0 0.0 103320 884 pts/0 S+ 21:07 0:00 grep fdfs
[root@CentOSX ~]#

FastDFS Shell-运维

上传文件

[root@CentOSX ~]# fdfs_upload_file /etc/fdfs/client.conf install.log
group1/M00/00/01/wKikgV0YtQ2AfmozAAAixZ60QfI792.log

下载

[root@CentOSX ~]# fdfs_download_file /etc/fdfs/client.conf
group1/M00/00/01/wKikgV0YtQ2AfmozAAAixZ60QfI792.log

信息

[root@CentOSX ~]# fdfs_file_info /etc/fdfs/client.conf
group1/M00/00/01/wKikgV0YtQ2AfmozAAAixZ60QfI792.log
source storage id: 0
source ip address: 192.168.164.129
file create timestamp: 2019-06-30 21:11:41
file size: 8901
file crc32: 2662613490 (0x9EB441F2)

删除文件

[root@CentOSX ~]# fdfs_delete_file /etc/fdfs/client.conf
group1/M00/00/01/wKikgV0YtQ2AfmozAAAixZ60QfI792.log

Nginx集成FastDFS

下载fastdfs-nginx-module

[root@CentOSX ~]# tar -zxf fastdfs-nginx-module.tar.gz
[root@CentOSX ~]# yum install -y pcre-devel
[root@CentOSX ~]# yum install -y openssl-devel
[root@CentOSX ~]# wget http://nginx.org/download/nginx-1.11.1.tar.gz
[root@CentOSX ~]# tar -zxf nginx-1.11.1.tar.gz
[root@CentOSX nginx-1.11.1]# ./configure --prefix=/usr/local/nginx-1.11.1/ --add-
module=/root/fastdfs-nginx-module/src
[root@CentOSX nginx-1.11.1]# make
[root@CentOSX nginx-1.11.1]# make install

拷贝配置

[root@CentOSX ~]# cp /root/fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs/
[root@CentOSX ~]# cd /root/fastdfs-5.11/conf/
[root@CentOSX conf]# cp http.conf mime.types anti-steal.jpg /etc/fdfs/

配置nginx.conf

[root@CentOSA ~]# vi /usr/local/nginx-1.11.1/conf/nginx.conf
server {
	listen 80;
	server_name localhost;
	#charset koi8-r;
	#access_log logs/host.access.log main;
	location ~ /group[0-9]+/M00 {
		root /data/fdfs/storage/store01;
		ngx_fastdfs_module;
	}
	location ~ /group[0-9]+/M01 {
		root /data/fdfs/storage/store02;
		ngx_fastdfs_module;
	}
	location / {
		root html;
		index index.html index.htm;
	}
	#error_page 404 /404.html;

修改mod_fastdfs.conf

[root@CentOSX ~]# vi /etc/fdfs/mod_fastdfs.conf
tracker_server=CentOSA:22122
tracker_server=CentOSB:22122
tracker_server=CentOSC:22122
group_name=group`[1,2,3]`
url_have_group_name = true
store_path_count=2
store_path0=/data/fdfs/storage/store01
store_path1=/data/fdfs/storage/store02

启动nginx

[root@CentOSX ~]# cd /usr/local/nginx-1.11.1/
[root@CentOSX nginx-1.11.1]# ./sbin/nginx -t
ngx_http_fastdfs_set pid=116305
ngx_http_fastdfs_set pid=116305
nginx: the configuration file /usr/local/nginx-1.11.1//conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx-1.11.1//conf/nginx.conf test is successful
[root@CentOSX nginx-1.11.1]# ./sbin/nginx

测试下载

[root@CentOSA ~]# fdfs_upload_file /etc/fdfs/client.conf install.log
group2/M01/00/00/wKikgl0p_4-AIqGJAAAixZ60QfI693.log

随便访问一个nginx服务查看效果

http://CentOS[A|B|C]/group2/M01/00/00/wKikgl0p_4-AIqGJAAAixZ60QfI693.log?
filename=install.log

FastDHT文件去重

1.安装BerkeleyDB 下载db-4.7.25.tar.gz

[root@CentOSX ~]# tar -zxf db-4.7.25.tar.gz
[root@CentOSX ~]# cd db-4.7.25
[root@CentOSX db-4.7.25]# cd build_unix/
[root@CentOSX build_unix]# ./../dist/configure
[root@CentOSX build_unix]# make
[root@CentOSX build_unix]# make install

2.安装FastDHT

[root@CentOSX ~]# tar zxf FastDHT_v2.01.tar.gz
[root@CentOSX ~]# cd FastDHT
[root@CentOSX FastDHT]# ./make.sh
[root@CentOSX FastDHT]# ./make.sh install

安装结束后会在/etc目录下产生fdht文件夹

[root@CentOSX FastDHT]# tree /etc/fdht/
/etc/fdht/
!"" fdht_client.conf
!"" fdhtd.conf
#"" fdht_servers.conf

3.修改fdhtd.conf

[root@CentOSX ~]# mkdir /data/fastdht
[root@CentOSX ~]# vi /etc/fdht/fdhtd.conf
base_path=/data/fastdht

4.修改fdht_servers.conf

[root@CentOSX ~]# vi /etc/fdht/fdht_servers.conf
group_count = 3
group0 = CentOSA:11411
group1 = CentOSB:11411
group2 = CentOSC:11411

5.修改fdht_client.conf配置⽂文件

[root@CentOSX ~]# vi /etc/fdht/fdht_client.conf
base_path=/tmp/

6.启动FDHT服务

[root@CentOSX ~]# fdhtd /etc/fdht/fdhtd.conf start|stop|restart
[root@CentOSX ~]# ps -axu| grep fd
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
root 29127 0.0 0.8 195200 17504 ? Sl 00:38 0:00 fdhtd
/etc/fdht/fdhtd.conf start
root 29193 0.0 0.0 103320 884 pts/0 S+ 00:39 0:00 grep fd
root 128672 0.2 0.1 275856 2204 ? Sl 00:14 0:03
/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf
root 128710 1.8 3.2 85584 67188 ? Sl 00:14 0:27
/usr/bin/fdfs_storaged /etc/fdfs/storage.conf

操作FastDHT服务

设置值

[root@CentOSX ~]# fdht_set /etc/fdht/fdht_client.conf jiangsiyu:user001
name='jiangsiyu',age=18;
This is FastDHT client test program v2.01
Copyright (C) 2008, Happy Fish / YuQing
FastDHT may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDHT source kit.
Please visit the FastDHT Home Page http://www.csource.org/
for more detail.
success set key count: 2, fail count: 0

读取值

[root@CentOSX ~]# fdht_get /etc/fdht/fdht_client.conf jiangsiyu:user001 name,age
This is FastDHT client test program v2.01
Copyright (C) 2008, Happy Fish / YuQing
FastDHT may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDHT source kit.
Please visit the FastDHT Home Page http://www.csource.org/
for more detail.
name=jiangsiyu
age=18
success get key count: 2, fail count: 0

删除值

[root@CentOSX ~]# fdht_delete /etc/fdht/fdht_client.conf jiangsiyu:user001 name;
This is FastDHT client test program v2.01
Copyright (C) 2008, Happy Fish / YuQing
FastDHT may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDHT source kit.
Please visit the FastDHT Home Page http://www.csource.org/
for more detail.
success delete keys: name
success delete key count: 1, fail count: 0

1.修改etc/fdfs/storage.conf配置文件

[root@CentOSX ~]# vi /etc/fdfs/storage.conf
check_file_duplicate=1
keep_alive=1
#include /etc/fdht/fdht_servers.conf

2.分别启动fdhtd服务、fastfs

[root@CentOSX usr]# /usr/local/bin/fdhtd /etc/fdht/fdhtd.conf restart
[root@CentOSX usr]# /etc/init.d/fdfs_trackerd restart
[root@CentOSX usr]# /etc/init.d/fdfs_storaged restart

上产文件测试

[root@CentOSA ~]# fdfs_upload_file /etc/fdfs/client.conf install.log
group2/M00/00/00/wKikgl0qC4KAErBTAAAixXWAIyY133.log
[root@CentOSA ~]# fdfs_upload_file /etc/fdfs/client.conf install.log
group2/M00/00/00/wKikgl0qDAqAa0XwAAAixWB5m1c851.log
[root@CentOSB ~]# ls -l /data/fdfs/storage/store01/data/00/00/
total 20
lrwxrwxrwx. 1 root root 72 Jul 14 00:49 wKikgl0qC4KAErBTAAAixXWAIyY133.log ->
/data/fdfs/storage/store01/data/00/00/wKikgl0qC4KARrYBAAAixZ60QfI755.log
-rw-r--r--. 1 root root 8901 Jul 14 00:49 wKikgl0qC4KARrYBAAAixZ60QfI755.log
lrwxrwxrwx. 1 root root 72 Jul 14 00:51 wKikgl0qDAqAa0XwAAAixWB5m1c851.log ->
/data/fdfs/storage/store01/data/00/00/wKikgl0qC4KARrYBAAAixZ60QfI755.log

可以看出系统产生了wKikgl0qC4KAErBTAAAixXWAIyY133.log的两个链接

引入依赖

<dependency>
<groupId>com.github.tobato</groupId>
<artifactId>fastdfs-client</artifactId>
<version>1.26.6</version>
</dependency>

配置application.properties

fdfs.tracker-list=CentOSA:22122,CentOSB:22122,CentOSC:22122
# 配置默认缩略略图
fdfs.thumb-image.height=80
fdfs.thumb-image.width=80

测试文件上传

@Autowired
private FastFileStorageClient fastFileStorageClient;

文件上传

FileInputStream inputStream = new FileInputStream("G:/素材资料料/61850.png");
FastImageFile fastImageFile=new
FastImageFile(inputStream,inputStream.available(),"png",new HashSet<MetaData>());
StorePath storePath = fastFileStorageClient.uploadImage(fastImageFile);
System.out.println(storePath.getFullPath());

图片上传（缩略略图）

FileInputStream inputStream = new FileInputStream("G:/素材资料料/61850.png");
FastImageFile fastImageFile=new
FastImageFile(inputStream,inputStream.available(),"png",new HashSet<MetaData>(),new
ThumbImage(150,150));
StorePath storePath = fastFileStorageClient.uploadImage(fastImageFile);
System.out.println(storePath.getFullPath());

删除文件

fastFileStorageClient.deleteFile("group3","M00/00/00/wKjvgl0prl6AX6ygAL26hh1kYdE312_80
x80.png");