集群主机的存储空间比HDFS所能识别/访问的要多？如何增加HDFS存储使用？

Question

HDFS（HDP v3.1.0)的存储空间不足（这也导致spark作业在接受模式下挂起时出现问题）。我假设有一些配置可以让HDFS使用节点主机上已经存在的更多存储空间，但通过快速搜索并不清楚具体是什么。有更多经验的人能帮上忙吗？

看看HDFS的磁盘使用情况，我看到...

[hdfs@HW001 root]$ hdfs dfs -du -h /
1.3 G    4.0 G    /app-logs
3.7 M    2.3 G    /apps
0        0        /ats
899.1 M  2.6 G    /atsv2
0        0        /datalake
39.9 G   119.6 G  /etl
1.7 G    5.2 G    /hdp
0        0        /mapred
92.8 M   278.5 M  /mr-history
19.5 G   60.4 G   /ranger
4.4 K    13.1 K   /services
11.3 G   34.0 G   /spark2-history
1.8 M    5.4 M    /tmp
4.3 G    42.2 G   /user
0        0        /warehouse

对于总共消耗的约269GB（也许设置一个较短的时间间隔来启动历史清理也会有所帮助？）。看看HDFS上的空闲空间，我看到了...

[hdfs@HW001 root]$ hdfs dfs -df -h /
Filesystem                        Size     Used  Available  Use%
hdfs://hw001.ucera.local:8020  353.3 G  244.1 G     31.5 G   69%

[root@HW001 ~]# clush -ab -x airflowet df -h /hadoop/hdfs/data
HW001: df: ‘/hadoop/hdfs/data’: No such file or directory
airflowetl: df: ‘/hadoop/hdfs/data’: No such file or directory
---------------
HW002
---------------
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/centos_mapr001-root  101G   93G  8.0G  93% /
---------------
HW003
---------------
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/centos_mapr001-root  101G   94G  7.6G  93% /
---------------
HW004
---------------
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/centos_mapr001-root  101G   92G  9.2G  91% /
---------------
HW005
---------------
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/centos_mapr001-root  101G   92G  9.8G  91% /

[hdfs@HW001 root]$ hdfs fsck / -files -blocks
.
.
.
Status: HEALTHY
 Number of data-nodes:  4
 Number of racks:               1
 Total dirs:                    8734
 Total symlinks:                0

Replicated Blocks:
 Total size:    84897192381 B (Total open files size: 10582 B)
 Total files:   43820 (Files currently being written: 10)
 Total blocks (validated):      42990 (avg. block size 1974812 B) (Total open file blocks (not validated): 8)
 Minimally replicated blocks:   42990 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       1937 (4.505699 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.045057
 Missing blocks:                0
 Corrupt blocks:                0
 Missing replicas:              11597 (8.138018 %)

Erasure Coded Block Groups:
 Total size:    0 B
 Total files:   0
 Total block groups (validated):        0
 Minimally erasure-coded block groups:  0
 Over-erasure-coded block groups:       0
 Under-erasure-coded block groups:      0
 Unsatisfactory placement block groups: 0
 Average block group size:      0.0
 Missing block groups:          0
 Corrupt block groups:          0
 Missing internal blocks:       0
FSCK ended at Tue May 26 12:10:43 HST 2020 in 1717 milliseconds


The filesystem under path '/' is HEALTHY

Answer 1

例如，您没有提到/tmp中是否存在可以清除的蹩脚数据。

每个数据阳极有88.33GB的存储空间？

如果是这样，您就不能仅仅创建新的HDD来连接到集群并突然创建空间。