Hadoop之hadoop fs命令

红砚文

2023-12-01

一、hadoop fs命令简介

调用文件系统(FS)Shell命令应使用 bin/hadoop fs <args>的形式。所有的的FS shell命令使用URI路径作为参数。URI格式是scheme://authority/path。对HDFS文件系统，scheme是hdfs，对本地文件系统，scheme是file。其中scheme和authority参数都是可选的，如果未加指定，就会使用配置中指定的默认scheme。一个HDFS文件或目录比如/parent/child可以表示成hdfs://namenode:namenodeport/parent/child，或者更简单的/parent/child（假设你配置文件中的默认值是namenode:namenodeport）。大多数FS Shell命令的行为和对应的Unix Shell命令类似，不同之处在于操作的是hdfs文件系统内的文件。出错信息会输出到stderr，其他信息输出到stdout。

二、使用示例

1、文件路径的增删查

1）、ls查看hdfs根目录下文件列表

[wuhs@s142 ~]$ hadoop fs -ls /
Found 4 items
drwxr-xr-x - wuhs supergroup 0 2021-12-20 22:24 /system
drwx------ - wuhs supergroup 0 2021-12-09 03:33 /tmp
drwxr-xr-x - wuhs supergroup 0 2021-12-09 03:33 /user
drwxr-xr-x - wuhs supergroup 0 2021-12-20 22:09 /wordcount
#如上system、tmp、user为默认创建的，wordcount是我自己手动创建的目录

2）、mkdir创建一个文件夹

[wuhs@s142 hadoop]$ hadoop fs -mkdir /wordcount/a
[wuhs@s142 hadoop]$ hadoop fs -ls -R /wordcount
drwxr-xr-x - wuhs supergroup 0 2021-12-22 16:30 /wordcount/a
drwxr-xr-x - wuhs supergroup 0 2021-12-16 17:44 /wordcount/input
…

3）、rm删除一个文件夹

[wuhs@s142 hadoop]$ hadoop fs -rm -r /wordcount/a
2021-12-22 16:34:17,048 INFO fs.TrashPolicyDefault: Moved: ‘hdfs://s142:9000/wordcount/a’ to trash at: hdfs://s142:9000/user/wuhs/.Trash/Current/wordcount/a1640162056100

4）、返回指定路径的统计信息

[wuhs@s142 hadoop]$ hadoop fs -stat /wordcount/input
2021-12-16 09:44:44

2、文件的属主属组修改

1）、chgrp改变文件目录的属组

[wuhs@s142 hadoop]$ ./bin/hadoop fs -chgrp -R bdsc /wordcount
[wuhs@s142 hadoop]$ hadoop fs -ls /
Found 4 items
drwxr-xr-x - wuhs supergroup 0 2021-12-21 11:24 /system
drwx------ - wuhs supergroup 0 2021-12-09 16:33 /tmp
drwxr-xr-x - wuhs supergroup 0 2021-12-09 16:33 /user
drwxr-xr-x - wuhs bdsc 0 2021-12-21 11:09 /wordcount

2、chown改变文件的所有者

[wuhs@s142 hadoop]$ ./bin/hadoop fs -chgrp -R supergroup /wordcount
[wuhs@s142 hadoop]$ hadoop fs -chown -R bdsc /wordcount
[wuhs@s142 hadoop]$ hadoop fs -ls /
Found 4 items
drwxr-xr-x - wuhs supergroup 0 2021-12-21 11:24 /system
drwx------ - wuhs supergroup 0 2021-12-09 16:33 /tmp
drwxr-xr-x - wuhs supergroup 0 2021-12-09 16:33 /user
drwxr-xr-x - bdsc supergroup 0 2021-12-21 11:09 /wordcount

3）、chmod修改文件的权限

[wuhs@s142 hadoop]$ hadoop fs -chmod 744 /wordcount/test.har
[wuhs@s142 hadoop]$ hadoop fs -ls /wordcount
Found 3 items
drwxr-xr-x - wuhs supergroup 0 2021-12-16 17:44 /wordcount/input
drwxr-xr-x - wuhs supergroup 0 2021-12-16 17:45 /wordcount/output
drwxr–r-- - wuhs supergroup 0 2021-12-21 11:10 /wordcount/test.har

3、文件空间大小查看

1）、du查看指定文件的大小

[wuhs@s142 hadoop]$ hadoop fs -du /wordcount/input/1.txt
96 192 /wordcount/input/1.txt

2）、du -h按照适合阅读的形式人性化显示文件大小

[wuhs@s142 hadoop]$ hadoop fs -du -h /wordcount
96 192 /wordcount/input
117 234 /wordcount/output
966 2.8 K /wordcount/test.har

4、文件上传下载、增删改查操作

1）、cat查看hdfs内的某个文件

[wuhs@s142 ~]$ hadoop fs -cat /wordcount/input/1.txt
123
321
This a test
hello hadoop
hi hadoop
Hadoop
Hadoop
…

2）、touchz创建一个空文件

[wuhs@s142 hadoop]$ hadoop fs -touchz /tmp/a.txt
[wuhs@s142 hadoop]$ hadoop fs -ls /tmp
Found 2 items
-rw-r–r-- 2 wuhs supergroup 0 2021-12-22 16:49 /tmp/a.txt
drwx------ - wuhs supergroup 0 2021-12-09 16:33 /tmp/hadoop-yarn

3）、rm删除一个文件

[wuhs@s142 hadoop]$ hadoop fs -rm /tmp/a.txt
2021-12-22 16:56:44,256 INFO fs.TrashPolicyDefault: Moved: ‘hdfs://s142:9000/tmp/a.txt’ to trash at: hdfs://s142:9000/user/wuhs/.Trash/Current/tmp/a.txt

4）、mv移动一个文件

#经常用于从回收站恢复文件
[wuhs@s142 hadoop]$ hadoop fs -mv hdfs://s142:9000/user/wuhs/.Trash/Current/tmp/a.txt hdfs://s142:9000/tmp/a.txt

5）、put从本地上传一个文件到hdfs

[wuhs@s142 hadoop]$ touch /tmp/2.txt
[wuhs@s142 hadoop]$ echo “this is a test” > /tmp/2.txt
[wuhs@s142 hadoop]$ hadoop fs -put /tmp/2.txt /wordcount/input/
[wuhs@s142 hadoop]$ hadoop fs -cat /wordcount/input/2.txt
this is a test

6）、get从hdfs复制文件到本地

[wuhs@s142 hadoop]$ hadoop fs -get /wordcount/input/1.txt /tmp/
[wuhs@s142 hadoop]$ cat /tmp/1.txt
123
321
This a test
hello hadoop
hi hadoop
Hadoop
Hadoop
…

7）、getmerge合并hdfs多个文件并传至本地

[wuhs@s142 hadoop]$ hadoop fs -getmerge /wordcount/input/1.txt /wordcount/input/2.txt 3.txt
[wuhs@s142 hadoop]$ cat 3.txt
123
321
This a test
hello hadoop
hi hadoop
Hadoop
Hadoop
hi
123
321
123
test
Test
123
test
Test
this is a test

8）、tail查看文件的最后1000字节

[wuhs@s142 hadoop]$ hadoop fs -tail /wordcount/input/1.txt
123
321
This a test
hello hadoop
hi hadoop
Hadoop
Hadoop
hi
123
321
123
test
Test
123
test
Test

9）、copyFromLocal 从本地拷贝一个文件到hdfs

[wuhs@s142 hadoop]$ hadoop fs -copyFromLocal 3.txt /wordcount/input/

10）、copyToLocal 从hdsf拷贝文件到本地

[wuhs@s142 hadoop]$ hadoop fs -copyToLocal /wordcount/input/3.txt /tmp/

11）、text查看文件

#将源文件输出为文本格式。允许的格式是zip和TextRecordInputStream，如果是压缩文件会先解压再查看
[wuhs@s142 hadoop]$ hadoop fs -text /wordcount/input/3.txt
123
321
This a test
hello hadoop
hi hadoop
Hadoop
Hadoop
hi
123
321
123
test
Test
123
test
Test
this is a test

5、expunge清空回收站

[wuhs@s142 hadoop]$ hadoop fs -ls -r hdfs://s142:9000/user/wuhs/.Trash
Found 2 items
drwx------ - wuhs supergroup 0 2021-12-22 17:34 hdfs://s142:9000/user/wuhs/.Trash/Current
drwx------ - wuhs supergroup 0 2021-12-22 16:56 hdfs://s142:9000/user/wuhs/.Trash/211222173413
[wuhs@s142 hadoop]$ hadoop fs -expunge
2021-12-22 17:35:11,436 INFO fs.TrashPolicyDefault: TrashPolicyDefault#deleteCheckpoint for trashRoot: hdfs://s142:9000/user/wuhs/.Trash
2021-12-22 17:35:11,436 INFO fs.TrashPolicyDefault: TrashPolicyDefault#deleteCheckpoint for trashRoot: hdfs://s142:9000/user/wuhs/.Trash
2021-12-22 17:35:11,454 INFO fs.TrashPolicyDefault: TrashPolicyDefault#createCheckpoint for trashRoot: hdfs://s142:9000/user/wuhs/.Trash
2021-12-22 17:35:11,479 INFO fs.TrashPolicyDefault: Created trash checkpoint: /user/wuhs/.Trash/211222173511