问题发现
突然发现机器重启后很多服务都出现了问题,不仅如此,还无法对文件进行操作:
>mkdir test
mkdir: cannot create directory ‘test’: Read-only file system
排查问题
查看mount发现根目录被挂载为了read only:
>mount
.......
/dev/mapper/vg0-root on / type ext4 (ro,relatime,stripe=256,data=ordered)
.......
dmesg日志记录
筛选dmesg的结果可以看到,EXT4-FS出现了问题(dmesg | grep "EXT4-fs error"):
[ 124.369135] systemd-readahead[724]: Failed to open pack file: Read-only file system
[ 303.068201] EXT4-fs (dm-1): error count since last fsck: 9
[ 303.068274] EXT4-fs (dm-1): initial error at time 1591123830: ext4_find_entry:1318: inode 25430455
[ 303.068412] EXT4-fs (dm-1): last error at time 1591123878: ext4_wait_block_bitmap:516: inode 25430531
[ 549.490580] EXT4-fs (dm-1): Couldn't remount RDWR because of unprocessed orphan inode list. Please umount/remount instead
[ 1050.216980] EXT4-fs (dm-1): Couldn't remount RDWR because of unprocessed orphan inode list. Please umount/remount instead
[ 1138.023493] EXT4-fs (dm-1): Couldn't remount RDWR because of unprocessed orphan inode list. Please umount/remount instead
[ 1179.341551] EXT4-fs (dm-1): Couldn't remount RDWR because of unprocessed orphan inode list. Please umount/remount instead
看来是磁盘出现了问题,使用smartctl查看是否有报错,但是没问题:
>smartctl --all /dev/sda
.......
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
.......
>smartctl --all /dev/sdb
.......
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
.......
那么很可能是最近服务器IO高负载导致崩溃,然后文件系统损坏了,需要进行修复。
开始修复
先尝试使用fsck来修复,报错版本太低:
>fsck.ext4 /dev/mapper/vg0-root
e2fsck 1.42.9 (28-Dec-2013)
/dev/vg0/root has unsupported feature(s): metadata_csum
e2fsck: Get a newer version of e2fsck!
但是尝试升级却会因为read only报错,因此直接接入rescue模式
Rescue模式下挂载lvm
>ls /dev/mapper/*
#查看lvm列表
mount /dev/mapper/vg0-root /mnt
#挂载
修复磁盘
root@rescue / # fsck /dev/mapper/vg0-root
fsck from util-linux 2.33.1
e2fsck 1.44.5 (15-Dec-2018)
/dev/mapper/vg0-root contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inode 4980739 extent tree (at level 2) could be narrower. Optimize? yes
Inode 4980740 extent tree (at level 1) could be narrower. Optimize? yes
Inode 4980741 extent tree (at level 2) could be narrower. Optimize? yes
Inode 4980743 extent tree (at level 2) could be narrower. Optimize? yes
Inode 4980744 extent tree (at level 2) could be narrower. Optimize? yes
Inodes that were part of a corrupted orphan linked list found. Fix? yes
Inode 25756179 was part of the orphaned inode list. FIXED.
Inode 25756180 was part of the orphaned inode list. FIXED.
Inode 25756181 was part of the orphaned inode list. FIXED.
Inode 25756182 was part of the orphaned inode list. FIXED.
Inode 25756183 was part of the orphaned inode list. FIXED.
Inode 25756184 was part of the orphaned inode list. FIXED.
Inode 25756211 was part of the orphaned inode list. FIXED.
Inode 25756233 was part of the orphaned inode list. FIXED.
Inode 25756648 was part of the orphaned inode list. FIXED.
Inode 25758032 was part of the orphaned inode list. FIXED.
Inode 25758033 was part of the orphaned inode list. FIXED.
Inode 25758034 was part of the orphaned inode list. FIXED.
Inode 25758035 was part of the orphaned inode list. FIXED.
Inode 26607626 was part of the orphaned inode list. FIXED.
Inode 26607637 was part of the orphaned inode list. FIXED.
Inode 26607638 was part of the orphaned inode list. FIXED.
Inode 26607639 was part of the orphaned inode list. FIXED.
Inode 26607640 was part of the orphaned inode list. FIXED.
Deleted inode 177733645 has zero dtime. Fix? yes
Inode 177733646 was part of the orphaned inode list. FIXED.
Inode 177733647 was part of the orphaned inode list. FIXED.
Inode 177733651 was part of the orphaned inode list. FIXED.
Inode 177733656 was part of the orphaned inode list. FIXED.
Deleted inode 177733664 has zero dtime. Fix? yes
Inode 177733665 was part of the orphaned inode list. FIXED.
Inode 177733666 was part of the orphaned inode list. FIXED.
Inode 177733667 was part of the orphaned inode list. FIXED.
Inode 177733668 was part of the orphaned inode list. FIXED.
Pass 1E: Optimizing extent trees
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Inode bitmap differences: -(25756179--25756184) -25756211 -25756233 -25756648 -(25758032--25758035) -26607626 -(26607637--26607640) -(177733645--177733647) -177733651 -177733656 -(177733664--177733668)
Fix? yes
Free inodes count wrong for group #6288 (1541, counted=1554).
Fix? yes
Free inodes count wrong for group #6496 (4070, counted=4075).
Fix? yes
Free inodes count wrong for group #43392 (4059, counted=4069).
Fix? yes
Free inodes count wrong (182861554, counted=182861582).
Fix? yes
/dev/mapper/vg0-root: ***** FILE SYSTEM WAS MODIFIED *****
/dev/mapper/vg0-root: 110834/182972416 files (0.3% non-contiguous), 860247893/1463754752 blocks
确认修复
root@rescue / # fsck /dev/mapper/vg0-root
fsck from util-linux 2.33.1
e2fsck 1.44.5 (15-Dec-2018)
/dev/mapper/vg0-root: clean, 110834/182972416 files, 860247893/1463754752 blocks
This article is under CC BY-NC-SA 4.0 license.
Please quote the original link:https://www.liujason.com/article/799.html