[翻译自mos文章] 当linux kernel和kmod-oracleasm升级之后,CRS不能发现voting disk.

来源于:  CRS Not Able to Discover Voting Disk after Kernel and Module kmod-oracleasm Upgrade (文档 ID 1994371.1)

适用于:
Linux 6.0以及更新版本
Linux x86-64bit

症状:

使用如下的参数来配置的asmlib:

ORACLEASM_USE_LOGICAL_BLOCK_SIZE=true 

当linux kernel和kmod-oracleasm从 2.0.6.rh1-2.el6.x86_64 升级到 2.0.8-4.el6_6.x86_64之后,CRS不能发现voting disk

2015-03-24 03:59:47.421: [ CSSD][2286941952]clssnmvDiskVerify: discovered a potential voting file
2015-03-24 03:59:47.421: [ SKGFD][2286941952]Handle 0x7feb7813ddf0 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:OCRVOTE11:

2015-03-24 03:59:47.422: [ CSSD][2286941952]clssnmvDiskVerify: TOC format mismatch expected(0x634c7373 0x546f636b), found(0x0000 0x0000)
2015-03-24 03:59:47.422: [ SKGFD][2286941952]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x7feb7813ddf0 for disk :ORCL:OCRVOTE11:

变化:
Linux kernel升级之后,kmod-oracleasm从 2.0.6.rh1-2.el6.x86_64 升级到 2.0.8-4.el6_6.x86_64.

原因:
对于disk来说,有两种block size:

•logical block size
•physical block size

对于传统的disk,在linux上,这两种block size均为512Bytes

现在越来越多的disk 的 physical block size等于4096,但是logical block size 依然是512

比如:

# grep "" /sys/block/sdag/queue/*block_size
/sys/block/sdag/queue/logical_block_size:512
/sys/block/sdag/queue/physical_block_size:4096

在老的配置中, kmod-oracleasm-2.0.6.rh1-2.el6.x86_64 不支持在 ORACLEASM_USE_LOGICAL_BLOCK_SIZE=true 这个选项(该选项在/etc/sysconfig/oracleasm文件中).
因此,即使 ORACLEASM_USE_LOGICAL_BLOCK_SIZE=true ,oracleasm的sector size是4096,而不是512

$ modinfo ./kmod-oracleasm-2.0.6.rh1-2.el6.x86_64/lib/modules/2.6.32-358.el6.x86_64/extra/oracleasm/oracleasm.ko
filename:       ./kmod-oracleasm-2.0.6.rh1-2.el6.x86_64/lib/modules/2.6.32-358.el6.x86_64/extra/oracleasm/oracleasm.ko
description:    Kernel driver backing the Generic Linux ASM Library.
author:         Joel Becker <[e-mail]>
version:        2.0.6
license:        GPL
srcversion:     B6C06FFEF593E8060B7BFEE
depends:        
vermagic:       2.6.32-358.el6.x86_64 SMP mod_unload modversions

但是,新的驱动kmod-oracleasm-2.0.8-4.el6_6.x86_64确实支持该参数

$ modinfo ./kmod-oracleasm-2.0.8-4.el6_6.x86_64/lib/modules/2.6.32-504.1.3.el6.x86_64/extra/oracleasm/oracleasm.ko
filename:       ./kmod-oracleasm-2.0.8-4.el6_6.x86_64/lib/modules/2.6.32-504.1.3.el6.x86_64/extra/oracleasm/oracleasm.ko
description:    Kernel driver backing the Generic Linux ASM Library.
author:         Joel Becker <[e-mail]>
version:        2.0.8
license:        GPL
srcversion:     223CD1891C9BE8AFC0A0E76
depends:        
vermagic:       2.6.32-504.1.3.el6.x86_64 SMP mod_unload modversions
parm:           use_logical_block_size:Prefer logical block size over physical (Y=logical, N=physical [default]) (bool)        <==== now it support

当升级完kernel之后,并且ORACLEASM_USE_LOGICAL_BLOCK_SIZE=true,这样的话,使用的sector size 是512.
因此,会由于sector size的不匹配导致 CRS 不能发现voting disk.

There are similar issues which are caused by opposite reason, please refer to:
Linux: After NetApp ONTAP Upgrade: CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds (Doc ID 1641344.1)
Alert: After SAN Firmware Upgrade, ASM Diskgroups ( Using ASMLIB) Cannot Be Mounted Due To ORA-15085: ASM disk "" has inconsistent sector size. (Doc ID 1500460.1)

解决方案:
第一步:重新配置oracleasm以便使用physical block size
# oracleasm configure -p

It's not recommended to edit the file /etc/sysconfig/oracleasm directly, as you might hit another issue described in:
Updates to ASMLib Configuration File "/etc/sysconfig/oracleasm" Not Effective (Doc ID 868352.1)

第二步:reboot 操作系统.

参考文档:

NOTE:1500460.1 - Alert: After SAN Firmware Upgrade, ASM Diskgroups ( Using ASMLIB) Cannot Be Mounted Due To ORA-15085: ASM disk "" has inconsistent sector size.
NOTE:1641344.1 - Linux: After NetApp ONTAP Upgrade: CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds
NOTE:868352.1 - Updates to ASMLib Configuration File "/etc/sysconfig/oracleasm" Not Effective
NOTE:1530578.1 - New Block Size Feature for OracleASM and oracleasm-support

译者总结:
其实这个文章说的比较绕,我个人总结如下:

前提:

1.这个asm diskgroup用了asmlib来实现的
2.升级前:/etc/sysconfig/oracleasm文件中的ORACLEASM_USE_LOGICAL_BLOCK_SIZE等于true (意味着asm使用底层disk的logical block size)
3.asm disk 的physical block size等于4096,logical block size等于512

升级前:
kmod-oracleasm-2.0.6.rh1-2.el6.x86_64,但是该版本的kmod-oracleasm不支持ORACLEASM_USE_LOGICAL_BLOCK_SIZE=true,
也就是说Oracleasm其实还是用的4096的physical sector size,512的logical block size.其实这是4k sector size的emulition mode.


升级后:
kmod-oracleasm-2.0.8-4.el6_6.x86_64,该版本的kmod-oracleasm支持ORACLEASM_USE_LOGICAL_BLOCK_SIZE=true,
也就是说Oracleasm其实还是用的4096的physical sector size,4096的logical sector size.


问题被解决之后的状态:
/etc/sysconfig/oracleasm文件中的ORACLEASM_USE_LOGICAL_BLOCK_SIZE等于false(意味着asm使用底层disk的physical block size)