ASM的普通盘转AFD

普通盘转AFD

 

ASMFD : How to Migrate ASM Diskgroups from ASMLIB to ASMFD (ASM Filter Driver) on Oracle Grid Infrastructure (RAC) (文档 ID 2172754.1)

1、确定当前的ASM扫描磁盘路径
[grid@ora19c01 ~]$ asmcmd dsget
parameter:/dev/asm-*
profile:/dev/asm-*

2、使用grid用户增加AFD磁盘扫描路径
[grid@ora19c01 ~]$ asmcmd dsset '/dev/asm-*','AFD:*'
[grid@ora19c01 ~]$ asmcmd dsget
parameter:/dev/asm-*, AFD:*
profile:/dev/asm-*,AFD:*
[grid@ora19c02 ~]$ asmcmd dsget
parameter:/dev/asm-*, AFD:*
profile:/dev/asm-*,AFD:*

在一个节点执行命令后，所有节点的信息都更新了。

3、关闭节点crs ，按官档要求是所有节点都要停下来的，经过测试，可以一个一个节点的配置生效。
[root@ora19c01 ~]#  /u01/app/19.0.0/grid/bin/crsctl stop crs 
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ora19c01'
CRS-2673: Attempting to stop 'ora.crsd' on 'ora19c01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'ora19c01'
…………
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ora19c01' has completed
CRS-4133: Oracle High Availability Services has been stopped.

Note : Repeat the above steps on all the nodes

4、查看当前环境已加载的oracle模块
[root@ora19c01 ~]# lsmod |grep oracle
oracleacfs           5581810  0 
oracleadvm           1231385  0 
oracleoks             721311  2 oracleacfs,oracleadvm   

可以看到，当前是没有加载AFD模块的。（oracleafd）

5、如果使用的是ASMLIB的模块，建议先删除，避免冲突。由于没有环境，过程摘自官档。
Note: It is recommended to remove the ASMLib related RPMs to avoid any conflict.
[root@cehaovmsp1129 trace]# rpm -qa | grep asm
oracleasm-support-2.1.8-3.el7.x86_64
[root@cehaovmsp1129 trace]# rpm -e oracleasm-support-2.1.8-3.el7.x86_64
warning: /etc/sysconfig/oracleasm saved as /etc/sysconfig/oracleasm.rpmsave

6、使用root用户执行配置AFD模块, 要求先export一下ORACLE_BASE，这实际为了asmcmd命令写日志用，不一定是grid_home。
[root@ora19c01 ~]# export ORACLE_BASE=/u01/app/grid
[root@ora19c01 ~]# /u01/app/19.0.0/grid/bin/asmcmd afd_configure
AFD-627: AFD distribution files found.
AFD-634: Removing previous AFD installation.
AFD-635: Previous AFD components successfully removed.
AFD-9294: updating file /etc/sysconfig/oracledrivers.conf
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies - this may take some time.
AFD-9154: Loading 'oracleafd.ko' driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device '/dev/oracleafd/admin'.
AFD-638: AFD installation correctness verified.
Modifying resource dependencies - this may take some time.

这个过程做了啥呢，可以在$ORACLE_BASE/diag/asmcmd/user_root/hostname/alert/alert.log看到。

[grid@ora19c01 alert]$ cat  alert.log |grep -v "reverted LANG/NLS_LANG" 
27-Mar-21 23:01 ASMCMD (PID = 18399) Given command - afd_configure
27-Mar-21 23:01 NOTE: asmcmdafd_is_exadata /etc/oracle/cell/network-config/cellip.ora not found
--检查是否是exadata平台，exadata平台使用的是用的IB的，不能启用AFD。
27-Mar-21 23:02 NOTE: Verifying AFD driver state : supported
--检查是否支持启动AFD模块，主要是做内核版本的匹配，如果当前内核不匹配AFD支持的内核版本，则不支持，相当于执行了，afddriverstate supported 命令
27-Mar-21 23:02 NOTE: Status of clusterware stack : Not online
--检查集群是否启动，需要关闭集群才能配置。
27-Mar-21 23:02 NOTE: command execution (/u01/app/19.0.0/grid/bin/afddriverstate loaded) returned : 1
27-Mar-21 23:02 Error: AFD-9206: AFD device driver loaded status: 'false'
27-Mar-21 23:02 NOTE: Verifying AFD driver state : Not loaded
--检查AFD模块是否已经加载了执行命令： /u01/app/19.0.0/grid/bin/afddriverstate loaded 
27-Mar-21 23:02 NOTE: Installing AFD... 
27-Mar-21 23:02 NOTE: Successfully installed AFD
--如果没有加载AFD的驱动模块，则加载 NOTE: Installing AFD...  ：相当于执行了afdroot install
27-Mar-21 23:02 Updating oracleafd.conf
27-Mar-21 23:02 Retrieved asm disk string from gpnp : '/dev/asm-*,AFD:*'
27-Mar-21 23:02 Created /etc/oracleafd.conf
--生成/etc/oracleafd.conf 文件，从asmcmd dsget获取除AFD以外的路径写入到文件/etc/oracleafd.conf，做为AFD的磁盘扫描路径（afd_diskstring='/dev/asm-*' ）,区别于我们ASM 的asm_diskstring
27-Mar-21 23:02 NOTE: Rescanning AFD disks; using afd_diskstring.
--使用afd_diskstring扫描AFD磁盘
27-Mar-21 23:02 Changing owership of /dev/oracleafd/disks/* to grid:oinstall
--将AFD绑定的磁盘路径权限设置为grid:oinstall
27-Mar-21 23:02 Label list under /dev/oracleafd/disks/* : 
--将/dev/oracleafd/disks/*目录下的磁盘加上LABEL
27-Mar-21 23:02 Creating AFD init scripts
27-Mar-21 23:02 NOTE: command execution (/bin/rpm -q sles-release 2>&1) returned : 1
27-Mar-21 23:02 NOTE: command execution (/bin/rpm -q sles-release 2>&1) returned : 1
27-Mar-21 23:02 Copied AFD init scripts to rcdirs
27-Mar-21 23:02 NOTE: command execution (/bin/rpm -q sles-release 2>&1) returned : 1
--创建init启动文件，/etc/init.d/afd  
27-Mar-21 23:02 NOTE: asmcmdafd_is_siha Opening /etc/oracle/ocr.loc
27-Mar-21 23:02 NOTE: asmcmdafd_is_siha Value (false) set for local_only
27-Mar-21 23:02 NOTE: Starting OHASD... 
27-Mar-21 23:02 NOTE: asmcmdafd_is_siha Opening /etc/oracle/ocr.loc
27-Mar-21 23:02 NOTE: asmcmdafd_is_siha Value (false) set for local_only
--检查ocr的配置信息
27-Mar-21 23:02 NOTE: Checking existence of AFD resource
27-Mar-21 23:02 NOTE: ora.driver.afd resource doesn't exist
27-Mar-21 23:02 NOTE: ora.driver.afd.type res type doesn't exist
27-Mar-21 23:02 NOTE: Registering resource type ora.driver.afd.type
27-Mar-21 23:02 NOTE: added resource type ora.driver.afd.type
27-Mar-21 23:02 NOTE: Added AFD resource
--将AFD的资源类型写入到集群
27-Mar-21 23:02 NOTE: asmcmdafd_is_siha Opening /etc/oracle/ocr.loc
27-Mar-21 23:02 NOTE: asmcmdafd_is_siha Value (false) set for local_only
27-Mar-21 23:02 cssd res modify command : /u01/app/19.0.0/grid/bin/crsctl modify res ora.cssd -attr "STOP_DEPENDENCIES='hard(shutdown:ora.gipcd,shutdown:ora.diskmon,intermediate:ora.cssdmonitor, shutdown:ora.gpnpd,shutdown:ora.driver.afd)'" -init
27-Mar-21 23:02 cssdmonitor res modify command : /u01/app/19.0.0/grid/bin/crsctl modify res ora.cssdmonitor -attr "START_DEPENDENCIES='hard(ora.gpnpd,intermediate:ora.driver.afd)'" -init
27-Mar-21 23:02 NOTE: Modified ora.cssd and ora.cssdmonitor resource attributes
--更新olr里面的资源依赖关系
27-Mar-21 23:02 NOTE: asmcmdafd_is_siha Opening /etc/oracle/ocr.loc
27-Mar-21 23:02 NOTE: asmcmdafd_is_siha Value (false) set for local_only
27-Mar-21 23:02 NOTE: Stopping OHASD... 

7、确认模块加载情况
[grid@ora19c01 ~]$ lsmod |grep oracle
oracleafd             214072  0 
oracleacfs           5581810  0 
oracleadvm           1231385  0 
oracleoks             721311  2 oracleacfs,oracleadvm
[grid@ora19c01 ~]$ 
[grid@ora19c01 ~]$ afddriverstate version 
AFD-9325:     Driver OS kernel version = 3.10.0-862.el7.x86_64.
AFD-9326:     Driver build number = 190222.
AFD-9212:     Driver build version = 19.0.0.0.0.
AFD-9547:     Driver available build number = 190222.
AFD-9548:     Driver available build version = 19.0.0.0.0.

[grid@ora19c01 ~]$ asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'ora19c01'

这里看到模块已经LOADED，filter 也是ENABLED 了，如果这里filter 是DISABLED的，可以通过root用户执行asmcmd afd_filter -e来启用。 相应的禁用filter命令就是asmcmd afd_filter -d

8、将原有的磁盘进行LABEL
为了保证磁盘不会错乱，建议参考disk_name配置
[grid@ora19c01 ~]$ kfed read /dev/asm-data2dg0 |grep name
kfdhdb.dskname:            DATA2DG_0000 ; 0x028: length=12
kfdhdb.grpname:                 DATA2DG ; 0x048: length=7
kfdhdb.fgname:             DATA2DG_0000 ; 0x068: length=12

[root@ora19c01 ~]# /u01/app/19.0.0/grid/bin/asmcmd afd_label DATA2DG00 /dev/asm-data2dg0 --migrate

< kfdhdb.driver.provstr:         ORCLDISK ; 0x000: length=8
< kfdhdb.driver.reserved[0]:            0 ; 0x008: 0x00000000
< kfdhdb.driver.reserved[1]:            0 ; 0x00c: 0x00000000
< kfdhdb.driver.reserved[2]:            0 ; 0x010: 0x00000000
---
> kfdhdb.driver.provstr:ORCLDISKDATA2DG00 ; 0x000: length=17
> kfdhdb.driver.reserved[0]:   1096040772 ; 0x008: 0x41544144
> kfdhdb.driver.reserved[1]:    809976882 ; 0x00c: 0x30474432
> kfdhdb.driver.reserved[2]:           48 ; 0x010: 0x00000030

由于这个盘是创建了磁盘组的，因此必须加上--migrate 参数，否则会报错
disk /dev/asm-data2dg0 is already provisioned for ASM
ASMCMD-9513: ASM disk label set operation failed.

转移了一块盘，发现并没有影响另一个节点的正常使用。尝试转移其他的盘。
好吧偷懒了 ，直接生成命令。
select '/u01/app/19.0.0/grid/bin/asmcmd afd_label '||regexp_replace(NAME,'_00','')||' '||PATH||'  --migrate' from v$asm_disk where name is not null order by group_number,name;

/u01/app/19.0.0/grid/bin/asmcmd afd_label DATA2DG00 /dev/asm-data2dg0  --migrate
/u01/app/19.0.0/grid/bin/asmcmd afd_label DATA2DG01 /dev/asm-data2dg1  --migrate
/u01/app/19.0.0/grid/bin/asmcmd afd_label DATA2DG02 /dev/asm-data2dg2  --migrate
/u01/app/19.0.0/grid/bin/asmcmd afd_label DATA2DG03 /dev/asm-data2dg3  --migrate
/u01/app/19.0.0/grid/bin/asmcmd afd_label DATA00 /dev/asm-data0  --migrate
/u01/app/19.0.0/grid/bin/asmcmd afd_label FRA00 /dev/asm-fra0  --migrate
/u01/app/19.0.0/grid/bin/asmcmd afd_label MGMT00 /dev/asm-mgmt0  --migrate
/u01/app/19.0.0/grid/bin/asmcmd afd_label OCR00 /dev/asm-ocr0  --migrate
/u01/app/19.0.0/grid/bin/asmcmd afd_label OCR01 /dev/asm-ocr1  --migrate
/u01/app/19.0.0/grid/bin/asmcmd afd_label OCR02 /dev/asm-ocr2  --migrate

9、查看LABEL后的磁盘
[root@ora19c01 ~]# /u01/app/19.0.0/grid/bin/asmcmd afd_lsdsk
--------------------------------------------------------------------------------
Label                     Filtering   Path
================================================================================
DATA00                      ENABLED   /dev/asm-data0
DATA2DG00                   ENABLED   /dev/asm-data2dg0
DATA2DG01                   ENABLED   /dev/asm-data2dg1
DATA2DG02                   ENABLED   /dev/asm-data2dg2
DATA2DG03                   ENABLED   /dev/asm-data2dg3
FRA00                       ENABLED   /dev/asm-fra0
MGMT00                      ENABLED   /dev/asm-mgmt0
OCR00                       ENABLED   /dev/asm-ocr0
OCR01                       ENABLED   /dev/asm-ocr1
OCR02                       ENABLED   /dev/asm-ocr2
[root@ora19c01 ~]# ls -ltr /dev/oracleafd/disks/
total 40
-rw-rw-r-- 1 grid oinstall 18 Mar 27 23:40 DATA2DG00
-rw-rw-r-- 1 grid oinstall 18 Mar 27 23:53 DATA2DG01
-rw-rw-r-- 1 grid oinstall 18 Mar 27 23:54 DATA2DG02
-rw-rw-r-- 1 grid oinstall 18 Mar 27 23:54 DATA2DG03
-rw-rw-r-- 1 grid oinstall 15 Mar 27 23:54 DATA00
-rw-rw-r-- 1 grid oinstall 14 Mar 27 23:54 FRA00
-rw-rw-r-- 1 grid oinstall 15 Mar 27 23:54 MGMT00
-rw-rw-r-- 1 grid oinstall 14 Mar 27 23:54 OCR00
-rw-rw-r-- 1 grid oinstall 14 Mar 27 23:54 OCR01
-rw-rw-r-- 1 grid oinstall 14 Mar 27 23:54 OCR02

看到AFD绑定的磁盘生成了。

10、启动CRS
[root@ora19c01 ~]# /u01/app/19.0.0/grid/bin/crsctl start crs 
CRS-4123: Oracle High Availability Services has been started.

**************************************************************
ora19c01:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
ora19c02:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

11、检查ASM磁盘状态
NAME                      PATH                    
------------------------- ------------------------
                          /dev/asm-fra0           
                          /dev/asm-data2dg3       
                          /dev/asm-data2dg4       
                          /dev/asm-ocr1           
                          /dev/asm-ocr0           
                          /dev/asm-ocr2           
                          /dev/asm-data2dg5       
                          /dev/asm-data2dg0       
                          /dev/asm-data0          
                          /dev/asm-data2dg1       
                          /dev/asm-data2dg2       
                          /dev/asm-mgmt0          
DATA2DG_0000              AFD:DATA2DG00           
DATA2DG_0001              AFD:DATA2DG01           
DATA2DG_0002              AFD:DATA2DG02           
DATA2DG_0003              AFD:DATA2DG03           
DATA_0000                 AFD:DATA00              
FRA_0000                  AFD:FRA00               
MGMT_0000                 AFD:MGMT00              
OCR_0000                  AFD:OCR00               
OCR_0001                  AFD:OCR01               
OCR_0002                  AFD:OCR02               

 DG_NUM NAME            STATE       TYPE     TOTAL_MB    FREE_MB USABLE_FREE_MB OFFLINE_DISKS USE_PER(%)
------- --------------- ----------- ------ ---------- ---------- -------------- ------------- ----------
      1 DATA2DG         CONNECTED   NORMAL       4096       3658           1317             0      10.69
      5 OCR             MOUNTED     NORMAL       6144       5228           1590             0      14.91
      3 FRA             CONNECTED   EXTERN      10240       8707           8707             0      14.97
      2 DATA            CONNECTED   EXTERN      20480      13453          13453             0      34.31
      4 MGMT            MOUNTED     EXTERN      30720       1840           1840             0      94.01

NAME                      PATH                          
------------------------- ------------------------------
                          /dev/asm-data2dg4             
                          /dev/asm-data2dg5             
DATA2DG_0000              /dev/asm-data2dg0             
DATA2DG_0001              /dev/asm-data2dg1             
DATA2DG_0002              /dev/asm-data2dg2             
DATA2DG_0003              /dev/asm-data2dg3             
DATA_0000                 /dev/asm-data0                
FRA_0000                  /dev/asm-fra0                 
MGMT_0000                 /dev/asm-mgmt0                
OCR_0000                  /dev/asm-ocr0                 
OCR_0001                  /dev/asm-ocr1                 
OCR_0002                  /dev/asm-ocr2                 

12 rows selected.
DG_NUM NAME            STATE       TYPE     TOTAL_MB    FREE_MB USABLE_FREE_MB OFFLINE_DISKS USE_PER(%)
------ --------------- ----------- ------ ---------- ---------- -------------- ------------- ----------
     1 DATA2DG         CONNECTED   NORMAL       4096       3658           1317             0      10.69
     5 OCR             MOUNTED     NORMAL       6144       5228           1590             0      14.91
     3 FRA             CONNECTED   EXTERN      10240       8707           8707             0      14.97
     2 DATA            CONNECTED   EXTERN      20480      13453          13453             0      34.31
     4 MGMT            MOUNTED     EXTERN      30720       1840           1840             0      94.01

可以看到，两边的CRS 都是正常的。而且两边的ASM磁盘path不同了。节点1启用了AFD，节点2还未配置。也就说明，实际上AFD只是一种新的绑定方式。实际上磁盘组通过扫描磁盘头挂载。并不要求名称一致。

至此一个节点完成。

12、其他节点做如下配置：
1）关闭crs 
[root@ora19c02 ~]#    /u01/app/19.0.0/grid/bin/crsctl stop crs 
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ora19c02'
…………
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ora19c02' has completed
CRS-4133: Oracle High Availability Services has been stopped.

2）如果使用的是ASMLIB的模块，先删除

3）使用root用户执行配置AFD模块
[root@ora19c02 ~]# export ORACLE_BASE=/u01/app/grid
[root@ora19c02 ~]# /u01/app/19.0.0/grid/bin/asmcmd afd_configure
AFD-627: AFD distribution files found.
AFD-634: Removing previous AFD installation.
AFD-635: Previous AFD components successfully removed.
AFD-9294: updating file /etc/sysconfig/oracledrivers.conf
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies - this may take some time.
AFD-9154: Loading 'oracleafd.ko' driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device '/dev/oracleafd/admin'.
AFD-638: AFD installation correctness verified.
Modifying resource dependencies - this may take some time.
[root@ora19c02 ~]# 

4）确认模块加载情况
[grid@ora19c02 ~]$ lsmod |grep oracle
oracleafd             214072  0 
oracleacfs           5581810  0 
oracleadvm           1231385  0 
oracleoks             721311  2 oracleacfs,oracleadvm
[grid@ora19c02 ~]$ afddriverstate version
AFD-9325:     Driver OS kernel version = 3.10.0-862.el7.x86_64.
AFD-9326:     Driver build number = 190222.
AFD-9212:     Driver build version = 19.0.0.0.0.
AFD-9547:     Driver available build number = 190222.
AFD-9548:     Driver available build version = 19.0.0.0.0.
[grid@ora19c02 ~]$ asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'ora19c02'

5）区别于第一个节点，需要LABEL磁盘为AFD模式。其他节点只需要scan磁盘。
确认磁盘是否存在，
[grid@ora19c02 ~]$ asmcmd afd_lsdsk
--------------------------------------------------------------------------------
Label                     Filtering   Path
================================================================================
DATA00                      ENABLED   /dev/asm-data0
DATA2DG00                   ENABLED   /dev/asm-data2dg0
DATA2DG01                   ENABLED   /dev/asm-data2dg1
DATA2DG02                   ENABLED   /dev/asm-data2dg2
DATA2DG03                   ENABLED   /dev/asm-data2dg3
FRA00                       ENABLED   /dev/asm-fra0
MGMT00                      ENABLED   /dev/asm-mgmt0
OCR00                       ENABLED   /dev/asm-ocr0
OCR01                       ENABLED   /dev/asm-ocr1
OCR02                       ENABLED   /dev/asm-ocr2
[grid@ora19c02 ~]$ 
[grid@ora19c02 ~]$ 
[grid@ora19c02 ~]$ ls -l /dev/oracleafd/disks/ 
total 40
-rw-rw-r-- 1 grid oinstall 15 Mar 28 00:42 DATA00
-rw-rw-r-- 1 grid oinstall 18 Mar 28 00:42 DATA2DG00
-rw-rw-r-- 1 grid oinstall 18 Mar 28 00:42 DATA2DG01
-rw-rw-r-- 1 grid oinstall 18 Mar 28 00:42 DATA2DG02
-rw-rw-r-- 1 grid oinstall 18 Mar 28 00:42 DATA2DG03
-rw-rw-r-- 1 grid oinstall 14 Mar 28 00:42 FRA00
-rw-rw-r-- 1 grid oinstall 15 Mar 28 00:42 MGMT00
-rw-rw-r-- 1 grid oinstall 14 Mar 28 00:42 OCR00
-rw-rw-r-- 1 grid oinstall 14 Mar 28 00:42 OCR01
-rw-rw-r-- 1 grid oinstall 14 Mar 28 00:42 OCR02

查看当前已经自动扫描出AFD磁盘了。如果未扫描出，执行一下扫描。
root:
/u01/app/19.0.0/grid/bin/afdtool -rescan

/u01/app/19.0.0/grid/bin/asmcmd afd_scan

6)启动CRS
[root@ora19c02 ~]# /u01/app/19.0.0/grid/bin/crsctl start crs 
CRS-4123: Oracle High Availability Services has been started.

13、所有节点都启用AFD之后，可以吧原来的路径从asmstring移除
[grid@ora19c01 ~]$ asmcmd dsget
parameter:/dev/asm-*, AFD:*
profile:/dev/asm-*,AFD:*
[grid@ora19c01 ~]$ asmcmd dsset 'AFD:*'
[grid@ora19c01 ~]$ asmcmd dsget
parameter:AFD:*
profile:AFD:*

NAME                      PATH                           
------------------------- -------------------------------
DATA2DG_0000              AFD:DATA2DG00                  
DATA2DG_0001              AFD:DATA2DG01                  
DATA2DG_0002              AFD:DATA2DG02                  
DATA2DG_0003              AFD:DATA2DG03                  
DATA_0000                 AFD:DATA00                     
FRA_0000                  AFD:FRA00                      
MGMT_0000                 AFD:MGMT00                     
OCR_0000                  AFD:OCR00                      
OCR_0001                  AFD:OCR01                      
OCR_0002                  AFD:OCR02                      

10 rows selected.

改成只有AFD后，原路径磁盘在v$asm_disk里面就查不到了。 
至此，AFD的改造完成。