Configuring Oracle AFD after Installation
Purpose of this post is to investigate on configuring Oracle AFD after Installation. for more information regarding Oracle AFD fundamentals and migrating to Oracle ASMLIB From AFD please visit links below:
Migrating to Oracle ASMLIB From AFD
Let me show you how to configure Oracle AFD after Installation within a scenario:
Existing setup:
Before start configuring Oracle AFD after Installation we check existing setup:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | [grid@mydb2 ~]$ oracleasm status Checking if ASM is loaded: yes Checking if /dev/oracleasm is mounted: yes [grid@mydb2 ~]$ asmcmd lsdsk -p Group_Num Disk_Num Incarn Mount_Stat Header_Stat Mode_Stat State Path 1 0 4041520447 CACHED MEMBER ONLINE NORMAL /dev/oracleasm/disks/ASMDISK01 1 1 4041520446 CACHED MEMBER ONLINE NORMAL /dev/oracleasm/disks/ASMDISK02 [grid@mydb2 ~]$ sqlplus / as sysasm SQL*Plus: Release 12.2.0.1.0 Production Copyright (c) 1982, 2016, Oracle. All rights reserved. Connected to: Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production SQL> col name for a20; SQL> col path for a50; SQL> select name, path from v$asm_disk; NAME PATH -------------------- -------------------------------------------------- ASMDISK02 /dev/oracleasm/disks/ASMDISK02 ASMDISK01 /dev/oracleasm/disks/ASMDISK01 [grid@mydb2 ~]$ crsctl stat res -t -------------------------------------------------------------------------------- Name Target State Server State details -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.DATA.dg ONLINE ONLINE mydb2 STABLE ora.LISTENER.lsnr ONLINE ONLINE mydb2 STABLE ora.asm ONLINE ONLINE mydb2 Started,STABLE ora.ons OFFLINE OFFLINE mydb2 STABLE -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.cssd 1 ONLINE ONLINE mydb2 STABLE ora.diskmon 1 OFFLINE OFFLINE STABLE ora.evmd 1 ONLINE ONLINE mydb2 STABLE -------------------------------------------------------------------------------- |
start configuring Oracle AFD
As you saw in preceding codes, ASMLIB used for accessing ASM Disks and the Clusterware stack is up. In the next step, we have to add Oracle AFD access path to ASM disk strings. If we do not specify the correct path, the Clusterware stack will not come up.
1 2 3 4 5 6 7 8 9 | [grid@mydb2 ~]$ asmcmd dsget parameter:/dev/oracleasm/disks/* profile:/dev/oracleasm/disks/* [grid@mydb2 ~]$ asmcmd dsset '/dev/oracleasm/disks/*','AFD:*' [grid@mydb2 ~]$ asmcmd dsget parameter:/dev/oracleasm/disks/*, AFD:* profile:/dev/oracleasm/disks/*,AFD:* |
Oracle AFD access path published globally in Clusterware stack, (the globally means that the configuration is propagated in many areas like OCR, ASM spfile or gpnp profile). To start the creation of Oracle AFD disks, we require root user permission and Clusterware stack needs to be down. As I mentioned in the introducing Oracle AFD, ASMLIB and Oracle AFD are mutually exclusive. Thus I am going to stop and disable ASMLIB and then create Oracle AFD disks.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | [grid@mydb2 ~]$ su - Password: [root@mydb2 ~]# /u01/app/grid/product/12.2/gi_1/bin/crsctl stop has CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'mydb2' CRS-2673: Attempting to stop 'ora.evmd' on 'mydb2' CRS-2673: Attempting to stop 'ora.DATA.dg' on 'mydb2' CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'mydb2' CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'mydb2' succeeded CRS-2677: Stop of 'ora.DATA.dg' on 'mydb2' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'mydb2' CRS-2677: Stop of 'ora.evmd' on 'mydb2' succeeded CRS-2677: Stop of 'ora.asm' on 'mydb2' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'mydb2' CRS-2677: Stop of 'ora.cssd' on 'mydb2' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'mydb2' has completed CRS-4133: Oracle High Availability Services has been stopped. [root@mydb2 ~]# /u01/app/grid/product/12.2/gi_1/bin/acfsload stop [root@mydb2 ~]# ll /dev/oracleasm/disks/* brw-rw----. 1 grid asmadmin 8, 17 Mar 4 09:43 /dev/oracleasm/disks/ASMDISK01 brw-rw----. 1 grid asmadmin 8, 33 Mar 4 09:43 /dev/oracleasm/disks/ASMDISK02 [root@mydb2 ~]# oracleasm exit Unmounting ASMlib driver filesystem: /dev/oracleasm Unloading module "oracleasm": oracleasm [root@mydb2 ~]# oracleasm status Checking if ASM is loaded: no Checking if /dev/oracleasm is mounted: no [root@mydb2 ~]# /u01/app/grid/product/12.2/gi_1/bin/asmcmd afd_configure AFD-636: Installing requested AFD software. AFD-637: Loading installed AFD drivers. AFD-9321: Creating udev for AFD. AFD-9323: Creating module dependencies - this may take some time. AFD-9154: Loading 'oracleafd.ko' driver. AFD-649: Verifying AFD devices. AFD-9156: Detecting control device '/dev/oracleafd/admin'. AFD-638: AFD installation correctness verified. Modifying resource dependencies - this may take some time. [root@mydb2 ~]# /u01/app/grid/product/12.2/gi_1/bin/asmcmd afd_label AFDDISK01 /dev/sdb1 --migrate [root@mydb2 ~]# /u01/app/grid/product/12.2/gi_1/bin/asmcmd afd_label AFDDISK02 /dev/sdc1 --migrate |
Let me check the correctness of Oracle AFD installation and configuration.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | [root@mydb2 ~]# /u01/app/grid/product/12.2/gi_1/bin/asmcmd afd_state ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'mydb2' [root@mydb2 ~]# /u01/app/grid/product/12.2/gi_1/bin/asmcmd afd_lslbl -------------------------------------------------------------------------------- Label Duplicate Path ================================================================================ AFDDISK01 /dev/sdb1 AFDDISK02 /dev/sdc1 [root@mydb2 ~]# /u01/app/grid/product/12.2/gi_1/bin/asmcmd afd_lsdsk -------------------------------------------------------------------------------- Label Filtering Path ================================================================================ AFDDISK01 ENABLED /dev/sdb1 AFDDISK02 ENABLED /dev/sdc1 [root@mydb2 ~]# ll /dev/oracleafd/disks/* -rwxrwx---. 1 grid oinstall 10 Mar 4 09:46 /dev/oracleafd/disks/AFDDISK01 -rwxrwx---. 1 grid oinstall 10 Mar 4 09:46 /dev/oracleafd/disks/AFDDISK02 [root@mydb2 ~]# cat /dev/oracleafd/disks/AFDDISK01 /dev/sdb1 |
Preceding codes verified the correctness of installation and configuration of Oracle AFD. As you saw labels and references do exist in Oracle AFD path. Be aware of an important point, we have to disable oracleasm service in the OS, otherwise, the Clusterware stack will not come up correctly.
You could find more information about Oracle AFD in MOS.
1 2 | [root@mydb2 ~]# systemctl disable oracleasm Removed symlink /etc/systemd/system/multi-user.target.wants/oracleasm.service. |
Bring Clusterware back up
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | [root@mydb2 ~]# /u01/app/grid/product/12.2/gi_1/bin/crsctl start has CRS-4123: Oracle High Availability Services has been started. [root@mydb2 ~]# /u01/app/grid/product/12.2/gi_1/bin/crsctl stat res -t -------------------------------------------------------------------------------- Name Target State Server State details -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.DATA.dg ONLINE ONLINE mydb2 STABLE ora.LISTENER.lsnr ONLINE ONLINE mydb2 STABLE ora.asm ONLINE ONLINE mydb2 Started,STABLE ora.ons OFFLINE OFFLINE mydb2 STABLE -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.cssd 1 ONLINE ONLINE mydb2 STABLE ora.diskmon 1 OFFLINE OFFLINE STABLE ora.driver.afd 1 ONLINE ONLINE mydb2 STABLE ora.evmd 1 ONLINE ONLINE mydb2 STABLE -------------------------------------------------------------------------------- [root@mydb2 ~]# /u01/app/grid/product/12.2/gi_1/bin/acfsload start ACFS-9391: Checking for existing ADVM/ACFS installation. ACFS-9392: Validating ADVM/ACFS installation files for operating system. ACFS-9393: Verifying ASM Administrator setup. ACFS-9308: Loading installed ADVM/ACFS drivers. ACFS-9154: Loading 'oracleoks.ko' driver. ACFS-9154: Loading 'oracleadvm.ko' driver. ACFS-9154: Loading 'oracleacfs.ko' driver. ACFS-9327: Verifying ADVM/ACFS devices. ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'. ACFS-9156: Detecting control device '/dev/ofsctl'. ACFS-9322: completed [root@mydb2 ~]# /u01/app/grid/product/12.2/gi_1/bin/crsctl check has CRS-4638: Oracle High Availability Services is online |
Verification:
As I expected, Clusterware came up again correctly. As I love tidy works, let’s clean up disk strings and do some verification.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 | [grid@mydb2 ~]$ asmcmd dsget parameter:/dev/oracleasm/disks/*, AFD:* profile:/dev/oracleasm/disks/*,AFD:* [grid@mydb2 ~]$ asmcmd dsset 'AFD:*' [grid@mydb2 ~]$ asmcmd dsget parameter:AFD:* profile:AFD:* [grid@mydb2 ~]$ asmcmd lsdsk -p Group_Num Disk_Num Incarn Mount_Stat Header_Stat Mode_Stat State Path 1 0 4036988647 CACHED MEMBER ONLINE NORMAL AFD:AFDDISK01 1 1 4036988648 CACHED MEMBER ONLINE NORMAL AFD:AFDDISK02 [grid@mydb2 ~]$ asmcmd afd_dsget AFD discovery string: /dev/sd* [grid@mydb2 ~]$ asmcmd afd_state ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'mydb2' [grid@mydb2 ~]$ srvctl config asm ASM home: <CRS home> Password file: +DATA/orapwasm Backup of Password file: ASM listener: LISTENER Spfile: +DATA/ASM/ASMPARAMETERFILE/registry.253.969188027 ASM diskgroup discovery string: AFD:* [grid@mydb2 ~]$ [grid@mydb2 ~]$ crsctl stat res ora.driver.afd NAME=ora.driver.afd TYPE=ora.driver.afd.type TARGET=ONLINE STATE=ONLINE on mydb2 [grid@mydb2 ~]$ crsctl stat res -t -------------------------------------------------------------------------------- Name Target State Server State details -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.DATA.dg ONLINE ONLINE mydb2 STABLE ora.LISTENER.lsnr ONLINE ONLINE mydb2 STABLE ora.asm ONLINE ONLINE mydb2 Started,STABLE ora.ons OFFLINE OFFLINE mydb2 STABLE -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.cssd 1 ONLINE ONLINE mydb2 STABLE ora.diskmon 1 OFFLINE OFFLINE STABLE ora.driver.afd 1 ONLINE ONLINE mydb2 STABLE ora.evmd 1 ONLINE ONLINE mydb2 STABLE -------------------------------------------------------------------------------- |
Observing Oracle AFD behaviour
Undoubtedly, Oracle AFD has many advantages over ASMLIB. The most impressive feature is rejecting non-Oracle I/Os, which prevents AFD disks from corruption. In upcoming codes, I am going to simulate intentional non-Oracle disks manipulation and we will observe Oracle AFD behaviour.
1 2 3 4 5 6 7 8 9 10 | [grid@mydb2 ~]$ echo "masoud" > /dev/sdb1 -bash: /dev/sdb1: Permission denied [grid@mydb2 ~]$ su - Password: [root@mydb2 ~]# dd if=/dev/zero of=/dev/sdb1 count=30 30+0 records in 30+0 records out 15360 bytes (15 kB) copied, 0.000211123 s, 72.8 MB/s |
In preceding codes, grid user received the permission denied because it has no direct access to block devices to manipulate Oracle AFD disks.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | [root@mydb2 ~]# /u01/app/grid/product/12.2/gi_1/bin/asmcmd afd_dsget AFD discovery string: /dev/sd* [root@mydb2 ~]# /u01/app/grid/product/12.2/gi_1/bin/asmcmd afd_lsdsk -------------------------------------------------------------------------------- Label Filtering Path ================================================================================ AFDDISK01 ENABLED /dev/sdb1 AFDDISK02 ENABLED /dev/sdc1 [root@mydb2 ~]# ll /dev/sdb1 brw-rw----. 1 root disk 8, 17 Mar 4 11:44 /dev/sdb1 [root@mydb2 ~]# ll /dev/sdc1 brw-rw----. 1 root disk 8, 33 Mar 4 10:02 /dev/sdc1 |
Although it seems that the root user could change disks but it is not true. Let’s take a look at /var/log/messages
1 2 3 4 5 6 7 8 9 10 11 | [root@mydb2 ~]# tail /var/log/messages Mar 4 11:44:49 mydb2 su: (to root) grid on pts/0 Mar 4 11:45:21 mydb2 kernel: F 4372115.736/180304081521 flush-8:16[32832] oracleafd:18:0894:Write IO to ASM managed device: [8] [17] Mar 4 11:45:21 mydb2 kernel: Buffer I/O error on device sdb1, logical block 0 Mar 4 11:45:21 mydb2 kernel: lost page write due to I/O error on sdb1 Mar 4 11:45:21 mydb2 kernel: Buffer I/O error on device sdb1, logical block 1 Mar 4 11:45:21 mydb2 kernel: lost page write due to I/O error on sdb1 Mar 4 11:45:21 mydb2 kernel: Buffer I/O error on device sdb1, logical block 2 Mar 4 11:45:21 mydb2 kernel: lost page write due to I/O error on sdb1 Mar 4 11:45:21 mydb2 kernel: Buffer I/O error on device sdb1, logical block 3 Mar 4 11:45:21 mydb2 kernel: lost page write due to I/O error on sdb1 |
In the above appearance, we could see an amazing Oracle AFD behaviour, which denied change on block devices respectively.
I hope you enjoyed reading this post.
