I had a very nasty experience with Oracle ASM recently. A long story short, ASM crashed on both our RAC nodes when removing a corrupted archive file on file and incarnation number (removing it on it’s name didn’t work). Starting the instance failed. When removing the failing diskgroup ASM was happy again. Luckily the failing diskgroup only housed archivelog files, except for one database it also held it’s index datafile. So that one wouldn’t come up. If you’re in such a situation, you can use the Oracle tools amdu in combination with kfed to extract the missing datafile or archivelogs (in our situation) from a dismounted diskgroup. If I knew that from the start it saved me alot of frustrating hours.
During the installation of Oracle 11gR2 on Red Hat Enterprise Linux 5 I stumbled upon a incorrect default setting for ASM disk discovery when using ASM lib. The discovery path is set to ‘/dev/raw/sd*’, which is incorrect because ASM lib creates the devices under ‘/dev/oracleasm/disks’.
Another thing to be aware of is the ownership of the disks. You need to run ‘/etc/init.d/oracleasm configure’ in order to configure the owner. If you just hit return the ownership of the disks is set to root. Because you will start the Oracle installation as user oracle the disks will not appear in the discovery step during the installation.