Today we upgrade our e-learning platform CLIX to the latest version and get some problems wihle upgrading the database in our Oracle RAC environment. In a Oracle-RAC with ASM setup, archive-logs will primary backuped to the Flash Recovery Area (+FRA). For redundancy we additionally save all archive-logs outside of the ASM on local disks. You can setting up this with the following command:
sqlplus SYS password/db-instance AS SYSDBA ALTER SYSTEM SET log_archive_dest_2='location="/path/on/disk"' SCOPE=BOTH SID='*'
Set SCOPE to BOTH make the change in memory and in the server parameter file. Setting the SID to * commit the change to all instances in an RAC environment. It has no effect in a single-instance database.
In our case the 2nd desitnation on local disk has to insufficient space available for all new created archive-logs (which were created during the database update process). So our backup partition was 100% full and the database upgrade process was stalling. To get the upgrade continue to work we misleadingly decide to delete some archive-logs manually on disk. A very bad outcome was that our RMAN backup did not work any more after this. In the logfiles found his lines:
RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of backup plus archivelog command at 01/29/2009 05:30:18 RMAN-06726: could not locate archivelog /oraarch/CLIX/1_12972_630768524.arc [...]
The first thing to solve the problem were to crosscheck and delete all our expired archives-logs with the RMAN command:
rman target SYS/password nocatalog crosscheck archivelog all ; delete expired archivelog ;
The backup seems now to to work but it still aborts with the error message:
RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of backup plus archivelog command at 01/29/2009 12:53:25 RMAN-03009: failure of backup command on c1 channel at 01/29/2009 12:52:55 ORA-19809: limit exceeded for recovery files ORA-19804: cannot reclaim 127926272 bytes disk space from 5368709120 limit [...]
After wasting some time with searching for a solution on google i solved the problem by increasing the DB_RECOVERY_FILE_DEST_SIZE parameter with the SQL statement:
ALTER SYSTEM SET DB_RECOVERY_FILE_DEST_SIZE = 10g SCOPE=BOTH SID='*';
Our FRA has about 15G capacity and was just 50% full, so i`m not sure why the RMAN backup works after increasing this value. I will update this article when i get more background information on this error. Please feel free to comment this post!
References:
