突然数据库出现了较高的enq: TX – row lock contention的等待事件，这是一个APPLICATION CLASS的wait event, 通常出现在DML事务级的行或块竞争，这次的MODE为6， 语句只是简单的delete xx where rowid=:bind;  理论上这个事务应该是非常快，但是查看数据库和应用日志都出现了ORA-600的错误， 判断问题就不再是简单的行事务竞争， 手动模拟语句也还原出问题，但并不是所有记录每次都能触发，现象为运行DELETE   SQL时失败并出现ora-600 [kcladdle_1].  数据库中DB Alert log 如下：

2017-06-03 19:43:50.563000 +08:00
Thread 1 advanced to log sequence 26913 (LGWR switch)
  Current log# 1 seq# 26913 mem# 0: +DATADG/anbob/onlinelog/group_1.257.923576165
2017-06-03 19:49:03.062000 +08:00
Thread 1 advanced to log sequence 26914 (LGWR switch)
  Current log# 2 seq# 26914 mem# 0: +DATADG/anbob/onlinelog/group_2.258.923576167
2017-06-03 19:53:45.570000 +08:00
ERROR: Unable to normalize symbol name for the following short stack (at offset 316):
dbgexProcessError()+180<-dbgeExecuteForError()+72<-dbgePostErrorKGE()+2048<-dbkePostKGE_kgsf()+68<-kgeadse()+380<-kgerinv_internal()+48<-kgerinv()+48<-kserin()+76
<-kclladdle()+128<-kclcls()+18764<-kclgrlk()+328<-kcbzib()+18480<-kcbgcur()+5440<-ktspgfblk3()+376<-ktspstchg1()+128<-ktspstchg()+20<-kddchg()+1212<-IPRA.$kdddrp()+1980<-IPRA.$kdddel()+472<-kdddel()+96<-kaudel()+120<-delrow()+1348<-qerdlFetch()+688<-delexe()+936<-opiexe()+14672<-opiodr()+720<-ttcpip()+1
028<-opitsk()+1508<-opiino()+940<-opiodr()+720<-opidrv()+1132<-sou2o()+136<-opimai_real()+608<-ssthrdmain()+268<-main()+204<-__start()+112
Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_60686692.trc  (incident=374057):
ORA-00600: internal error code, arguments: [kcladdle_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_374057/anbob1_ora_60686692_i374057.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Non critical error ORA-48913 caught while writing to trace file "/oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_374057/anbob1_ora_60686692_i374057.trc"
Error message: ORA-48913: Writing into trace file failed, file size limit [1024000] reached
Writing to the above trace file is disabled for now on...
2017-06-03 19:54:01.034000 +08:00
Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_66978102.trc  (incident=374073):
ORA-00600: internal error code, arguments: [kcladdle_1], [], [], [], [], [], [], [], [], [], [], []
Dumping diagnostic data in directory=[cdmp_20170603195401], requested by (instance=1, osid=60686692), summary=[incident=374057].
Incident details in: /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_374073/anbob1_ora_66978102_i374073.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2017-06-03 19:54:02.219000 +08:00
Sweep [inc][374057]: completed
Non critical error ORA-48913 caught while writing to trace file "/oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_374073/anbob1_ora_66978102_i374073.trc"
Error message: ORA-48913: Writing into trace file failed, file size limit [1024000] reached
Writing to the above trace file is disabled for now on...
Sweep [inc2][374057]: completed
Sweep [inc][374073]: completed
Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_61473276.trc  (incident=374033):
ORA-00600: internal error code, arguments: [kcladdle_1], [], [], [], [], [], [], [], [], [], [], []
Dumping diagnostic data in directory=[cdmp_20170603195403], requested by (instance=1, osid=66978102), summary=[incident=374073].
Incident details in: /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_374033/anbob1_ora_61473276_i374033.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2017-06-03 19:54:03.461000 +08:00
Sweep [inc2][374073]: completed
Sweep [inc][374033]: completed
Non critical error ORA-48913 caught while writing to trace file "/oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_374033/anbob1_ora_61473276_i374033.trc"
Error message: ORA-48913: Writing into trace file failed, file size limit [1024000] reached
Writing to the above trace file is disabled for now on...
2017-06-03 19:54:04.567000 +08:00
Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_5440550.trc  (incident=374081):
ORA-00600: internal error code, arguments: [kcladdle_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_374081/anbob1_ora_5440550_i374081.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Dumping diagnostic data in directory=[cdmp_20170603195404], requested by (instance=1, osid=61473276), summary=[incident=374033].
Sweep [inc2][374033]: completed
Sweep [inc][374081]: completed
Non critical error ORA-48913 caught while writing to trace file "/oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_374081/anbob1_ora_5440550_i374081.trc"
Error message: ORA-48913: Writing into trace file failed, file size limit [1024000] reached
Writing to the above trace file is disabled for now on...
2017-06-03 19:54:06.212000 +08:00
Dumping diagnostic data in directory=[cdmp_20170603195406], requested by (instance=1, osid=5440550), summary=[incident=374081].
Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_1902024.trc  (incident=374065):
ORA-00600: internal error code, arguments: [kcladdle_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_374065/anbob1_ora_1902024_i374065.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Non critical error ORA-48913 caught while writing to trace file "/oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_374065/anbob1_ora_1902024_i374065.trc"
...
...

错误时的CALL Stack，确认BUG常用的

----- Call Stack Trace -----
skdstdst <- ksedst1 <- ksedst <- dbkedDefDump <- ksedmp
<- ksfdmp <- dbgexPhaseII <- dbgexProcessError <- dbgeExecuteForError <- dbgePostErrorKGE
<- 2048 <- dbkePostKGE_kgsf <- kgeade <- kgefec <- kgefic
<- ksefic <- ksupop <- opiodr <- ttcpip <- opitsk
<- opiino <- opiodr <- opidrv <- sou2o <- opimai_real
<- ssthrdmain <- main <- start

TIP:
日志错误一致在持续，根据错误函数kcladdle_1判断， KCL=>Kernel Cache, (RAC) Lock,判断是cache级错误，MOS中也没有关于该错误的更多的信息只有一条BUG描述。判断CACHE级错误，尝试把在1节点delete报错的SQL去2节点执行无任何错误，后建议应用调整之2节点，同时刷新2个节点的Buffer cache, 问题现象得到了解决。后期经SR确认为Bug 12644270，当转换buffer cache k中的block PastImages 锁时发生 。

ORA-00600: [kcladdle_1]

Rediscovery Notes:

  The problem can occur while downconverting a lock to NULL because   there is just a PI attached – the PI may be written to disk and be   removed from the lock.

 Versions confirmed as being affected

11.2.0.3

Platforms affected Generic (all / most platforms affected)

Fixed:

The fix for 12644270 is first included in

12.1.0.1 (Base Release)

11.2.0.4 (Server Patch Set)

How to release still “killed“ status session in v$session?(释放killed的session) (一)
How to release still “killed“ status session in v$session? (释放killed的session) (二)
昨天数据库有3个session记录在v$session一直是KILLED状态，且已经好几个小时，当前的等待事件为”db file sequential read”, 其中有一个session是DML SQL,所以最后在事务结束前的行级锁一直未能释放，影响了操作相同记录的业务， 从数据库尝试kill [immeidate]没报错但是无法清理，手动weekup PMON检查PMON trace无报错， 通过操作系统层KILL -9 无报错仍无法清理,环境是11.2.0.3 2-NODES RAC ON AIX, 记录一下分析思路。

SQL>@ase

USERNAME      SID       SEQ# EVENT                MACHINE    MODULE      STATUS   LAST_CALL_ET SQL_ID          WAI_SECINW 
---------- ------ ---------- -------------------- ---------- ----------- -------- ------------ --------------- ---------- 
...                                                                    
ANBOB      3510      28110 gc buffer busy acqui kdbxxxx     SQL*xxxxxxx ACTIVE           3458 3dpbdtsg82768   0:0        
ANBOB      6024      24828 enq: TX - row lock c kycxxxx     evenxxxxxxx ACTIVE           9761 drdanx3hhzwc6   0:9760     
ANBOB      6121       1224 read by other sessio kycxxxx     evenxxxxxxx ACTIVE           9767 drdanx3hhzwc6   0:9762     
ANBOB      7180       1768 read by other sessio qmzxxxx     AccSxxxxxxx ACTIVE          29170 bazvmr4hmyprb   0:28806    
ANBOB      5276      19993 db file sequential r kycxxxx     evenxxxxxxx KILLED          29183 drdanx3hhzwc6   0:29183    
ANBOB      2284      38595 db file sequential r kycxxxx     evenxxxxxxx KILLED          29183 3yv2fann81bq7   0:29183    
ANBOB      6959      26745 db file sequential r qmzxxxx     AccSxxxxxxx KILLED          29184 cntmwhgmv5wuw   0:29183    
...


[oracle@weejar1:/home/oracle]$ kill -9 39518840
[oracle@weejar1:/home/oracle]$ ps -ef|grep 39518840
  oracle 44892446 59769246   0 15:33:54  pts/9  0:00 grep 39518840
    grid 39518840        1   0   Jun 07      - 199:30 oracleanbob1 (LOCAL=NO)
[oracle@weejar1:/home/oracle]$


truss -p  pid 
-- hang
procestack  pid
-- hang

oradebug setorapid xx
oradebug errorstack 
-- hang

oracebug short_stack
-- hang

SQL> oradebug setmypid;
Statement processed.
SQL> oradebug unlimit;
Statement processed.
SQL> oradebug dump systemstate 266;
Statement processed.

SQL> oradebug tracefile_name
/oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_65995280.trc

[oracle@weejar1:/home/oracle]$ grep -n 39518840 /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_65995280.trc
5891344:    O/S info: user: grid, term: UNKNOWN, ospid: 39518840 
5891345:    OSD pid info: Unix process pid: 39518840, image: oracle@weejar1
5891348:    Short stack dump: ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 39518840, image: oracle@weejar1'; prior command execution time exceeds 30000 ms
5899983:                       owner: 700000db18e19c0 - ospid: 39518840
5900005:    pid               : 39518840


[oracle@weejar1:/home/oracle]$  sed -n 5891330,5891364p  /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_65995280.trc
  (process) Oracle pid:2474, ser:178, calls cur/top: 0x700000d43d97f58/0x700000be07a5dd0
            flags : (0x0) -
            flags2: (0x0),  flags3: (0x10) 
            intr error: 24761, call error: 0, sess error: 0, txn error 0
            intr queue: 24761 2147483676 2147483676 2147483676 2147483676 2147483676 2147483676 
    ksudlp FALSE at location: 0
  (post info) last post received: 0 0 178
              last post received-location: kcl2.h LINE:4340 ID:kclcget
              last process to post me: 700000dc96f2df8 1 6
              last post sent: 0 0 159
              last post sent-location: kcb2.h LINE:4139 ID:kcbzww
              last process posted by me: 700000db9866cc8 93 0
    (latch info) wait_event=0 bits=0
    Process Group: DEFAULT, pseudo proc: 0x700000dd9a86230
    O/S info: user: grid, term: UNKNOWN, ospid: 39518840 
    OSD pid info: Unix process pid: 39518840, image: oracle@weejar1

*** 2017-06-13 17:08:33.300
    Short stack dump: ORA-32515: cannot issue ORADEBUG command 'SHORT_STACK' to process 'Unix process pid: 39518840, image: oracle@weejar1'; prior command execution time exceeds 30000 ms
    ----------------------------------------
    SO: 0x700000d7912cdc8, type: 10, owner: 0x700000db18e19c0, flag: INIT/-/-/0x00 if: 0x1 c: 0x1
     proc=0x700000db18e19c0, name=FileOpenBlock, file=ksfd.h LINE:6448 ID:, pg=0
    (FOB) 700000d7912cdc8 flags=2050 fib=700000da3a2c340 incno=0 pending i/o cnt=0
     fname=/dev/rzwb_lv15_104
     fno=377 lblksz=16384 fsiz=983039
    ----------------------------------------
    SO: 0x700000d7b9b2ea0, type: 10, owner: 0x700000db18e19c0, flag: INIT/-/-/0x00 if: 0x1 c: 0x1
     proc=0x700000db18e19c0, name=FileOpenBlock, file=ksfd.h LINE:6448 ID:, pg=0
    (FOB) 700000d7b9b2ea0 flags=2050 fib=700000da39b38b0 incno=0 pending i/o cnt=0
     fname=/dev/rzwb_lv15_106
     fno=61 lblksz=16384 fsiz=983039
    ----------------------------------------
    SO: 0x700000d7c53d2b8, type: 10, owner: 0x700000db18e19c0, flag: INIT/-/-/0x00 if: 0x1 c: 0x1
     proc=0x700000db18e19c0, name=FileOpenBlock, file=ksfd.h LINE:6448 ID:, pg=0
    (FOB) 700000d7c53d2b8 flags=2050 fib=700000da39dfd60 incno=0 pending i/o cnt=0

Note:
尝试trace进程调用均hang. 下一步使用PS检查进程的STAT列，如果进程状态为D，并且如strace,truss, tuss等工具无输出时，我们是无法kill或恢复进程，表未进程在wait内核调用，KILL 命令无权终止，只能重启操作系统恢复。通常数据库出现该问题时，如果系统资源充足时可能是I/O请求挂起引起， 会一直等待下去。

$ ps aux | grep ” D”

SQL> @sw "select sid from v$session where status='KILLED'"

    SID STATE   EVENT                                          SEQ# SEC_IN_WAIT P1                  P2                  P3               
------- ------- ---------------------------------------- ---------- ----------- ------------------- ------------------- -----------------
   6959 WAITING db file sequential read                       26745       39163 file#= 621          block#= 266395      blocks= 1
   2284 WAITING db file sequential read                       38595       39163 file#= 655          block#= 569108      blocks= 1
   5276 WAITING db file sequential read                       19993       39163 file#= 764          block#= 1948676     blocks= 1

SQL> @sw "select sid from v$session where status='KILLED'"

    SID STATE   EVENT                                          SEQ# SEC_IN_WAIT P1                  P2                  P3               
------- ------- ---------------------------------------- ---------- ----------- ------------------- ------------------- -----------------
   6959 WAITING db file sequential read                       26745       39165 file#= 621          block#= 266395      blocks= 1
   2284 WAITING db file sequential read                       38595       39165 file#= 655          block#= 569108      blocks= 1
   5276 WAITING db file sequential read                       19993       39165 file#= 764          block#= 1948676     blocks= 1

SQL> @sw "select sid from v$session where status='KILLED'"

    SID STATE   EVENT                                          SEQ# SEC_IN_WAIT P1                  P2                  P3               
------- ------- ---------------------------------------- ---------- ----------- ------------------- ------------------- -----------------
   6959 WAITING db file sequential read                       26745       39166 file#= 621          block#= 266395      blocks= 1
   2284 WAITING db file sequential read                       38595       39166 file#= 655          block#= 569108      blocks= 1
   5276 WAITING db file sequential read                       19993       39166 file#= 764          block#= 1948676     blocks= 1

SQL> select name from v$datafile where file#=621;
NAME
--------------------------------------------------------------
/dev/rzwb_lv15_330

SQL> select name from v$datafile where file#=655;
NAME
--------------------------------------------------------------
/dev/rzwb_lv30_411

SQL> select name from v$datafile where file#=764;
NAME
--------------------------------------------------------------
/dev/rzwb_lv30_525

[oracle@weejar1:/home/oracle]$ lslv zwb_lv15_330
0516-1201 lslv: Warning: Volume group zwb_datavg09 is locked. This command
        will continue retries until lock is free.  If lock is inadvertent
        and needs to be removed, execute 'chvg -u zwb_datavg09'.

[oracle@weejar1:/home/oracle]$ errpt
IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
4B436A3D   0613172517 T H fscsi1         LINK ERROR
4B436A3D   0613172517 T H fscsi1         LINK ERROR
4B436A3D   0613172417 T H fscsi1         LINK ERROR
4B436A3D   0613172417 T H fscsi1         LINK ERROR
4B436A3D   0613172417 T H fscsi1         LINK ERROR
4B436A3D   0613172317 T H fscsi1         LINK ERROR

Note:
从数据库等待事件的数据文件查起，确认了该LV及所在的VG已被锁定，同步有4个VG受到影响， 第二个节点存储（11个VG）正常。 从节点一主机日志及系统确认该主机存储目前有4条存储链路， 其中1条存储链路故障（sicsi1 hba卡，且使用POWER存储命令查看该链路有3个读写I/O请求未完成）引起读写异常进而引发VG被锁，进程处于等待I/O资源的核心态而不是用户态，不能接受任何外部信号(包括KILL)。

当然这种情况下最安全的方法是重启主机， 当然这个情况数据库需要ABORT才能停下来，且DB INSTANCE和CRS LISTENER停掉后，KILLED状态的OS进程（LOCAL=NO）还一直存在，重启操作系统后，链路和数据库恢复正常。

env: 11.2.0.3 RAC Dataguard ON Exadata x2, a standby instance crash, alert log show ora-600 internal error.

# db alert log

2017-06-12 08:39:10.148000 +08:00
Archived Log entry 70956 added for thread 1 sequence 247664 ID 0x94a6c6e9 dest 1:
2017-06-12 10:04:32.837000 +08:00
Media Recovery Waiting for thread 2 sequence 236820 (in transit)
Recovery of Online Redo Log: Thread 2 Group 32 Seq 236820 Reading mem 0
  Mem# 0: +DATA/rptstby/onlinelog/group_32.556.906801599
2017-06-12 10:11:11.510000 +08:00
Errors in file /oracle/app/oracle/diag/rdbms/rptstby/oradb1/trace/oradb1_dbwa_14567.trc  (incident=224233):
ORA-00600: internal error code, arguments: [H_MK_WRITING_CR_BG], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/rptstby/oradb1/incident/incdir_224233/oradb1_dbwa_14567_i224233.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2017-06-12 10:11:12.599000 +08:00
Dumping diagnostic data in directory=[cdmp_20170612101112], requested by (instance=1, osid=14567 (DBWA)), summary=[incident=224233].
Errors in file /oracle/app/oracle/diag/rdbms/rptstby/oradb1/trace/oradb1_dbwa_14567.trc:
ORA-00600: internal error code, arguments: [H_MK_WRITING_CR_BG], [], [], [], [], [], [], [], [], [], [], []
DBWa (ospid: 14567): terminating the instance due to error 471
System state dump requested by (instance=1, osid=14567 (DBWA)), summary=[abnormal instance termination].
System State dumped to trace file /oracle/app/oracle/diag/rdbms/rptstby/oradb1/trace/oradb1_diag_14516.trc
2017-06-12 10:11:13.759000 +08:00
ORA-1092 : opitsk aborting process
License high water mark = 66
2017-06-12 10:11:17.917000 +08:00
Instance terminated by DBWa, pid = 14567
USER (ospid: 10154): terminating the instance
Instance terminated by USER, pid = 10154
2017-06-12 10:11:19.223000 +08:00
Starting ORACLE instance (normal)


adrci> show trace /oracle/app/oracle/diag/rdbms/rptstby/oradb1/trace/oradb1_dbwa_14567.trc
Output the results to file: /tmp/utsout_22055_13995_5.ado
/oracle/app/oracle/diag/rdbms/rptstby/oradb1/trace/oradb1_dbwa_14567.trc
----------------------------------------------------------
LEVEL PAYLOAD
----- ------------------------------------------------------------------------------------------------------------------------------------------------
      Trace file /oracle/app/oracle/diag/rdbms/rptstby/oradb1/trace/oradb1_dbwa_14567.trc
      Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
      With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
      Data Mining and Real Application Testing options
      ORACLE_HOME = /oracle/app/oracle/product/11.2.0.3/dbhome_1
      System name:      Linux
      Node name:        qdexa1db01.anbob.com
      Release:  2.6.32-400.35.2.el5uek
      Version:  #1 SMP Thu Feb 13 15:00:30 PST 2014
      Machine:  x86_64
      Instance name: oradb1
      Redo thread mounted by this instance: 1
      Oracle process number: 29
      Unix process pid: 14567, image: oracle@qdexa1db01.anbob.com (DBWA)


      *** 2017-06-12 10:11:11.522
      *** SESSION ID:(632.1) 2017-06-12 10:11:11.522
      *** CLIENT ID:() 2017-06-12 10:11:11.522
      *** SERVICE NAME:(SYS$BACKGROUND) 2017-06-12 10:11:11.522
      *** MODULE NAME:() 2017-06-12 10:11:11.522
      *** ACTION NAME:() 2017-06-12 10:11:11.522

1>     ***** Incident 224233 created, dump file:  *****
       /oracle/app/oracle/diag/rdbms/rptstby/oradb1/incident/incdir_224233/oradb1_dbwa_14567_i224233.trc
1<     ***** incident_file *****
1>     ***** Error Stack *****
       ORA-00600: internal error code, arguments: [H_MK_WRITING_CR_BG], [], [], [], [], [], [], [], [], [], [], []
1<     ***** Error Stack *****
      error 471 detected in background process
      ORA-00600: internal error code, arguments: [H_MK_WRITING_CR_BG], [], [], [], [], [], [], [], [], [], [], []
      kjzduptcctx: Notifying DIAG for crash event
      ----- Abridged Call Stack Trace -----
      ksedsts()+461<-kjzdssdmp()+267<-kjzduptcctx()+232<-kjzdicrshnfy()+53<-ksuitm()+1332<-ksbrdp()+3344<-opirip()+623<-opidrv()+603
      <-sou2o()+103<-opimai_real()+266<-ssthrdmain()+252<-main()+201<-__libc_start_main()+244<-_start()+36
      ----- End of Abridged Call Stack Trace -----

      *** 2017-06-12 10:11:12.765
      DBWa (ospid: 14567): terminating the instance due to error 471
      ksuitm: waiting up to [5] seconds before killing DIAG(14516)
      
adrci> show trace  /oracle/app/oracle/diag/rdbms/rptstby/oradb1/incident/incdir_224233/oradb1_dbwa_14567_i224233.trc
Output the results to file: /tmp/utsout_22055_13995_7.ado
/oracle/app/oracle/diag/rdbms/rptstby/oradb1/incident/incdir_224233/oradb1_dbwa_14567_i224233.trc
----------------------------------------------------------
LEVEL PAYLOAD
----- ------------------------------------------------------------------------------------------------------------------------------------------------
      Dump file /oracle/app/oracle/diag/rdbms/rptstby/oradb1/incident/incdir_224233/oradb1_dbwa_14567_i224233.trc
      Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
      With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
      Data Mining and Real Application Testing options
      ORACLE_HOME = /oracle/app/oracle/product/11.2.0.3/dbhome_1
      System name:      Linux
      Node name:        qdexa1db01.anbob.com
      Release:  2.6.32-400.35.2.el5uek
      Version:  #1 SMP Thu Feb 13 15:00:30 PST 2014
      Machine:  x86_64
      Instance name: oradb1
      Redo thread mounted by this instance: 1
      Oracle process number: 29
      Unix process pid: 14567, image: oracle@qdexa1db01.anbob.com (DBWA)


      *** 2017-06-12 10:11:11.522
      *** SESSION ID:(632.1) 2017-06-12 10:11:11.522
      *** CLIENT ID:() 2017-06-12 10:11:11.522
      *** SERVICE NAME:(SYS$BACKGROUND) 2017-06-12 10:11:11.522
      *** MODULE NAME:() 2017-06-12 10:11:11.522
      *** ACTION NAME:() 2017-06-12 10:11:11.522

      Dump continued from file: /oracle/app/oracle/diag/rdbms/rptstby/oradb1/trace/oradb1_dbwa_14567.trc
1>     ***** Error Stack *****
       ORA-00600: internal error code, arguments: [H_MK_WRITING_CR_BG], [], [], [], [], [], [], [], [], [], [], []
1<     ***** Error Stack *****
1>     ***** Dump for incident 224233 (ORA 600 [H_MK_WRITING_CR_BG]) *****
2>      ***** Beginning of Customized Incident Dump(s) *****
        BH (0x867cbf798) file#: 1093 rdba: 0x130caad3 (76/830163) class: 1 ba: 0x865298000
          set: 59 pool: 3 bsz: 16384 bsi: 0 sflg: 1 pwc: 0,0
          dbwrid: 10 obj: 2541940 objn: -1 tsn: 11 afn: 1093 hint: f
          hash: [0x227d2d950,0x707b5dcd0] lru: [0x60fcf9188,0x507bd2408]
          lru-flags: hot_buffer
          ckptq: [NULL] fileq: [NULL] objq: [0x2dfb2e1b0,0x9e07cfa20] objaq: [0x60fcf91c0,0x507bd2440]
          use: [0xa45bfc838,0xa45bfc838] wait: [NULL]
          st: MEDIA_WRITING md: SHR rsop: 0x9fd89d2e0 tch: 77 atm: 3732566919,3577170450 rlscn: 0x0dec.039476d9
          flags: only_sequential_access block_written_once affinity_lock
        Dump of buffer cache at level 10 for tsn=11 rdba=319597267
        BH (0x707b5dc18) file#: 1093 rdba: 0x130caad3 (76/830163) class: 1 ba: 0x7017a4000
          set: 59 pool: 3 bsz: 16384 bsi: 0 sflg: 1 pwc: 0,0
          dbwrid: 10 obj: 2541940 objn: -1 tsn: 11 afn: 1093 hint: f
          hash: [0x867cbf850,0xa59dea248] lru: [0x13fbfad88,0x567c1d288]
          obj-flags: object_ckpt_list
          ckptq: [0x13fbfac98,0x567c1d198] fileq: [0x567c1d1a8,0x13fbfaca8] objq: [0x13fbfadb0,0x567c1d2b0] objaq: [0x13fbfadc0,0x567c1d2c0]
          st: MEDIA_RCV md: NULL rsop: 0x9fd89d2e0 tch: 1 atm: 3300241033,3577178658 rlscn: 0x0dec.039477d6
          flags: buffer_dirty only_sequential_access block_written_once affinity_lock
          buffer tsn: 11 rdba: 0x130caad3 (76/830163)
          scn: 0x0dec.039477d7 seq: 0x01 flg: 0x04 tail: 0x77d70601
          frmt: 0x02 chkval: 0xa4a8 type: 0x06=trans data
        Hex dump of block: st=0, typ_found=1
      
       ----- Abridged Call Stack Trace -----
       ksedsts()+461<-kjzdssdmp()+267<-kjzduptcctx()+232<-kjzdpcrshnfy()+43<-kstdmp()+282<-dbkedDefDump()+10477<-ksedmp()+41<-ksfdmp()+69<-dbgexPhaseI
       I()+1764<-dbgexExplicitEndInc()+755<-dbgeEndDDEInvocationImpl()+772<-dbgeEndDDEInvocation()+52<-kcbbic2()+9165<-kcbbic1()+1216
       <-kcbbiop()+1042<-kcbbdrv()+733<-ksbabs()+771<-ksbrdp()+971<-opirip()+623<-opidrv()+603
       ----- End of Abridged Call Stack Trace -----
       

Search in MOS, that’s easy to found a node hit a bug.
Cause

This issue was further investigated and fixed in unpublished BUG 16915139.

Solution

Apply Patch 16915139 if it is available for your platform and version.The final fix is included in RDBMS Versions: 11.2.0.4, 12.1.0.2, 12.2.

ASM DISK 的Discovery PATH

ASM实例的ASM_DISKSTRING初始化参数使用一个逗号分割的字符串限制ASM实例发现的DISK可以用于ASM DISK, 该字符串支持通配符如使用星号（*）表示LIKE，只有匹配了该字符串中的路径，ASM disk才会被发现；同样支持如果问号(?)为该字符串的第一个字符时，像sqlplus中一样表示的是ORACLE_HOME的路径。注意同一DISK不会因为路径多次匹配而显示多次。

ASM_DISKSTRING的格式依赖于使用的是ASM library还是操作系统路径，如使用了ASM library,在安装和配置时asm_diskstring只需要配置ASM路径如ORCL:*路径即可，不要再配置ASMLIB中指定的映射前的DISK的路径如/dev/oracleasm/asmdisk*, 如果配置了会导致在v$asm_disk显示2种类型的路径，可以使用ASMCMD> dsget 查看，并使用ASMCMD> dsset –profile ‘ORCL:*’修正GPNPprofile中的配置。

ASM_DISKSTRING的默认值是NULL,NULL值表示ORACLE ASM会查找当前系统中默认路径中可以读写访问的所有磁盘。默认的路径是平台中指定的，依赖于操作系统，已知如下：

----------------------------------------------------------|
|Platform	                       |Discovery Path    |
-----------------------------------------------------------
|AIX 	                               |/dev/rhdisk*      |
-----------------------------------------------------------
|HP-UX 	                               |/dev/rdisk/*      |
-----------------------------------------------------------
|Solaris 	                       |/dev/rdsk/*       |
-----------------------------------------------------------
|WINDOWS 	                       |\\.\ORCLDISK*     |
-----------------------------------------------------------
|Linux 	                               |/dev/raw/*        |
-----------------------------------------------------------
|On Linux if using ASMLib Kernel Driver| ORCL:*           |
----------------------------------------------------------|

除非同一个ASM instance集群中的每个ASM实例Node都能通过自己的ASM_DISKSTRING发现,否则磁盘将不可用。 在同一ASM集群中所有节点的磁盘名不一定非要相同，但是必须都要被发现，因为ASM DG信息在DISK的磁盘头中， 可以动态的改变ASM_DISKSTRING参数增加新存储。

如何发现ASM DISK？
当ASM 实例启动时，会扫描并审核在参数ASM_DISKSTRING指定的路径中的所有磁盘，是的，所有。除了在实例启动时还有如下情况同样也会发生读取ASM_DISKSTRING查找磁盘：
* MOUNT一个ASM磁盘组时
* online一个ASM磁盘组中的磁盘时
* 创建一个ASM磁盘组时
* 增加一个ASM 磁盘到到ASM磁盘组时
* resize 一个asm 磁盘大小时
* 查询 v$asm_diskgroup 或者 v$asm_disk视图时
注： 以上行为不限于SQL或asmcmd或EM ,GC，CC等其它等同工具。

当ASM instance成功发现了ASM DISK后，会出现在V$ASM_DISK视图中，同时在DISK盘头上会记录磁盘的状态信息，可以在视图中或使用kfed查看磁盘的状态(kfdhdb.hdrsts)。

|——————————————————————————|
|状态                         |描述                                                                             |
——————————————————————————–
|MEMBER              |属于当前diskgroup的disk                                       |
——————————————————————————–
|FORMER              |这个disk以前属于一个diskgroup,现在这个diskgroup被删除了       |
——————————————————————————–
|CANDIDATE        |当使用裸设备，一个新的可以被diskgroup所用的disk               |
——————————————————————————–
|PROVISIONED    |特定平台的功能提供的可用asmdisk，如WIN asmtool或LINUX asmlib  |
——————————————————————————-|

ASM DISK发现的规则

* asm 最多可以发现1万个asm disk(10G,11g,12c R1),如果ASM disk超过了1万个，也是只显示前1万个。
* asm 只会发现磁盘分区，不会发现包含分区表中的分区
* 对于ASM DISK状态为condidate,provisioned,former的DISK不用force选项加入DISKGROUP, 而对于foreign状态只能使用force选项填加。
* 对于member状态的disk,如果不属于当前的任何mount的diskgroup,可以使用force强制填加。
* 同一个磁盘不要显示多个路径，如使用多路径软件时只显示伪路径。
* 如果使用了copy disk,可能会导致多个asm disk同一个磁盘头，填加时会失败
对于ASM的限制请查看#370921.1 ASM – Scalability and Limits

提升ASM DISK发现的时间
ASM的asm_diskstring参数是依赖操作系统值，用于限制ORACLE ASM查找ASM磁盘的路径集。当一个磁盘被加进DISKGROUP时， 每个ASM实例mounted的diskgroup中的磁盘路径必须匹配自己实例中的asm_diskstring路径。 多数情况时使用asm_diskstring默认的值足够， 但是使用更严格的值可能会减少oracle asm扫描磁盘时不必要的时间。提升asm diskgroup mount时间或add disk时的时间。 默认的asm_diskstring值(NULL)可能不会发现所有的磁盘， 比如如果使用了ASMLIB或第三方的多路径软件时， 必须使用asm_diskstring;

在ORACLE ASM 10gR1时查询v$asm_disk和v$asm_diskgroup是一个排它操作，因为每一次执行都要执行disk discovery.为了减少开销，允许轻量级的访问数据集，ORACLE ASM 10g R2及以后引入了新的视图，v$asm_disk_stat 和v$asm_diskgroup_stat，查询这两个视图数据来源内存并且不需要执行disk discovery. 所以在OEM中也通常使用该视图。

删除asm disk并且不想在v$asm_disk中看到，可以在drop diskgroup或drop disk后修改该disk的owner和权限禁止ASM访问，如果删除asm_diskstring中的部分disk也可以动态的修改asm_diskstring参数。

Reference  Esteban D. Bernal [improving oracle asm discovery time best practices]

ORACLE 12c有些小特性非常的实用，如Oracle 12c New Feature: Last Login Time for Non-Sys Users，可以列出非SYS用户的最后登录时间，该数据可以做为清理用户里的依据，同时前段时间应对安全检查， 数据库中扫出了一些弱口令，需要清理一部分长期不登录的用户或找到对应的责任人才可以修改密码，以评估修改修改对系统的影响。 如果找出哪些用户是ORACLE 系统用户在12C之前还是相对麻烦一些，因为我们可能知道像sys, system这些系统默认创建的用户，其它如果安装时选用较多的DB option时，往往不容易查找自动在创建数据库里脚本中创建的哪些用户。

在12c 中dba_user 字典视图引入了ORACLE_MAINTAINED 字段， 当数据库安装options时，oracle自己创建的用户会赋值为”Y”,我们自己创建的用户默认为”N”， 所以像之前的需求在12c中就变的简单，即可以使用LAST_LOGIN判断数据库的最后一次登录，又可以使用ORACLE_MAINTAINED =’N’ 查找我们创建的用户。

同时也可以查出这些schema是否可以exp,expdp,logical standby… , 下面是在我的12.2 CDB测试环境中查询结果

  SQL> SELECT created,
         username,
         oracle_maintained,
         common,
         no_exp,
         no_expdp,
         no_sby,
         default_password,
         sysaux,
         occupant_desc
    FROM dba_users
         LEFT OUTER JOIN (SELECT DISTINCT name username, 'Y' no_expdp
                            FROM sys.ku_noexp_tab
                           WHERE obj_type = 'SCHEMA')
            USING (username)
         LEFT OUTER JOIN
         (SELECT DISTINCT name username, 'Y' no_exp FROM sys.exu8usr)
            USING (username)
         LEFT OUTER JOIN (SELECT DISTINCT name username, 'Y' no_sby
                            FROM SYSTEM.logstdby$skip_support
                           WHERE action IN (0, -1))
            USING (username)
         LEFT OUTER JOIN
         (SELECT DISTINCT user_name username, 'Y' default_password
            FROM sys.default_pwd$)
            USING (username)
         LEFT OUTER JOIN
         (  SELECT schema_name username,
                   'Y' sysaux,
                   DECODE (COUNT (*), 1, MIN (occupant_desc)) occupant_desc
              FROM v$sysaux_occupants
          GROUP BY schema_name)
            USING (username)
ORDER BY created, username;

别外也整理了一些在12c之前的版本中可能会创建的SCHEMA及用途， 如下:

默认数据库中的数据文件都是未加密的，或者说是可以直接使用如OS strings命令读取数据文件，明文显示的部分内容，如有些敏感信息都是明文存储，这样就可以直接从存储中读取此类数据如手机号、身份证、信用卡、或密码。Tablespace encryption 使用的是TDE技术， Transparent Data Encryption (以下简称TDE) 是Oracle Advanced Security Option中引入从存储层对数据文件加密的技术 . TDE把存储密钥存储在数据库外部，加密表空间不会给数据存储带来额外的空间。

TDE是从Oracle 10g R2版本时引入，Transparent Tablespace Encryption对表空间的加密是在ORACLE 11G R1版本引入，但是只支持对新建表空间时指定加密选项，虽然可以先创建加密表空间逐个移入表对象对其加密，但是那样对于Huge数据库就意味着申请很多的时间维护窗口，从12C R2版本引入了online tablespace encryption的新特性，同时还引入了新的管理密钥的方法，下面是我对12.2 TDE新特性的一些总结。

以下Demo环境中是12.2 的多租户环境，TDE对是否为多租户都是支持的， ORACLE表示非多租户架构已经deprecated, 当然目前还不是Desupported，但是多租户已是趋势，对于不存在多PDB需求的用户，可以使用1CDB和1PDB的组合，同样不需要购买Multitenant license。

12.2 TDE 特性

12.2 TDE引入了新的密钥管理方法，之前的版本中一直叫Wallet(钱包)，在12C版本中叫Keystore(密钥库)。Keystore可以分软件或硬件，关于硬件的不在这篇的描述范围可以查看这里。 过去版本中的PKI已经deprecated，引入 ADMINISTER KEY MANAGEMENT 新的命令替换过去的 ALTER SYSTEM SET ENCRYPTION WALLET 和 ALTER SYSTEM SET ENCRYPTION KEY命令；在多租户环境中ROOT container (CDB$ROOT)必须有1个打开的Keystore (Wallet)和1个可用的Master Encryption Key；同时12.2 中online tablespace encryption功能就意味着可以在不停业务的情况下，对已存在的表空间进行加密，原理应该和ONLINE tablespace Move一样，转换中DBWR写两份，完成后remove原文件，切到新文件；另外在12.2中可以加密整个数据库所有表空间，如SYSTEM、SYSAUX 、TEMP 、 UNDO tablespace同样支持，但是加密这些系统表空间不能指定 encryption key，但是如果system等系统表空间已加密，那KEYSOTRY就不能再关闭了，否则会报ORA-28439. 但是注意ORACLE是不建议对这些系统表空间加密的，除非有业务数据在这些表空间或需要有些PL/SQL对象中的信息. 注意于Temp 表空间的加密，也只能使用增加新的删除旧的方式。

TDE在公有云不是一个OPTION,另外有个UNDOCUMENT参数encrypt_new_tablespace在DBaaS Cloud环境中默认会加密码新建的表空间，参数描述如下:

The ENCRYPT_NEW_TABLESPACES initialization parameter controls default encryption of new tablespaces. In Database as a Service databases, this parameter is set to CLOUD_ONLY.
Any tablespace created will be transparently encrypted with the AES128 algorithm unless a different algorithm is specified on the ENCRYPTION clause.

Step 1: Setup a Keystore（密钥库） Location:

密钥库存储是的TDE的密钥信息，路径可以在sqlnet.ora中ENCRYPTION_WALLET_LOCATION参数指定，查询Keystore的方法顺序是:

if  ENCRYPTION_WALLET_LOCATION in sqlnet.ora
else WALLET_LOCATION in sqlnet.ora
else $ORACLE_BASE/admin/DB_UNIQUE_NAME/wallet or $ORACLE_HOME/admin/DB_UNIQUE_NAME/wallet

Oracle 建议将 Oracle Keysotre(Wallet) 放置在 $ORACLE_BASE 目录树之外，以避免意外将钱包与加密数据一起存储在备份磁带上, 如果是RAC 建议放在ASM OR  ACFS 共享存储上。Demo使用的本地文件系统。

# mkdir -pv /etc/ORACLE/anbob/encryption_keystore
# cd /etc
# chown -R oracle:oinstall ORACLE
# chmod -R 700 ORACLE
编辑"$ORACLE_HOME/network/admin/sqlnet.ora"文件, 增加下面的记录：

ENCRYPTION_WALLET_LOCATION =
(SOURCE =(METHOD = FILE)
(METHOD_DATA =(DIRECTORY = /etc/ORACLE/anbob/encryption_keystore/))
)

如果使用ASM 路径如(DIRECTORY=+DATA/PRODCDB/WALLET)

Step 2: Create a Keystore（密钥库）:

Keystore在12C的多租户架构下必须创建在root container, 创建可以使用sysdba或syskm . software keystore因为是file 类型，一旦创建就会在localtion对应的目录中创建一个ewallet.p12的文件，Keystore的状态可 查询v$encryption_wallet

oracle@anbob ~]$ sqlplus / as  syskm
SQL*Plus: Release 12.2.0.1.0 Production on Thu Jul 27 15:56:51 2017
Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to:
Oracle Database 12c EE Extreme Perf Release 12.2.0.1.0 - 64bit Production

SQL> ADMINISTER KEY MANAGEMENT CREATE KEYSTORE '/etc/ORACLE/anbob/encryption_keystore/' IDENTIFIED BY "Anbob.com";
keystore altered.

---- 
---- keystore_location : Keystore Location mentioned in SQLNET.ORA file
---- keystore_password : Password for opening the Keystore

SQL> host ls -l /etc/ORACLE/anbob/encryption_keystore
total 4
-rw------- 1 oracle oinstall 2408 Jul 27 15:57 ewallet.p12

SQL> col WRL_PARAMETER for a45
SQL> select * from v$encryption_wallet;

WRL_TYPE    WRL_PARAMETER                                 STATUS     WALLET_TYPE   WALLET_OR FULLY_BAC     CON_ID
----------- --------------------------------------------- ---------- ------------- --------- --------- ----------
FILE        /etc/ORACLE/anbob/encryption_keystore/        CLOSED     UNKNOWN       SINGLE    UNDEFINED          1

Step 3: Open the Keystore（密钥库）:

需要在root container打开密钥库，如果没有使用CONTAINER=ALL 只影响当前的container.状态发生改变

SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY "Anbob.com" CONTAINER=ALL;
keystore altered.

SQL> select * from v$encryption_wallet;

WRL_TYPE             WRL_PARAMETER                                 STATUS                         WALLET_TYPE          WALLET_OR FULLY_BAC     CON_ID
-------------------- --------------------------------------------- ------------------------------ -------------------- --------- --------- ----------
FILE                 /etc/ORACLE/anbob/encryption_keystore/        OPEN_NO_MASTER_KEY             PASSWORD             SINGLE    UNDEFINED          1

如果CLOSE 使用
ADMINISTER KEY MANAGEMENT SET KEYSTORE CLOSE IDENTIFIED BY "Anbob.com" CONTAINER=ALL;

Step 4: Create TDE Master Encryption Key:

打开密钥库后还必须要在root container和每个PDB创建一个主密钥，可以使用CONTAINER=ALL一条命令创建，如果没带还需要在每个PDB中创建，创建后可以在 V$ENCRYPTION_KEYS view查询，同时密钥库状态改变，密钥一定要保管好，每次修改记的备份和异地保存

语法：
ADMINISTER KEY MANAGEMENT SET KEY [USING TAG 'tag'] IDENTIFIED BY password [WITH BACKUP [USING 'backup_identifier']] [CONTAINER = ALL | CURRENT];
--- 
--- tag : It is the associated attributes and infomration that we define.
--- password : It is the mandatory Keystore password defined while creating the Keystore.
--- WITH BACKUP : This optional parameter creates a backup of the Keystore. We must use this option for a password based Keystore. Optionally,we can use the 'USING' clause to add a brief description about the backup.
--- CONTAINER: This parameter is for use in a multi-tenant environment (CDB). We can use ALL option for creating Master Encryption Key in all associated PDBs and CURRENT for the current PDB or ROOT (CDB$ROOT) container. If omitted the default is the CURRENT container.

SQL> ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY "Anbob.com" WITH BACKUP CONTAINER=ALL;
keystore altered.

SQL> SELECT * FROM v$encryption_wallet;
WRL_TYPE             WRL_PARAMETER                                 STATUS                         WALLET_TYPE          WALLET_OR FULLY_BAC     CON_ID
-------------------- --------------------------------------------- ------------------------------ -------------------- --------- --------- ----------
FILE                 /etc/ORACLE/anbob/encryption_keystore/        OPEN                           PASSWORD             SINGLE    NO                 1

SQL> SELECT con_id, key_id FROM v$encryption_keys;
    CON_ID KEY_ID
---------- ------------------------------------------------------------------------------
         3 AftVI2YmxE/Uv9MrhgEDqBMAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
         1 Aco82tv4Rk9rv4/L7JSWusMAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
如果不创建密钥，加密时会提示
SQL> create tablespace user_enc datafile '/u02/app/oracle/oradata/anbob/pdbanbob/user_enc.dbf' size 10m encryption using 'AES128' encrypt;
create tablespace user_enc datafile '/u02/app/oracle/oradata/anbob/pdbanbob/user_enc.dbf' size 10m encryption using 'AES128' encrypt
*
ERROR at line 1:
ORA-28374: typed master key not found in wallet

Step 5: Encrypt the Tablespace

在创建了密钥后并打开了密钥库后，下面可以测试加密表空间了,在12.2中可以在创建表空间时加密，同时也可以对已存在的表空间在线加密解密，首先创建个不加密表空间，我们尝试使用strings从数据库外读取数据，再加密后尝试。

SQL> alter session set container=pdbanbob;
Session altered.

SQL> create tablespace user_enc datafile '/u02/app/oracle/oradata/anbob/pdbanbob/user_enc.dbf' size 10m ;
Tablespace created.

SQL> select tablespace_name,status,contents,encrypted from dba_tablespaces;

TABLESPACE_NAME                STATUS    CONTENTS              ENC
------------------------------ --------- --------------------- ---
SYSTEM                         ONLINE    PERMANENT             NO
SYSAUX                         ONLINE    PERMANENT             NO
UNDOTBS1                       ONLINE    UNDO                  NO
TEMP                           ONLINE    TEMPORARY             NO
USERS                          ONLINE    PERMANENT             NO
LOWER                          ONLINE    PERMANENT             NO
USER_ENC                       ONLINE    PERMANENT             NO

SQL> @ls USER_ENC

TABLESPACE_NAME                   FILE_ID FILE_NAME                                           EXT         MB      MAXSZ
------------------------------ ---------- --------------------------------------------------- --- ---------- ----------
USER_ENC                               55 /u02/app/oracle/oradata/anbob/pdbanbob/user.dbf     NO          10

SQL> create table t_user (id int, name varchar2(10), phone varchar2(100)) tablespace USER_ENC;
Table created.

begin
 for i in 1..100 loop
 insert into t_user values (i,'anbob'||i,'138'||lpad(i,8,'0'));
 dbms_lock.sleep(5);
 end loop;
 commit;
end;
/

[oracle@anbob admin]$  strings /u02/app/oracle/oradata/anbob/pdbanbob/user.dbf  
}|{z
Yt"NANBOB
USER_ENC
AAAAAAAA
anbob100
13800000100,
anbob99
13800000099,
anbob98
13800000098,
anbob97
13800000097,
anbob96
13800000096,
anbob95
13800000095,
anbob94
13800000094,
...

Note:
在加密前可以直接从操作系统读取到数据库内的敏感信息。下面在新开一个会话，在不停的insert数据，模仿在线应用，另一个会话直接对存在的表空间加密。

SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY "Anbob.com" CONTAINER=ALL;
keystore altered.

begin
 for i in 101..150 loop
 insert into t_user values (i,'anbob'||i,'138'||lpad(i,8,'0'));
 dbms_lock.sleep(5);
 end loop;
 commit;
end;
/

ONLINE  ENCRYPT TABLESPACE

SQL> ALTER TABLESPACE USER_ENC ENCRYPTION ONLINE using 'aes192'  encrypt file_name_convert = ('user.dbf','user_enc.dbf');
Tablespace altered.

加密算法可以使用下面的：
3DES168
AES128
AES192 (default)
AES256

SQL>  select tablespace_name,status,contents,encrypted from dba_tablespaces;
TABLESPACE_NAME                STATUS    CONTENTS              ENC
------------------------------ --------- --------------------- ---
SYSTEM                         ONLINE    PERMANENT             NO
SYSAUX                         ONLINE    PERMANENT             NO
UNDOTBS1                       ONLINE    UNDO                  NO
TEMP                           ONLINE    TEMPORARY             NO
USERS                          ONLINE    PERMANENT             NO
LOWER                          ONLINE    PERMANENT             NO
USER_ENC                       ONLINE    PERMANENT             YES

SQL> select * from t_user;

        ID NAME       PHONE
---------- ---------- ---------------------------------------------------
         1 anbob1     13800000001
         2 anbob2     13800000002
         3 anbob3     13800000003
...
       142 anbob142   13800000142
       143 anbob143   13800000143
       144 anbob144   13800000144
       145 anbob145   13800000145
       146 anbob146   13800000146
       147 anbob147   13800000147
       148 anbob148   13800000148
       149 anbob149   13800000149
       150 anbob150   13800000150

[oracle@anbob ~]$ strings /u02/app/oracle/oradata/anbob/pdbanbob/user_enc.dbf|head
}|{z
Yt"NANBOB
USER_ENC
k+4d
U#f&
R\kBy
'7&A
qleT
I(U"
NaHS

ONLINE DECRYPT TABLESPACE

SQL>  ALTER TABLESPACE USER_ENC ENCRYPTION ONLINE DECRYPT file_name_convert = ('user_enc.dbf', 'user.dbf');
Tablespace altered.

Note:

加密后的表空间，数据文件已经在存储层加密，无法再使用strings读取数据，但在密钥库打开的情况下，应用可以透明的读取，同时是在线加密，对当前业务不需要中断， 因当前的表空间较小所以时间很快，如果几百G的空间可以需要几小时，所以不要在业务忙和对该表空间写操作较多时在在线加密的操作。 如果不是OMF模式管理的数据文件需要使用file_name_convert转换前后文件名。据其它用户测试在线加密表空间中原SQL最终用户响应时间的性能影响为 4% 至 8%，CPU 利用率提高了 1% 至 5%。 对整个实例的性能影响可能在50%左右，相关的等待事件主要是I/O类，时间mode中可以看到Tablespace encryption elapsed time较高。

Setup Auto logon KEYSTORE

我们可以创建自动登录或本地登录密钥库; 以避免每次手动打开Keystore。一旦创建了自动登录的密钥会在密钥库路径下创建’cwallet.sso’文件。要使密钥库自动打开，请使用以下命令:

ADMINISTER KEY MANAGEMENT CREATE [LOCAL] AUTO_LOGIN KEYSTORE FROM KEYSTORE 'keystore_location' IDENTIFIED BY keystore_password;

---  
--- LOCAL: enables us to create a local auto-login keystore. Otherwise, skip this clause if you want the keystore to be accessible by other computers.
--- keystore_location: Location of the Keystore 
--- keystore_password: password of the password-based keystore for which we want to create the auto-login keystore

SQL> ADMINISTER KEY MANAGEMENT CREATE LOCAL AUTO_LOGIN KEYSTORE FROM KEYSTORE '/etc/ORACLE/anbob/encryption_keystore/' IDENTIFIED BY "Anbob.com";
keystore altered.

SQL> ho ls -lrt /etc/ORACLE/anbob/encryption_keystore/
total 20
-rw------- 1 oracle oinstall 2408 Jul 27 16:10 ewallet_2017072708103106.p12
-rw------- 1 oracle oinstall 5328 Jul 27 16:10 ewallet.p12
-rw------- 1 oracle oinstall 5379 Aug  1 15:21 cwallet.sso

SQL> SELECT * FROM v$encryption_wallet

WRL_TYPE             WRL_PARAMETER                  STATUS                         WALLET_TYPE          WALLET_OR FULLY_BAC     CON_ID
-------------------- ------------------------------ ------------------------------ -------------------- --------- --------- ----------
FILE                                                OPEN                           LOCAL_AUTOLOGIN      SINGLE    NO                 3

Export\Import encryption keys

迁移PDB包含加密表空间时需要导出密钥，同时使用一个密码再次加密密钥文件，这里的密码是mySecre,然后在新的CDB像上面的方法创建密钥库，再使用密码导入密钥。

ADMINISTER KEY MANAGEMENT EXPORT ENCRYPTION KEYS WITH SECRET "mySecret" TO '/tmp/export.p12' IDENTIFIED BY "Anbob.com";

# CREATE KEYSTORE AND ROOT LEVEL
CONN / AS SYSDBA
HOST mkdir -p /u01/app/oracle/admin/cdb2/encryption_keystore/
ADMINISTER KEY MANAGEMENT CREATE KEYSTORE '/u01/app/oracle/admin/cdb2/encryption_keystore/' IDENTIFIED BY "Anbob.com";
ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY "Anbob.com";

# PLUGIN PDB ,IMPORT KEYS
ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY ""Anbob.com"";
ADMINISTER KEY MANAGEMENT IMPORT ENCRYPTION KEYS WITH SECRET "mySecret" FROM '/tmp/export.p12' IDENTIFIED BY ""Anbob.com"" WITH BACKUP;

EXPDP Table in encrypted Tablespace

导出或导入加密的表数据时，需要使用数据泵，增加ENCRYPTION parameter 和 ENCRYPTION_PWD_PROMPT parameter ，如ENCRYPTION_PWD_PROMPT=YES 这样在导出里就要人为交互输入密钥。导入时一样，同时要先打开密钥库。

另外TDE 有一些限制条件，如加密列类型和加密列索引和外键时需要注意，更多请查询官方文档。

— over —

从Oracle 12.2版本发布以后原只以为DBBP就已是关于和以前版本PSU相关最大的改变，但是并不是， 关注我WEIBO（@weejar）的朋友知道我在6月初发布过2条关于Oracle可能将改变ORACLE的版本和补丁称呼。其中一个改季度更新为RU (Release Update)和原PSU 更名 RUR (Release Update Revision)，但不适用于12.1及以前版本，这点已见证，Oracle已经于201707发布了12.2的一个RU；另一个改变可能是ORACLE12C的是18.1(12.2.0.2)，在MOS 742060.1中已证实。 叫什么不重要，这里我要分享的这个新特性是12.2 已经悄悄引入但undocument的，也可能会在下个RU公布，因为RU是可以引入小的特性的(RUR不会)。

在之前的版本或现在，对于表上的主键列或唯一约束的列的填充时会使用sequence, 利用sequnece的特性防止ID编号的重复， 但是在现实中如果高并发多会话insert表只是单纯的调 用sequnect nextval时，会发现主键列或唯一列上的索引争用非常严重，造成了数据库加载时的瓶颈。 数据库中TOP event可能是enq: tx- index contention, buffer busy wait，如果是在两个节点上的insert还有可能会显示”gc”类的 热块争用事件。原因很简单就是因为索引是一种按KEY顺序存放的物理结构，在insert时所有会话问是在频繁的更新索引最后一个块（或几个块），才产生的因sequence顺序值填充index产生的索引块热块。相关的statistics是索引右侧的分裂leaf node 90-10 splits.

常用的优化手段都是围绕打散索引值，如hash partition index , reverse index. 如果使用hash也只是把1个段上的争用分摊给hash 的数量，但不能完全避免， 但是也知道对于索引如范围扫描等将不可用。对于reverse 是可以避免该类争用，但是又增加了索引维护里的I/O 次数，增加了存储的IOPS。 当然如果有一种方法可以让每个session有自己的sequence区间，这样就可以解决 该问题， 当然如果前期有考虑到该问题， 在优化时就建议应用使用如instance number+ sid + sequence的组合手动拼接数值就可以了。Oracle的强大就是与庞大的用户群合作互赢，把用户的最佳实践和需求引入ORACLE， 这个特性因为没有某些原因在12.2中都没有公开，但是SQL语法已经可用，暂时称做Sequence SCALE EXTEND 。注意公开前不建议在用户应用中使用。

anbob@pdbanbob:anbob> @desc dba_sequences;
           Name                            Null?    Type
           ------------------------------- -------- ----------------------------
    1      SEQUENCE_OWNER                  NOT NULL VARCHAR2(128)
    2      SEQUENCE_NAME                   NOT NULL VARCHAR2(128)
    3      MIN_VALUE                                NUMBER
    4      MAX_VALUE                                NUMBER
    5      INCREMENT_BY                    NOT NULL NUMBER
    6      CYCLE_FLAG                               VARCHAR2(1)
    7      ORDER_FLAG                               VARCHAR2(1)
    8      CACHE_SIZE                      NOT NULL NUMBER
    9      LAST_NUMBER                     NOT NULL NUMBER
   10      SCALE_FLAG                               VARCHAR2(1)
   11      EXTEND_FLAG                              VARCHAR2(1)
   12      SESSION_FLAG                             VARCHAR2(1)
   13      KEEP_VALUE                               VARCHAR2(1)
   

anbob@pdbanbob:anbob> create sequence seq_scale maxvalue 100000 scale;
Sequence created.

anbob@pdbanbob:anbob> select seq_scale.nextval from dual;
select seq_scale.nextval from dual
       *
ERROR at line 1:
ORA-64603: NEXTVAL cannot be instantiated for SEQ_SCALE. Widen the sequence by
1 digits or alter sequence with SCALE EXTEND.

anbob@pdbanbob:anbob> alter sequence seq_scale SCALE EXTEND;
Sequence altered.

anbob@pdbanbob:anbob> set numw 30
anbob@pdbanbob:anbob>  select seq_scale.nextval from dual;
                       NEXTVAL
------------------------------
                  101147000001

1 row selected.

anbob@pdbanbob:anbob> /
                       NEXTVAL
------------------------------
                  101147000002
1 row selected.

anbob@pdbanbob:anbob> /
                       NEXTVAL
------------------------------
                  101147000003
1 row selected.


anbob@pdbanbob:anbob> col SEQUENCE_OWNER for a10
anbob@pdbanbob:anbob> col SEQUENCE_NAME for a15
anbob@pdbanbob:anbob> @seq seq_scale

SEQUENCE_O SEQUENCE_NAME    MIN_VALUE  MAX_VALUE INCREMENT_BY C O CACHE_SIZE LAST_NUMBER S E S K
---------- --------------- ---------- ---------- ------------ - - ---------- ----------- - - - -
ANBOB      SEQ_SCALE                1     100000            1 N N         20          21 Y Y N N

1 row selected.

Note:
注意到在12.2的dba_sequences里就增加了这个特性的标识列，在启用了该特性后sequnece为2大部分组成，第1部分为系统扩展区间区，第2部分和以前一样是sequece值，长度为sequence的最大长前，前补0。 对于第1部分的扩展区间因为文档未公开还不确认其数的资源来源，但可以猜。

anbob@pdbanbob:anbob> select INSTANCE_NUMBER from v$instance;
INSTANCE_NUMBER
---------------
              1

anbob@pdbanbob:anbob> select sid from v$mystat where rownum<2; SID ------------------------------ 147 1 row selected. sys@pdbanbob:anbob> @pd sequence
Show all parameters and session values from x$ksppi/x$ksppcv...

       NUM N_HEX NAME                        VALUE        DESCRIPTION
---------- ----- --------------------------- ------------ ------------------------------------------------------
      2608   A30 _pdb_use_sequence_cache     TRUE         Use sequence cache in PDB mode
      2919   B67 _kqdsn_instance_digits      2            number of instance digits in scalable sequence
      2920   B68 _kqdsn_cpu_digits           3            number of cpu digits in scalable sequence

Note:
第1部分是1位是1，目前不知道其意义
第2部分是2位的实例号，如实例1是01,实例2是02
第3部分是3位的session id，如果SID为4位以上时可以测是否是后3位还是前3位？
第4部分是sequnece最大数位前用0补齐。
并且控制第2和3部分长度的参数应该是“_kqdsn_instance_digits”和“_kqdsn_cpu_digits”。
以上只是猜测。

如果启用了SCALE而未启用EXTEND,NO EXTEND区别是会把第一部分系统扩展区的长度算在最大长度内。

anbob@pdbanbob:anbob> create sequence seq_scale_noext maxvalue 1000000 scale;
Sequence created.

anbob@pdbanbob:anbob> select seq_scale_noext.nextval from dual;
   NEXTVAL
----------
   1011471
1 row selected.

anbob@pdbanbob:anbob> select seq_scale_noext.nextval from dual;
   NEXTVAL
----------
   1011472
...
...
anbob@pdbanbob:anbob> select seq_scale_noext.nextval from dual;
   NEXTVAL
----------
   1011479

anbob@pdbanbob:anbob> select seq_scale_noext.nextval from dual;
select seq_scale_noext.nextval from dual
*
ERROR at line 1:
ORA-64603: NEXTVAL cannot be instantiated for SEQ_SCALE_NOEXT. Widen the sequence by 1 digits or alter sequence with SCALE EXTEND.

anbob@pdbanbob:anbob> create sequence seq_scale_noext1 maxvalue 10000000 scale;
Sequence created.

anbob@pdbanbob:anbob> select seq_scale_noext1.nextval from dual;
   NEXTVAL
----------
  10114701

1 row selected.

Summary:
scale extend sequence这个功能非常的适用，可以有效的避免使用sequnece大并发insert时并生的索引块上的争用，包含了实例号和sid的组合及sequence的组合，只是出于某些原因并未把方法和使用公开到12.2的document中，相信在后续的版本中会公开，在公开前不建议使用在应用中。可以手动实现如

create sequence seq_scale_nosca maxvalue 10000000;

select  to_number(1||lpad(instance_number,2,0)||sid||lpad(seq_scale_nosca.nextval,2,0))
from dual,(select instance_number from v$instance) ins,(select sid from v$mystat where rownum=1) ses

其实早在2个月前就从一些国外OUG得知，第一个是从2017年开始改变了季度更新的方式，改变了过去的PSU为RUR (Release Update Revision) ，和改变 ProactiveBP 为 RU (Release Update)， BP(not Windows BP)的这12.1才出新的补丁形式又这么快消失了，前不久《Oracle 补丁那些事儿(PS、PSU、CPU、SPU、BP、DBBP&》整理过ORACLE的补丁相关的名词，没想到这么快又得更新;  第二个是oracle 12c的下一个版本不再延续12.2.0.2 和12.2.0.3的形式发布，从201708月更新MOS note#742060.1确认了计划分别与2018年年第1季度和2019年第1季度发现未来的两个版本oracle 18.1 和oracle 19.1,目前支持到2025年, 似乎更像MS 发布SQL Server的版本号，只不过不是叫2018只是18。

这种发布方式似乎像是从过去的瀑布式开发方式变成了迭代式开发
1，降低一次版本升级带来的特性改变的数量来提高质量
2，客户可以在未来8年中持续更新和修复bug

关于RUs和RURs

1，RUs和12.1时DBBP一样是主动的，经过高强度测试修改了客户已知的关键问题，并有可能引入小特性，代替BP
2，RURs包含了对安全和上个版本RUs的修正
3，RUs和RURs即提供了PSUs的稳定性好处，又具有BPs维护的主动性
4，RUs和RURs从12.2.0.1开始适用，从2017年7月发布了第一个RU(12.2.0.1.170718 没发布多久就又更新为12.2.0.1.170730)
5，可以简单的理解从12.2起RU代替了过去的BP,RUR代替了过去的PSU

以后如何选择季度补丁？

1， 如查使用是Oracle Engineered System如EXADATA Machine安装Bundle Patches for Engineered Systems
2， 如果使用是12.2.0.1及以后版本安装Release Upgrades (RU)
3， 如果使用是12.1.0.x安装Bundle Patches (BP)
4，如果使用是11.2.0.4安装Patch Set Updates (PSU)
5，如果使用提更老的版本应尽快计划升级已过支持期，如果不升级还是安装原PSU，并不再提供新补丁。

关于NEXT RELEASE和RUs 、RURs发布计划

1， 12.2.0.1没有计划改变版本号
2，下一个版本是oracle 18(12.2.0.2) 2018年发布, oracle 19(12.2.0.3)与2019年发布
3，12c R1和11G R2没有RU和RUR的计划，继续使用PSU,SPU,BP
4,   季度发布时间和以前一样，每年1、4、7、10月份
5， Interim (one-off) patches继续存在
6， 不再发布PSU,BP为12.2.0.1
7， 第一个RU与201707发布（40MB左右），第二个201710发布，第三个201801发布
8，第一个RUR计划于201710发布，第二个RUR于201801发布
9，计划每个RU只发布2个RUR（最近）

安装RU

安装RU的方式同样是使用之前的OPatch工具，对于RAC可以滚动安装。不过个人感觉第一个RU发布有点仓促，发布没几天因为BUG再次发布，并且readme txt or html都无内容。

安装方法:

[oracle@anbob ~]$ unzip p26549748_122010_Linux-x86-64.zip

[oracle@anbob 26549748]$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph .

SQL> shutdown immediate;

[oracle@anbob 26549748]$ $ORACLE_HOME/OPatch/opatch apply

SQL> alter pluggable database all open;

[oracle@anbob 26549748]$ $ORACLE_HOME/OPatch/datapatch -verbose

-- verify
SET lines 300 
COLUMN action_time FORMAT A20
COLUMN action FORMAT A10
COLUMN bundle_series FORMAT A10
COLUMN comments FORMAT A30
COLUMN description FORMAT A40
COLUMN namespace FORMAT A20
COLUMN status FORMAT A10
COLUMN version FORMAT A10

SELECT TO_CHAR(action_time, 'DD-MON-YYYY HH24:MI:SS') AS action_time,
 action,
 status,
 description,
 version,
 patch_id,
 bundle_series
 FROM   sys.dba_registry_sqlpatch
 ORDER by action_time;

sys@pdbanbob:anbob> /
ACTION_TIME          ACTION     STATUS     DESCRIPTION                              VERSION      PATCH_ID BUNDLE_SER
-------------------- ---------- ---------- ---------------------------------------- ---------- ---------- ----------
07-AUG-2017 17:52:16 APPLY      SUCCESS    DATABASE RELEASE UPDATE 12.2.0.1.170730  12.2.0.1     26549748 DBRU

— over —

朋友那一套12C R2多组户架构3节点RAC数据库反应数据库响应缓慢，找到我，因远程不方便，只拿到事后一份AWR，数据库出现较高的cursor: mutex X 和cursor: mutex S， 12.2目前的资料较少，这个问题个人感觉BUG的可能性大，但MOS中未发现匹配的NOTE.这里仅记录一下。

DB Name	DB Id	Unique Name	Role	Edition	Release	RAC	CDB
ANBOB	3056822707	ANBOB	PRIMARY	EE	12.2.0.1.0	YES	YES


			Snap Id	Snap Time	Sessions	Cursors/Session	Instances	Pluggable Databases Open
Begin Snap:	3264	03-Aug-17 11:00:16	230	1.9				3			4
End Snap:	3265	03-Aug-17 12:00:04	1034	2.1			3			4
Elapsed:	 	59.79 (mins)	 	 	 	 
DB Time:	 	57,548.34 (mins)	 


	Per Second	Per Transaction	Per Exec	Per Call
DB Time(s):	962.4	248.5	9.51	9.73
DB CPU(s):	5.1	1.3	0.05	0.05
Background CPU(s):	0.6	0.1	0.01	0.00
Redo size (bytes):	190,081.8	49,081.9	 	 
Logical read (blocks):	369,483.3	95,406.1	 	 
Block changes:	1,458.0	376.5	 	 
Physical read (blocks):	3,537.2	913.4	 	 
Physical write (blocks):	1,765.4	455.8	 	 
Read IO requests:	126.6	32.7	 	 
Write IO requests:	89.7	23.2	 	 
Read IO (MB):	27.6	7.1	 	 
Write IO (MB):	13.8	3.6	 	 
IM scan rows:	0.0	0.0	 	 
Session Logical Read IM:	0.0	0.0	 	 
Global Cache blocks received:	3,082.4	795.9	 	 
Global Cache blocks served:	2,789.8	720.4	 	 
User calls:	98.9	25.5	 	 
Parses (SQL):	53.2	13.7	 	 
Hard parses (SQL):	10.3	2.7	 	 
SQL Work Area (MB):	11.0	2.9	 	 
Logons:	1.7	0.4	 	 
Executes (SQL):	101.2	26.1	 	 
Rollbacks:	0.1	0.0	 	 
Transactions:	3.9	 	 

time model

SQL ordered by Version Count

• Only Statements with Version Count greater than 20 are displayed

Mutex Sleep Summary
ordered by number of sleeps desc

f0h5rpzmhju11
select SYS_CONTEXT('USERENV', 'SERVER_HOST'), SYS_CONTEXT('USERENV', 'DB_UNIQUE_NAME'), SYS_CONTEXT('USERENV', 'INSTANCE_NAME'), SYS_CONTEXT('USERENV', 'INSTANCE'), STARTUP_TIME, SYS_CONTEXT('USERENV', 'DB_DOMAIN'), SYS_CONTEXT('USERENV', 'SERVICE_NAME') from v$instance
awxuv71j1wzbh
select distinct s.sowner, s.vname from sys.snap$ s, sys.snap_reftime$ r where r.masobj# in (73811) and s.sowner = r.sowner and s.vname = r.vname and bitand(s.flag3, 67108864) = 67108864

gtuz08rcbk69y
select distinct s.sowner, s.vname from sys.snap$ s, sys.snap_reftime$ r where r.masobj# in (73811) and s.sowner = r.sowner and s.vname = r.vname and s.mowner = r.mowner and s.master = r.master and bitand(s.flag3, 67108864) = 67108864

note:
从AWR可以看出数据库的解析出现了严重的问题，cursor: mutex X占了94%的DB TIME同时cursor: mutex S伴随出现。同时mutex 函数kkscsAddChildNode [KKSPRTLOC28]和kkscsPruneChild [KKSPRTLOC27] 增加和裁剪子游标时产生了问题，数据库存在high version SQL, version高于200的都是数据库的递归SQL，最大的达到了7000多。同时ASH 中显示也是该上面的3个SQL.

关于MUTEX

mutex很像latch，mutex 是一种低级的粒度串行机制控制访问SGA中的共享数据结构， 但是相比mutex更小更轻量并且更快， get时少几倍的指令调用， 在10g r2 版本引入，属于KGX (Kernel Generic Mutex Module). 每个结构对象都有自己的mutex保护，父游标和子游标都有各自的mutex; 每个结构又被多个的mutex保护，在11G R2中有超过60种原因会导致不能重用已存在的child cursor，hard parse产生新的child cursor；soft parse选择已存在的child cursor，查找已存在的child cursor时，持有的是cursor: mutex S,  而增加新的Child Cursor持有的是cursor: mutex X.

分析此类问题需要收集的信息:

• AWR Report
• ASH Report
• Hanganalyze trace
• Systemstate dump

What are Wait Events (on Mutexes)?
•Somebody requests a Mutex
•Can not get it by spinning
•And thus, sleeps
•Sleeping is recorded as wait time
•Spinning is not recorded as wait-, but as CPU time

Cursor: Mutex X情景：

Wants: Exclusive mode on Mutex of Parent / Child
• Building a new cursor under a parent
• Although this operation is cheaper, building many cursors under a parent cursor is not recommended.
• Capture SQL bind data(PEEK)
• Build or Update statistics blocks,Modify cursor-related statistics
• Mutex is in the parent cursor.

Cursor: mutex S情景：

Wants: Shared mode on Mutex of Parent / Child

• Change the reference count (“in flux”)= “new guy is interested / spinning”
• Parent examination
• When finding a cursor to execute, the parent must be examined. The examination of the parent is performed using the mutex, cursor: mutex S.
• When the parent cursor has many child cursors involved, this waits will come as the server process has to traverse the entire list of child cursors under the parent to find a match.
• Mutex is in the parent cursor.

Complex – cursor: mutex S/X情景：
•Root-caused by invalidated child cursor(s) => Too many cursor objects in Library Cache
Diagnosis:
•Oracle Wait Interface
•10046 Level 12 session trace (=sql_trace event)  根据bind oacdty检查绑定变量的类型是否存在bind mismatch
•v$sql_shared_cursor plus cursor trace [296377.1]

SQL不能共享的原因:

 select * from
  (select sql_id, nonshared_reason, count(*) from v$sql_shared_cursor
  unpivot
  (nonshared_value for nonshared_reason in (
  UNBOUND_CURSOR as 'UNBOUND_CURSOR',
  SQL_TYPE_MISMATCH as 'SQL_TYPE_MISMATCH',
  OPTIMIZER_MISMATCH as 'OPTIMIZER_MISMATCH',
  OUTLINE_MISMATCH as 'OUTLINE_MISMATCH',
  STATS_ROW_MISMATCH as 'STATS_ROW_MISMATCH',
  LITERAL_MISMATCH as 'LITERAL_MISMATCH',
  FORCE_HARD_PARSE as 'FORCE_HARD_PARSE',
  EXPLAIN_PLAN_CURSOR as 'EXPLAIN_PLAN_CURSOR',
  BUFFERED_DML_MISMATCH as 'BUFFERED_DML_MISMATCH',
  PDML_ENV_MISMATCH as 'PDML_ENV_MISMATCH',
  INST_DRTLD_MISMATCH as 'INST_DRTLD_MISMATCH',
  SLAVE_QC_MISMATCH as 'SLAVE_QC_MISMATCH',
  TYPECHECK_MISMATCH as 'TYPECHECK_MISMATCH',
  AUTH_CHECK_MISMATCH as 'AUTH_CHECK_MISMATCH',
  BIND_MISMATCH as 'BIND_MISMATCH',
  DESCRIBE_MISMATCH as 'DESCRIBE_MISMATCH',
  LANGUAGE_MISMATCH as 'LANGUAGE_MISMATCH',
  TRANSLATION_MISMATCH as 'TRANSLATION_MISMATCH',
  BIND_EQUIV_FAILURE as 'BIND_EQUIV_FAILURE',
  INSUFF_PRIVS as 'INSUFF_PRIVS',
  INSUFF_PRIVS_REM as 'INSUFF_PRIVS_REM',
  REMOTE_TRANS_MISMATCH as 'REMOTE_TRANS_MISMATCH',
  LOGMINER_SESSION_MISMATCH as 'LOGMINER_SESSION_MISMATCH',
  INCOMP_LTRL_MISMATCH as 'INCOMP_LTRL_MISMATCH',
  OVERLAP_TIME_MISMATCH as 'OVERLAP_TIME_MISMATCH',
  EDITION_MISMATCH as 'EDITION_MISMATCH',
  MV_QUERY_GEN_MISMATCH as 'MV_QUERY_GEN_MISMATCH',
  USER_BIND_PEEK_MISMATCH as 'USER_BIND_PEEK_MISMATCH',
  TYPCHK_DEP_MISMATCH as 'TYPCHK_DEP_MISMATCH',
  NO_TRIGGER_MISMATCH as 'NO_TRIGGER_MISMATCH',
  FLASHBACK_CURSOR as 'FLASHBACK_CURSOR',
  ANYDATA_TRANSFORMATION as 'ANYDATA_TRANSFORMATION',
  PDDL_ENV_MISMATCH as 'PDDL_ENV_MISMATCH',
  TOP_LEVEL_RPI_CURSOR as 'TOP_LEVEL_RPI_CURSOR',
  DIFFERENT_LONG_LENGTH as 'DIFFERENT_LONG_LENGTH',
  LOGICAL_STANDBY_APPLY as 'LOGICAL_STANDBY_APPLY',
  DIFF_CALL_DURN as 'DIFF_CALL_DURN',
  BIND_UACS_DIFF as 'BIND_UACS_DIFF',
  PLSQL_CMP_SWITCHS_DIFF as 'PLSQL_CMP_SWITCHS_DIFF',
  CURSOR_PARTS_MISMATCH as 'CURSOR_PARTS_MISMATCH',
  STB_OBJECT_MISMATCH as 'STB_OBJECT_MISMATCH',
  CROSSEDITION_TRIGGER_MISMATCH as 'CROSSEDITION_TRIGGER_MISMATCH',
  PQ_SLAVE_MISMATCH as 'PQ_SLAVE_MISMATCH',
  TOP_LEVEL_DDL_MISMATCH as 'TOP_LEVEL_DDL_MISMATCH',
  MULTI_PX_MISMATCH as 'MULTI_PX_MISMATCH',
  BIND_PEEKED_PQ_MISMATCH as 'BIND_PEEKED_PQ_MISMATCH',
  MV_REWRITE_MISMATCH as 'MV_REWRITE_MISMATCH',
  ROLL_INVALID_MISMATCH as 'ROLL_INVALID_MISMATCH',
  OPTIMIZER_MODE_MISMATCH as 'OPTIMIZER_MODE_MISMATCH',
  PX_MISMATCH as 'PX_MISMATCH',
  MV_STALEOBJ_MISMATCH as 'MV_STALEOBJ_MISMATCH',
  FLASHBACK_TABLE_MISMATCH as 'FLASHBACK_TABLE_MISMATCH',
  LITREP_COMP_MISMATCH as 'LITREP_COMP_MISMATCH',
  PLSQL_DEBUG as 'PLSQL_DEBUG',
  LOAD_OPTIMIZER_STATS as 'LOAD_OPTIMIZER_STATS',
  ACL_MISMATCH as 'ACL_MISMATCH',
  FLASHBACK_ARCHIVE_MISMATCH as 'FLASHBACK_ARCHIVE_MISMATCH',
  LOCK_USER_SCHEMA_FAILED as 'LOCK_USER_SCHEMA_FAILED',
  REMOTE_MAPPING_MISMATCH as 'REMOTE_MAPPING_MISMATCH',
  LOAD_RUNTIME_HEAP_FAILED as 'LOAD_RUNTIME_HEAP_FAILED',
  HASH_MATCH_FAILED as 'HASH_MATCH_FAILED',
  PURGED_CURSOR as 'PURGED_CURSOR',
  BIND_LENGTH_UPGRADEABLE as 'BIND_LENGTH_UPGRADEABLE',
  USE_FEEDBACK_STATS as 'USE_FEEDBACK_STATS'
  ))
  where nonshared_value = 'Y'
  group by sql_id, nonshared_reason
  )
where sql_id = '&sqlid' ;

模拟一种 cursor: mutex S/X wait 情况：
•Generating 64 child cursors for one SQL_ID
•Accessing them 20x parallel
➔ Delay to create new children (X)
➔ Delay to select good child (S)

Similar Problem with CHAR binds
•Bind buffers are size 32, 128, 2000 or 4000 bytes
•Changing CHAR bind length invalidates
•Reason BIND_LENGTH_UPGRADEABLE= 4^n cursor versions

情况较复杂应该根据自己的实际情况分析，可能的一些解决方案如下:

• One Quick Fix for cursor: mutex S/X System is loaded with heavy mutex waits due to high number of cursors (=version count)
=> frequently flush this cursor with dbms_shared_pool.purge (look out for new parsing issues = CPU)
•  One solution for cursor: mutex S/X Application uses jdbc setter methods now properly on INTEGER column (=2)
#•setNUMBER(2) => Bind Var. is NUMBER
#setNULL(2, java.sql.Types.INTEGER)
=> Bind Var. is NUMBER
•Check how the application does handle bind variables Avoid BIND MISMATCH at (nearly) any cost
•Reduce the number of cursor versions below 100 More will lead to overhead
•Look for matching Oracle bugs in your RDBMS release
•Upgrade to 11.2.0.3 or higher 11.2.0.2 is worst version for cursor issues IMHO
• Increase the SGA if the shared pool is too small and heavy hard parse occurs due to reloads.
• For 11g, make sure cursor_sharing is not similar, as it has been deprecated. This may also cause cursor: mutex S waits:Recommended to use FORCE.
• Increase SESSION_CACHED_CURSORS to higher value if there are too many parses and waits on cursor: pin S.
• Use CURSOR_SHARING to FORCE to avoid waits on cursor: mutex S & cursor: mutex X due to huge version counts.
• If possible run with _LIBRARY_CACHE_ADVICE = FALSE and see if that helps
• On HP platforms ensure HPUX_SCHED_NOAGE parameter is set to 178.
• Obsoleting parent cursors if VERSION_COUNT exceeds a threshold (cursor: mutex S).
• When the child cursors grow beyond certain count be it 20 or 100, it obsoletes the parent cursors. In order to activate this enhancement bug set following:

For 11.2.0.2, apply the 11.2.0.2.2 PSU 12431716 atleast and do the following.
• _CURSOR_FEATURES_ENABLED to 1026
• event 106001 with value 100 (as the parameter _CURSOR_OBSOLETE_THRESHOLD is not present)
• Set , event= 106001 trace name context forever, level 100 in init.ora
For 11.2.0.3 and above, do the following.
• _CURSOR_OBSOLETE_THRESHOLD to 100 (default. this is the number of child cursor after which we obsolete it)

在有些情况下是存在真的hot对象，可以上面的方法都无法解决，需要手动打散：

To identify if the issue is a hot mutex use the following SQL :

SQL> select KGLNAOBJ, KGLNAOWN, KGLHDNSP,
KGLOBTYP, KGLNAHSV, KGLOBT23, KGLOBT24 
from X$KGLOB
where KGLOBT23 > 100000 or KGLOBT24 > 100000 
order by KGLOBT24;

Note:
相隔几分钟运行上面的SQL，如果KGLOBT23 或KGLOBT24 短时间内暴涨就说明该对象是HOT OBJ.或者从ASH或v$session数据根据当时的p1值得到。

Mutex wait Parameter Description
P1 Hash value of cursor
P2 Mutex value (top 2 bytes contain SID holding mutex in exclusive mode, and bottom two bytes usually hold the value 0)
P3 Mutex where (an internal code locator) OR'd with Mutex Sleeps

-- find hot obj
select KGLNAOBJ, KGLNAOWN, KGLHDNSP, KGLOBTYP,
KGLNAHSV 
from X$KGLOB
 where KGLNAHSH= {value of P1};

-- find blocker
SELECT count (*),
to_number(substr(to_char(rawtohex(p2raw)), 1, 8),'XXXXXXXX') blocking_sid FROM v$session
WHERE event = 'library cache: mutex X‘
group by to_number(substr(to_char(rawtohex(p2raw)),1, 8), 'XXXXXXXX');

-- hot copy
 You can apply the hot copy fix from 11.2.0.2 onwards w/o anyinterim patches.
• Set the following parameters in init.ora:-
_KGL_HOT_OBJECT_COPIES={a}


_KGL_DEBUG="name='{b}' schema='{c}' namespace={d} debug=33554432"
or (where the object is a cursor)
_KGL_DEBUG="hash='{e}' debug=33554432"

NOTE: Only with Oracle Support Guidance

Where:
{a} - half the CPU count
{b} - KGLNAOBJ from SQL
{c} - KGLNAOWN from SQL
{d} - KGLHDNSP from SQL
{e} - KGLNAHSV from SQL.

The same can be performed using dbms_shared_pool APIs in 11.2.0.2+.
SQL> EXEC DBMS_SHARED_POOL.MARKHOT('username','object_name',KGLHDNSP);
<or, for a cursor>
SQL> EXEC DBMS_SHARED_POOL.MARKHOT('KGLNAHSV', 0);

NOTE: dbms_shared_pool.markhot只是实例周期的改变，重启后失效，需要每次重启后重复手动配置.

12.2 这个案例的临时解决方案

上面介绍了那么多，再回到这个12.2的案例， 这么高的mutex等待与跟high version脱离不了关系，简单咨询客户后确认了没有对12.2默认参数做调整，sql verison能达到7000多，我查询了12.2的隐藏参数_CURSOR_OBSOLETE_THRESHOLD原来现在默认已加大到了8192, 该参数在11G中也只是100，12C R1版本是也在1024. 并且high version的SQL都是递归内部SQL,无法调应用解决，于是让客户调整_CURSOR_OBSOLETE_THRESHOLD参数调到200，并重启实例生效。问题消失了，并且上面的那个high version的SQL从TOP中消失。

当然如果再深入查，至少要在当时做SSD或ash 中找到进程信息,或尝试禁用_optim_peek_user_binds, sql ACS\ECS这些特性，或者是个未知的BUG, 我相信未来肯定会有客户同样遇到，希望提供一个分析的方向。

oracle官方网站在2017/8/18已提供了Ogg最新版本12.3下载，在Oracle 数据库12.2版本中提供了大量的新特性，OGG12.3配合ORACLE 数据库12c R2版本的新特性，提供更高的性能和吞吐量，最新发布的OGG12.3也是支持Oracle 12c R2第一个版本，引入了新的微服务架构和众多特性。OGG下载地址：

http://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html

1， 支持oracle rdbms 12c R2

我以前写过《注意： GoldenGate12.2不支持Oracle RDBMS 12.2 error OGG-06535》，oracle 12.2提供了大量的新功能，OGG12.3是支持oracle 12.2的第一个版本，可以更佳有效的利用oracle 12.2的功能提供更高的性能和吞吐量。

2,   提供微服务架构

一种基于服务的新架构，可简化大型和云部署的配置、管理和监控。 RESTFul服务使用基于HTTPS和Websocket（流）协议的基于角色的授权实现安全远程访问。

3,   支持并行Replicat

Parallel Replicat提供集成复制模式在数据库外执行依赖关系计算和并行性的所有好处。 它并行化跟踪文件的读取和映射，并提供在Oracle Database 11g（11.2.0.4）及更高版本中快速应用大型事务的能力。

4， 支持自动冲突检测Automatic Conflict-Detection-Resolution (CDR)

在OGG 应用的 active/active replication环境中，同时确保一致的冲突（如主键值冲突）检测（CDR），而无需修改应用程序或数据库结构。以便在Oracle Database 12c Release 2（12.2）及更高版本中配置时使用自动CDR自动执行冲突检测和解决。

5， 支持存储过程同步

允许在Oracle GoldenGate中procedure复制，复制Oracle提供的PL / SQL过程，避免通常由这些操作生成的大量记录的传送和应用。

6， 支持sharding

Oracle Sharding是一种为客户提供的可定制OLTP应用程序，提供了数据库的可伸缩性和可用性功能，它可以在不共享任何硬件或软件的离散Oracle数据库池中进行数据分发和复制，是ORACLE的一种分库解决方案。数据冗余可由Oracle GoldenGate通过Active-Active复制管理，该复制通过调用RestFul API的数据库引擎自动配置和编排。

7， 引入Fast Start Capture

Fast Start Capture是OGG集成模式中引入的新功能，能够快速的开始捕获和复制事务，提升整体性能。

8， 增强其它数据库支持

支持 Sql Server 2016 ,IBM i7.3

Oracle GoldenGate (OGG) 支持DDL capture 和apply 只有 Oracle和Teradata databases在以前的版本.OGG12.3支持MySQL 间的 DDL 同步

前几日在一客户搭建ASM环境，存储工程师为每个LUN划分了1TB大小，但是当创建ASM DISKGROUP时显示为每个ASM DISK约为4PB 大小， 当然不是存储工程师开的玩笑，也不是上帝的馈赠，实际是ASM在AIX 平台上的当ASM DISK  LUN 在1T及以上的BUG.

如果使用ASMCA 选中ASM  Disk强行创建磁盘组时会提示：

ORA-15099: disk '/dev/rhdiskXX' is larger 
than maximum size of 2097152 MBs

这里每个ASM DISK的限制2TB,如果从V$ASM_DISK 查看total_mb时4293918720，还好这里就没有多么大的存储。通过操作系统

bootinfo -s [diskname]  --从操作系统可以显示正确的大小1T。

根据这个4P大小这个关键字，找到了一个类似的Bug 9495887， MOS中描述的是影响的是11.2.0.1版本，因10.2.0.5与该版本较接近，怀疑同样是该BUG引起。

解决方法：
1， 使用命令行方式创建，并指定大小;
如
create diskgroup datadg external redundancy
disk ‘/dev/rhdiskxxx’ size 1000G, ‘/dev/rhdiskyyy’ size 1000G;

OR

2, 重新划分为小于1T 的LUN

OR

3,升级版本或修复BUG

library cache: mutex X等待事件是当一个会话以 exclusive mode持有library cache mutex时，另一个会话当前无论何时申请此mutex时必须等待其释放。很多时候对 library cache做不同的操作时都需要申请一个mutex, 所以最重要的是确认mutex的位置”location”,  该位置有助于分析该类等待事件的原因。

对应参数：

P1 = “idn” = Unique Mutex Identifier
P2 = Mutex “value” = in order to find the mutex value, read note 1298015.1
P3 = “where” = location in code (internal identifier) where mutex is being waited for

mutex的问题排查较为复杂，通常一个现象有多个原因，并且mutex问题又会引起其它级连问题。常见原因：

• 频繁的hard parse
• High sql version count,需要检查一个很长的chain of versions
• 引为某些原因导致的Invalidations and reloads
• shared pool配置过小
• cursor_sharing=similar和session_cached_cursors配置不当
• BUGs

该类问题时通常需要引集的信息：

• AWR Report
• ASH Report
• Hanganalyze trace
• Systemstate dump (contain call stack level)

系统级可用的相关视图：GV$MUTEX_SLEEP 和GV$MUTEX_SLEEP_HISTORY

分析该事件的相关SQL:

-- which library cache objects are the target of the operation.
select * 
from 
   (select 
      case when (kglhdadr = kglhdpar) 
      then 'Parent' 
      else 'Child '||kglobt09 end cursor,  
      kglhdadr ADDRESS, 
      substr(kglnaobj,1,20) NAME, 
      kglnahsh HASH_VALUE, 
      kglobtyd TYPE,  
      kglobt23 LOCKED_TOTAL, 
      kglobt24 PINNED_TOTAL,
      kglhdexc EXECUTIONS, 
      kglhdnsp NAMESPACE  
   from 
      x$kglob 
     -- where kglobtyd != 'CURSOR' 
   order by 
      kglobt24 desc) 
where 
   rownum <= 10; 

-- to find blocker session 
SELECT count (*), to_number(substr(to_char(rawtohex(p2raw)), 1, 8),'XXXXXXXX') blocking_sid 
FROM v$session 
WHERE event = 'library cache: mutex X' 
group by to_number(substr(to_char(rawtohex(p2raw)),1, 8), 'XXXXXXXX'); 

-- to find mutex from ash 
select event, p1, count(1) 
from v$active_session_history 
where sample_time > (sysdate - 20/1440)
and   event = 'library cache: mutex X' 
group by 
   event, p1 
order by 3;  

最近遇到过一个案例数据库几个session 在执行相同的insert values sql时library cache: mutex X争用严重，其它SQL和session正常，SQL文本中使用了46个绑定变量值。

SQL> select child_number,count(*)  from v$sql where sql_id='b92jmb1qngsyd' group by child_number;

CHILD_NUMBER   COUNT(*)
------------ ----------
           0         47
           1         47
           2         47
           3         47
           4         47
           5         46
           6         46
           7         47
           8         48
           9         46
          10         47
          11         47
          12         47
          13         47
          14         47
          15         48
          16         47
          17         47
 ...
          83         47
          84         47
          85         47
          86         47
          87         47
          88         47
          89         47
          90         47
          91         47
          92         47
          93         46
          94         47
          95         47
          96         47
          97         47
          98         47
          99         47

100 rows selected.

sql> @no_shared

SQL_ID        NONSHARED_REASON                COUNT(*)
------------- ----------------------------- ----------
b92jmb1qngsyd BIND_MISMATCH                       4730
b92jmb1qngsyd PURGED_CURSOR                          2
b92jmb1qngsyd HASH_MATCH_FAILED                      1
b92jmb1qngsyd BIND_LENGTH_UPGRADEABLE              952

select /*+rule*/m.position,m.bind_name , m.max_length,count(*) child_cursor_count
     from v$sql s, v$sql_bind_metadata m
     where s.sql_id =  'b92jmb1qngsyd'
     and s.child_address = m.address group by m.position,m.bind_name , m.max_length
     order by 1, 2;

-- 确认存在部分变量varchar长度区间问题

SQL> select mutex_type,location_id,location,sleeps from x$mutex_sleep where location_id=85;

MUTEX_TYPE                       LOCATION_ID LOCATION                                     SLEEPS
-------------------------------- ----------- ---------------------------------------- ----------
Library Cache                             85 kgllkdl1  85                               15001254

当时有做SSD 但级别不够（10）， 确认了blocker 是no in wait, 并且是on cpu, 执行的是相同的insert sql, 该sql是应用同步数据使用执行频率较高。当时也有尝试flush shared pool无效果，后尝试切换另一节点临时解决该现象。

因为当时也有做errorstack

  ----- Abridged Call Stack Trace -----
146297  ksedsts()+544<-kjzdssdmp()+400<-kjzdpcrshnfy()+512<-kstdmp()+416<-dbkedDefDump()+6032<-ksedmp()+64<-ksdxfdmp()+1360<
-ksdxcb()+3216<-sspuser()+688<-<kernel><-kglrdtin()+1121<-kglrdti()+80<-kksAllocCursorStat()+1152<-kksLoadChild()+14400<-kkslod()+112<-kglobld()+1872
146298  <-kglobpn()+1744<-kglpim()+832<-kglpin()+2496<-kxsGetRuntimeLock()+1840
146299  
----- End of Abridged Call Stack Trace -----

以call stack和event为关键字，在MOS中发现一篇Session Spin on Kglrdtin Holding ‘Library Cache: Mutex X’ (文档 ID 2219897.1)
和当前的版本现象较为相似因当前的平台hpux无相关one off patch,且无升级11.2.0.4计划记划，这里只记录一下该问题。

The V$backup_corruption view shows corrupted blocks discovered during an RMAN backup. But once the corruption on this blocks has been fixed this view is not updated. It can be said that this view stores the blocks that have been corrupted in past.

There is another view to identify the corrupted blocks: V$DATABASE_BLOCK_CORRUPTION. This view will reflect the blocks that were found to be corrupted after the last RMAN backup and is updated after each RMAN backup or backup validate command.

The corrupted blocks found in a backup must be checked in V$DATABASE_BLOCK_CORRUPTION.

In 10.2.0.4, the v$database_block_corruption view is based on v$copy_corruption and v$backup_corruption.The rows for dropped datafile doesn’t go away until the datafile# is reused by database and a backup of that file# is taken.This issue was fixed in 11g. To workaround the problem in 10.2.0.4, you will have to clear the v$backup_corruption and v$copy_corruption view on target database.

SQL> execute dbms_backup_restore.resetCfileSection(17); /** clear v$backup_corruption
SQL> execute dbms_backup_restore.resetCfileSection(18); /**clear v$copy_corruption

dbms_backup_restore.resetCfileSection(section_id),    How to get section_id?  In 11g Release 2, if you query the v$controlfile_record_section use the following sql:

SQL> select rownum-1, type from v$controlfile_record_section;

The V$BACKUP_CORRUPTION & V$COPY_CORRUPTION views list corrupt blocks in the backups, not the database itself.

The V$DATABASE_BLOCK_CORRUPTION lists corrupt blocks in the database detected during a variety of RMAN operations. Recovered blocks will still be listed until the next backup is performed.

RMAN keeps corruption information in the control file (v$database_block_corruption, v$backup_corruption). DBV does not.

In addition to the above method used to clear the v$backup_coruption record, you can also use   recreate control file or set control_file_record_keep_time=0 and wait auto clear;

Note Although dbms_backup_restore.resetCfileSection can clean up the records in the control file, But do not do that!, at least  do not execute on the production database before the test, because there are other cases that might make the database crash  .

rman>run{
set MAXCORRUPT for datafile 4 to 10;
backup database; -- or backup datafile 4
}

The SET MAXCORRUPT command specifies the total number of physical and logical corruptions permitted in a datafile during a backup job.The default limit is zero, If the sum of physical and logical corruptions detected for a datafile is no more than its MAXCORRUPT setting, then the BACKUP command completes. If more than MAXCORRUPT corrupt blocks exist, then RMAN terminates without creating output files. when set MAXCORRUPT  and   detect corrupted block then will generate record in v$backup_corruption, the case is meaning that RMAN tolerates datafile 4# detected 10 corrupt blocks .

朋友一套12.2的RAC一节点CRASH, 发我一份alert log让帮着看看，crash的原因以后有时间再写，但是alert log中出现了大量的”WARNING: too many parse errors”，是以前的版本中从未见过，这也是12.2的新特性，自动生成解析失败的信息写入db alert log, 即使没有在数据库启用event 10035, 以前版本可以通过启用10035 event分析解决失败信息写入alert log.

一段db alert log

2017-11-21T04:22:55.526449+08:00
XXHCPDB(12):WARNING: too many parse errors, count=2996 SQL hash=0x117d3781
XXHCPDB(12):PARSE ERROR: ospid=56905, error=1446 for statement: 
2017-11-21T04:22:55.526595+08:00
XXHCPDB(12):select rowid as "__Oracle_JDBC_internal_ROWID__", inv_objs from (select count(*) as inv_objs from dba_objects where status='INVALID')
XXHCPDB(12):Additional information: hd=0x43f493be0 phd=0x4368401a0 flg=0x20 cisid=0 sid=0 ciuid=0 uid=0 
XXHCPDB(12):WARNING: too many parse errors, count=2996 SQL hash=0x92a117b7
XXHCPDB(12):PARSE ERROR: ospid=56905, error=918 for statement: 
2017-11-21T04:22:55.563449+08:00
XXHCPDB(12):select rowid as "__Oracle_JDBC_internal_ROWID__", ses,ac_ses,waits,ofn_files,fai_jobs,inv_objs,lck_objs from (select count(*) as ses from v$session),(select count(*) as ac_ses from v$session where status='ACTIVE'),(select count(*) as waits from v$session_wait),(select count(*) as ofn_files from v$datafile where status = 'OFFLINE'),(select count(*) as fai_jobs from dba_jobs where failures > 0),(select count(*) as lck_objs from v$locked_object),(select count(*) as inv_objs from dba_objects where status='INVALID')
XXHCPDB(12):Additional information: hd=0x479b14608 phd=0xfc3e4728 flg=0x20 cisid=0 sid=0 ciuid=0 uid=0 

Note:
从以上信息中可以看到的信息：
1， WARNING: too many parse errors
2， SQL HASH
3,    SQL TEXT
4,    SQL ERROR CODE
5,    __Oracle_JDBC_internal_ROWID__

[oracle@anbob ~]$ oerr ora 1446
01446, 00000, “cannot select ROWID from, or sample, a view with DISTINCT, GROUP BY, etc.”
// *Cause:
// *Action:
[oracle@anbob ~]$ oerr ora 918
00918, 00000, “column ambiguously defined”
// *Cause:

SQL解析失败是一种常见现象，但是同一SQL频繁的解析失败会给数据库带来性能问题， 从AWR中的时间模型中同样可以发现parse, failed parse 占用较高的DB TIME,  以上DB Alert log中的输出是一种诊断信息的增强，可以默认把经常解决失败的信息写入db alert, 多数都是应用程序问题，解决方法自然是调整应用或SQL.

WARNING: too many parse errors 并不是每次解析都提示，默认是在同一SQL在60分钟内每出现100次就会提示一次worning到DB ALERT LOG. 对于这个阀值应该是有隐藏参数”_kks_parse_error_warning” 控制的。

SQL> @pd parse_error
Show all parameters and session values from x$ksppi/x$ksppcv...

       NUM N_HEX NAME                         VALUE         DESCRIPTION
---------- ----- ---------------------------- ------------- -------------------------
      3080   C08 _kks_cached_parse_errors     0             KKS cached parse errors
      3082   C0A _kks_parse_error_warning     100           Parse error warning

对于SQL文本是可以手动执行还原问题，ERROR 就是解析的错误ORACE的提示编号，这里的1446 是提示rowid 不能用于count这类汇总操作中。

对于SQL文本中的ROWID 和“__Oracle_JDBC_internal_ROWID__”应该不是程序员人为加入，ALERT中提示中出现了大量__Oracle_JDBC_internal_ROWID__字样，判断应该是ORACLE JDBC驱动自动附加的。

当JDBC 中scroll sensitive resultset 启用时，JDBC驱动就会在用户SQL中自动增加ROWID ，但是并不是所有的查询都适合附加rowid, 该功能是JDBC中对于结果集是否在数据改变后自动刷新，有TYPE_SCROLL_SENSITIVE和TYPE_SCROLL_INSENSITIVE两种，区别可以看这里。
–JAVA CODE Demo

try {
        Class.forName(driverName).newInstance();
        connection = DriverManager.getConnection(url+dbName, userName, password);
           Statement stmt = connection.createStatement(
              ResultSet.TYPE_SCROLL_SENSITIVE,
              ResultSet.CONCUR_READ_ONLY);
...
if (res.getType() == ResultSet.TYPE_SCROLL_SENSITIVE) {
          System.out.println("ResultSet TYPE_SCROLL_SENSITIVE.");
        }
'''
}

解决该类问题需要程序员了解TYPE_SCROLL_SENSITIVE的限制，修改SQL解析失败处代码ResultSet型的类

We limit the default display of database character sets that can be used for a new database in Oracle Database 11 via DBCA (Database Creation Assistant) based on the recommended list.

In the release 11gR1 and later of the Oracle database, we reduce the number of character sets on offer trough the DBCA (Database Creation Assistant) to customers to a list of recommended charactersets.
If it is required to create a database using one of the “non recommended” character sets in the 11g DBCA (US7ASCII or WE8ISO8859P1 for example) start the 11g DBCA and in step 9 of 11 of the DBCA choose the “Character Sets” tab, under “Database Character Set” header select the “Choose from the list of character sets” button and deselect the “show recommended character sets only” box .

In the 12c the location of choosing an “non recommended” character set is somewhat changed:
Start the 12c DBCA ,  choose ” create a database” , in step 2 choose the “advanced mode” and in step 10 of 14 of the 12c DBCA go to the the “Character Sets” tab, select the “Choose from the list of character sets” button and deselect the “show recommended character sets only” box . An “non recommended” characterset (US7ASCII or WE8ISO8859P1 for example) can then be used as Database Character Set (NLS_CHARACTERSET).

Note that ALL character sets are still SUPPORTED and “non recommended” character sets can be used if needed.

As mentioned above the CREATE DATABASE COMMAND still allows legacy, non-recommended character sets. This is unless and until any given character set is placed on our obsolete/deprecated list. Note the change the default behavior in Oracle Database 11g when customers don’t specify the character set for the CREATE DATABASE COMMAND. The default is changed from US7ASCII to AL32UTF8.

To know what characters are known in a certain characterset then please see Note 282336.1 Charts of most current mono-byte Character sets

Japanese, Korean and Chinese character sets

JA16SJIS
JA16SJISTILDE
JA16EUC
JA16EUCTILDE
KO16MSWIN949
ZHS16GBK
ZHT16MSWIN950
ZHT16HKSCS
ZHT16HKSCS31
ZHT32EUC

Comments on some popular database character sets that are not recommended

US7ASCII: better to migrate to WE8MSWIN1252, or WE8ISO8859P15 etc.
WE8ISO8859P1: WE8MSWIN1252 is a superset
UTF8: better to migrate to AL32UTF8
ZHS16CGB231280: ZHS16GBK is a superset
KO16KSC5601: KO16MSWIN949 is a superset
ZHT16BIG5: ZHT16MSWIN950 solves various problems of ZHT16BIG5

References ORACLE Docs.

前段时间有个朋友遇到的问题，让我协助分析，现象是一个地市的数据库与省级数据库通过DBLINK连接时提示ORA-600 2252, 但是其它地市与省级的DBLINK正常。错误如下：

SQL> select sysdate from dual@ANBOB_RMT

ORA-00600: 内部错误代码, 参数: [2252], [3985], [1364216517], [], [], [], [], []

ORA-600 [2252] [A] [B] [] [] []

Cause:
————
Oracle compares the given SCN value  with the reasonable upper limit value calculated based on the
system date. If Oracle detects the provided scn is too large,  ORA-600[2252] would be raised.

ARGS:
————
arguments A: SCN WRAP
arguments B: SCN BASE

Fix:
————
symptom: System date differs between the two machines
symptom: Query that uses a database link between two machines fails
cause: Argument [2252] for the ORA-00600 indicates that the System Change Number (SCN) Oracle is calculating for a transaction is an unreasonable number.
The SCN is calculated based in part on the host system date.
If the system date is a LONG way off, the maximum possible value calculated for the SCN may be impossibly large which would result in an ORA-600[2252].

简而言之：数据库当前的请求SCN大于当前最大允许SCN时会提示ORA-600[2252], 最大允许SCN是有本地系统时间决定。一个可能性是本地库主机时间向前调了，还有个可能性是通过DBLINK 分布式事务同步SCN时，远程库 SCN大于本地允许的最大scn.

SQL@ANBOB> select 3985*power(2,32)+1364216517 from dual;

3985*POWER(2,32)+1364216517
---------------------------
17116808891077

SQL@ANBOB> select dbms_flashback.get_system_change_number scn from dual;

SCN
----------
15756464714

sys@ANBOB>select current_scn,dbms_flashback.get_system_change_number scn from v$database;

CURRENT_SCN                  SCN
-------------------- --------------------
15756464716          15756464716

-- 确认时间,时区
select CURRENT_TIMESTAMP, LOCALTIMESTAMP FROM DUAL;

-- 现场的人整理了本地(DBa)、远程(DBb)库的系统时间和SCN.
DBa          DBb
--------      ----------
OS date: 20171204 15:50        20171204 15:48
sysdate: 20171204 ..        20171204 ..
scn: 15756464722        17117005290806
DB ver: 11.2.0.1            11.2.0.3
OS Plat: Windows            Aix

Note:
本地库和远程库的SCN不在一个数量级，相差1000倍，其实我们可以根据当前的时间计算一下SCN 的最大允许值，远程库的SCN 远大于本地库的最大允许SCN(简称RSL，下面会补充相关知识)。 关于SCN可以查阅 MOS note <<System Change Number (SCN), Headroom, Security and Patch Information (文档 ID 1376995.1)>>

— 查询远程库的SCN Headroom

SQL> select
2   version,
3   to_char(SYSDATE, 'YYYY/MM/DD HH24:MI:SS') DATE_TIME,
4   ((((
5    ((to_number(to_char(sysdate, 'YYYY')) - 1988) * 12 * 31 * 24 * 60 * 60) +
6    ((to_number(to_char(sysdate, 'MM')) - 1) * 31 * 24 * 60 * 60) +
7    (((to_number(to_char(sysdate, 'DD')) - 1)) * 24 * 60 * 60) +
8    (to_number(to_char(sysdate, 'HH24')) * 60 * 60) +
9    (to_number(to_char(sysdate, 'MI')) * 60) +
10    (to_number(to_char(sysdate, 'SS')))
11   ) * (16 * 1024)) - dbms_flashback.get_system_change_number)
12   / (16 * 1024 * 60 * 60 * 24)
13   ) indicator
14    from v$instance
15  ;

VERSION           DATE_TIME            INDICATOR
----------------- ------------------- ----------
11.2.0.3.0        2017/12/04 17:15:26 -959.18726

Note:
Oops!!!  上面的脚本也较常见来自官方的scnhealthcheck.sql， INDICATOR是距离SCN Headroom(天花板)的天数，是负数说明已经超过天花板上天了。当然scn限制是决定不会也不允许超过天花板的。 那会不会是远程库有问题？为什么其它地市的库可以跟这个远程库查询？负数的原因是什么？

基实上面的脚本对于11.2.0.2以后的数据库还需要确认另一处，就是每秒16K的限制从11G R2(11.2.0.2)起已经改变为32K（我查了11.2.0.3 11.2.0.4 12.2.0.1默认都是32K）, 有隐藏参数”_max_reasonable_scn_rate”控制，同时需要使用下面的SQL语句实际确认是16K还是32K(只对11.2.0.2以后的版本有意义)，因为我发现有些11.2的数据库仍然使用的是16K(也许是低版本直接升级原因，也许是某个PSU临时回归了16K的增速):

sys@ANBOB_RMT>@pd _max_reasonable_scn_rate
Show all parameters and session values from x$ksppi/x$ksppcv...

INDX I_HEX NAME                         VALUE      DESCRIPTION
-------------------- ----- ---------------------------- ---------- ---------------------------
978   3D2 _max_reasonable_scn_rate     32768      Max reasonable SCN rate

sys@ANBOB_RMT>select decode(bitand(DI2FLAG,65536),65536,'Y','N') using16 from x$kccdi2;
U
-
N

sys@ANBOB_RMT>select
version,
to_char(SYSDATE, 'YYYY/MM/DD HH24:MI:SS') DATE_TIME,
((((
((to_number(to_char(sysdate, 'YYYY')) - 1988) * 12 * 31 * 24 * 60 * 60) +
((to_number(to_char(sysdate, 'MM')) - 1) * 31 * 24 * 60 * 60) +
(((to_number(to_char(sysdate, 'DD')) - 1)) * 24 * 60 * 60) +
(to_number(to_char(sysdate, 'HH24')) * 60 * 60) +
(to_number(to_char(sysdate, 'MI')) * 60) +
(to_number(to_char(sysdate, 'SS')))
) * (32 * 1024)) - dbms_flashback.get_system_change_number)
/ (32 * 1024 * 60 * 60 * 24)
) indicator
from v$instance;

VERSION           DATE_TIME            INDICATOR
----------------- ------------------- ----------
11.2.0.3.0        2017/12/04 17:35:26 5087.41908

现在总结一下这个问题：

1，本地库是11.2.0.1 scn 的频率限制还是16K
2, 远程库是11.2.0.3，并且从x$kccdi2确认了当前使用的频率限制是32K
3, 远程库当前的SCN已经超过了本地库16K允许的上限所以使用DBLINK 同步SCN 会出现ora-600 [2252]
4, 远程库查询天花板需要修改脚本中16为32
5，其它地市能访问远程库是因为他们的数据库也是11.2.0.2版本以后，且同样使用的是32K的限制。

SCN增长速率加快了，如果32k的速度使用6bytes的scn总上限也就不是过去说的可用500年了，所以从12.2又引入了big scn加到了8bytes.且在12.2版本中对于SCN传播跳跃， 增加了2个视图可以定位源头， 使用DBA_EXTERNAL_SCN_ACTIVITY  DBA_DB_LINK_SOURCES and DBA_DB_LINK关连就可以，2012年1月以后的PSU起或在11G的部份版本中提供了控制SCN相关参数：

A.1) SCN rejected due to request for high SCN increment (controlled by _external_scn_rejection_threshold_hours) 限制最多用到多少，保留时间
A.2) SCN rejected due to request in certain DELTA of changes (controlled by _external_scn_rejection_delta_threshold_minutes) 限制一次最多变化多少，如果请求超过会失败，提示ORA-19706。
A.3) SCN accepted but with a warning (controlled by _external_scn_logging_threshold_seconds) 增长超过一定阀值时，写ALERT LOG。

SCN相关知识点：

1， SCN 是Oracle数据库单向增长的”时钟”,广泛用于数据库一致性恢复和分布式事务(如dblink)
2,  SCN 有两部分组成 wrap.base， 在数据库中占用8bytes, 在12c r2前预留2bytes, 是一个6bytes(48bit)的Integer类型的数字，[16bit SCN Wrap].[32bit SCN Base], 在12c R2起引入big scn, 启用了原来预留的2bytes, 总长限有原来的2^48增长到2^64。
3， 为了限制SCN无限增长，在程序的代码级设计了一个当前时间点的允许的最大SCN(Maximum Reasonable SCN)的软限制, Reasonable SCN Limit简称RSL, 这个值是有一个工式计算出来的RSL=(从1988年1月1日起到当前时间) * 24 * 3600 * 每秒允许的最大增长率， 需要注意的是并不是简单的当前时间和1988-1-1两个时间点相减，可能是出于计算的简单，每个月是按31天计算的，从上面MOS中提供的脚本也可以看出。 每秒允许的最大增长率在11.2.0.2之前是16384（16K）11.2.0.2及以后的版本是32768（32K），有隐藏参数_max_reasonable_scn_rate控制。
4， SCN Headroom是一个最重要的检查项，值是当前时间点的允许的最大SCN与当前数据库SCN的差值，为了方便阅读以”天“为单位，SCN Headroom 天数=(Maximum Reasonable SCN-Current SCN) / (每秒允许的最大增长率) /3600/24
5， SCN异常增长通常是有DBLINK、人为前推SCN、oracle bug。SCN的历史变化可以从V$ARCHIVED_LOG得到， 最近5天的SCN也可以尝试从smon_scn_time得到。 数据库自身的增长可以从从dba_hist_sysstat得到’calls to kcmgas’的变化， SCN通过DBLINK 的传播可以通过上面提到的参数控制，和12c 中提供的新视图。
6， 数据库11.2.0.2及以后的版本默认是允许32K的增长速率，所以就会像本案例以上产生较大的SCN, 这就意味着11.2.0.2可能不能再与低版本的数据库或是使用16K增长速率的数据库通过DBLINK。

数据库中做分区表DDL是较为常见的维护，如运营商中大量以月份为分区键的分区表，前不久就在例行了大量的分区表几个小时后数据库双实例hang死，下面分享这个案例希望有相同环境的DBA们早日预防。

环境是11.2.0.3.7的2Nodes RAC  ON HPUX 11.31, 当晚在2节点做了几万次的分区维护但是在2：00前就已做完，还好在营业厅开门前，7：00有人反应数据库实例无法连接，开始是发现节点2连接数满，后维护人员KILL了所有的前台进程反应当时全是Library cache lock等待事件，但是KILL前没有解决问题，不久节点1出现了同样的症状。我登录上查看节点1的活动会话已接近4000，几乎全是Library cache lock的会话，blocker session 在等待DFS lock handle ， DFS lock handle 是一种OPS时分布式文件系统的一种等待事件通常是RAC相关，可以转换为具体的event.

为了尽快恢复业务，开始KILL 节点1 的会话，但是会话变成KILLED状态未释放，随后决定重启该实例1，shutdown immediate无法停止，alert log 一直提示“SHUTDOWN: waiting for active calls”，等待了10多分钟后决定kill pmon 强行终止(不到最后非常不建议这种手段)， 实例重启后数据库一切恢复正常。 下面只能使用日志和ASH信息分析原因。

-- node1 
SQL> select * from (
  2      select etime,nvl(event,'on cpu') events, dbtime, round(100*ratio_to_report(dbtime) OVER (partition by etime ),2) pct,row_number() over(partition by etime order by dbtime  desc) rn
  3   from (
  4  select substr(to_char(SAMPLE_TIME,'yyyymmdd hh24:mi'),1,13)||'0' etime,event,count(*)*10 dbtime
  5   from dbmt.dash1027
  6   where INSTANCE_NUMBER=1
  7   and sample_time between  to_date('2017-10-27 07:00:00','yyyy-mm-dd hh24:mi:ss') AND  to_date('2017-10-27 08:50:00','yyyy-mm-dd hh24:mi:ss')
  8   group by substr(to_char(SAMPLE_TIME,'yyyymmdd hh24:mi'),1,13),event
  9  )
 10  ) where rn<=5;

ETIME          EVENTS                              DBTIME                            PCT                             RN
-------------- ------------------------------ ----------- ------------------------------ ------------------------------
20171027 07:00 enq: SQ - contention                147580                          45.71                              1
20171027 07:00 library cache lock                  134870                          41.77                              2
20171027 07:00 on cpu                               11810                           3.66                              3
20171027 07:00 gc buffer busy acquire                8210                           2.54                              4
20171027 07:00 db file sequential read               4070                           1.26                              5
20171027 07:10 on cpu                               13110                          42.45                              1
20171027 07:10 library cache lock                    6580                          21.31                              2
20171027 07:10 db file sequential read               5230                          16.94                              3
20171027 07:10 db file scattered read                1660                           5.38                              4
20171027 07:10 read by other session                  800                           2.59                              5
20171027 07:20 enq: SQ - contention                 69190                          44.18                              1
20171027 07:20 library cache lock                   62140                          39.68                              2
20171027 07:20 on cpu                               12660                           8.08                              3
20171027 07:20 db file sequential read               4330                           2.76                              4
20171027 07:20 db file scattered read                1960                           1.25                              5

--node 2
SQL>  select * from (
  2      select etime,nvl(event,'on cpu') events, dbtime, round(100*ratio_to_report(dbtime) OVER (partition by etime ),2) pct,row_number() over(partition by etime order by dbtime  desc) rn
  3   from (
  4  select substr(to_char(SAMPLE_TIME,'yyyymmdd hh24:mi'),1,13)||'0' etime,event,count(*)*10 dbtime
  5   from dbmt.dash1027
  6   where INSTANCE_NUMBER=2
  7   and sample_time between  to_date('2017-10-27 07:00:00','yyyy-mm-dd hh24:mi:ss') AND  to_date('2017-10-27 08:50:00','yyyy-mm-dd hh24:mi:ss')
  8   group by substr(to_char(SAMPLE_TIME,'yyyymmdd hh24:mi'),1,13),event
  9  )
 10  ) where rn<=5;

ETIME          EVENTS                                        DBTIME                            PCT                             RN
-------------- ---------------------------------------------------- ------------------------------ ------------------------------
20171027 07:00 latch: row cache objects                       27250                          23.92                              1
20171027 07:00 enq: SQ - contention                           27140                          23.82                              2
20171027 07:00 library cache load lock                        18460                           16.2                              3
20171027 07:00 gc buffer busy release                         11270                           9.89                              4
20171027 07:00 on cpu                                          8130                           7.14                              5
20171027 07:10 on cpu                                          6720                          34.62                              1
20171027 07:10 Backup: MML write backup piece                  2620                           13.5                              2
20171027 07:10 SQL*Net message from dblink                     2390                          12.31                              3
20171027 07:10 db file sequential read                         1700                           8.76                              4
20171027 07:10 latch: row cache objects                        1430                           7.37                              5
20171027 07:20 enq: SQ - contention                           16230                          34.24                              1
20171027 07:20 latch: row cache objects                       10240                           21.6                              2
20171027 07:20 on cpu                                          5910                          12.47                              3
20171027 07:20 library cache: mutex X                          3410                           7.19                              4
20171027 07:20 Backup: MML write backup piece                  2490                           5.25                              5

SQL> select INSTANCE_NUMBER,sql_id,p2,
  2  --blocking_session,
  3  blocking_inst_id,
  4  count(*)
  5  from dbmt.dash1027 where
  6    sample_time between  to_date('2017-10-27 7:00:00','yyyy-mm-dd hh24:mi:ss') AND  to_date('2017-10-27 07:10:00','yyyy-mm-dd hh24:mi:ss')
  7   and event='enq: SQ - contention' group by INSTANCE_NUMBER,sql_id,p2
  8   --,blocking_session
  9   ,blocking_inst_id
 10   order by 5 desc;

               INSTANCE_NUMBER SQL_ID                                    P2               BLOCKING_INST_ID                       COUNT(*)
------------------------------ ------------- ------------------------------ ------------------------------ ------------------------------
                             1 2783h5bwb20f8                        8260071                              1                          11718
                             2                                          362                              2                           2706
                             1 4rhpkb0fh39nc                        8260071                              1                           2509
                             1 08324f71s5na3                        8260071                              1                            386
                             1 6fa71k5cafbh4                        8260071                              1                             77

SQL> @oid 8260071

owner                     object_name                    object_type        SUBOBJECT_NAME                 CREATED           LAST_DDL_TIME     status                    DATA_OBJECT_ID
------------------------- ------------------------------ ------------------ ------------------------------ ----------------- ----------------- --------- ------------------------------
ANBOB                      SEQ_OID                        SEQUENCE                                          20150324 02:47:42 20150807 15:08:24 VALID

1 row selected.
SQL> @seq anbob.SEQ_OID

SEQUENCE_OWNER                 SEQUENCE_NAME                                       MIN_VALUE                      MAX_VALUE                   INCREMENT_BY C O                     CACHE_SIZE                    LAST_NUMBER
------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------ - - ------------------------------ ------------------------------
ANBOB                           SEQ_OID                                        88100000000000   1000000000000000000000000000                              4 N N                          10000                 88507311739938

SQL> @sqlid 2783h5bwb20f8 %
Show SQL text, child cursors and execution stats for SQLID 2783h5bwb20f8 child %

                    HASH_VALUE                PLAN_HASH_VALUE  CH# SQL_TEXT
------------------------------ ------------------------------ ---- ------------------------------------------------------------------------------------------------------------------------------------------------------
                    4172349896                      207917406    0 select to_char(seq_oid.nextval) oid from dual

1 row selected.


 CH# PARENT_HANDLE    OBJECT_HANDLE                         PLAN_HASH                         PARSES                       H_PARSES                     EXECUTIONS                        FETCHES                 ROWS_PROCESSED                 ROWS_PER_FETCH                        CPU_SEC                   CPU_SEC_EXEC                        ELA_SEC                           LIOS                           PIOS                          SORTS                USERS_EXECUTING
---- ---------------- ---------------- ------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------ ------------------------------
   0 C00000153DA00458 C00000153D9FF2F8                      207917406                           5891                              1                       12580228                       12580235                       12580073 .99998712265708867918604064233                        1174.11 .00009332978702770728797602078                    1362.309233                           9239                              0                              0                              0

Note:
当时的ENQ:SQ 事件也较为严重，结合AWR，根据SQL 和sequnece相关业务和执行频率无异常，以当前的SEQ调用频率，从AWR “Wait Event Histogram Detail”查看enq:sq每次平均wait时间太长，sq问题应该发生在row cache填充时， 判断当时shared pool 应该存在性能问题。

下面查询当时的session wait chains

SQL> select INSTANCE_NUMBER,sql_id,p1,
  2  blocking_session,blocking_session_serial#,
  3  blocking_inst_id,
  4  count(*)
  5  from dbmt.dash1027 where
  6    sample_time between  to_date('2017-10-27 7:00:00','yyyy-mm-dd hh24:mi:ss') AND  to_date('2017-10-27 07:10:00','yyyy-mm-dd hh24:mi:ss')
  7   and event='library cache lock' group by INSTANCE_NUMBER,sql_id,p1
  8  ,blocking_session,blocking_session_serial#
  9   ,blocking_inst_id
 10   order by 7 desc;

 INSTANCE_NUMBER SQL_ID                            P1   BLOCKING_SESSION       BLOCKING_SESSION_SERIAL#  BLOCKING_INST_ID COUNT(*)
---------------- ------------- ---------------------- ------------------ ------------------------------ ----------------- --------
               1                 13835058141231219944               8502                           6515                 1    13372
               2                 13835058145195890800                                                                          131
               1                 13835058141231219944                                                                           52
               1                 13835058141231219944               9694                          58887                 1       20
               1                 13835058141231219944              13307                          40389                 1       13
               1                 13835058141231219944               7143                          57041                 1       10
               1                 13835058141231219944              14432                          64427                 1        8

SQL> select sample_time,program,machine,sql_id,event,p1text,p1,p2text,p2,p3text,p3,session_state,
blocking_session,blocking_session_serial#,blocking_inst_id,current_obj#,delta_time
    from dbmt.dash1027 
	where  sample_time between  to_date('2017-10-27 7:00:00','yyyy-mm-dd hh24:mi:ss') AND  to_date('2017-10-27 07:12:00','yyyy-mm-dd hh24:mi:ss')
     and  session_id=8502 and session_serial#=6515 and instance_number=1 order by 1;
	 

SAMPLE_TIME                    PROGRAM                    MACHINE   SQL_ID        EVENT               P1TEXT                  P1 P2TEXT   BLOCKING_SESSION BS_SERIAL# BLOCKING_INST_ID 
------------------------------ -------------------------- --------- ------------- ------------------- ----------- -------------- ------- -----------------  -------- ---------------- 
27-OCT-17 07.03.35.009 AM      Deal@anbob (TNS V1-V3)     anbob                   DFS lock handle     type|mode       1230372869 id1                                                   
27-OCT-17 07.03.45.459 AM      Deal@anbob (TNS V1-V3)     anbob                   DFS lock handle     type|mode       1230372869 id1                  10717         1                2 
27-OCT-17 07.03.56.219 AM      Deal@anbob (TNS V1-V3)     anbob                   DFS lock handle     type|mode       1230372869 id1                  10717         1                2 
27-OCT-17 07.04.06.639 AM      Deal@anbob (TNS V1-V3)     anbob                   DFS lock handle     type|mode       1230372869 id1                  10717         1                2 
27-OCT-17 07.04.17.168 AM      Deal@anbob (TNS V1-V3)     anbob                   DFS lock handle     type|mode       1230372869 id1                  10717         1                2 
27-OCT-17 07.04.27.678 AM      Deal@anbob (TNS V1-V3)     anbob                   DFS lock handle     type|mode       1230372869 id1                  10717         1                2 
...
... have truncated
...
27-OCT-17 07.06.47.259 AM      Deal@anbob (TNS V1-V3)     anbob                   DFS lock handle     type|mode       1230372869 id1                  10717         1                2 
27-OCT-17 07.06.58.509 AM      Deal@anbob (TNS V1-V3)     anbob                   DFS lock handle     type|mode       1230372869 id1                  10717         1                2 
27-OCT-17 07.07.09.389 AM      Deal@anbob (TNS V1-V3)     anbob                   DFS lock handle     type|mode       1230372869 id1                  10717         1                2 
27-OCT-17 07.07.20.399 AM      Deal@anbob (TNS V1-V3)     anbob                   DFS lock handle     type|mode       1230372869 id1                  10717         1                2 
27-OCT-17 07.07.31.429 AM      Deal@anbob (TNS V1-V3)     anbob                   DFS lock handle     type|mode       1230372869 id1                  10717         1                2 


SQL> select chr(bitand(&&p11,-16777216)/16777215) ||
  2         chr(bitand(&&p11,16711680)/65535) type,
  3         mod(&&p11, 16) md
  4         from dual;
Enter value for p11: 1230372869

TY                             MD
-- ------------------------------
IV                              5

SQL> select * from v$lock_type where type='IV';

TYPE                                                             NAME                                                             ID1_TAG                                                          ID2_TAG                                                          IS_
---------------------------------------------------------------- ---------------------------------------------------------------- ---------------------------------------------------------------- ---------------------------------------------------------------- ---
DESCRIPTION
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
IV                                                               Library Cache Invalidation                                       object #                                                         time stamp                                                       NO
Synchronizes library cache object invalidations across instances

SQL> select sample_time,program,machine,sql_id,event,p1text,p1,p2text,p2,p3text,p3,session_state,blocking_session,blocking_session_serial#,blocking_inst_id,current_obj#,delta_time
  2      from dbmt.dash1027 where
  3        sample_time between  to_date('2017-10-27 7:00:00','yyyy-mm-dd hh24:mi:ss') AND  to_date('2017-10-27 07:12:00','yyyy-mm-dd hh24:mi:ss')
  4     and session_id=10717 and session_serial#=1 and instance_number=2 order by 1;

SAMPLE_TIME                         PROGRAM                       MACHINE         SQL_ID        EVENT                          BLOCKING_SESSION BS_SERIAL# BS_INST_ID
----------------------------------- ----------------------------- --------------- ------------- ------------------------------ ---------------- ---------- ----------
27-OCT-17 07.00.05.868 AM           oracle@anbob2 (LCK0)          anbob2                        latch: shared pool                        11596      14855          2
27-OCT-17 07.00.26.527 AM           oracle@anbob2 (LCK0)          anbob2                        latch: shared pool                         2329      15239          2
27-OCT-17 07.00.47.258 AM           oracle@anbob2 (LCK0)          anbob2                        latch: shared pool                         3132       7895          2
27-OCT-17 07.01.49.888 AM           oracle@anbob2 (LCK0)          anbob2                                                                                             
27-OCT-17 07.02.00.367 AM           oracle@anbob2 (LCK0)          anbob2                        latch: shared pool                         7055      45921          2
27-OCT-17 07.02.10.667 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                   6283      57835          2
27-OCT-17 07.02.21.028 AM           oracle@anbob2 (LCK0)          anbob2                        latch: shared pool                         2329      15239          2
27-OCT-17 07.02.52.059 AM           oracle@anbob2 (LCK0)          anbob2                        latch: shared pool                        13863      58359          2
27-OCT-17 07.03.02.540 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                   6283      57835          2
27-OCT-17 07.03.23.199 AM           oracle@anbob2 (LCK0)          anbob2                        latch: shared pool                         5987      63625          2
27-OCT-17 07.03.33.569 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                                             
27-OCT-17 07.03.43.870 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                                             
27-OCT-17 07.03.54.239 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                                             
27-OCT-17 07.04.04.719 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                  11303      22599          2
27-OCT-17 07.04.15.039 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                                             
27-OCT-17 07.04.25.429 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                   9384      46843          2
27-OCT-17 07.04.35.820 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                  17485          3          2
27-OCT-17 07.04.46.199 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                  17485          3          2
27-OCT-17 07.04.56.519 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                                             
27-OCT-17 07.05.07.030 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                                             
27-OCT-17 07.05.17.419 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                                             
27-OCT-17 07.05.27.740 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                                             
27-OCT-17 07.05.38.140 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                                             
27-OCT-17 07.05.48.589 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                   5925         61          2
27-OCT-17 07.05.58.979 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                   5925         61          2
27-OCT-17 07.06.09.609 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                                             
27-OCT-17 07.06.20.060 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                   7339       3825          2
27-OCT-17 07.06.30.529 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                   3692      39497          2
27-OCT-17 07.06.40.930 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                                             
27-OCT-17 07.06.51.380 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                                             
27-OCT-17 07.07.02.030 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                  12996      63675          2
27-OCT-17 07.07.12.452 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                                             
27-OCT-17 07.07.22.920 AM           oracle@anbob2 (LCK0)          anbob2                        latch: row cache objects                                             
27-OCT-17 07.10.20.190 AM           oracle@anbob2 (LCK0)          anbob2                        latch: shared pool                                                   
27-OCT-17 07.11.22.280 AM           oracle@anbob2 (LCK0)          anbob2                        latch: shared pool                        11584      41211          2
                                                                                                                               
35 rows selected.

Note：
堵塞链条chains: 大部分session等待事件library cache lock
==>1:8502等待事件等待事件 “DFS lock handle”是在等待其它实例invalidation lock，也就是释放锁资源，用p1的值可以转换成具体的enq, 这里是一种IV, 一种跨实例的串行使library cache object InValidations的Library cache锁等待，通常发生在当发生了DDL或收集了统计信息后使相关的cursor失效。(FG process)
==> 2:10717 (LCK BG process) 此进程是lck0的后台进程，LCK进程出现latch:row cache objects的时间段，正是2个实例第一个负载高峰的时间段，Row cache 也叫dictionary cache，存储的是表、列、索引、用户等元数据。Lck 的wait通常都是很短暂，从LCK所全时段的wait event分析， 两次负载高峰时间段lck进程都是在等待latch:row cache objects, 是在加载或清理row cache object时申请的latch.

分析row cache objects
当lck0等待latch: row cache objects时，P1的值为13835058147926472600，因为篇幅原因不再附上

SQL> @dec 13835058147926472600

                                DEC                  HEX
----------------------------------- --------------------
        13835058147926472600.000000     C00000159207F798

SQL> select latch#,child# ,name,gets,misses from  v$latch_children where addr like 'C00000159207F798';

         LATCH#          CHILD# NAME                                                                        GETS          MISSES
--------------- --------------- ---------------------------------------------------------------- --------------- ---------------
            280              13 row cache objects                                                     2458650848         8927867
            
SQL>  select s.kqrstcln latch#,kqrstcid cache#,kqrsttxt name from x$kqrst s where kqrstcln=13;

         LATCH#          CACHE# NAME
--------------- --------------- --------------------------------
             13              16 dc_histogram_defs
             13              16 dc_histogram_data
             13              16 dc_histogram_data

TIP:
根据P1的值可以判断当时LCK进程等待持有latch对象是dc_histogram_*, 与计算表列上的柱状图相关。
从ASH后来的时间发现lck0出现了row cache cleaup, 用于清理row cache中的对象，结合IV enq，可以判断是在清理InValidations的对象，通常发现在当一个节点做了DDL或者DCL或shared pool内存紧张时清理row cache object，其它实例同样也需要清理与其相关的在row cache中的对象。

另外从DB ALERT LOG中发现该时间段出现过ora-4031

Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob2/trace/anbob2_ora_26816.trc  (incident=1565538):
ORA-04031: unable to allocate 20504 bytes of shared memory ("shared pool","unknown object","sga heap(4,0)","KTI SGA freea")
Incident details in: /oracle/app/oracle/diag/rdbms/anbob/anbob2/incident/incdir_1565538/anbob2_ora_26816_i1565538.trc


-- anbob2_ora_26816.trc
==============================
Memory Utilization of Subpool 4
================================
      Allocation Name            Size
___________________________  ____________
"free memory              "     223004992
"miscellaneous            "          1024
...

但从当时的trace文件中显示当时Subpool 4还有200MB左右的空闲空间，所以当时应该是shared pool中应该是存在碎片.同时从AWR的历史记录看shared pool中的较大的组件”KQR L PO”从晚2点做的分区维护以后一直从300MB增长到出问题时1200MB，最后成为shared pool中最大的资源内存区。同时对比前两个月例行分区维护时该组件不存在该现象最大到700M 然后逐渐释放，以上说明本月的row cache的内存管理出现问题。
KQR ==> [K]ernel [Q]uery Layer [R]ow Cache Management 是指ROW Cache Object管理的相关资源.

RAC中每个节点可能都有某张表引用信息在ROW(dictionary) cache中,当任何一个节点上修改了表的结构,都需要把所有节点上把与该表相关的对象置为invaild, 所以在传统的library cache lock外每个节点上的lck0 进程会对本 instance library cache中的对象加一个,shared-mode的iv(invalidation)instance lock,如果想修改对象定义,必须获取一个exclusive的 iv lock,会通知本地的lck0释放shared-mode lock,本地lck0在释放这个lock之前,会通知其它节点上的lck0,其它节点上的lck0收到这个消息后,就会将本地library cache中的所有相关对象置为invalidate属于广播机制有进程lmd 完成。所以oracle对这些资源采取了广播机制,每个节点上的变化都通知所有节点。

查看当时LCK进程的trace 文件。

loadavg : 0.09 0.12 0.17
Swapinfo :
        Avail = 629429.25Mb Used = 205304.09Mb
        Swap free = 424074.81Mb Kernel rsvd = 27521.14Mb
        Free Mem  = 316882.34Mb
  F S      UID   PID  PPID  C PRI NI             ADDR   SZ            WCHAN    STIME TTY       TIME COMD
3401 S   oracle 10708     1  0 178 20 e0000012722b7700 92499 e0000013cad17240  Jul 13  ?        1108:05 ora_lck0_tbcsb2
Short stack dump:
ksedsts()+544<-ksdxfstk()+48<-ksdxcb()+3216<-sspuser()+688<-<-_pw_wait()+48<-pw_wait()+112<-sskgpwwait()+432<-skgpwwait()+320<-kslges()+2208<-kslgetl()+784
<-kqrbip()+1136<-kqrbfr()+1232<-kqrbgl()+368<-ksbabs()+1008<-kclabs()+400<-ksbrdp()+2736<-opirip()+1296<-opidrv()+1152
<-sou2o()+256<-opimai_real()+352<-ssthrdmain()+576<-main()+336<-main_opd_entry()+80

 
 SO: 0xc000001624721918, type: 4, owner: 0xc000001642e31a08, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
     proc=0xc000001642e31a08, name=session, file=ksu.h LINE:12624 ID:, pg=0
    (session) sid: 10717 ser: 1 trans: 0x0000000000000000, creator: 0xc000001642e31a08
              flags: (0x51) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
              flags2: (0x409) -/-/INC
              DID: , short-term DID:
              txn branch: 0x0000000000000000
              oct: 0, prv: 0, sql: 0x0000000000000000, psql: 0x0000000000000000, user: 0/SYS
    ksuxds FALSE at location: 0
    service name: SYS$BACKGROUND
    Current Wait Stack:
     1: waiting for 'latch: row cache objects'
        address=0xc00000158216b238, number=0x118, tries=0x0
        wait_id=2575238331 seq_num=6566 snap_id=1
        wait times: snap=8.825235 sec, exc=8.825235 sec, total=8.825235 sec
        wait times: max=infinite, heur=8.825235 sec
        wait counts: calls=0 os=0
        in_wait=1 iflags=0x520
     0: waiting for 'row cache cleanup'
        =0x0, =0x0, =0x0
        wait_id=2575235501 seq_num=6565 snap_id=2830
        wait times: snap=0.000000 sec, exc=0.491436 sec, total=12 min 50 sec
        wait times: max=infinite, heur=12 min 50 sec
        wait counts: calls=0 os=0
        in_wait=1 iflags=0x15a2
    There are 2618 sessions blocked by this session.
    Dumping one waiter:
      inst: 1, sid: 9422, ser: 23985
      wait event: 'DFS lock handle'
        p1: 'type|mode'=0x49560005
        p2: 'id1'=0x4c4f434b
        p3: 'id2'=0x21
      row_wait_obj#: 4294967295, block#: 0, row#: 0, file# 0
      min_blocked_time: 631 secs, waiter_cache_ver: 447
    Wait State:
      fixed_waits=0 flags=0x22 boundary=0x0000000000000000/-1
    Session Wait History:
        elapsed time of 0.000000 sec since current wait
     0: waited for 'row cache cleanup'
        =0x0, =0x0, =0x0
        wait_id=2575235501 seq_num=6565 snap_id=2830
        wait times: snap=0.000126 sec, exc=0.491436 sec, total=12 min 41 sec
        wait times: max=infinite
        wait counts: calls=0 os=0
        occurred after 0.000000 sec of elapsed time
     1: waited for 'latch: shared pool'
        address=0xc00000006fee4438, number=0x133, tries=0x0
        wait_id=2575238330 seq_num=6564 snap_id=1
        wait times: snap=0.058322 sec, exc=0.058322 sec, total=0.058322 sec
        wait times: max=infinite
        wait counts: calls=0 os=0
        occurred after 0.000000 sec of elapsed time
     2: waited for 'row cache cleanup'
        =0x0, =0x0, =0x0
        wait_id=2575235501 seq_num=6563 snap_id=2829
        wait times: snap=0.000074 sec, exc=0.491310 sec, total=12 min 41 sec
        wait times: max=infinite
        wait counts: calls=0 os=0
        occurred after 0.000000 sec of elapsed time
     3: waited for 'latch: row cache objects'
        address=0xc00000158216b238, number=0x118, tries=0x0
        wait_id=2575238329 seq_num=6562 snap_id=1
        wait times: snap=14.639462 sec, exc=14.639462 sec, total=14.639462 sec
        wait times: max=infinite
        wait counts: calls=0 os=0
        occurred after 0.000000 sec of elapsed time
     4: waited for 'row cache cleanup'
        =0x0, =0x0, =0x0
        wait_id=2575235501 seq_num=6561 snap_id=2828
        wait times: snap=0.002836 sec, exc=0.491236 sec, total=12 min 27 sec
        wait times: max=infinite
        wait counts: calls=0 os=0
        occurred after 0.000000 sec of elapsed time
     5: waited for 'latch: shared pool'
        address=0xc00000006fee4438, number=0x133, tries=0x0
        wait_id=2575238328 seq_num=6560 snap_id=1
        wait times: snap=0.073927 sec, exc=0.073927 sec, total=0.073927 sec
        wait times: max=infinite
        wait counts: calls=0 os=0
        occurred after 0.000000 sec of elapsed time
     6: waited for 'row cache cleanup'
        =0x0, =0x0, =0x0
        wait_id=2575235501 seq_num=6559 snap_id=2827
        wait times: snap=0.000105 sec, exc=0.488400 sec, total=12 min 27 sec
        wait times: max=infinite
        wait counts: calls=0 os=0
        occurred after 0.000000 sec of elapsed time
     7: waited for 'latch: shared pool'
        address=0xc00000006fee4438, number=0x133, tries=0x0
        wait_id=2575238327 seq_num=6558 snap_id=1
        wait times: snap=0.029721 sec, exc=0.029721 sec, total=0.029721 sec
        wait times: max=infinite
        wait counts: calls=0 os=0
        occurred after 0.000000 sec of elapsed time
     8: waited for 'row cache cleanup'
        =0x0, =0x0, =0x0
        wait_id=2575235501 seq_num=6557 snap_id=2826
        wait times: snap=0.000074 sec, exc=0.488295 sec, total=12 min 27 sec
        wait times: max=infinite
        wait counts: calls=0 os=0
        occurred after 0.000000 sec of elapsed time
     9: waited for 'latch: row cache objects'
        address=0xc00000158216b238, number=0x118, tries=0x0
        wait_id=2575238326 seq_num=6556 snap_id=1
        wait times: snap=7.106811 sec, exc=7.106811 sec, total=7.106811 sec
        wait times: max=infinite
        wait counts: calls=0 os=0
        occurred after 0.000000 sec of elapsed time

*** 2017-10-27 07:03:44.749
LCK0 (ospid: 10708) has not moved for 35 sec (1509059024.1509058989)
--
*** 2017-10-27 07:04:04.789
LCK0 (ospid: 10708) has not moved for 55 sec (1509059044.1509058989)
--
*** 2017-10-27 07:04:24.829
LCK0 (ospid: 10708) has not moved for 74 sec (1509059063.1509058989)
--
*** 2017-10-27 07:04:44.869
LCK0 (ospid: 10708) has not moved for 95 sec (1509059084.1509058989)
--
*** 2017-10-27 07:05:04.909
LCK0 (ospid: 10708) has not moved for 115 sec (1509059104.1509058989)
--
...
--
*** 2017-10-27 07:06:45.110
LCK0 (ospid: 10708) has not moved for 215 sec (1509059204.1509058989)
--
*** 2017-10-27 07:07:05.150
LCK0 (ospid: 10708) has not moved for 235 sec (1509059224.1509058989)
--
*** 2017-10-27 07:07:25.190
LCK0 (ospid: 10708) has not moved for 255 sec (1509059244.1509058989)
--
*** 2017-10-27 07:22:52.475
LCK0 (ospid: 10708) has not moved for 39 sec (1509060172.1509060133)
--
*** 2017-10-27 07:26:14.056
LCK0 (ospid: 10708) has not moved for 25 sec (1509060373.1509060348)
--
*** 2017-10-27 07:26:34.096
LCK0 (ospid: 10708) has not moved for 45 sec (1509060393.1509060348)
--
*** 2017-10-27 07:26:54.167
...
...
...
*** 2017-10-27 08:07:20.780
LCK0 (ospid: 10708) has not moved for 1689 sec (1509062840.1509061151)
--
*** 2017-10-27 08:07:40.820
LCK0 (ospid: 10708) has not moved for 1709 sec (1509062860.1509061151)

Note:
期间连续累积的等待时间(已截取了部分输出)可以看出其间lck0进程出现了2次 not moved 超过100秒的时间段，和数据库负载高峰对应，最后一次是从7：26到8：07之间半小时左右lck0 都未活动。
LCK: This lock process manages requests that are not cache-fusion requests. Requests like row cache requests and library cache requests. Only a single LCK process is allowed for each instance.
PMON is waiting for an LCK process to cleanup the lock context after a foreground process died while doing a global cache lock operation.
The shared pool is stressed and memory need to be freed for the new cursors. As a consequence, the dictionary cache is reduced in size by the LCK process causing a temporal hang of the instance since the LCK can’t do other activity during that time. Since the dictionary cache is a memory area protected clusterwide in RAC, the LCK is responsible to free it in collaboration with the dictionary cache users (the sessions using cursors referenced in the dictionary cache). This process can be time consuming when the dictionary cache is big.

当shared pool里的对象需要为新的对象释放空间时如sql cursor, LCK进程降低Row Cache 大小期间使数据库临时hang， 因为在RAC环境中LCK进程负责释放持有row cache的用户进程协调工作及Library cache 的请求， 如果LCK出现性能问题也就会导致library cache object无法请求和会话补KILL后的释放row cache堵塞。Pmon 进程也需要等待LCK进程清理died进程后才能清理锁上下文。这个是为什么大量的等待library cache lock, library cache object load的事件和kill 会话无法使问题得到缓解的原因。 甚至因为中间件的session 重连机制，在kill 会话后大批量的新会话请求 也会加剧shared pool 占用的原因，在原资源未释放，新资源再大量请求时， shared pool内存管理出现问题 就更容易出现ora-4031。而且如要row cache越大后期LCK进程释放资源时就需要消耗更长的时间(每个instance 只有一个lck进程)。

该类现象相关的BUG：
Bug 8215444 Many processes hang in RAC with wait event “DFS lock handle”
Bug 16529874 Frequent “LCK(ospid: ****) has not moved for 20 sec” trace messages
BUG:8666117 – LCK0 PROCESS STUCK AT WAITING FOR “LATCH: ROW CACHE OBJECTS”
Bug 18199537 – RAC database becomes almost hung when large amount of row cache are used in shared pool
Bug 13814739 Excessive KQR X PO” allocations in a RAC environment (can cause ORA-4031)

只有18199537和当前的版本符合，同时也存在16529874提到的现象，开SR后也同样确认是该BUG。在Oracle最佳实践中也建议安装该Oneoff patch。

解决方案：
1，安装 one off patch 18199537 or upgrade 11.2.0.4.4 or above
2，同时建议修复16529874

前段时间朋友一套12C R2   3Nodes RAC on Linux using multipath 环境的crash总是不定时的crash(几天一次),  使用的是LINUX的多路径软件的默认配置。

— alert log

2017-11-21T21:52:48.474246+08:00
Thread 3 advanced to log sequence 26159 (LGWR switch)
  Current log# 17 seq# 26159 mem# 0: +DATADG/anbob/ONLINELOG/group_17
2017-11-21T21:55:42.614168+08:00
IMR0 (ospid: 53353) waits for event 'control file sequential read' for 58 secs.
2017-11-21T21:55:42.614240+08:00
IMR0 (ospid: 53353) is hung in an acceptable location (cfio 0x11.00).
2017-11-21T21:55:55.374988+08:00
IMR0 (ospid: 53353) waits for event 'control file sequential read' for 81 secs.
2017-11-21T21:55:58.554970+08:00
IMR has experienced some problems and can't re-enable, retry count 1
2017-11-21T21:56:20.580385+08:00
CKPT (ospid: 53295) waits for event 'control file parallel write' for 95 secs.
2017-11-21T21:56:22.776564+08:00
Errors in file /u01/app/oracle/diag/rdbms/anbob/anbob3/trace/anbob3_lmhb_53275.trc  (incident=824802) (PDBNAME=CDB$ROOT):
ORA-29770: global enqueue process CKPT (OSID 53295) is hung for more than 70 seconds
Incident details in: /u01/app/oracle/diag/rdbms/anbob/anbob3/incident/incdir_824802/anbob3_lmhb_53275_i824802.trc
2017-11-21T21:56:25.265258+08:00
LOCK_DBGRP: GCR_SYSTEST debug event locked group GR+DB_anbob by memno 0 
ERROR: Some process(s) is not making progress.
LMHB (ospid: 53275) is terminating the instance.
Please check LMHB trace file for more details.
Please also check the CPU load, I/O load and other system properties for anomalous behavior
ERROR: Some process(s) is not making progress.
LMHB (ospid: 53275): terminating the instance due to error 29770
2017-11-21T21:56:25.763179+08:00
System state dump requested by (instance=3, osid=53275 (LMHB)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/anbob/anbob3/trace/anbob3_diag_53235_20171121215625.trc
2017-11-21T21:56:27.521552+08:00
License high water mark = 172
2017-11-21T21:56:31.945109+08:00
Instance terminated by LMHB, pid = 53275
2017-11-21T21:56:31.945889+08:00
Warning: 2 processes are still attach to shmid 458758:
 (size: 24576 bytes, creator pid: 53153, last attach/detach pid: 53235)
2017-11-21T21:56:32.530103+08:00
USER (ospid: 34072): terminating the instance
2017-11-21T21:56:32.535249+08:00
Instance terminated by USER, pid = 34072

— lmnb trace anbob3_lmhb_53275_i824802.trc

----- Beginning of Customized Incident Dump(s) -----
==================================================
=== CKPT (ospid: 53295) Heartbeat Report
==================================================
CKPT (ospid: 53295) has no heartbeats for 103 sec. (threshold 70) 
  : heartbeat state 0x11.00 (cfio) pso-flag 0x100
  : waiting for event 'control file parallel write' for 97 secs with wait_id 4255818.
===[ Wait Chain ]===
Wait chain is empty.
=[SNAPSHOT SUMMARY]=
Process moved according to collected snapshots
===[NO-HB PROCESS CALLSTACK]===
ksedsts()+346<-ksdxfstk()+71<-ksdxcb()+912<-sspuser()+217<-__sighandler()<-io_queue_run()+100<-skgfrwat()+196
<-ksfdwtio()+306<-ksfdwat_internal()+234<-kfk_reap_ufs_async_io()+166<-kfk_reap_ios_from_subsys()+1000<-kfk_reap_ios()
+409<-kfk_io1()+1449<-kfk_transitIO()+451<-kfioReapIO()+633<-kfioRequestPriv()+216<-kfioRequest()+337<-ksfdafRequest()
+742<-ksfdafWait()+371<-ksfdwtio()+773<-ksfdwat_internal()+234<-ksfdblock()+325<-ksfdbio()+2910<-ksfdbio()+3035<-kccwbp()
+1035<-kccbmp_wdl()+257<-kccfhb()+144<-kcccmt()+244<-kccecx()+80<-kctmttrest()+303<-kctrcp()+1942<-kcvcca()+189<-ksbcti()
+247<-ksbabs()+11524<-ksbrdp()+1079<-opirip()+609<-opidrv()+602<-sou2o()+145<-opimai_real()+202<-ssthrdmain()+417<-main()
+262<-__libc_start_main()+253
==============================
==============================
Dumping PROCESS CKPT (ospid: 53295) States
==============================
===[ System Load State ]===
  CPU Total 56 Raw 56 Core 28 Socket 2
  Load normal: Cur 1144 Highmark 71680 (4.46 280.00)
===[ Latch State ]===
  Not in Latch Get
===[ Session State Object ]===
  ----------------------------------------
  SO: 0x23251ba70, type: 4, owner: 0x1f1d4c328, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
   proc=0x1f1d4c328, name=session, file=ksu.h LINE:15737, pg=0, conuid=1
   SGA version=(1,0)
  (session) sid: 4720 ser: 27225 trans: (nil), creator: 0x1f1d4c328
            flags: (0x51) USR/- flags2: (0x409) -/-/INC
            flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/-
            DID: 0003-0027-0000000A0000-0000-00000000, short-term DID: 
            txn branch: (nil)
            con_id/con_uid/con_name: 1/1/CDB$ROOT
            con_logonuid: 1 con_logonid: 1
            con_scuid: 1 con_scid: 1
            edition#: 0            user#/name: 0/SYS
            oct: 0, prv: 0, sql: (nil), psql: (nil)
            stats: 0x1ffa5a0b0, PX stats: 0x1101de44
  service name: SYS$BACKGROUND
  Current Wait Stack:
   0: waiting for 'control file parallel write'
      files=0x1, block#=0x27, requests=0x1
      wait_id=4255818 seq_num=61824 snap_id=1
      wait times: snap=1 min 37 sec, exc=1 min 37 sec, total=1 min 37 sec
      wait times: max=infinite, heur=1 min 37 sec
      wait counts: calls=0 os=0
      in_wait=1 iflags=0x5a0
  There are 4 sessions blocked by this session.
  Dumping one waiter:
    inst: 3, sid: 3389, ser: 36254
    wait event: 'enq: CF - contention'
      p1: 'name|mode'=0x43460004
      p2: '0'=0x0
      p3: 'operation'=0x0
    row_wait_obj#: 213, block#: 1969, row#: 0, file# 1
    min_blocked_time: 92 secs, waiter_cache_ver: 59413
  Wait State:
    fixed_waits=0 flags=0x22 boundary=(nil)/-1
  Session Wait History:
      elapsed time of 0.000027 sec since current wait
   0: waited for 'control file parallel write'
      files=0x1, block#=0x29, requests=0x1
      wait_id=4255817 seq_num=61823 snap_id=1
      wait times: snap=0.000462 sec, exc=0.000462 sec, total=0.000462 sec
      wait times: max=infinite
      wait counts: calls=0 os=0
      occurred after 0.000029 sec of elapsed time
   1: waited for 'control file parallel write'
      files=0x1, block#=0x372, requests=0x1
      wait_id=4255816 seq_num=61822 snap_id=1
      wait times: snap=0.000589 sec, exc=0.000589 sec, total=0.000589 sec
      wait times: max=infinite
      wait counts: calls=0 os=0
      occurred after 0.000051 sec of elapsed time
   2: waited for 'control file sequential read'
      file#=0x0, block#=0x371, blocks=0x1
      wait_id=4255815 seq_num=61821 snap_id=1
      wait times: snap=0.000279 sec, exc=0.000279 sec, total=0.000279 sec
      wait times: max=infinite
      wait counts: calls=0 os=0
      occurred after 0.000010 sec of elapsed time
   3: waited for 'control file sequential read'
      file#=0x0, block#=0x2a, blocks=0x1
      wait_id=4255814 seq_num=61820 snap_id=1
      wait times: snap=0.000272 sec, exc=0.000272 sec, total=0.000272 sec
      wait times: max=infinite
      wait counts: calls=0 os=0
      occurred after 0.000004 sec of elapsed time
   4: waited for 'control file sequential read'
      file#=0x0, block#=0x28, blocks=0x1
      wait_id=4255813 seq_num=61819 snap_id=1
      wait times: snap=0.000280 sec, exc=0.000280 sec, total=0.000280 sec
      wait times: max=infinite
      wait counts: calls=0 os=0
      occurred after 0.000009 sec of elapsed time
   5: waited for 'control file sequential read'
      file#=0x0, block#=0x1, blocks=0x1
      wait_id=4255812 seq_num=61818 snap_id=1
      wait times: snap=0.000319 sec, exc=0.000319 sec, total=0.000319 sec
      wait times: max=infinite
      wait counts: calls=0 os=0
      occurred after 0.000070 sec of elapsed time
   6: waited for 'enq: CF - contention'
      name|mode=0x43460005, 0=0x0, operation=0x0
      wait_id=4255811 seq_num=61817 snap_id=1
      wait times: snap=0.000257 sec, exc=0.000257 sec, total=0.000257 sec
      wait times: max=0.000000 sec
      wait counts: calls=3 os=3
      occurred after 0.000800 sec of elapsed time
   7: waited for 'rdbms ipc message'
      timeout=0x12c, =0x0, =0x0
      wait_id=4255810 seq_num=61816 snap_id=1
      wait times: snap=3.002193 sec, exc=3.002193 sec, total=3.002193 sec
      wait times: max=3.000000 sec
      wait counts: calls=1 os=1
      occurred after 0.000124 sec of elapsed time
   8: waited for 'control file sequential read'
      file#=0x0, block#=0x34a, blocks=0x1
      wait_id=4255809 seq_num=61815 snap_id=1
      wait times: snap=0.000284 sec, exc=0.000284 sec, total=0.000284 sec
      wait times: max=infinite
      wait counts: calls=0 os=0
      occurred after 0.000005 sec of elapsed time
   9: waited for 'control file sequential read'
      file#=0x0, block#=0x2a, blocks=0x1
      wait_id=4255808 seq_num=61814 snap_id=1
      wait times: snap=0.000341 sec, exc=0.000341 sec, total=0.000341 sec
      wait times: max=infinite
      wait counts: calls=0 os=0
      occurred after 0.000006 sec of elapsed time
	  
	  ...
	  ...
*** 2017-11-21T21:56:24.918643+08:00
loadavg : 4.47 2.09 1.21
System user time: 0.00 sys time: 0.00 context switch: 19791
Memory (Avail / Total) = 107148.16M / 257929.37M
Swap (Avail / Total) = 32768.00M /  32768.00M
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
0 S oracle   53295     1  0  80   0 - 19975755 read_e Nov13 ?     00:06:16 ora_ckpt_anbob3
Name:	ora_ckpt_anbob3
State:	S (sleeping)
Tgid:	53295
Pid:	53295
PPid:	1
TracerPid:	0
Uid:	602	602	602	602
Gid:	601	604	604	604
Utrace:	0
FDSize:	512
Groups:	601 602 603 604 606 
VmPeak:	79907024 kB
VmSize:	79903020 kB
VmLck:	       0 kB
VmHWM:	 3446908 kB
VmRSS:	 3438240 kB
VmData:	   25352 kB
VmStk:	     164 kB
VmExe:	  342688 kB
VmLib:	   30328 kB
VmPTE:	   14816 kB
VmSwap:	       0 kB
Threads:	1
SigQ:	1/1031607
SigPnd:	0000000000000000
ShdPnd:	0000000000000000
SigBlk:	0000000000000000
SigIgn:	0000000016400207
SigCgt:	00000003c9887cf8
CapInh:	0000000000000000
CapPrm:	0000000000000000
CapEff:	0000000000000000
CapBnd:	ffffffffffffffff
Cpus_allowed:	ffffff,ffffffff
Cpus_allowed_list:	0-55
Mems_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
Mems_allowed_list:	0-1
voluntary_ctxt_switches:	3914186
nonvoluntary_ctxt_switches:	43390
Short stack dump: 
ksedsts()+346<-ksdxfstk()+71<-ksdxcb()+912<-sspuser()+217<-__sighandler()<-io_queue_run()+100<-skgfrwat()+196<-ksfdwtio()
+306<-ksfdwat_internal()+234<-kfk_reap_ufs_async_io()+166<-kfk_reap_ios_from_subsys()+1000<-kfk_reap_ios()+409<-kfk_io1()
+1449<-kfk_transitIO()+451<-kfioReapIO()+633<-kfioRequestPriv()+216<-kfioRequest()+337<-ksfdafRequest()+742<-ksfdafWait()
+371<-ksfdwtio()+773<-ksfdwat_internal()+234<-ksfdblock()+325<-ksfdbio()+2910<-ksfdbio()+3035<-kccwbp()+1035<-kccbmp_wdl()
+257<-kccfhb()+144<-kcccmt()+244<-kccecx()+80<-kctmttrest()+303<-kctrcp()+1942<-kcvcca()+189<-ksbcti()+247<-ksbabs()+11524
<-ksbrdp()+1079<-opirip()+609<-opidrv()+602<-sou2o()+145<-opimai_real()+202<-ssthrdmain()+417<-main()+262<-__libc_start_main()+253
 

Note:
从日志看是CKPT进程hang超过了70秒所以LMON进程终止了实例， 从CKPT的trace文件中Call Stack看I/O的可能性较大， 开始有说这个数据库使用的是linux的多路径软件，使用的默认配置， 通常来说I/O错误对于ASM镜像和多路径以上的应用进程在做I/O操作时是通明的。但是在I/O重试期间可能会导致进程hang, 对于像CKPT\LMON 长时间的HANG是致命的， 在MOS中没有找到相关的BUG个NOTE但有一篇Extend RAC因远程IO失败导致数据库crash的case， 参照尝试性的调整了多路径的配置参数如下：

polling_interval= 10
no_path_retry = 3
/sys/module/scsi_transport_fc/parameters/dev_loss_tmo = 30

polling_interval: Specifies the interval between two path checks in seconds. For properly functioning paths, the interval between checks will gradually increase to (4 * polling_interval). The default value is 5.

no_path_retry: A numeric value for this attribute specifies the number of times the system should attempt to use a failed path before disabling queueing.
A value of fail indicates immediate failure, without queuing.A value of queue indicates that queuing should not stop until the path is fixed.

The “dev_loss_tmo” option controls how long the SCSI layer waits after a SCSI device fails before marking it as failed.

未提供OS,和多路径日志。 但做了以前调整重启了多路径服务后，观察了1个月时间数据库实例没有再crash过。

前段时间有套Oracle  Exadata Machine x2-2 2-nodes RAC 环境数据库两个db instance 前后crash, 并且自动启动失败，ASM 存储使用Normal 冗余，每个DG 3个CELL, 理论上坏一个CELL 应该是不会影响到数据库，简单记录分享一下这个故障和处理思路。

[oracle@kdexa1db01 (anbob1)oracle]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Fri Dec 1 09:51:38 2017
Copyright (c) 1982, 2011, Oracle.  All rights reserved.
Connected to an idle instance.

SQL> startup
ORA-00304: requested INSTANCE_NUMBER is busy

Note:
当前数据库是RAC环境，且当时节点2是活动状态，此时检查了与当前实例相关的进程是否存在，共享内存是否已释放，CRS是否在自动拉起DB instance, 参数文件是否真的istance_number重复。就在此时节点2也突然crash, 手动启动挂起.

数据库实例检查

-- db1 alert log 

Immediate Kill Session: sess: 0xa59411b58  OS pid: 32215
Fri Dec 01 09:40:17 2017
opiodr aborting process unknown ospid (32520) as a result of ORA-28
Fri Dec 01 09:40:22 2017
Errors in file /oracle/app/oracle/diag/rdbms/orarpt/anbob1/trace/anbob1_dia0_16581.trc  (incident=2085703):
ORA-32701: Possible hangs up to hang ID=109905 detected
Incident details in: /oracle/app/oracle/diag/rdbms/orarpt/anbob1/incident/incdir_2085703/anbob1_dia0_16581_i2085703.trc
DIA0 requesting termination of session sid:2334 with serial # 64947 (ospid:13691) on instance 2
    due to a LOCAL, HIGH confidence hang with ID=109905.
    Hang Resolution Reason: Although the number of affected sessions did not
    justify automatic hang resolution initially, this previously ignored
    hang was automatically resolved.
DIA0: Examine the alert log on instance 2 for session termination status of hang with ID=109905.
Fri Dec 01 09:40:28 2017
DSKM process appears to be hung. Initiating system state dump.
Fri Dec 01 09:40:28 2017
System state dump requested by (instance=1, osid=16565 (GEN0)), summary=[system state dump request (ksz_check_ds)].
System State dumped to trace file /oracle/app/oracle/diag/rdbms/orarpt/anbob1/trace/anbob1_diag_16567.trc
Fri Dec 01 09:40:45 2017
Immediate Kill Session#: 2661, Serial#: 19795
Immediate Kill Session: sess: 0xa493a7190  OS pid: 1420
...
...
..
Fri Dec 01 09:41:02 2017
ORA-00020: maximum number of processes (2000) exceeded
 ORA-20 errors will not be written to the alert log for
 the next minute. Please look at trace files to see all
 the ORA-20 errors.
Fri Dec 01 09:41:22 2017
Errors in file /oracle/app/oracle/diag/rdbms/orarpt/anbob1/trace/anbob1_dia0_16581.trc  (incident=2085704):
ORA-32701: Possible hangs up to hang ID=109909 detected
Incident details in: /oracle/app/oracle/diag/rdbms/orarpt/anbob1/incident/incdir_2085704/anbob1_dia0_16581_i2085704.trc
Fri Dec 01 09:41:40 2017
DIA0 terminating blocker (ospid: 1842 sid: 130 ser#: 63021) of hang with ID = 109906
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Although the number of affected sessions did not
    justify automatic hang resolution initially, this previously ignored
    hang was automatically resolved.
    by terminating session sid: 130 ospid: 1842
DIA0 successfully terminated session sid:130 ospid:1842 with status 31.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=109906.
DIA0 terminating blocker (ospid: 3355 sid: 2586 ser#: 14507) of hang with ID = 109907
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Although the number of affected sessions did not
    justify automatic hang resolution initially, this previously ignored
    hang was automatically resolved.
    by terminating session sid: 2586 ospid: 3355
DIA0 successfully terminated session sid:2586 ospid:3355 with status 31.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=109907.
Fri Dec 01 09:41:51 2017
DIA0 terminating blocker (ospid: 3311 sid: 2938 ser#: 12823) of hang with ID = 109908
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Although the number of affected sessions did not
    justify automatic hang resolution initially, this previously ignored
    hang was automatically resolved.
    by terminating session sid: 2938 ospid: 3311
DIA0 successfully terminated session sid:2938 ospid:3311 with status 31.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=109908.
DIA0 terminating blocker (ospid: 3333 sid: 899 ser#: 12861) of hang with ID = 109909
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Although the number of affected sessions did not
    justify automatic hang resolution initially, this previously ignored
    hang was automatically resolved.
    by terminating session sid: 899 ospid: 3333
Fri Dec 01 09:41:57 2017
Errors in file /oracle/app/oracle/diag/rdbms/orarpt/anbob1/trace/anbob1_pmon_16539.trc  (incident=2069655):
ORA-00600: internal error code, arguments: [ksz_cln_proc1], [0x94B89D2A0], [3], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/orarpt/anbob1/incident/incdir_2069655/anbob1_pmon_16539_i2069655.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /oracle/app/oracle/diag/rdbms/orarpt/anbob1/trace/anbob1_pmon_16539.trc:
ORA-00600: internal error code, arguments: [ksz_cln_proc1], [0x94B89D2A0], [3], [], [], [], [], [], [], [], [], []
PMON (ospid: 16539): terminating the instance due to error 472

Fri Dec 01 09:42:04 2017
ORA-1092 : opitsk aborting process
Instance terminated by PMON, pid = 16539
USER (ospid: 23180): terminating the instance
Termination issued to instance processes. Waiting for the processes to exit
Fri Dec 01 09:42:13 2017
Instance terminated by USER, pid = 23180
Fri Dec 01 09:43:48 2017
Starting ORACLE instance (normal)

...
...
...
Fri Dec 01 09:55:15 2017
MMON started with pid=48, OS id=18350
Fri Dec 01 09:55:15 2017
MMNL started with pid=50, OS id=18356
Fri Dec 01 09:55:28 2017
USER (ospid: 14048): terminating the instance due to error 304
Instance terminated by USER, pid = 14048
Fri Dec 01 09:57:23 2017
ORA-01565: Unable to open Spfile ?/dbs/spfile@.ora.

Note:
从db alert log可以看出， DSKM( DiskMon)进程开始出现了HANG,并且数据库因无法处理事务，逐渐积压导致数据库连接数满，随后出现了ORA-600 [ksz_cln_proc1] 内部错误，通常是因为ASM 反应慢或因为其它原因导致DSKM进程hang时，当PMON 清理终止的会话资源申请隔离请求失败也会提示这个600错误，当前版本有2个相关BUG。 还有能显示启动失败ORA-304，和spfile不可读，种种说明当前的ASM 出现问题，DB已经无法从ASM读写数据。在上面我的检查instance number是否重复时，create pfile from spfile就报出了spfile读取错误。

–DB1  ASM ALERT LOG

ri Dec 01 09:42:00 2017
NOTE: ASM client anbob1:anbob disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Trace records dumped in trace file /oracle/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_16769.trc
Fri Dec 01 09:45:33 2017
WARNING: client [anbob1:anbob] cleanup delayed; waited 213s, pid 16769 mbr 0x1
Fri Dec 01 09:46:33 2017
WARNING: client [anbob1:anbob] cleanup delayed; waited 273s, pid 16769 mbr 0x1
Fri Dec 01 09:47:34 2017
WARNING: client [anbob1:anbob] cleanup delayed; waited 333s, pid 16769 mbr 0x1
Fri Dec 01 09:48:34 2017
WARNING: client [anbob1:anbob] cleanup delayed; waited 393s, pid 16769 mbr 0x1
Fri Dec 01 09:48:55 2017
NOTE: timeout waiting for prior umbilllicus process to exit; 300s
Fri Dec 01 09:49:34 2017
WARNING: client [anbob1:anbob] cleanup delayed; waited 453s, pid 16769 mbr 0x1
Fri Dec 01 09:50:34 2017
WARNING: client [anbob1:anbob] cleanup delayed; waited 513s, pid 16769 mbr 0x1
Fri Dec 01 09:51:34 2017
WARNING: client [anbob1:anbob] cleanup delayed; waited 573s, pid 16769 mbr 0x1
Fri Dec 01 09:52:34 2017
WARNING: client [anbob1:anbob] cleanup delayed; waited 633s, pid 16769 mbr 0x1
Fri Dec 01 09:53:34 2017
WARNING: client [anbob1:anbob] cleanup delayed; waited 693s, pid 16769 mbr 0x1
Fri Dec 01 09:54:34 2017
WARNING: client [anbob1:anbob] cleanup delayed; waited 753s, pid 16769 mbr 0x1
Fri Dec 01 09:54:48 2017
NOTE: timeout waiting for prior umbilllicus process to exit; 300s
Fri Dec 01 09:55:34 2017
WARNING: client [anbob1:anbob] cleanup delayed; waited 813s, pid 16769 mbr 0x1
Fri Dec 01 09:56:34 2017
WARNING: client [anbob1:anbob] cleanup delayed; waited 873s, pid 16769 mbr 0x1
Fri Dec 01 09:57:32 2017
Errors in file /oracle/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14282.trc:
ORA-27603: Cell storage I/O error, I/O failed on disk o/192.168.160.37/DATA_CD_05_kdexa1cel03 at offset 0 for data length 4096
ORA-27626: Exadata error: 201 (Generic I/O error)
WARNING: Read Failed. group:1 disk:45 AU:0 offset:0 size:4096
path:o/192.168.160.37/DATA_CD_05_kdexa1cel03
         incarnation:0xe9681366 asynchronous result:'I/O error'
         subsys:OSS iop:0x7f163de0fa80 bufp:0x7f163ddbe400 osderr:0xc9 osderr1:0x0
Errors in file /oracle/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14282.trc:
ORA-27603: Cell storage I/O error, I/O failed on disk o/192.168.160.37/DATA_CD_07_kdexa1cel03 at offset 0 for data length 4096
ORA-27626: Exadata error: 201 (Generic I/O error)
WARNING: Read Failed. group:1 disk:41 AU:0 offset:0 size:4096
path:o/192.168.160.37/DATA_CD_07_kdexa1cel03
         incarnation:0xe9681362 asynchronous result:'I/O error'
         subsys:OSS iop:0x7f163de13680 bufp:0x7f163de78800 osderr:0xc9 osderr1:0x0
Errors in file /oracle/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14282.trc:
ORA-27603: Cell storage I/O error, I/O failed on disk o/192.168.160.37/DATA_CD_10_kdexa1cel03 at offset 0 for data length 4096
ORA-27626: Exadata error: 201 (Generic I/O error)
WARNING: Read Failed. group:1 disk:43 AU:0 offset:0 size:4096
path:o/192.168.160.37/DATA_CD_10_kdexa1cel03

...
...
...
Fri Dec 01 09:58:45 2017
NOTE: PST update grp = 1 completed successfully!
NOTE: initiating PST update: grp = 1, dsk = 38/0xe968135f, mask = 0x7e, op = clear
GMON updating disk modes for group 1 at 105 for pid 16, osid 13436
NOTE: group DATA: updated PST location: disk 0012 (PST copy 0)
NOTE: group DATA: updated PST location: disk 0008 (PST copy 1)
NOTE: cache closing disk 38 of grp 1: DATA_CD_03_KDEXA1CEL03
Fri Dec 01 09:58:45 2017
NOTE: process _b000_+asm1 (6940) initiating offline of disk 36.3915912029 (DATA_CD_08_KDEXA1CEL03) with mask 0x7e[0x15] in group 1
NOTE: process _b000_+asm1 (6940) initiating offline of disk 37.3915912030 (DATA_CD_00_KDEXA1CEL03) with mask 0x7e[0x15] in group 1
NOTE: process _b000_+asm1 (6940) initiating offline of disk 39.3915912032 (DATA_CD_09_KDEXA1CEL03) with mask 0x7e[0x15] in group 1
NOTE: process _b000_+asm1 (6940) initiating offline of disk 40.3915912033 (DATA_CD_04_KDEXA1CEL03) with mask 0x7e[0x15] in group 1
NOTE: process _b000_+asm1 (6940) initiating offline of disk 41.3915912034 (DATA_CD_07_KDEXA1CEL03) with mask 0x7e[0x15] in group 1
NOTE: process _b000_+asm1 (6940) initiating offline of disk 42.3915912035 (DATA_CD_11_KDEXA1CEL03) with mask 0x7e[0x15] in group 1
NOTE: process _b000_+asm1 (6940) initiating offline of disk 43.3915912036 (DATA_CD_10_KDEXA1CEL03) with mask 0x7e[0x15] in group 1
NOTE: process _b000_+asm1 (6940) initiating offline of disk 44.3915912037 (DATA_CD_01_KDEXA1CEL03) with mask 0x7e[0x15] in group 1
NOTE: process _b000_+asm1 (6940) initiating offline of disk 45.3915912038 (DATA_CD_05_KDEXA1CEL03) with mask 0x7e[0x15] in group 1
NOTE: process _b000_+asm1 (6940) initiating offline of disk 46.3915912039 (DATA_CD_06_KDEXA1CEL03) with mask 0x7e[0x15] in group 1
NOTE: process _b000_+asm1 (6940) initiating offline of disk 47.3915912040 (DATA_CD_02_KDEXA1CEL03) with mask 0x7e[0x15] in group 1
WARNING: Disk 36 (DATA_CD_08_KDEXA1CEL03) in group 1 in mode 0x15 is now being taken offline on ASM inst 1
WARNING: Disk 37 (DATA_CD_00_KDEXA1CEL03) in group 1 in mode 0x15 is now being taken offline on ASM inst 1
WARNING: Disk 39 (DATA_CD_09_KDEXA1CEL03) in group 1 in mode 0x15 is now being taken offline on ASM inst 1
WARNING: Disk 40 (DATA_CD_04_KDEXA1CEL03) in group 1 in mode 0x15 is now being taken offline on ASM inst 1
WARNING: Disk 41 (DATA_CD_07_KDEXA1CEL03) in group 1 in mode 0x15 is now being taken offline on ASM inst 1
WARNING: Disk 42 (DATA_CD_11_KDEXA1CEL03) in group 1 in mode 0x15 is now being taken offline on ASM inst 1
WARNING: Disk 43 (DATA_CD_10_KDEXA1CEL03) in group 1 in mode 0x15 is now being taken offline on ASM inst 1
WARNING: Disk 44 (DATA_CD_01_KDEXA1CEL03) in group 1 in mode 0x15 is now being taken offline on ASM inst 1
WARNING: Disk 45 (DATA_CD_05_KDEXA1CEL03) in group 1 in mode 0x15 is now being taken offline on ASM inst 1
WARNING: Disk 46 (DATA_CD_06_KDEXA1CEL03) in group 1 in mode 0x15 is now being taken offline on ASM inst 1
WARNING: Disk 47 (DATA_CD_02_KDEXA1CEL03) in group 1 in mode 0x15 is now being taken offline on ASM inst 1
WARNING: Disk 38 (DATA_CD_03_KDEXA1CEL03) in group 1 in mode 0x1 is now being taken offline on ASM inst 1
NOTE: initiating PST update: grp = 1, dsk = 36/0xe968135d, mask = 0x6a, op = clear

Note:
ASM在处理client进程清理时延时，随后报出了Cell storage I/O error，CELL03(192.168.160.37)读写错误，及cell03相关的DISK全部OFFLINE。v$asm_disk视图中mount_status显示为”MISSING” , 可以在下面的脚本在数据库中查询：

col PATH for a20
col DG_NAME for a15
col DG_STATE for a10
col FAILGROUP for a15
col name for a20
set lines 200 pages 10000

select dg.name dg_name, dg.state dg_state, dg.type, d.disk_number dsk_no,
d.path, d.mount_status, d.FAILGROUP,d.name, d.state 
from v$asm_diskgroup dg, v$asm_disk d
where dg.group_number=d.group_number
order by dg_name, dsk_no;

— db2 db alert log

-- node2 db alert log 
Fri Dec 01 09:45:14 2017
ORA-00020: maximum number of processes (2000) exceeded
 ORA-20 errors will not be written to the alert log for
 the next minute. Please look at trace files to see all
 the ORA-20 errors.
Fri Dec 01 09:45:38 2017
Immediate Kill Session#: 191, Serial#: 8141
...
Immediate Kill Session: sess: 0xa51130b58  OS pid: 18573
Immediate Kill Session#: 2808, Serial#: 13305
Immediate Kill Session: sess: 0xa31411a00  OS pid: 21909
Immediate Kill Session#: 2948, Serial#: 11987
Immediate Kill Session: sess: 0xa4140b1d8  OS pid: 21913
Fri Dec 01 09:45:39 2017
opiodr aborting process unknown ospid (14209) as a result of ORA-28
Fri Dec 01 09:46:39 2017
DSKM process appears to be hung. Initiating system state dump.
Fri Dec 01 09:47:15 2017
IPC Send timeout detected. Sender: ospid 21743 [oracle@kdexa1db02.hebei.mobile.com (TNS V1-V3)]
Receiver: inst 1 binc 429466025 ospid 16596
Fri Dec 01 09:47:16 2017
IPC Send timeout detected. Sender: ospid 29713 [oracle@kdexa1db02.hebei.mobile.com]
Receiver: inst 1 binc 429465961 ospid 16585
...
IPC Send timeout detected. Sender: ospid 29739 [oracle@kdexa1db02.hebei.mobile.com]
Receiver: inst 1 binc 429465961 ospid 16585
Fri Dec 01 09:47:16 2017
IPC Send timeout detected. Sender: ospid 29709 [oracle@kdexa1db02.hebei.mobile.com]
Receiver: inst 1 binc 429466025 ospid 16591
IPC Send timeout to 1.0 inc 4 for msg type 12 from opid 201
IPC Send timeout to 1.1 inc 4 for msg type 36 from opid 148
IPC Send timeout: Terminating pid 201 osid 29739
IPC Send timeout: Terminating pid 148 osid 29709
...
IPC Send timeout detected. Sender: ospid 29785 [oracle@kdexa1db02.hebei.mobile.com]
Receiver: inst 1 binc 429465961 ospid 16585
Fri Dec 01 09:47:16 2017
IPC Send timeout detected. Sender: ospid 29783 [oracle@kdexa1db02.hebei.mobile.com]
Receiver: inst 1 binc 429465961 ospid 16585
IPC Send timeout to 1.0 inc 4 for msg type 12 from opid 230
Fri Dec 01 09:47:16 2017
IPC Send timeout detected. Sender: ospid 29789 [oracle@kdexa1db02.hebei.mobile.com]
Receiver: inst 1 binc 429466025 ospid 16596
IPC Send timeout to 1.0 inc 4 for msg type 12 from opid 227
IPC Send timeout to 1.2 inc 4 for msg type 32 from opid 232
IPC Send timeout: Terminating pid 230 osid 29785
IPC Send timeout: Terminating pid 227 osid 29783
IPC Send timeout: Terminating pid 232 osid 29789
...
Receiver: inst 1 binc 429466025 ospid 16591
IPC Send timeout to 1.1 inc 4 for msg type 32 from opid 321
IPC Send timeout: Terminating pid 321 osid 29832
Fri Dec 01 09:47:18 2017
IPC Send timeout detected. Sender: ospid 29836 [oracle@kdexa1db02.hebei.mobile.com]
Receiver: inst 1 binc 429465961 ospid 16585
IPC Send timeout to 1.0 inc 4 for msg type 12 from opid 340
IPC Send timeout: Terminating pid 340 osid 29836
Fri Dec 01 09:47:37 2017
IPC Send timeout to 1.2 inc 4 for msg type 32 from opid 55
IPC Send timeout: Terminating pid 55 osid 21743
Fri Dec 01 09:48:08 2017
Errors in file /oracle/app/oracle/diag/rdbms/orarpt/orarpt2/trace/orarpt2_pmon_18433.trc  (incident=594684):
ORA-00600: internal error code, arguments: [ksz_cln_proc1], [0x9FD5C5C10], [3], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/orarpt/orarpt2/incident/incdir_594684/orarpt2_pmon_18433_i594684.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /oracle/app/oracle/diag/rdbms/orarpt/orarpt2/trace/orarpt2_pmon_18433.trc:
ORA-00600: internal error code, arguments: [ksz_cln_proc1], [0x9FD5C5C10], [3], [], [], [], [], [], [], [], [], []
PMON (ospid: 18433): terminating the instance due to error 472
Fri Dec 01 09:48:09 2017
ORA-1092 : opitsk aborting process
Fri Dec 01 09:48:10 2017
ORA-1092 : opitsk aborting process
Fri Dec 01 09:48:10 2017
License high water mark = 1902
Instance terminated by PMON, pid = 18433
USER (ospid: 3194): terminating the instance
Instance terminated by USER, pid = 3194

登录CellSever03 查看Alert log

[root@kdexa1cel03 localhost]# cd trace
/var/log/oracle/diag/asm/cell/localhost/trace
[root@kdexa1cel03 trace]# ls -l alert.log 
-rw-rw---- 1 root celladmin 773 Feb  2  2012 alert.log
[root@kdexa1cel03 trace]# cat alert.log 
cat: alert.log: Input/output error
[root@kdexa1cel03 trace]# more alert.log 
Bus error

[root@kdexa1cel03 ~]# ls
ls: reading directory .: Input/output error

Note:
当前的CellServer03 本地文件系统出现了异常， 此时为了尽快恢复业务， 因为当前DB和ASM之间无法通信，决定重启了节点1的CRS。 重启节点1 CRS后，数据库实例1恢复正常并且实例2之前启动hang也恢复正常， 只是因为少了一个CELL，ASM Diskgroup的总存储和可用存储少了一些,注意当前ASMDG 并不会立即DROP或rebalance， 需要调整DISK_REPAIR_TIME参数。

DISK_REPAIR_TIME

The DISK_REPAIR_TIME attribute of the disk group controls the maximum acceptable outage duration. Once one or more disks become unavailable to ASM, it will wait for up to the interval specified for DISK_REPAIR_TIME for the disk(s) to come online. If the disk(s) come back online within this interval, a resync operation will occur, where only the extents that were modified while the disks were offline are written to the disks once back online. If the disk(s) do not come back within this interval, ASM will initiate a forced drop of the disk(s), which will trigger a rebalance, in order to restore redundancy using the surviving disks. Once the disk(s) are back online, they will be added to the diskgroup, with all existing extents on those disks being ignored/discarded, and another rebalance will begin. In other words, the DISK_REPAIR_TIME value is the acceptable time of duration during you need to fix the failure. This setting is also the countdown timer of ASM to drop the disk(s) that have been taken offline. The default setting for DISK_REPAIR_TIME is 3.6 hours.

(1). 检查所有磁盘组的disk_repair_time，登录到ASM实例并执行以下查询：

SQL> select dg.name, a.value from v$asm_diskgroup dg, v$asm_attribute a where dg.group_number=a.group_number and a.name='disk_repair_time';

(2). 修改所有磁盘组的disk_repair_time：

SQL> ALTER DISKGROUP <DISKGROUP NAME> SET ATTRIBUTE 'DISK_REPAIR_TIME'='8.5H';

如果属于CELL的磁盘都已经OFFLINE像本案例这样，就不能再使用上面的方法，因为Cell本身已经不可用, 需要使用下面的方法延长disk_repair_time:

SQL>   ALTER DISKGROUP <DISKGROUP NAME> OFFLINE DISKS IN FAILGROUP <FAILGROUP NAME> DROP AFTER 24H;

验证所有磁盘是否都已修改

SQL>  select dg.name ,a.value from v$asm_diskgroup
  2          dg, v$asm_attribute a where dg.group_number=a.group_number and
  3          a.name ='disk_repair_time';

NAME                           VALUE
------------------------------ -------------------------------
DATA                           24h
DBFS_DG                        24h
FLASHTS                        24h
RECO                           24h

CellServer03 检查

1，机器面板无报错；
2，登录ILOM

[root@kdexa1db01 ~]# ssh 192.168.60.xx
Password: 
Oracle(R) Integrated Lights Out Manager
Version 3.0.16.10.d r74499
Copyright (c) 2012, Oracle and/or its affiliates. All rights reserved.


-> show /SP/faultmgmt

 /SP/faultmgmt
    Targets:
        shell
    Properties:
    Commands:
        cd
        show

-> start /SP/faultmgmt/shell
Are you sure you want to start /SP/faultmgmt/shell (y/n)? y

faultmgmtsp> fmadm faulty -a
No faults found
faultmgmtsp> 

3, 检查OS layer log

[root@kdexa1cel03 /]# dmesg
sd 0:2:1:0: rejecting I/O to offline device
sd 0:2:1:0: rejecting I/O to offline device
sd 0:2:1:0: rejecting I/O to offline device
sd 0:2:1:0: rejecting I/O to offline device
sd 0:2:1:0: rejecting I/O to offline device
sd 0:2:1:0: rejecting I/O to offline device
...

[root@kdexa1cel03 /]# lsscsi
[0:0:20:0]   enclosu SUN      HYDE12           0341  -       
[0:2:0:0]    disk    LSI      MR9261-8i        2.13  /dev/sda
[0:2:1:0]    disk    LSI      MR9261-8i        2.13  /dev/sdb
[0:2:2:0]    disk    LSI      MR9261-8i        2.13  /dev/sdc
[0:2:3:0]    disk    LSI      MR9261-8i        2.13  /dev/sdd
[0:2:4:0]    disk    LSI      MR9261-8i        2.13  /dev/sde
[0:2:5:0]    disk    LSI      MR9261-8i        2.13  /dev/sdf
[0:2:6:0]    disk    LSI      MR9261-8i        2.13  /dev/sdg
[0:2:7:0]    disk    LSI      MR9261-8i        2.13  /dev/sdh
[0:2:8:0]    disk    LSI      MR9261-8i        2.13  /dev/sdi
[0:2:9:0]    disk    LSI      MR9261-8i        2.13  /dev/sdj
[0:2:10:0]   disk    LSI      MR9261-8i        2.13  /dev/sdk
[0:2:11:0]   disk    LSI      MR9261-8i        2.13  /dev/sdl
[1:0:0:0]    disk    Unigen   PSA4000          1100  /dev/sdm
[8:0:0:0]    disk    ATA      MARVELL SD88SA02 D21Y  /dev/sdn
[8:0:1:0]    disk    ATA      MARVELL SD88SA02 D21Y  /dev/sdo
[8:0:2:0]    disk    ATA      MARVELL SD88SA02 D21Y  /dev/sdp
[8:0:3:0]    disk    ATA      MARVELL SD88SA02 D21Y  /dev/sdq
[9:0:0:0]    disk    ATA      MARVELL SD88SA02 D21Y  /dev/sdr
...

[root@kdexa1cel03 mptsas]# cat /proc/mdstat 
Personalities : [raid1] 
md4 : active raid1 sda1[2](F) sdb1[1]
      120384 blocks [2/1] [_U]
      
md5 : active raid1 sda5[0] sdb5[2](F)
      10485696 blocks [2/1] [U_]
      
md6 : active raid1 sda6[2](F) sdb6[1]
      10485696 blocks [2/1] [_U]
      
md7 : active raid1 sda7[0] sdb7[2](F)
      3145664 blocks [2/1] [U_]
      
md8 : active raid1 sda8[2](F) sdb8[1]
      3145664 blocks [2/1] [_U]
      
md2 : active raid1 sda9[0] sdb9[2](F)
      2097088 blocks [2/1] [U_]
      
md11 : active raid1 sda11[2](F) sdb11[1]
      5242752 blocks [2/1] [_U]
      
md1 : active raid1 sda10[2](F) sdb10[1]
      714752 blocks [2/1] [_U]
      
unused devices: 
	   
[root@kdexa1cel03 mptsas]# mdadm --detail /dev/md6
/dev/md6:
        Version : 0.90
  Creation Time : Thu Feb  2 15:16:27 2012
     Raid Level : raid1
     Array Size : 10485696 (10.00 GiB 10.74 GB)
  Used Dev Size : 10485696 (10.00 GiB 10.74 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 6
    Persistence : Superblock is persistent

    Update Time : Fri Dec  1 13:49:39 2017
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 1
  Spare Devices : 0

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       22        1      active sync   /dev/sdb6

       2       8        6        -      faulty spare   /dev/sda6

当前的文件系统出现异常，好多日志无法查看，当前情况解决方法也只能是重启操作系统。

通常LINUX中出现了BUS Error错误，意为的可能无法正常关机， 尝试了各种关机命令后果然， 需要去机房按电源。但是主机上如果无标签，就需要确认一下哪台是对应CELL03主机，可以使用点亮CELL主机前置灯的方法，同样更换硬盘时，也有点亮硬盘灯的方法。

使用点亮灯识别CELL主机的方法：

方法1

CellCLI> alter cell led on;
Cell kdexa1cel03 Locator LED switched on

CellCLI> alter cell led off;
Cell kdexa1cel03 Locator LED switched off

方法2

[root@kdexa1db01 ~]# ssh kdexa1cel03-ilom
Password:

Oracle(R) Integrated Lights Out Manager

Version 3.0.16.10.d r74499
Copyright (c) 2012, Oracle and/or its affiliates. All rights reserved.
-> set /SYS/LOCATE value=fast_blink
Set 'value' to 'fast_blink'

-> set /SYS/LOCATE value=off
Set 'value' to 'off'

然后按主机电源重启cell03 主机后，主机恢复正常，cell server也恢复正常，自动回归ASM diskgroup。– 前提未超过DISK_REPAIR_TIME

收集日志，提交SR

1, cell alert log
# /var/log/oracle/diag/asm/cell/localhost/trace

2, sundiag
# /opt/oracle.SupportTools/sundiag.sh

3, OS hardware log
# dmesg or cat /var/log/dmesg
# cat /var/log/messages

4, ILOM Snapshot
使用web 登录ILOM 省略
使用shell :

# ssh kdexa1cel03-ilom
Password:
Oracle(R) Integrated Lights Out Manager

Version 3.0.16.10.d r74499
Copyright (c) 2012, Oracle and/or its affiliates. All rights reserved.
-> set /SP/diag/snapshot dataset=normal
-> /SP/diag/snapshot dump_uri=sftp://root:welcome@192.168.60.42/tmp
-> cd /SP/diag/snapshot
-> show

多次检查状态，几分钟后会显示”Snapshot Complete” 会有个压缩文件生成并已FTP 到上面远程主机的/tmp 目录中。

5, sos report as root user
# sosreport

Summary:

因为cell03存储主机的文件系统异常，导致ASM Hang,数据库实例crash, 虽然是NORMAL级别的冗余，但是数据库实例此时不能于ASM通信，重启CRS进程恢复，可使用剩余的2条CELL继续为数据库提供服务。 在延长了disk_repair_time时间后，等待时间后强置重启CELL03主机操作系统后，一切恢复。 后EXADATA hardware部门并未发现硬件问题，Oracle Linux部门怀疑是文件系统问题， 说是可以使用e2fsck解决，未尝试. 但也未确认是否是因为MD层的软RAID问题。

Oracle ILOM服务快照工具用于收集Oracle服务人员使用的数据来诊断系统问题，除非Oracle Service要求收集，否则平时我们不用使用该工具。

创建ILOM Snapshot有两种方法，1. 通过WEB 接口使用浏览器登录ILOM; 2. 使用ILOM 命令行方式； 这里记录如何使用ILOM命令行:

从DB server查看ILOM 的IP
eg.

$ cat /etc/hosts

登录ILOM
eg.

$ ssh  exadcell03-ilom

Password: 
Oracle(R) Integrated Lights Out Manager
Version 3.0.16.10.d r74499
Copyright (c) 2012, Oracle and/or its affiliates. All rights reserved.

->

配置Snapshot选项和生成路径

->set /SP/diag/snapshot dataset=<data>
->set /SP/diag/snapshot dump_uri=<URI>
eg.

-> set /SP/diag/snapshot dataset=normal
-> /SP/diag/snapshot dump_uri=sftp://root:welcome@192.168.60.42/tmp

Oracle ILOM service processor (SP)

生成Snapshot

-> cd /SP/diag/snapshot
-> show

等待Snapshot Complete完成或查看show 的输出result:

取Snapshot 文件

上一步的命令执行成功后会在URI指定的程序中生成上面的zip包文件,生成如下格式：Machine name-ilom_192.168.60.42_2017-12-05T01-17-45.zip。unzip 查看CONFIG文件中有目录结构。