DB alert log frequently show the following error:

 Wed Dec 27 17:33:09 2017
Errors in file /ora/weejar/weejardb/9.2.0/admin/weejar_anbob1/udump/weejar_ora_4458.trc:
ORA-00600: internal error code, arguments: [kwqidrdq: loop], [0], [0], [0], [0], [], [], []
Wed Dec 27 17:34:14 2017
Errors in file /ora/weejar/weejardb/9.2.0/admin/weejar_anbob1/udump/weejar_ora_4458.trc:
ORA-00600: internal error code, arguments: [kwqidrdq: loop], [0], [0], [0], [0], [], [], []
Wed Dec 27 17:35:19 2017
Errors in file /ora/weejar/weejardb/9.2.0/admin/weejar_anbob1/udump/weejar_ora_4458.trc:
ORA-00600: internal error code, arguments: [kwqidrdq: loop], [0], [0], [0], [0], [], [], []
Wed Dec 27 17:36:24 2017
Errors in file /ora/weejar/weejardb/9.2.0/admin/weejar_anbob1/udump/weejar_ora_4458.trc:
ORA-00600: internal error code, arguments: [kwqidrdq: loop], [0], [0], [0], [0], [], [], []
Wed Dec 27 17:37:29 2017
Errors in file /ora/weejar/weejardb/9.2.0/admin/weejar_anbob1/udump/weejar_ora_4458.trc:

trace file show

Dump file /ora/weejar/db/weejardb/9.2.0/admin/weejar_anbob1/udump/weejar_ora_4458.trc
Oracle9i Enterprise Edition Release 9.2.0.6.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.6.0 - Production
ORACLE_HOME = /ora/weejar/db/weejardb/9.2.0
System name:	SunOS
Node name:	anbob
Release:	5.10
Version:	Generic_148888-01
Machine:	sun4v
Instance name: weejar
Redo thread mounted by this instance: 1
Oracle process number: 345
Unix process pid: 4458, image: oracle@anbob (TNS V1-V3)

*** 2017-12-26 18:04:09.055
*** SESSION ID:(81.52806) 2017-12-26 18:04:09.049
Txn id , Step# 213
*** 2017-12-26 18:04:09.055
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [kwqidrdq: loop], [0], [0], [0], [0], [], [], []
Current SQL statement for this session:
BEGIN WF_EVENT_OJMSTEXT_QH.DEQUEUE(:1, :2, :3); END;
----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
f00588bb0       221  package body SYS.DBMS_AQ
f08f4d660      1015  package body APPS.WF_EVENT_OJMSTEXT_QH
f86a0cc70         1  anonymous block
----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
ksedmp()+328         CALL     ksedst()             00000000B ? 000000000 ?
                                                   000000000 ? 00000004A ?
                                                   FFFFFFFF7FFF1B48 ?
                                                   10331A258 ?
kgerinv()+184        PTR_CALL 0000000000000000     00010373F ? 10373F000 ?
                                                   10373F068 ? 103744000 ?
                                                   000102C00 ? 000000000 ?
kgeasnmierr()+28     CALL     kgerinv()            10373F2C8 ? 1038D32F8 ?
                                                   0000013C8 ? 000000004 ?
                                                   1037413D4 ? 103740698 ?
kwqidrdq()+6480      CALL     kgeasnmierr()        10373F2C8 ? 1038D32F8 ?
                                                   1033DD6A8 ? 000000004 ?
                                                   000000004 ? 000000000 ?
kwqididq()+4728      CALL     kwqidrdq()           10373F068 ? 000000020 ?
                                                   FFFFFFFF7FFF5CA0 ?
                                                   000103744 ? 10373F068 ?
                                                   FFFFFFFF7FFF6408 ?
kwqiocideq()+3140    CALL     kwqididq()           10373F408 ? 000103400 ?
                                                   FFFFFFFF7FFF6408 ?
                                                   FFFFFFFF7CA7E2F8 ?
                                                   00010373F ? 00010373F ?
kwqidxdeq()+1816     CALL     kwqiocideq()         000000089 ? 103894838 ?
                                                   000000000 ? 10373F290 ?
                                                   000000540 ?
                                                   FFFFFFFFFFFFFFFF ?
kwqiadtdeq()+780     CALL     kwqidxdeq()          000000000 ? 1038ABF16 ?
                                                   000000089 ? 000000000 ?
                                                   000000000 ? 1038978C8 ?
spefcmpa()+1028      PTR_CALL 0000000000000000     10373F068 ? 000000000 ?
                                                   000000000 ? 1038979F8 ?
                                                   103897960 ? 103897858 ?
spefmccallstd()+184  CALL     spefcmpa()           FFFFFFFF7CA5B288 ?
                                                   103894838 ? 100DDE7A0 ?
                                                   FFFFFFFF7FFF7EB8 ?
                                                   000000000 ? 000000000 ?
peftrusted()+132     CALL     pextproc()           FFFFFFFF7FFF8C78 ?
                                                   FFFFFFFF7FFF8880 ?
                                                   FFFFFFFF7FFF8950 ?
                                                   FFFFFFFF7FFF88C0 ?
                                                   000000000 ? 102EF2000 ?

To search in MOS is matched Note 1070715.6

kwqid==》Streams Advanced Queuing Dequeue Call function

[R]eported on [D]equeue of a messages from an Advanced [Q]ueueing.

ORA-00600: [kwqidrdq: loop] reported on dequeue of a message from an Advanced Queueing (AQ) Multi-Consumer Queue Table.

Cause
In some cases rows have been manually deleted from the Queue Table

In Bug 9593216 the ORA-00600 [kwqidrdq: loop] is raised because a message enqueued with delay specified has been dequeued via msgid before the message state has been switched to READY.

The entry is no longer in the queue table and any other dequeue session which attempts to dequeue the message will report the ORA-00600 [kwqidrdq: loop].

WORKAROUND:
———–
Do not dequeue message with delay using message-id. Alternately the delay
period may be extended so that the race between the aq background and dequeue
can be avoided.

more to read MOS note 1070715.6

朋友有套数据库出现了ora-600 [kclfadd_1]引起数据库实例crash, 数据库版本10.2.0.2 2-nodes RAC, 找我帮分析原因，定位BUG后，在不升级版本并安装oneoff patch后，节点2实例无法启动，在startup 过程种从alert log看在提示了一些后台进程启动后，到LCK0 进程启动后没有任何报错挂起，节点1正常，记录一下分析过程。

ora-600 [kclfadd_1] 错误

Tue Dec 19 09:23:52 2017
Completed: ALTER DATABASE OPEN
Tue Dec 19 09:24:46 2017
Shutting down archive processes
Tue Dec 19 09:24:51 2017
ARCH shutting down
ARC2: Archival stopped
Tue Dec 19 09:53:07 2017
Errors in file /u01/admin/anbob/udump/anbob1_ora_389530.trc:
ORA-00600: 内部错误代码, 参数: [kclfadd_1], [], [], [], [], [], [], []
Tue Dec 19 09:53:17 2017
Errors in file /u01/admin/anbob/udump/anbob1_ora_389530.trc:
ORA-00600: 内部错误代码, 参数: [504], [0x700000010026AD8], [10], [2], [compile environment latch], [0], [0], [0x000000000]
ORA-00600: 内部错误代码, 参数: [kclfadd_1], [], [], [], [], [], [], []
Tue Dec 19 09:53:29 2017
Trace dumping is performing id=[cdmp_20171219095329]
Tue Dec 19 09:53:31 2017
Errors in file /u01/admin/anbob/udump/anbob1_ora_389530.trc:
ORA-00600: 内部错误代码, 参数: [kclfadd_1], [], [], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [kclfadd_1], [], [], [], [], [], [], []
Tue Dec 19 09:53:41 2017
Errors in file /u01/admin/anbob/udump/anbob1_ora_389530.trc:
ORA-00600: 内部错误代码, 参数: [504], [0x700000010026AD8], [10], [2], [compile environment latch], [0], [0], [0x000000000]
ORA-00600: 内部错误代码, 参数: [kclfadd_1], [], [], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [kclfadd_1], [], [], [], [], [], [], []
Tue Dec 19 09:53:53 2017
Trace dumping is performing id=[cdmp_20171219095353]
Tue Dec 19 09:53:56 2017
Errors in file /u01/admin/anbob/bdump/anbob1_pmon_516246.trc:
ORA-00600: internal error code, arguments: [kclfadd_1], [], [], [], [], [], [], []
Tue Dec 19 09:53:57 2017
Errors in file /u01/admin/anbob/bdump/anbob1_pmon_516246.trc:
ORA-00600: internal error code, arguments: [kclfadd_1], [], [], [], [], [], [], []
Tue Dec 19 09:53:57 2017
PMON: terminating instance due to error 472

MOS 确认为BUG,
ORA-00600: internal error code, arguments: [kclfadd_1], [], [], [], [], [], [], []
Cause
This problem only affects 64bit releases of Oracle.
In a RAC environment a global cache element structure can
become corrupted leading to various ORA-600 and instance crashes.

Solution
It is recommended to:
1) Apply a Patch Set where this issue is fixed (10.2.0.3 onwards)
or
2) Apply one off patch for your platform if it’s available, please check PATCH 5071492.

安装PATCH 5071492
下面是apply oneoff patch的日志

oracle@ibmp55a1[/u01/5077508]$opatch apply
Invoking OPatch 10.2.0.2.0

Oracle interim Patch Installer version 10.2.0.2.0
Copyright (c) 2005, Oracle Corporation.  All rights reserved..

Oracle Home       : /u01/product
Central Inventory : /u01/oraInventory
   from           : /u01/product/oraInst.loc
OPatch version    : 10.2.0.2.0
OUI version       : 10.2.0.2.0
OUI location      : /u01/product/oui
Log file location : /u01/product/cfgtoollogs/opatch/opatch-00_Dec_21_14-38-26-GMT+08_Thu.log

ApplySession applying interim patch '5077508' to OH '/u01/product'
Invoking fuser to check for active processes.
Invoking fuser on "/u01/product/bin/oracle"

OPatch detected the node list and the local node from the inventory.  OPatch will patch the local system then propagate the patch to the remote nodes.

This node is part of an Oracle Real Application Cluster.
Remote nodes: 'ibmp55a2' 
Local node: 'ibmp55a1'
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/product')
Is the local system ready for patching?
Do you want to proceed? [y|n]
y
User Responded with: Y
Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '5077508' for restore. This might take a while...
Backing up files affected by the patch '5077508' for rollback. This might take a while...
Patching component oracle.rdbms, 10.2.0.2.0...
Updating archive file "/u01/product/lib/libserver10.a"  with "lib/libserver10.a/kjb.o"
Running make for target ioracle
ApplySession adding interim patch '5077508' to inventory
Verifying the update...
Inventory check OK: Patch ID 5077508 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 5077508 are present in Oracle Home.

The local system has been patched.  You can restart Oracle instances on it.
Patching in rolling mode.
The node 'ibmp55a2' will be patched next.
Please shutdown Oracle instances running out of this ORACLE_HOME on 'ibmp55a2'.
(Oracle Home = '/u01/product')
Is the node ready for patching?
Do you want to proceed? [y|n]
n
User Responded with: N
ApplySession exits on request
You may exit the patching session and patch remaining nodes later from an un-patched node.  Do you want to continue?
Do you want to proceed? [y|n]
y
User Responded with: Y
Updating nodes 'ibmp55a2' 
   Apply-related files are:
     FP = "/u01/product/.patch_storage/5077508_Mar_3_2006_18_19_39/rac/copy_files.txt"
     DP = "/u01/product/.patch_storage/5077508_Mar_3_2006_18_19_39/rac/copy_dirs.txt"
     MP = "/u01/product/.patch_storage/5077508_Mar_3_2006_18_19_39/rac/make_cmds.txt"
Propagating files to remote nodes...
Propagating directories to remote nodes...
Running command on remote node 'ibmp55a2': cd /u01/product/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/product || echo REMOTE_MAKE_FAILED::>&2 
--------------------------------------------------------------------------------
WARNING for re-link on remote node 'ibmp55a2':
OPatch completed the command 'cd /u01/product/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/product || echo REMOTE_MAKE_FAILED::>&2 ' with warnings.
This command is from the file '/u01/product/.patch_storage/5077508_Mar_3_2006_18_19_39/rac/make_cmds.txt.instantiated', line number '1'
Probable cause:         chmod 755 /u01/product/bin - Linking Oracle     rm -f /u01/product/rdbms/lib/oracle     ld -b64 -o /u01/product/rdbms/lib/oracle -L/u01/product/rdbms/lib/ -L/u01/product/lib/  -bbigtoc -bnoipath -bI:/u01/product/lib/ksms.imp /u01/product/rdbms/lib/opimai.o /u01/product/rdbms/lib/ssoraed.o /u01/product/rdbms/lib/ttcsoi.o  -lperfsrv10 /u01/product/lib/nautab.o /u01/product/lib/naeet.o /u01/product/lib/naect.o /u01/product/lib/naedhs.o /u01/product/rdbms/lib/config.o -bI:/usr/lib/aio.exp   -lserver10 /u01/product/lib/libodm10.so -lnnet10  -lskgxp10 -lsthasgen10 /u01/product/has/lib/clssgc.o /u01/product/lib/libstskgxn2.a -lstocr10 -lstocrb10  -lstocrutl10 -lsthasgen10 /u01/product/has/lib/clssgc.o /u01/product/lib/libstskgxn2.a   -lclient10  -lvsn10  -lcommon10 -lgeneric10  /u01/product/rdbms/lib/defopt.o -lknlopt  `if /bin/ar -X64 tv /u01/product/rdbms/lib/libknlopt.a | grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap10 -bE:/u01/product/rdbms/lib/olap.exp" ; fi`  -lslax10 -lpls10  -lplp10 -bE:/u01/product/rdbms/lib/plsqlncomp.exp  /u01/product/lib/libstclsra10.a -lstdbcfg10 -lserver10 -lclient10  -lvsn10  -lcommon10 -lgeneric10  -lknlopt -lslax10 -lpls10  -lplp10  -ljox10 -bE:/u01/product/rdbms/lib//oracle.exp   `sed -e 's/-ljava//g' /u01/product/lib/ldflags`      -lncrypt10 -lnsgr10 -lnzjs10 -ln10  -lnnz10 -lnl10 -lnzjs10 -lnro10 `sed -e 's/-ljava//g' /u01/product/lib/ldflags`      -lncrypt10 -lnsgr10 -lnzjs10 -ln10  -lnnz10 -lnl10 -lnzjs10 -lclient10  -lvsn10  -lcommon10 -lgeneric10   -lmm -lsnls10 -lnls10  -lcore10 -lsnls10  -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10  -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `sed -e 's/-ljava//g' /u01/product/lib/ldflags`      -lncrypt10 -lnsgr10 -lnzjs10 -ln10  -lnnz10 -lnl10 -lnzjs10 -lnro10 `sed -e 's/-ljava//g' /u01/product/lib/ldflags`      -lncrypt10 -lnsgr10 -lnzjs10 -ln10  -lnnz10 -lnl10 -lnzjs10 -lclient10  -lvsn10  -lcommon10 -lgeneric10 -lpls10   -lsnls10 -lnls10  -lcore10 -lsnls10  -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10  -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10  -lclient10  -lvsn10  -lcommon10 -lgeneric10 -lsnls10 -lnls10  -lcore10 -lsnls10  -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10  -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lserver10 `if /bin/ar -X64 tv /u01/product/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo10"; fi` -lctxc10 -lctx10 -lzx10 -lgx10 -lctx10 -lzx10 -lgx10 -lordimt10  -lsnls10 -lnls10  -lcore10 -lsnls10  -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10  -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lsnls10 -lunls10  -bE:/u01/product/rdbms/lib//libcorejava.exp  -lld -lm `cat /u01/product/lib/sysliblist`  -lm  `if [ "\`/usr/bin/uname -v\`" = "4" ]; \        then echo "-bI:/u01/product/lib/pw-syscall.exp"; fi;`  `if /bin/ar -X64 t /u01/product/rdbms/lib/libknlopt.a | grep '^'kcsm.o > /dev/null 2>&1 ; then echo "-lha_gs_r -lha_em_r -lpthreads"; fi` -locijdbcst10  -lwwg  -bpT:0x100000000 -bpD:0x110000000 -bforceimprw       mv -f /u01/product/bin/oracle /u01/product/bin/oracleO  mv /u01/product/rdbms/lib/oracle /u01/product/bin/oracle        chmod 6751 /u01/product/bin/oracleld: 0711-415 WARNING: Symbol plzcls is already exported.ld: 0711-415 WARNING: Symbol plzexe is already exported.ld: 0711-415 WARNING: Symbol plzopn is already exported.ld: 
...
0711-783 WARNING: TOC overflow. TOC size: 141728        Maximum size: 65536     Extra instructions are being generated for each reference to a TOC      symbol if the symbol is in the TOC overflow area. :failed
The node 'ibmp55a2' has been patched.  You can restart Oracle instances on it.

There were relinks on remote nodes.  Remember to check the binary size and timestamp on the nodes 'ibmp55a2' .
The following make commands were invoked on remote nodes:
'cd /u01/product/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/product
'
--------------------------------------------------------------------------------
The following warnings have occurred during OPatch execution:
1) OUI-67212:
--------------------------------------------------------------------------------
WARNING for re-link on remote node 'ibmp55a2':
OPatch completed the command 'cd /u01/product/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/product || echo REMOTE_MAKE_FAILED::>&2 ' with warnings.
This command is from the file '/u01/product/.patch_storage/5077508_Mar_3_2006_18_19_39/rac/make_cmds.txt.instantiated', line number '1'
Probable cause:         chmod 755 /u01/product/bin - Linking Oracle     rm -f /u01/product/rdbms/lib/oracle     ld -b64 -o /u01/product/rdbms/lib/oracle -L/u01/product/rdbms/lib/ -L/u01/product/lib/  -bbigtoc -bnoipath -bI:/u01/product/lib/ksms.imp /u01/product/rdbms/lib/opimai.o /u01/product/rdbms/lib/ssoraed.o /u01/product/rdbms/lib/ttcsoi.o  -lperfsrv10 /u01/product/lib/nautab.o /u01/product/lib/naeet.o /u01/product/lib/naect.o /u01/product/lib/naedhs.o /u01/product/rdbms/lib/config.o -bI:/usr/lib/aio.exp   -lserver10 /u01/product/lib/libodm10.so -lnnet10  -lskgxp10 -lsthasgen10 /u01/product/has/lib/clssgc.o /u01/product/lib/libstskgxn2.a -lstocr10 -lstocrb10  -lstocrutl10 -lsthasgen10 /u01/product/has/lib/clssgc.o /u01/product/lib/libstskgxn2.a   -lclient10  -lvsn10  -lcommon10 -lgeneric10  /u01/product/rdbms/lib/defopt.o -lknlopt  `if /bin/ar -X64 tv /u01/product/rdbms/lib/libknlopt.a | grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap10 -bE:/u01/product/rdbms/lib/olap.exp" ; fi`  -lslax10 -lpls10  -lplp10 -bE:/u01/product/rdbms/lib/plsqlncomp.exp  /u01/product/lib/libstclsra10.a -lstdbcfg10 -lserver10 -lclient10  -lvsn10  -lcommon10 -lgeneric10  -lknlopt -lslax10 -lpls10  -lplp10  -ljox10 -bE:/u01/product/rdbms/lib//oracle.exp   `sed -e 's/-ljava//g' /u01/product/lib/ldflags`      -lncrypt10 -lnsgr10 -lnzjs10 -ln10  -lnnz10 -lnl10 -lnzjs10 -lnro10 `sed -e 's/-ljava//g' /u01/product/lib/ldflags`      -lncrypt10 -lnsgr10 -lnzjs10 -ln10  -lnnz10 -lnl10 -lnzjs10 -lclient10  -lvsn10  -lcommon10 -lgeneric10   -lmm -lsnls10 -lnls10  -lcore10 -lsnls10  -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10  -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `sed -e 's/-ljava//g' /u01/product/lib/ldflags`      -lncrypt10 -lnsgr10 -lnzjs10 -ln10  -lnnz10 -lnl10 -lnzjs10 -lnro10 `sed -e 's/-ljava//g' /u01/product/lib/ldflags`      -lncrypt10 -lnsgr10 -lnzjs10 -ln10  -lnnz10 -lnl10 -lnzjs10 -lclient10  -lvsn10  -lcommon10 -lgeneric10 -lpls10   -lsnls10 -lnls10  -lcore10 -lsnls10  -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10  -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10  -lclient10  -lvsn10  -lcommon10 -lgeneric10 -lsnls10 -lnls10  -lcore10 -lsnls10  -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10  -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lserver10 `if /bin/ar -X64 tv /u01/product/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo10"; fi` -lctxc10 -lctx10 -lzx10 -lgx10 -lctx10 -lzx10 -lgx10 -lordimt10  -lsnls10 -lnls10  -lcore10 -lsnls10  -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10  -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lsnls10 -lunls10  -bE:/u01/product/rdbms/lib//libcorejava.exp  -lld -lm `cat /u01/product/lib/sysliblist`  -lm  `if [ "\`/usr/bin/uname -v\`" = "4" ]; \        then echo "-bI:/u01/product/lib/pw-syscall.exp"; fi;`  `if /bin/ar -X64 t /u01/product/rdbms/lib/libknlopt.a | grep '^'kcsm.o > /dev/null 2>&1 ; then echo "-lha_gs_r -lha_em_r -lpthreads"; fi` -locijdbcst10  -lwwg  -bpT:0x100000000 -bpD:0x110000000 -bforceimprw       mv -f /u01/product/bin/oracle /u01/product/bin/oracleO  mv /u01/product/rdbms/lib/oracle /u01/product/bin/oracle        chmod 6751 /u01/product/bin/oracle ld: 0711-415 WARNING: Symbol plzcls is 
...
...
/lib//libserver10.a[dmbu.o], imported symbol _DBLINF       Symbol was expected to be local. Extra instructions     are being generated to reference the symbol.ld: 0711-783 WARNING: TOC overflow. TOC size: 141728        Maximum size: 65536     Extra instructions are being generated for each reference to a TOC      symbol if the symbol is in the TOC overflow area. :failed
--------------------------------------------------------------------------------
OPatch Session completed with warnings.

OPatch completed with warnings.
oracle@ibmp55a1[/u01/5077508]$opatch lsinventory
Invoking OPatch 10.2.0.2.0

Oracle interim Patch Installer version 10.2.0.2.0
Copyright (c) 2005, Oracle Corporation.  All rights reserved..

Oracle Home       : /u01/product
Central Inventory : /u01/oraInventory
   from           : /u01/product/oraInst.loc
OPatch version    : 10.2.0.2.0
OUI version       : 10.2.0.2.0
OUI location      : /u01/product/oui
Log file location : /u01/product/cfgtoollogs/opatch/opatch-00_Dec_21_14-40-04-GMT+08_Thu.log

Lsinventory Output file location : /u01/product/cfgtoollogs/opatch/lsinv/lsinventory-00_Dec_21_14-40-04-GMT+08_Thu.txt
--------------------------------------------------------------------------------
Installed Top-level Products (2): 
Oracle Database 10g                                                  10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 1                            10.2.0.2.0
There are 2 products installed in this Oracle Home.
Interim patches (1) :
Patch  5077508      : applied on Thu Dec 21 14:38:42 GMT+08:00 2017
   Created on 3 Mar 2006, 18:19:39 hrs US/Pacific
   Bugs fixed:
     5071492
Rac system comprising of multiple nodes
  Local node = ibmp55a1
  Remote node = ibmp55a2
--------------------------------------------------------------------------------
OPatch succeeded.
oracle@ibmp55a1[/u01/5077508]$

附这么大篇幅的日志为了更加准确的描述这个问题，这也是我了解到的唯一数据， 在节点1应用补丁后，再去2节点应用，包括后期的 — local 方式修复补丁都无法启动2节点。
1， 首先尝试停掉1节点，重启2节点， 问题现象一样
2， rollback 这个oneoff patch, 问题现象一样
3， 检查oracle 二进制执行文件大小和最后修改日期及权限，两个节点一致
4， 尝试手动relink问题节点

先对比$ORACLE_HOME/rdbms/lib 下类文件数量和大小是否一致？如果不一致把正常节点的lib下的文件复制到问题节点。

在当前版本有个bug 5128575  文件libknlopt.a不一致。尝试复制$ORACLE_HOME/rdbms/lib/libknlopt.a 从正常节点到问题节点，这个文件包含了oracle二进制文件中option的启用情况。如检查RAC 是否已启用：
On Linux/UNIX except AIX:

ar -t $ORACLE_HOME/rdbms/lib/libknlopt.a|grep kcsm.o

On AIX:

ar -X32_64 -t $ORACLE_HOME/rdbms/lib/libknlopt.a|grep kcsm.o

如禁用RAC option,可以使用：

$ cd $ORACLE_HOME/rdbms/lib
$ make -f ins_rdbms.mk rac_off  ioracle

从11.2起增加了一个新工具chopt同样可以启停option,如禁用分区option:

$ chopt disable partitioning

本案例虽然libknlopt.a在两个节点的文件一样大，但仍旧把正常节点的libknlopt.a文件复制到了问题节点后编译：

cd $ORACLE_HOME/rdbms/lib 
make -f ins_rdbms.mk ioracle

再次启动问题节点，恢复正常。 并且ora-600[kclfadd_1]的问题解决。

众所周知, oracle临时表空间是用于存储数据库的排序数据和全局临时表数据，tempfile不同于datafile因为不存在控制文件中，在恢复数据库后需要重建tempfile,也可以在standby环境中配置不同的temp表空间。用监控数据表空间间的脚本也许看到临时表空间使用率100%，但并且有时并非真实，DBA_FREE_SPACE 不会记录临时表空间的可用空间，使用v$sort_segments,V$TEMP_SPACE_HEADER和v$tempseg_usage监控临时表空间的使用，如果临时表空间不足会提示ORA-1652错误。对于临时表空间组特性是没有用的，为不同的用户分配不同的temp tablespace并且以用户分配到不同的实例可以减少temp空间争用。

临时表空间是被分成不同的extent,在11g r2中每个extent为1MB，extent maps是cache在本地的SGA中，并在再重启时不会保存extent map所以在实例重启后可以完全释放临时表空间的扩展， 临时表空间(cache extent map)是按需扩展的，用完后会标记为free但不会回收(如果不重启)，可以使用v$tempseg_usege查看当前实际使用。本实例cache的extent即使为free,对于其他实例也不可以立即使用，另一个实例需要申请新的uncache的extent, free extent只重用于本实例请求。当一个实例需要更多的extent时可以从另一个实例偷窃，另一个实例会uncache extent map，可以从gv$temp_extent_pool中观察,在11g中为一次请求100 extents,当本实例窃取够了足够的extent就可以继续使用temp extent，临时段的清理或合并是有每5分钟smon进程完成的。

SQL> select * from gv$temp_extent_pool;

   INST_ID TABLESPACE_NAME      FILE_ID EXTENTS_CACHED EXTENTS_USED BLOCKS_CACHED BLOCKS_USED BYTES_CACHED BYTES_USED RELATIVE_FNO
---------- ----------------- ---------- -------------- ------------ ------------- ----------- ------------ ---------- ------------
         1 TEMP                       1          11916            8       1525248        1024   1.2495E+10    8388608            1
         1 TEMP                       3           2884            1        369152         128   3024093184    1048576            3
         1 TEMP                       4           5990            3        766720         384   6280970240    3145728            4
         1 TEMP                       5           9422            1       1206016         128   9879683072    1048576            5
         1 TEMP                       6          10659            3       1364352         384   1.1177E+10    3145728            6
         2 TEMP                       1          18347            0       2348416           0   1.9238E+10          0            1
         2 TEMP                       3          12474            2       1596672         256   1.3080E+10    2097152            3
         2 TEMP                       4           9369            0       1199232           0   9824108544          0            4
         2 TEMP                       5           5937            3        759936         384   6225395712    3145728            5

近期有一套数据库出现了ORA-1652,extent map都已cache，显示使用率为100%，但当前真实使用的并不多，重启数据库不现实，我想通过drop tempfile再填加的方式释放已扩展的temp空间，结果发现了一个有意思的事情。

1, Drop Tempfile

SQL> @df

TABLESPACE_NAME                   TotalMB     UsedMB     FreeMB % Used Ext Used
------------------------------ ---------- ---------- ---------- ------ --- ----------------------
...
SYSAUX                              24318      17012       7306    70% NO  |##############      |
SYSTEM                              20478       2674      17804    14% NO  |###                 |
TEMP                               107517     107474         43   100% YES |####################|
UNDOTBS1                            61438      35678      25760    59% NO  |############        |
UNDOTBS2                            61438      13743      47695    23% NO  |#####               |
USERS                              330226     283772      46454    86% NO  |##################  |

SQL> @ls temp

TABLESPACE_NAME                   FILE_ID FILE_NAME                                EXT         MB      MAXSZ
------------------------------ ---------- ---------------------------------------- --- ---------- ----------
TEMP                                    3 /dev/yyc_oravg02/ryyc_lv15_128           NO       15359
TEMP                                    1 /dev/yyc_oravg02/ryyc_temp               YES      30719   32767.98
TEMP                                    2 /dev/yyc_oravg03/ryyc_lv15_240
TEMP                                    4 /dev/yyc_oravg04/ryyc_lv15_281           NO    15359.99
TEMP                                    5 /dev/yyc_oravg04/ryyc_lv15_282           NO    15359.99
TEMP                                    6 /dev/yyc_oravg04/ryyc_lv15_283           NO    15359.99
TEMP                                    7 /dev/yyc_oravg04/ryyc_lv15_284           NO    15359.99

SQL> alter tablespace temp drop tempfile '/dev/yyc_oravg04/ryyc_lv15_284';
Tablespace altered.

SQL> alter tablespace temp drop tempfile '/dev/yyc_oravg04/ryyc_lv15_283';
Tablespace altered.

SQL> alter tablespace temp add tempfile '/dev/yyc_oravg04/ryyc_lv15_284';
alter tablespace temp add tempfile '/dev/yyc_oravg04/ryyc_lv15_284'
*
ERROR at line 1:
ORA-01537: cannot add file '/dev/yyc_oravg04/ryyc_lv15_284' - file already part of database

SQL> @ls temp
TABLESPACE_NAME                   FILE_ID FILE_NAME                                                                        EXT         MB      MAXSZ
------------------------------ ---------- -------------------------------------------------------------------------------- --- ---------- ----------
TEMP                                    3 /dev/yyc_oravg02/ryyc_lv15_128                                                   NO       15359
TEMP                                    1 /dev/yyc_oravg02/ryyc_temp                                                       YES      30719   32767.98
TEMP                                    2 /dev/yyc_oravg03/ryyc_lv15_240
TEMP                                    4 /dev/yyc_oravg04/ryyc_lv15_281                                                   NO    15359.99
TEMP                                    5 /dev/yyc_oravg04/ryyc_lv15_282                                                   NO    15359.99
TEMP                                    6 /dev/yyc_oravg04/ryyc_lv15_283
TEMP                                    7 /dev/yyc_oravg04/ryyc_lv15_284

SQL> @df

TABLESPACE_NAME                   TotalMB     UsedMB     FreeMB % Used Ext Used
------------------------------ ---------- ---------- ---------- ------ --- ----------------------
...
SYSAUX                              24318      17014       7304    70% NO  |##############      |
SYSTEM                              20478       2674      17804    14% NO  |###                 |
TEMP                                76797      76754         43   100% YES |####################|
UNDOTBS1                            61438      35678      25760    59% NO  |############        |
UNDOTBS2                            61438      13726      47712    23% NO  |#####               |
USERS                              330226     283772      46454    86% NO  |##################  |

NOTE:
使用alter tablespace无法drop tempfile(从数据库释放tempfile),但是可用空间被收回。 正确的方法是使用alter database方式如下：

SQL> alter database tempfile '/dev/yyc_oravg03/ryyc_lv15_240'  drop including datafiles;
Database altered.

SQL> @ls temp

TABLESPACE_NAME                   FILE_ID FILE_NAME                                                                        EXT         MB      MAXSZ
------------------------------ ---------- -------------------------------------------------------------------------------- --- ---------- ----------
TEMP                                    3 /dev/yyc_oravg02/ryyc_lv15_128                                                   NO       15359
TEMP                                    1 /dev/yyc_oravg02/ryyc_temp                                                       YES      30719   32767.98
TEMP                                    4 /dev/yyc_oravg04/ryyc_lv15_281                                                   NO    15359.99
TEMP                                    5 /dev/yyc_oravg04/ryyc_lv15_282                                                   NO    15359.99
TEMP                                    6 /dev/yyc_oravg04/ryyc_lv15_283
TEMP                                    7 /dev/yyc_oravg04/ryyc_lv15_284

Note:
drop tempfile或drop temporary tablespace前提是当前无会话使用该文件和表空间，如果删除了所有的tempfile,在使用时会提示ora-25153错误，如果有会话使用该tempfile将报ORA-25152错误。

SQL> alter database tempfile '/dev/yyc_oravg04/ryyc_lv15_284'  drop including datafiles;
alter database tempfile '/dev/yyc_oravg04/ryyc_lv15_284'  drop including datafiles
*
ERROR at line 1:
ORA-25152: TEMPFILE cannot be dropped at this time

SQL> select u.inst_id,s.sid,s.status,tablespace,u.sql_id,segfile#,segfile#-p.value rfile 
from gv$tempseg_usage u,v$parameter p,gv$session s 
where u.session_addr = s.saddr AND u.inst_id = s.inst_id and p.NAME='db_files';

   INST_ID        SID STATUS   TABLESPACE                      SQL_ID            SEGFILE#      RFILE
---------- ---------- -------- ------------------------------- --------------- ---------- ----------
         1      17813 INACTIVE TEMP                            cbpmfx12jzj4r         2025          1
         1      17530 INACTIVE TEMP                            64tcqftuuwaan         2028          4
         1      17285 INACTIVE TEMP                            cbpmfx12jzj4r         2025          1
...
         2       4814 INACTIVE TEMP                            cbpmfx12jzj4r         2025          1
         2       4806 INACTIVE TEMP                            6h214g82rxb7x         2027          3
         1       4319 INACTIVE TEMP                            64tcqftuuwaan         2030          6
         1       1776 INACTIVE TEMP                            a6kb0pf8x0hqc         2029          5
         2         51 INACTIVE TEMP                            cbpmfx12jzj4r         2025          1
         1        285 INACTIVE TEMP                            6h214g82rxb7x         2031          7
         1          4 INACTIVE TEMP                            a6kb0pf8x0hqc         2028          4



SQL> select inst_id,tablespace,segtype,sum(blocks),count(*),sum(blocks)/count(*) from gv$tempseg_usage group by inst_id,tablespace,segtype;

   INST_ID TABLESPACE                      SEGTYPE   SUM(BLOCKS)   COUNT(*) SUM(BLOCKS)/COUNT(*)
---------- ------------------------------- --------- ----------- ---------- --------------------
         1 TEMP                            LOB_DATA         2176         17                  128
         2 TEMP                            LOB_DATA         1536         12                  128

Note:
上面持用tempfile rfile #7的会话是connect pool的长连接，在当前的数据库版本11.2.0.3(其实在9i-12c的版本都存在)，持有的temp直到该session退出才会释放。如删除本案例中的tempfile rfile 7#,Rfile# 7的tempfile正在被实例1的sid=285的会话使用，kill 掉实例1的285会话即可删除7# tempfile。

并且当前2个实例上一共29个会话，占用的都是temp类型都是temp lob data.每个session占用1M(128*8k). 注意该视图中的SQL_ID并不是产生temp seg的SQL，而是该会话最后一次执行的SQL, 在12C版本中v$tempseg_usage新增加列”SQL_ID_TEMPSEG”可以确认产生temp segment的SQLID.如下：

— 12.2

SQL> select username,session_num,sql_id,tablespace,CONTENTS,SEGTYPE,SEGRFNO#,BLOCKS,SQL_ID_TEMPSEG from v$tempseg_usage;
USERNAME SESSION_NUM SQL_ID TABLESPACE CONTENTS SEGTYPE SEGRFNO# BLOCKS SQL_ID_TEMPSE
-------------------- ----------- ------------- ------------------------------ --------- --------- ---------- ---------- -------------
18783 g44b5rqw7xwq7 TEMP TEMPORARY LOB_DATA 1 128 0w26sk6t6gq98

SQL> select sql_text from v$sqlarea where sql_id='0w26sk6t6gq98';

SQL_TEXT
--------------------------
SELECT XMLTYPE(DBMS_REPORT.GET_REPORT_WITH_SUMMARY(:B1 )) FROM DUAL

Note: SQL_ID_TEMPSEG 是产生temp的SQL，XML 也是LOB一种形式。

According to Bug 5723140,since 9i onwards we have been using only one temp segment for temp lobs in a session and all these temp lobs share the same temp segment. When there is a request to cleanup those temp lobs we do free space from temp tablespace and the same session can reuse it, but the temp segment is not released which means it is not available for use to other sessions. The actual cleanup happens only on session exit thru the callback that was registered for this purpose. Hence multiple sessions using temp lobs will have these temp segments active though unused, thus leading to temp segment growth.

正如note中描述的从9i以后，session使用一个临时段存在所有temp lobs,但是用完后free空间，同一会话可以reuse, 临时段只有到exit时才会释放，否则一直占用。如果session 使用了connect pool的长连接，并且会话数增长，temp segment会逐渐增长，不会让其它会话使用。 从10.2.0.4提供了一个event 60025,当系统级或会话级启用了该event,会在session中的temp lob不活动时(cache temp和nocache temp lobs都为0时)，即使不退出会话也会自己释放空间供其它会话使用。如果启用了60025 event像下面这引起temp lob 会被释放掉:

SQL> select * from gv$temporary_lobs;

   INST_ID        SID CACHE_LOBS NOCACHE_LOBS ABSTRACT_LOBS
---------- ---------- ---------- ------------ -------------
         2       4806          0            0             0
         2       5159          0            0             0
         2       6843          0            0             0
         2       7975          0            0             0
         2       8192          0            0             0
         2      10728          0            0             0
         1          4          0            0             0
         1       1776          0            0             0
         1       3460          0            0             0
         1       4319          0            0             0
         1       4799          0            0             0
         1       4887          0            0             0
         1       8220          0            0             0
         1       8544          0            0             0
         1      10478          0            0             0
         1      11031          0            0             0
         1      11110          0            0             0
         1      11369          0            0             0
         1      12983          0            0             0
         1      14120          0            0             0
         1      17285          0            0             0
         1      17530          0            0             0
         1      17813          0            0             0
         1      17872          0            0             0

60025 event不可以动态在系统级修改生效，在spfile级修改重启生效；session级启用60025后，也不会立即释放之前的temp lobs.下面做个测试:

[oracle@weejar1:/home/oracle] ora

SQL*Plus: Release 11.2.0.3.0 Production on Wed Jan 17 14:27:24 2018
Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options


USERNAME             INST_NAME            HOST_NAME                 SID   SERIAL#  VERSION    STARTED  SPID            OPID  CPID            SADDR            PADDR
-------------------- -------------------- ------------------------- ----- -------- ---------- -------- --------------- ----- --------------- ---------------- ----------------
SYS                  anbob1               weejar1                    17872 50597    11.2.0.3.0 20171114 19239           3327  19235           C0000016152D86D8 C000001652FD99F0

-ne 

SQL> declare
  2  b blob;
  3  begin
  4  dbms_lob.CREATETEMPORARY(b,true);
  5  dbms_lob.freeTEMPORARY(b);
  6  end;
  7  /

PL/SQL procedure successfully completed.

SQL> select * from gv$temporary_lobs;

   INST_ID        SID CACHE_LOBS NOCACHE_LOBS ABSTRACT_LOBS
---------- ---------- ---------- ------------ -------------
...
         1      14120          0            0             0
         1      17285          0            0             0
         1      17530          0            0             0
         1      17813          0            0             0
         1      17872          0            0             0 

24 rows selected.

SQL> select u.inst_id,s.sid,s.status,tablespace,u.sql_id,segfile#,u.blocks,segfile#-p.value rfile from gv$tempseg_usage u,v$parameter p,gv$session s where u.session_addr = s.saddr AND u.inst_id = s.inst_id and p.NAME='db_files';

   INST_ID        SID STATUS   TABLESPACE                      SQL_ID            SEGFILE#     BLOCKS      RFILE
---------- ---------- -------- ------------------------------- --------------- ---------- ---------- ----------
         1      17872 INACTIVE TEMP1                           g0wf0fkkqwaya         2033        128          9
         1      17813 INACTIVE TEMP                            cbpmfx12jzj4r         2025        128          1
...

SQL> alter session set events '60025 trace name context forever';
Session altered.

SQL> select u.inst_id,s.sid,s.status,tablespace,u.sql_id,segfile#,u.blocks,segfile#-p.value rfile from gv$tempseg_usage u,v$parameter p,gv$session s where u.session_addr = s.saddr AND u.inst_id = s.inst_id and p.NAME='db_files';

   INST_ID        SID STATUS   TABLESPACE                      SQL_ID            SEGFILE#     BLOCKS      RFILE
---------- ---------- -------- ------------------------------- --------------- ---------- ---------- ----------
         1      17872 INACTIVE TEMP1                           g0wf0fkkqwaya         2033        128          9
         1      17813 INACTIVE TEMP                            cbpmfx12jzj4r         2025        128          1
...

SQL> declare
  2  b blob;
  3  begin
  4  dbms_lob.CREATETEMPORARY(b,true);
  5  end;
  6  /

PL/SQL procedure successfully completed.

SQL> select * from gv$temporary_lobs;
-- sid 17872 released

Note:
因为temp seg同会话可以reuse, 再配置了60025 event后，再次执行lob操作后， temp lob自动清理。

系统级修改

alter system set event='60025 trace name context forever' scope=spfile;

对于已存在的会话可以使用oradebug 启用指定sid

oradebug setorapid <oracle pid>
oradebug event 60025 trace name context forever
-- wait the session next call lob to clean temp lob by itself

Oracle RAC环境中有时常因为LMS进程取不到足够的CPU而hang,最终在lmhb监控进程发现lms hung超过70秒后而终止实例。lmhb是11G R2引入的RAC环境中新的后台进程，用于监控LMON、LMD、LMSn等RAC关键的后台进程，确认本地以上background process不被阻塞或spin， LMHB是Lock Manager HeartBeat的缩写。LMHB进程的跟踪日志也成为诊断RAC故障的主要的日志文件。

主要收集的日志:
– Alert logs form all instances ( Cluster alert.log, Rdbms alert.log, ASM alert.log )
– ocssd.logs for all instances ( note older files are named ocssd.l01, .. )
– LMON, LMSn, LMD0 traces from all instances
– Any other traces mentioned in any alert.log
– lmhb traces ( LMHB monitors LMON, LMD, and LMSn processes to ensure they are running normally without blocking or spinning )
– CHM and OSWatcher logs from the eviction time
– OS message logs form all nodes ( /var/log/messages for Linux )

这里记录一种现象：
1, db alert log 显示ora-29770 和lms has not moved for xx sec.
2, lmhb 显示loadavg 负载高
3， top 显示lms进程占用CPU高
4， top or mpstat 显示sys cpu高
5， gc cr failure 状态值高
6， UDP error包增长
7， lms 进程显示call stack如下

__poll()+47<-ssskgxp_poll()+40<-sskgxp_select()+263<-skgxpiwait()+3680<-skgxpwaiti()+1544<-skgxpwait()+162<-ksxpwait()+2501<-ksliwat()+12852<-kslwaitctx()+1
63<-kslwait()+141<-ksxprcv_int()+6092<-ksxprcvimd()+36<-kjctr_rksxp()+313<-kjctrcv()+398<-kjcsrmg()+102<-kjmsm()+4953<-ksbrdp()+971<-opirip()+623<-opidrv()+
603<-sou2o()+103<-opimai_real()+266<-ssthrdmain()+252<-main()+201<-__libc_start_main()+230<-_start()+36

sendmsg()+16<-sskgxp_sndmsg()+444<-skgxpsegsnda()+146<-skgxpimcpy()+5195<-skgxpmcpy()+238<-ksxpmcpy_with_bcb()+1115<-kclbcpy2()+324<-kcl_snd_cur()+2376

可能的原因
因为操作系统在刷新大量的dirty pages使用了大量的sys cpu.

解决方法
调整内核参数
vm.dirty_background_ratio = 1
vm.dirty_ratio = 3

可以修改/etc/sysctl.conf 执行sysctl -p生效
或
# echo 1 > /proc/sys/vm/dirty_background_ratio
# echo 3 > /proc/sys/vm/dirty_ratio

oracle 18c当前很受关注的数据库，Oracle OOW上宣传的第一个自治数据库也是基于oracle 18c（实际oracle 18c不是自治数据库）。当前oracle 18c是只发布了在Oracle Cloud 和Oracle Exadata等Engineered Systems上版本，这也是之前我weibo（@weejar）调侃到以后的大版本第一版都是在云端和Engineered Systems首发，也就解决了出.1 版普遍不敢使用的尴尬^_^。
PS.因当前oracle 18c on-premise 还没正式发布，我只是用XD版简单测试几个小功能:

sys@cdb$root:anbob18c>  SELECT BANNER FROM V$VERSION;
BANNER
——————————————————————————–
Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 – Production

• Feedback sqlid
• private temporary table
• CANCEL SQL statement
• SEQUENCE增强
• Read-only Oracle Home (ROOH)
• DBMS_SESSION.SLEEP
• Inline External Table

1， Feedback sqlid

在SQL执行成功后，sqlplus 中返回SQL id.

sys@cdb$root:anbob18c> SET FEEDBACK ON SQL_ID
sys@cdb$root:anbob18c> select * from dual;
D
-
X

SQL_ID: a5ks9fhw2v9s1

2,  CANCEL SQL statement

18c 引入CANCEL SQL statement ，终止正在运行的SQL，而不是会话。

-- session 1
SQL> select count(*) from dba_objects,dba_objects,dba_objects;
running ....

-- session 2
sys@cdb$root:anbob18c> @usid 32

USERNAME                SID                 AUDSID OSUSER           MACHINE
----------------------- -------------- ----------- ---------------- ------------------
PROGRAM              SPID             OPID CPID                     SQL_ID         HASH_VALUE
-------------------- -------------- ------ ------------------------ ------------- -----------
  LASTCALL STATUS   SADDR            PADDR            TADDR            LOGON_TIME
---------- -------- ---------------- ---------------- ---------------- --------------------
SYS                      '32,52057'     4294967295 oracle           anbob
(TNS V1-V3)          5238               33 5237                     f5kskn9df2h2p  1524711509
         9 ACTIVE   000000006B2FC9E8 000000006CC99308                  03-MAR-2018 22:16:14


--语法：ALTER SYSTEM CANCEL SQL 'SID, SERIAL[, @INST_ID][, SQL_ID]';

sys@cdb$root:anbob18c> ALTER SYSTEM CANCEL SQL '32,52057,f5kskn9df2h2p'; 
System altered. 
-- session 1 
SQL> select count(*) from dba_objects,dba_objects,dba_objects; 
select count(*) from dba_objects,dba_objects,dba_objects 
* ERROR at line 1: 
ORA-01013: user requested cancel of current operation 

SQL> select distinct sid from v$mystat; 
SID 
---------- 
32

3,  private temporary table

在内存中创建的会话级和事务级临时表， 在会话或事务结束时会根据设置而丢弃。 有点像SQL server Tsql存储过程中的的临时表。有别于global temporary table，私有临时表在其它会话对象都不存在,表名只能是指定的参数开头。

sys@cdb$root:anbob18c> show parameter prefix

PARAMETER_NAME                                               TYPE        VALUE
------------------------------------------------------------ ----------- ------------------
common_user_prefix                                           string      C##
os_authent_prefix                                            string      ops$
private_temp_table_prefix                                    string      ORA$PTT_

SQL> create private temporary table ORA$PTT_MY_TT ( x int ) ;

SQL> select count(*)  
from   all_objects  
where  object_name = 'ORA$PTT_MY_T1' 
-- NONE

4, SEQUENCE增强

Sequence SCALE EXTEND ? 去年8月份就写过一篇不再描述 可以移步这里

sequence 可以reset了，不用删了重建。

sys@cdb$root:anbob18c> create sequence seq_1 start with 100;
Sequence created.
sys@cdb$root:anbob18c> select seq_1.nextval from dual connect by rownum<=3; 
NEXTVAL 
---------- 
100 
101 
102 
sys@cdb$root:anbob18c> alter sequence seq_1 restart;
Sequence altered.

sys@cdb$root:anbob18c> select seq_1.nextval from dual;

   NEXTVAL
----------
         1
sys@cdb$root:anbob18c> alter sequence seq_1 restart start with 100;
Sequence altered.

sys@cdb$root:anbob18c> select seq_1.nextval from dual;
   NEXTVAL
----------
       100

5, Read-only Oracle Home (ROOH)

只读ORACLE HOME是oracle 软件安装一大改进，ORACLE BASE不再允许和HOME同一个目录，同样也简化了安装。 对于Docker容器类环境非常合适， 把ORACLE HOME只读而把配置文件到放外面oraclebasehome下，更容易管理空间， 通过克隆Oracle Home轻松部署。
缺省默认ORACLE_HOME是read-write 模式，如果要启动read-only OH, 需要在安装完软件而创建数据库前用roohctl 工具改变，否则会提示
[oracle@anbob oracle]$ which roohctl
/u01/app/oracle/bin/roohctl
[oracle@anbob oracle]$ roohctl -enable
Enabling Read-Only Oracle home.
Cannot enable Read-Only Oracle home in a configured Oracle home.
The Oracle Home is configured with databases ‘anbob18c’.
虽然是roohctl 有一个force 选项(undocumented),但是转换后参数文件也不会转移到orabasehome下。roohctl工具在12.2时就已悄悄引入，只是到了18c -disable补去掉实际功能还存在。

[oracle@VM122 ~]$ roohctl -help
Usage:  roohctl [] [ ]
Following are the possible flags:
        -help
 
Following are the possible commands:
        -enable Enable Read-only Oracle Home
        -disable Disable Read-only Oracle Home

[oracle@anbob18c ~]$ roohctl -help
Usage:  roohctl [] []
Following are the possible flags: 
        -help 

Following are the possible commands: 
        -enable Enable Read-only Oracle Home
                [-nodeList List of nodes in a cluster environment]

启动方法
[oracle@VM181 18c]$ roohctl -enable
Enabling Read-Only Oracle home.
Update orabasetab file to enable Read-Only Oracle home.
Orabasetab file has been updated successfully.
Create bootstrap directories for Read-Only Oracle home.
Bootstrap directories have been created successfully.
Bootstrap files have been processed successfully.
Read-Only Oracle home has been enabled successfully.
Check the log file /u01/app/oracle/cfgtoollogs/roohctl/roohctl-180217PM111551.log.

禁用方法
[oracle@VM181 18c]$ roohctl -disable

启动了Read Only Oracle Home后，配置文件会在ORACLEBASEHOME /dbs下和以前一样保证唯一性。只不过oraclebasehome 在只读模式下ORACLE_BASE,而读写模式下是ORACLE_HOME 路径。 可以使用orabaseconfig查看orabasehome配置路径。同样可以查看orabasetab文件，如果最后一位是Y 说明是ROOH。

[oracle@anbob ~]$ cat $ORACLE_HOME/install/orabasetab
#orabasetab file is used to track Oracle Home associated with Oracle Base
/u01/app/oracle:/u01/orabase:OraDB18Home1:N:

The $ORACLE_HOME/install/orabasetab records the ORACLE_HOME, ORACLE_BASE and ORACLE_HOME_NAME:

6， DBMS_SESSION.sleep

在oracle 18c中sleep 存储过程填加到DBMS_SESSION PACKAGE中， 这样所有会话都可以直接使用，不需要再授权DBMS_LOCK PACKAGE。

sys@cdb$root:anbob18c> SET SERVEROUTPUT ON

sys@cdb$root:anbob18c> BEGIN
 DBMS_OUTPUT.put_line('Time 1: ' || TO_CHAR(SYSTIMESTAMP, 'HH24:MI:SS.FF'));
 -- Pause for 1.5 second.
 DBMS_SESSION.sleep(1.5);
 DBMS_OUTPUT.put_line('Time 2: ' || TO_CHAR(SYSTIMESTAMP, 'HH24:MI:SS.FF'));
END;
sys@cdb$root:anbob18c>
Time 1: 08:47:38.445748000
Time 2: 08:47:40.181196000
 PL/SQL procedure successfully completed.

7， Inline External Table

inline external table允许把外部表的定义写在SQL中，而不用先创建外部表对象。

[oracle@anbob ~]$ rm /tmp/et.txt
[oracle@anbob ~]$ rm /tmp/et1.txt
[oracle@anbob ~]$ for i in {1..5}; do echo $i',anbob'$i >> /tmp/et.txt; done; 
[oracle@anbob ~]$ cat /tmp/et.txt
1,anbob1
2,anbob2
3,anbob3
4,anbob4
5,anbob5
[oracle@anbob ~]$ for i in {6..10}; do echo $i',anbob'$i >> /tmp/et1.txt; done;  
[oracle@anbob ~]$ cat /tmp/et1.txt
6,anbob6
7,anbob7
8,anbob8
9,anbob9
10,anbob10

sys@cdb$root:anbob18c> CREATE OR REPLACE DIRECTORY tmp_dir1 AS '/tmp/';

Directory created.

sys@cdb$root:anbob18c>  SELECT *
  2  FROM   EXTERNAL (
  3           (
  4             id  number,
  5             name    VARCHAR2(128)
  6           )
  7           TYPE oracle_loader
  8           DEFAULT DIRECTORY tmp_dir1
  9           ACCESS PARAMETERS (
 10             RECORDS DELIMITED BY NEWLINE
 11             BADFILE tmp_dir1
 12             LOGFILE tmp_dir1:'inline_ext_tab_%a_%p.log'
 13             DISCARDFILE tmp_dir1
 14             FIELDS CSV WITH EMBEDDED TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
 15             MISSING FIELD VALUES ARE NULL (
 16               id,
 17   name
 18             )
 19          )
 20          LOCATION ('et.txt', 'et1.txt')
 21          REJECT LIMIT UNLIMITED
 22        ) inline_ext_tab;

        ID NAME
---------- ---------------------------------------------
         1 anbob1
         2 anbob2
         3 anbob3
         4 anbob4
         5 anbob5
         6 anbob6
         7 anbob7
         8 anbob8
         9 anbob9
        10 anbob10

What is the meaning of “_enable_shared_pool_durations” parameter?
With sga_target set, the shared pool and the streams pool have of subpools subpools for 4 durations.
The durations are “instance”, “session”, “cursor”, and “execution”.
The main advantage of “_enable_shared_pool_durations = false” is that all the durations are combined into one pool and so a duration will not run out while another duration has free memory. This is true for both the shared pool and the streams pool.

The disadvantage of the setting is that neither the streams pool nor the shared pool can shrink. (Without the setting, only the execution duration of the shared pool is eligible to shrink.)

The setting does not affect the large pool because the large pool is not divided into durations.

With sga_target set, all the pools grow by transferring granules from the buffer cache. Shrinks if enabled go back to the buffer cache. There is no direct transfer from one pool to another pool or from one duration to another duration. All transfers involve the buffer cache as source or target.

Note that “_kghdsidx_count=4” creates four “list-set” pools (standard subpools) for the each of the shared pool, large pool, and streams pool. Each list-set has its own kgh latch. The list-set pools are orthogonal to the duration subpools. So if sga_target is set and durations are not disabled, the shared pool and the streams pool have 16 subpool/duration combinations, i.e. 4 subpools * 4 durations.

Impact of setting _enable_shared_pool_durations = false

This will change the architecture of memory in the pools. When set to FALSE, subpools within the SGA will no longer have 4 durations. Instead, each subpool will have only a single duration. This mimics the behavior in 9i, and the shared pool will no longer be able to shrink.
The advantage of this is that the performance issues documented in this note can be avoided. A duration will not encounter memory exhaustion while another duration has free memory.
The disadvantage is that the shared pool (and streams pool) are not able to shrink, mostly negating the benefits of ASMM.

前段时间为了尽快体验18C的功能，在安装时少修改了一个文件参数，结果安装的数据库无法创建新的PDB(使用pdb$seed)，提示ORA-65036，并且db$seed Plugable database状态也有问题，这篇简单的记录一下如何修复PDB$SEED,先看一下我的场景。

SQL> CREATE PLUGGABLE DATABASE 
  2     anbob_pdb1 
  3  ADMIN USER pdbadmin IDENTIFIED BY oracle 
  4  FILE_NAME_CONVERT=('/u01/orabase/oradata/ANBOB18C/pdbseed/','/u01/orabase/oradata/ANBOB18C/anbob_pdb1/');
CREATE PLUGGABLE DATABASE
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []

Note: 查看trace文件是在查询temp file时有问题，新创建的CDB没有TEMP FILE, 解决方法很简单增加tempfile.

SQL> alter tablespace temp add tempfile '/u01/orabase/oradata/ANBOB18C/temp01.dbf' size 30m ;
Tablespace altered.

SQL> CREATE PLUGGABLE DATABASE 
  2     anbob_pdb1 
  3  ADMIN USER pdbadmin IDENTIFIED BY oracle 
  4  FILE_NAME_CONVERT=('/u01/orabase/oradata/ANBOB18C/pdbseed/','/u01/orabase/oradata/ANBOB18C/anbob_pdb1/');
CREATE PLUGGABLE DATABASE
*
ERROR at line 1:
ORA-65036: pluggable database PDB$SEED not open in required mode

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       MOUNTED

SQL> alter pluggable database pdb$seed open;
alter pluggable database pdb$seed open
*
ERROR at line 1:
ORA-65104: operation not allowed on an inactive pluggable database

SQL> ho oerr ora 65104
65104, 00000, "operation not allowed on an inactive pluggable database"
// *Cause:  The pluggable database status was INACTIVE. It was still being
//          created or there was an error during the create operation.
// *Action: Wait until the status is changed to CREATED or, in case of
//          errors, drop the pluggable database and re-create it.

Note:
当前的PDB$SEED状态有问题，正常情况下PDB$SEED应该是open read-only。 根据错误提示需要重建PDB$SEED。

Oracle 12c为Oracle数据库引入了多租户架构，其中一个容器数据库（CDB $ ROOT）可以具有多个可插拔数据库（PDB），在多租户体系结构中，理想情况下，我们使用SEED模板可插入数据库（PDB $ SEED）在容器数据库（CDB $ ROOT）内创建任何新的可插入数据库。 SEED可插入数据库（PDB $ SEED）充当创建新的可插拔数据库的模板，默认情况下不允许更改SEED可插入数据库的配置（默认情况下以READ ONLY模式打开），除非使用“_oracle_script”隐藏参数。

重建PDB$SEED的常见方法：

1. 使用建库时保存的脚本
2. 使用rman 恢复pdb$seed(前提有备份)
3. 使用同一个CDB的其它较干净的已存在的PDB创建PDB$SEED(不用备份)
4. 使用其它CDB的PDB$SEED复制PDB$SEED

使用建库时保存的脚本

SQL> drop pluggable database pdb$seed;
drop pluggable database pdb$seed
                        *
ERROR at line 1:
ORA-65017: seed pluggable database may not be dropped or altered

SQL> alter session set "_oracle_script"=true;
Session altered.

SQL> drop pluggable database pdb$seed;
Pluggable database dropped.

vi /u01/orabase/admin/anbob18c/scripts/initanbob18cTemp.ora
-- append 
_exadata_feature_on=true

[oracle@anbob scripts]$ cd /u01/orabase/oradata/ANBOB18C/pdbseed
[oracle@anbob pdbseed]$ ls
sysaux01.dbf  system01.dbf  temp012018-03-03_18-27-17-324-PM.dbf  undotbs01.dbf

[oracle@anbob pdbseed]$ rm *
[oracle@anbob pdbseed]$ ls

SQL> @plug_PDBSeed.sql 

SQL> CREATE PLUGGABLE DATABASE 
  2     anbob_pdb1 
  3  ADMIN USER pdbadmin IDENTIFIED BY oracle 
  4  FILE_NAME_CONVERT=('/u01/orabase/oradata/ANBOB18C/pdbseed/','/u01/orabase/oradata/ANBOB18C/anbob_pdb1/');

Pluggable database created.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ WRITE YES
         3 ANBOB_PDB1                     MOUNTED

使用rman 恢复pdb$seed

$ rman target /
RMAN> backup database "pdb$seed" ;
...

SQL> alter session set "_oracle_script"=true;
Session altered.

SQL> alter pluggable database "pdb$seed" close;
Pluggable database altered.

RMAN> restore datafile xxx;  -- pdb$seed datafile 

RMAN> alter session set "_oracle_script"=true;
Statement processed

RMAN> alter pluggable database "pdb$seed" open read only;
Statement processed

RMAN> alter session set "_oracle_script"=false;
Statement processed

已存在的PDB创建PDB$SEED

SQL>  alter session set "_oracle_script"=true;
Session altered.

SQL> alter pluggable database "pdb$seed" close;
Pluggable database altered.

SQL> drop pluggable database "pdb$seed" including datafiles;
Pluggable database dropped.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         3 ANBOB_PDB1                     READ WRITE NO

SQL> alter pluggable database ANBOB_PDB1 close;
Pluggable database altered.

SQL> alter pluggable database ANBOB_PDB1 open read only;
Pluggable database altered.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         3 ANBOB_PDB1                     READ WRITE NO

SQL> create pluggable database "PDB$SEED" from CDB2_PDB_2
   file_name_convert=('/u01/orabase/oradata/ANBOB18C/anbob_pdb1/','/u01/orabase/oradata/ANBOB18C/pdbseed/') ;
Pluggable database created.

SQL> alter session set "_oracle_script"=true;
Session altered.

SQL>  alter pluggable database PDB$SEED open read only;
 alter pluggable database PDB$SEED open read only
*
ERROR at line 1:
ORA-65085: cannot open pluggable database in read-only mode

SQL> alter pluggable database PDB$SEED open read write;
Pluggable database altered.

SQL> alter pluggable database PDB$SEED close;
Pluggable database altered.

SQL> alter pluggable database PDB$SEED open read only;
Pluggable database altered.

SQL> alter session set "_oracle_script"=false;
Session altered.

使用其它CDB的PDB$SEED复制PDB$SEED

创建一个dblink指向远程CDB
SQL> create database link rmt_seed_link xxxxxx ;

SQL> select con_id,name,open_mode from v$pdbs@rmt_seed_link where name='PDB$SEED';

    CON_ID NAME            OPEN_MODE
---------- --------------- ----------
         2 PDB$SEED        READ ONLY

SQL> exec DBMS_PDB.DESCRIBE(pdb_descr_file => '/home/oracle/seed_orpcdb1.xml', pdb_name => 'pdb$seed@REMOTE_SEED_LINK');
PL/SQL procedure successfully completed.

SQL> !ls -lrt /home/oracle/seed_orpcdb1.xml
-rw-r--r-- 1 oracle dba 5344 Apr 29 01:04 /home/oracle/seed_orpcdb1.xml

SQL> select name from v$datafile@RMT_SEED_LINK where con_id=2;
NAME
------------------------------------------------------------
/u01/orabase/oradata/ANBOB18C_R/pdbseed/system01.dbf
/u01/orabase/oradata/ANBOB18C_R/sysaux01.dbf
/u01/orabase/oradata/ANBOB18C_R/users01.dbf


SQL> select name from v$tempfile@RMT_SEED_LINK where con_id=2;
NAME
------------------------------------------------------------
/u01/orabase/oradata/ANBOB18C_R/temp01.dbf

$> scp 复制所有pdb$seed 数据文件从远程到本地

-- plug-in
SQL> create pluggable database "PDB$SEED" using '/home/oracle/seed_orpcdb1.xml'
  2  source_file_name_convert=('/u01/orabase/oradata/ANBOB18C_R/pdbseed/','/u01/orabase/oradata/ANBOB18C/pdbseed/')
  3  NOCOPY
  4  TEMPFILE REUSE
  5  ;

Pluggable database created.

— 另外12.2 以后提供的online pdb clone,也可以测试一下，参考之前的blog不再演示.

今天有朋友问她们的/var/log/messages (Linux syslog)文件有很多ogg的info信息，影响了操作系统日志的可读性，信息如下:

Apr 15 03:13:01 ccwlracnode01 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="21123" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Apr 16 15:43:32 ccwlracnode01 Oracle GoldenGate Command Interpreter for Oracle[188615]: 2018-04-16 15:43:32  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (ogguser): info all.
Apr 16 16:08:57 ccwlracnode01 Oracle GoldenGate Command Interpreter for Oracle[188615]: 2018-04-16 16:08:57  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (ogguser): info all.
Apr 16 16:09:12 ccwlracnode01 Oracle GoldenGate Command Interpreter for Oracle[188615]: 2018-04-16 16:09:12  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (ogguser): dblogin userid ogguser password ***.
Apr 16 16:09:23 ccwlracnode01 Oracle GoldenGate Command Interpreter for Oracle[188615]: 2018-04-16 16:09:23  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (ogguser): delete e_mysql.
Apr 16 16:09:43 ccwlracnode01 Oracle GoldenGate Command Interpreter for Oracle[188615]: 2018-04-16 16:09:43  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (ogguser): add ext e_mysql  tranlog, begin now, threads 2.
Apr 16 16:09:43 ccwlracnode01 Oracle GoldenGate Command Interpreter for Oracle[188615]: 2018-04-16 16:09:43  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (ogguser): add exttrail ./dirdat/my  ext e_mysql, megabytes 100.
Apr 16 16:09:49 ccwlracnode01 Oracle GoldenGate Command Interpreter for Oracle[188615]: 2018-04-16 16:09:49  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (ogguser): info all.

显然这和OGG_HOME目录中oggerr log文件中的内容相同，从ORACLE DOC上记录OGG所有日志是默认启用的。

On UNIX and Linux, Oracle GoldenGate messaging to thesyslog is enabled by default.

之前一直没有注意到这个问题，还不确认是从哪个版本来开引入的，但是从OGG v11.1.1.0.0 版本开始引入syslog 来控制OGG的消息写入操作系统日志(WIN&UNIX).

syslog用法
SYSLOG {[ALL | NONE] | [, INFO] [, WARN] [, ERROR]}

写日志的粒度写所有信息、禁止所有信息、和信息、告警、错误三个等级的任意组全。

可以配置在GLOBAL, mgr和不同的OGG 进程参数中， 在GLOBAL中配置应用于所有进程，如果在GLOBAL和进程参数中都配置了syslog, OGG进程参数的配置会覆盖GLOBAL中的配置.

ALL 是所有日志信息， NONE 是禁止写系统日志，剩下三个选项顾名思义。

如我们只希望把告警和错误信息写入系统日志，可以配置如下:
SYSLOG WARN, ERROR
— or:
SYSLOG ERROR, WARN

如果禁用OGG写系统日志配置如下：
SYSLOG NONE

在OGG v11.1.1.0.0版本以前没有办法控制OGG消息的传递，如果禁止OGG写SYSLOG, 只能通过操作系统层上的/etc/syslog.conf 中把user.info 或 *.info 去除。

新版本变化

在OGG V12.3 版本中syslog 已经弃用(deprecated),但不是不支持(unsupported), 可以使得另一种方法控制syslog的写入。修改$OGG_HOME/diretc/logging/目录下的ogg-ggserr.xml文件，去掉下面这部份的注释，重启MGR进程，又可以重新启用把ogg的消息写入syslog.

<!--
!- The ggserr.Syslog adapter will write all messages to the local
!- system log.
!-
<appender name="ggserr.Syslog" class="SyslogAppender">
<layout class="PatternLayout">
<param name="Pattern" value="%-19.19d{%Y-%m-%d %H:%M:%S} %m%n"/>
</layout>
</appender>
! -->

有朋友问我她的db alert日志里经常有下面的错误信息有没有影响？ 是一套12.1.0.2 的oracle RAC环境， 错误信息如下：

Trace file /u01/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29118.trc
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/12.1.0/db_1
System name:	Linux
Node name:	ccwlracnode01
Release:	3.8.13-68.3.4.el6uek.x86_64
Version:	#2 SMP Tue Jul 14 15:03:36 PDT 2015
Machine:	x86_64
Instance name: anbob1
Redo thread mounted by this instance: 1
Oracle process number: 115
Unix process pid: 29118, image: oracle@ccwlracnode01 (TNS V1-V3)


*** 2018-03-26 09:23:57.675
*** SESSION ID:(2397.39210) 2018-03-26 09:23:57.675
*** CLIENT ID:() 2018-03-26 09:23:57.675
*** SERVICE NAME:(SYS$USERS) 2018-03-26 09:23:57.675
*** MODULE NAME:(esfdaemon@ccwlracnode01 (TNS V1-V3)) 2018-03-26 09:23:57.675
*** CLIENT DRIVER:() 2018-03-26 09:23:57.675
*** ACTION NAME:() 2018-03-26 09:23:57.675
 
2018-03-26 09:23:57.674: [    GPNP]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:469] (:GPNP01002:)SlfFopen2
Internal Error Information: 
  Category: SLF_SYSTEM(-8)
  Operation: lstat failed
  Location: slsfopen3
  Other: 
  Dep: 13
  Dep Message: Permission denied
2018-03-26 09:23:57.675: [    GPNP]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:471] Result: (7) CLSGPNP_IO. (:GPNP01002:)Failed to open wallet file. dir 'peer' in '/u01/app/12.1.0/grid/gpnp/ccwlracnode01/wallets/', cannot check wallet home.
2018-03-26 09:23:57.675: [    GPNP]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:469] (:GPNP01002:)SlfFopen2
Internal Error Information: 
  Category: SLF_SYSTEM(-8)
  Operation: lstat failed
  Location: slsfopen3
  Other: 
  Dep: 13
  Dep Message: Permission denied

Note:
这是一个可以被忽略的提示信息， 只是当前版本中错误的提示“错误”信息， 已经确定为BUG 18893463 ，并在oracle 12.2 版本中修复。 当前的12.1 版本因影响较小未提供修复补丁，忽略即可。

Oracle MOS中新发布预警对于版本和对应版本的已安装PSU高于11.1.0.7.20，11.2.0.7.57(for Windows), 11.2.0.3.9，11.2.0.3.29 (for Windows), 11.2.0.3 BP22(for Exadata),  11.2.0.4，12.1.0.2 ，12.2.0.1 的Oracle 数据库， 将在2019年06月23日后自动调整SCN的增长速率为更大的上限，DBLINK 在不同数据库之间访问时会同步SCN， 为了避免不同SCN 上限数据库之间因SCN拒接访问，建议所有DBLINK访问的数据库升级为相同的SCN 兼容性。

Note: 如果不使用DBLINK，或也不存在scn headroom问题，可以不安装该补丁。

背景

Oracle的MOS上发布了两篇与DB Link（数据库链接）相关的预警文章，它们分别是：

• Mandatory Patching Requirement for Database Versions 11.2.0.3 or Earlier, Using DB Links (文档1)
• Oracle Databases Need to be Patched to a Minimum Patch set/PSU/RU level before April 2019 (文档1)

注意里面用到了Mandatory（必须的）这个词，引起一片恐慌，但是没隔几天，Oracle又重新修改了用词，把Mandatory（必须的）换成了Recommended（建议的）。 建议在2019年4月份前安装上对应的补丁和升级版本，实际的SCN新兼容性变化时间是2019年06月24日。

Oracle如何自动触发的?

在高版本的数据库中引入了SCN 兼容性特性( Compatibility)，而且在这个特性中设置了时间限制。在Oracle数据库软件内核中， 引入了一个：Auto-RollOver 的机制，也就是说Oracle 为不同 SCN 兼容性设定了触发时间，随着时间推移自动迭代，用户会在不知情的情况下自动应用了新的SCN 兼容性。

应用了补丁在2019年6月23后会发生什么？

不同的SCN兼容性又有不同的SCN速率限制，会有不同的SCN RSL(当前时间点的允许的最大SCN ).Oracle 为每个 SCN 的兼容性设置了时间点, 2019年6月23日直接跳级到兼容性 3， 3 级允许更高的 SCN 增长率, 但并不改变系统中SCN自身变化的速率，也不会修改当前的SCN，只是让系统在当前时间点可以生成更大的SCN，应对更忙的数据库，更高的事务速率。

如果所有数据库的事务处理速率没有任何重大变化, SCN都低于低速率的当前最大SCN允许值，应用补丁和未应用补丁的数据之间使用DBLINK还是可以正常的访问，不受应用补丁后允许更大速率的影响。

如果应用了补丁数据允许更大的增长速率，同时因为数据库SCN使用较快比如超过了32K每秒， 那当前SCN如果超过了未打补丁数据库的最大SCN，两个库通过DBLINK访问时就会因为无法同步SCN，而访问会被拒绝。

SCN 兼容级别

上面介绍了SCN兼容性级别的调整会改变SCN允许的最大增长速率限制和RSL（允许的SCN最大限制）， 从11.2.0.4版本数据库提供了一个dbms_scn package ，可以查看当前的SCN兼容级别，在$ORACLE_HOME/rdbms/admin/dbmsscnc.sql中可以找到其定义。使用这个package也可以启停Auto-rollover特性，和下一次调整的行为。

如何查看当前的SCN兼容性级别？

当前scn 兼容级别是1，最大级别为3

如何查看下一次SCN兼容性级别的调整？

当前auto rollover是启用的， 会在2019-6-23自动调整SCN兼容级别为3

SCN兼容级别和Auto-RollOver的关系？

SCN 兼容级别是限制SCN增速和SCN RSL， Auto -rollover 是一种类例JOB的任务，定时修改SCN 兼容性级别， 禁用了AUTO-ROLLOVER 到2019-06-23后SCN兼容级别就不会自动调整，还保持原来的限制。 禁用了Auto-RollOver，可以手动调整scn兼容性(前提是应用了补丁)。

如何禁用、启用Auto-RollOver?

同样使用DBMS_SCN包，禁用执行

exec DBMS_SCN.DISABLEAUTOROLLOVER;

启用执行

EXEC DBMS_SCN. ENABLEAUTOROLLOVER;

如何禁用SCN兼容级别自动调整，手动调整级别?

目前禁用Auto-RollOver就可以禁用SCN兼容性的自动调整，这样已安装补丁的数据库在2019年6月23日不会有新变化。

手动可以调整SCN兼容性，执行

ALTER DATABASE SETSCN COMPATIBILITY N;   — N可以是1，2，3

不同SCN Compatibility的区别

以2018-4-20一个11.2.0.4的数据库为例，不同SCN兼容性的区别。

为什么引入这样的修改和补丁？

SCN 是 Oracle 的一致性的核心，为了应对更高的事务率，和解决SCN Headroom问题，需要引入新的SCN算法。

本次预警影响的数据库版本？

在本次预警涉及的变化，在11.1.0.7.20+ /11.2.0.3.9+ /11.2.0.4+/ 12.1.0.2+/ 12.2.0.1+ 版本已经包含对应的补丁，实现了关于SCN兼容性自动修改的新特性，默认将会在2019年6月23日之后改变SCN的兼容性为3， 增强了SCN的增长速率限制和最大允许SCN限制。

对于11g之前的版本需要升级，和11G版本安装PSU 来达到SCN 兼容级别的统一，避免因不同SCN兼容性级别，在后期SCN 访问时的限制而拒绝连接。

该补丁实现了什么功能？

补丁本身并不针对某个具体bug进行修复，是内部特性的增强，意在调整SCN的增速上限和当前最大允许SCN的策略， 规避SCN增长过快的系统可能遇到的“SCN Headroom（SCN天花板）”问题，注：这个补丁并不改变系统中SCN本身的变化的速率，只是可允许的系统中SCN的变化率和当前SCN最大值较之前增长数倍。

这个补丁是怎么影响到DBLink的？

打了补丁的系统与未打补丁的系统的SCN不同的增长速率限制不同，如果打了补丁的系统确实非常繁忙，SCN增长快且最大允许SCN限制高，而未打补丁系统SCN增长上限不变，这样在新旧系统间通过DB Link同步SCN时，可能因为打补丁的系统SCN超过未打补丁系统的“RSL”而出现问题；但是，上述问题出现的前提是：新系统运行确实非常繁忙，SCN增长速度确实足够快，这与补丁本身无关, 有效的解决办法还是找到SCN增长的根源。

注： 如果不使用DBLINK，而且也不存在scn headroom，可以不安装该补丁。

不升级/不打补丁现有DB Link或数据库就必然会有问题吗？

不是！系统SCN的变化是基于系统的繁忙情况，事务的多少和DBLINK的同步， 在打上该补丁后，系统SCN的变化速度并不改变，只是允许系统上支持更繁忙事务和当前SCN允许更大的值，这样在通过DBLINK同步到其他低SCN又未打补丁的系统上时，才会有可能造成DB link访问拒绝或其它未知问题。

升级/打补丁之后DB Link因SCN拒绝或SCN headroom就不存在了么？

不是。 该补丁本质只是增大了SCN增长速度和上限限制，如果某系统上依然存在bug或其他原因导致SCN的异常增长， 即使是所有系统都打上了该补丁，DB Link间的SCN同步依然会发生，同样会将SCN的异常传播到整个DB Link网络，SCN HEADROOM依然存在。

对于10.2 版本以及更早版本影响

如果不存在SCN headroom问题和也不存在DBLINK 指向已安装补丁的数据库，可以不在任何改变；

2019年6月之后，如果老版本数据库和已安装了补丁并使用了新的SCN兼容性的数据库存在dblink，如果已安装补丁数据库SCN 使用速度没有变化（虽然已允许更快的速率），老版本的DBLINK 仍可以正常访问； 如果DBLINK 另一端已安装补丁的数据库SCN 增长超过了16K, 可能就会因为DBLINK 同步老版本的数据库SCN导致SCN headroom问题甚至拒绝链接，并且那时需要断开这些dblink 连接或升级。Oracle研发目前正在评估为10.2.0.5提供补丁的需求和可行性.

对于11.2.0.4,12.1.0.2和12.2.0.1版本数据库需要做什么？

什么都不用做， 所有需要的修补程序已包含在这些版本中，但是并不是SCN就不会有SCN headroom，只是概率非常低，很少有数据库事务率会使用SCN每秒增长超过90多K.

未安装补丁的数据库dblink间是否会有问题？

不会， 两个未修复的数据库或两个旧版本的数据库之间，如10204,9i的版本数据dblink连接不受此更改的影响。

应该优先处理什么样的系统？

综上，我们应该优先处理环境中目前是否存在SCN增长过快的系统和SCN headroom天数较小的系统。

建议检查目前环境中的所有数据库的SCN值和headroom是否都大于30天。

如何查看具体的DBLINK信息？

所有的数据库版本可以使用DBA_DB_LINKS视图查看现在数据库中存在的DBLINK.

对于12.1以后的数据库可以查询dba_db_link_sources视图查看。

从12.2 版本起数据库提供了DBA_EXTERNAL_SCN_ACTIVITY 视图可以排查SCN 的跳跃信息，更新信息查看MOS note ID 2171090.1

如果需要升级，升级方案

对于老版本的数据库，建议升级，升级列表汇总如下:

PSU 安装方法

略

— 如果有疑问和升级需求可以QQ联系我 —

oracle数据库软件是获得过最高级别的安全认证，完全超越其它所有数据库软件，并且在可维护性上基于大量的实践需要稳步前行，这里简单整理几个12c 关于password几个新特性。

1. 新的password hash算法
2. 新的password verify function
3. 密码文件可以存储到ASM
4. 密码自动从primary同步到standby端在Dataguard环境中
5. 新的密码认证协议

1. 新的password hash算法

Oracle对Oracle Database 12c中的用户密码哈希进行了改进， 通过使用基于PBKDF2的SHA512哈希算法，而不是简单的SHA1哈希，密码哈希更安全, 从11g起user$.spare4列存储着密码的哈希值。在12.1.0.2版本时spare4列有3部分组成（S:H:T）.12.2时只剩下（S:T ）2部分。

S部分和11g时的算法一样，长度60 chars,是基于SHA1的哈希。

H部分是基于MD5的哈希，长度为32 chars, 也可能是因为MD5 hash更方便入侵者的暴力破解，在12.2 版本时从Spare4列去掉该部分。

T部分是从12.1.0.2版本增加，长度为160 chars, 使用的是基于PBKDF2-based SHA512的算法。该算法后部分32chars 的验证数据部分是随机生成。更加安全。

2. 新的password verify function

在12c中default profile除了password verify function和11g不同，其它资源限制都是相同的。
不过国防安全要求有个建议值：
RESOURCE NAME LIMIT
PASSWORD_LIFE_TIME 60
PASSWORD_REUSE_TIME 365
PASSWORD_REUSE_MAX 5
FAILED_LOGIN_ATTEMPTS 3
PASSWORD_VERIFY_FUNCTION ora12c_strong_verify_function

12c新引入了verify function 有ora12c_verify_function, ora12c_strong_verify_function (ora12c_stig_verify_function) ，而且可能在12.1.0.1 ,12.1.0.2, 12.2.0.1     profile默认的verify function名都不一样。

比如STIG有以下限制：

• Password at least 9 characters
• at least 2 capital letters
• at least 2 small letters
• at least 2 digits
• at least 2 special characters
• password must differ by at least 4 characters from the old password

3.密码文件可以存储到ASM

在Oracle Database 12c之前，密码文件始终位于$ ORACLE_HOME / dbs文件下，即使对于RAC实例和RAC ASM集群也是如此。 对于RAC而言，DBA必须设法保持这一点密码文件在每个节点上同步。现在，在Oracle 12c中，可以将密码文件存储在ASM上。 这意味着一个共享密码文件Oracle RAC数据库为集群中的所有实例共享，与ASM spfile不同，ASM密码文件的访问只有在启动ASM ，并且在磁盘组mount后才可以访问。用于创建密码文件的命令实用程序仍然是相同的：“orapwd”

asmcmd同样也可以创建密码文件
$ asmcmd
ASMCMD> pwcreate

ASM存储密码文件前提条件 COMPATIBLE.ASM>= 12.1

-- create new password in ASM
orapwd file='+data/ASM/orapwasm' asm=y
-- create new password in ASM from location
orapwd input_file='/oraclegrid/dbs/orapwasm' file='+data/ASM/orapwasm'[asm=y]
-- move password file from A asm diskgroup to another
ASMCMD> pwmove --asm +CRS/asm/password/orapwasm +data/orapwasm

4. 密码自动从primary同步到standby端在Dataguard环境中

12cR2中一个有用的增强功能是密码文件在Data Guard环境中自动同步，如果在primary database上更改了sys的密码，并且它自动也在standby database上进行了更改。 在早期版本中必须手动完成的，通常通过使用primary database 密码文件替换standby database上的密码文件。

5. 新的密码认证协议

升级12C注意事项： 连接失败 ORA-28040 ORA-1017

之前写过关于认证的协议的。如果你的数据库是从12c以前的版本升上来的，在升级时会提示去掉SEC_CASE_SENSITIVE_LOGON=FALSE参数如下，但是如果你没有去掉SEC_CASE_SENSITIVE_LOGON=FALSE，那么升级后登录可以会遇到ora-1017错误，即使密码是正确的，甚至修改在升级后再修改密码(新的密码版本将为”11G 12C”)。解决方法是：change SEC_CASE_SENSITIVE_LOGON=TRUE, –the default.另外注意如果是Data Pump导入创建的用户，密码版本会和导出时版本一致。

RECOMMENDED ACTIONS
===================
+ Consider removing the following DEPRECATED initialization parameters.
They are not OBSOLETE in version 12.2.0.1.0
but probably will be OBSOLETE in a future release.

Parameter
——————————
sec_case_sensitive_logon

Note:

Oracle notes that the term “version” in the allowed_logon_version_server parameter name refers to the version of the authentication protocol. It does NOT refer to the Oracle release version.

sqlnet.allowed_logon_version_server, depending on the version of the authentication protocol.

sqlnet.allowed_logon_version_server=12a: For Oracle Database 12c Release 1 (12.1) release 12.1.0.2 or later.
sqlnet.allowed_logon_version_server=12: For the critical patch updates CPUOct2012 and later Oracle Database 11g authentication protocols (recommended).
sqlnet.allowed_logon_version_server=11: For Oracle Database 11g authentication protocols (default).
sqlnet.allowed_logon_version_server=10: For Oracle Database 10g authentication protocols.
sqlnet.allowed_logon_version_server=8: For Oracle8i authentication protocol

前段时间遇到的一个案例，朋友开始给我说是数据库连接数很高，让我帮分析一下。这是一套 11.2.0.4 2nodes RAC on Sun OS 5.11环境， 问题发现已经过去几个小时， 等我到时客户反应现在连接数已经平稳，刚才看到的大多数会话等待enq: sq – contention.  相信根据这点信息，不知你是否觉的这就是一个高并发的sequence cache不足问题？当然没有这么简单。不过还了解到节点1 有问题，节点2正常。

常常我们会问题现象所迷惑，如果这已经是个病入膏肓的人，当然病因不会只是当前症状， 现在所见多数是果。先说说什么情况下会enq:sq – contention.

1.  什么是ENQ:SQ -CONTENTION

ENQ 是数据库一种队列机制，SQ 是SeQuence相关的队列， 当进程从内存中申请SEQUENCE的nextval时，如果发现内存已用完，需要从磁盘的字典表seq$ 读一批到内存的sequence cache. 当填充或初始化SEQUENCE CACHE序列缓存时需要申请该SQ Enqueue, 同时更新SEQ$基表， 这也就是为什么当时看到的SQL 是update seq$的内部递归SQL.

产生enq:sq -contention的原因有：
sequence cache过小，应用使用频繁导致Seq 对象需要频繁填充cache;
RAC 上的order sequence;
shared pool 过小导致sequence cache被刷出shared pool;
申请seq enq时间过长被其它进程堵塞等。
排查seq: sq- contention的sequence的具体对象可以从v$session 该事件的P2值得到。 P2值就是sequence 的object_id .

wait event: 'enq: SQ - contention'
p1: 'name|mode'=0x53510006
p2: 'object #'=0x16a
p3: '0'=0x0

SQL> @hex 16a
                                DEC                  HEX
----------------------------------- --------------------
                         362.000000                  16A
SQL> select owner,object_name from dba_objects where object_id=362;
OWNER OBJECT_NAME
----- -----------
SYS   AUDSES$

Note:
可以看出当时应用是在以排它方式申请sys.audses$ 序列时遇到堵塞。 该序列是会话登录时在没有禁用审计时调用。11.2的审计默认是启用的。当然会想是不是这个sequence过小，又遇到LOGON STORM呢？不是的，这个可以从listener log确认。最快的方法是找一下blocker chains (堵塞链条)。

2. 查找阻塞链条

SQL> select event,BLOCKING_SESSION,BLOCKING_INSTANCE,count(*)
 from v$session 
 where status<>'INVALID' 
 group by event,BLOCKING_SESSION,BLOCKING_INSTANCE;

EVENT                          BLOCKING_SESSION BLOCKING_INSTANCE        COUNT(*)
------------------------------ ---------------- ----------------- ---------------
VKTM Logical Idle Wait                                                          1
gc buffer busy release                      608                 1               1
Streams AQ: waiting for time m                                                  1
anagement or cleanup tasks
Space Manager: slave idle wait                                                  1
write complete waits                        871                 1               1
rdbms ipc message                                                              54
SQL*Net message to client                                                       1
enq: SQ - contention                        608                 1             605
PING                                                                            1
buffer busy waits                          4685                 1               1
SQL*Net message from client                                                     4
enq: SQ - contention                       7382                 1               7
DIAG idle wait                                                                  2
log file switch (checkpoint in             1771                 1               5
complete)
...
GCR sleep                                                                       1
VKRM Idle                                                                       1
enq: SQ - contention                                                            1
buffer busy waits                          7298                 1               2

26 rows selected.

SQL> select BLOCKING_SESSION,BLOCKING_INSTANCE from v$session where sid=608;
BLOCKING_SESSION BLOCKING_INSTANCE
---------------- -----------------
            1771                 1

SQL> select BLOCKING_SESSION,BLOCKING_INSTANCE from v$session where sid=1771;
BLOCKING_SESSION BLOCKING_INSTANCE
---------------- -----------------

Note:
block chaints:   enq:SQ – contention (N个)>>> enq:SQ – contention(sid=608)>>> log file switch (checkpoint incomplete)(sid=1711 LGWR),
大部分enq:SQ – contention（600多个）会话是被sid: 608堵塞 ，608又是被sid:1771 堵塞， sid:1771是后台进程LGWR 日志写进程, 等待事件是log file switch (checkpoint incomplete)。

3. 什么是log file switch (checkpoint incomplete)

这个等待事件几年前log file switch (checkpoint incomplete) event 简单总结过， 当Oracle会话更改buffer cache中的Oracle buffer block时，它们的进程将足够的redo 信息复制到redo buffer中。 出于多种原因(3s\commit\…)，LGWR会将其redo 从redo buffer写入到联机重做日志文件中。 在线重做日志文件是串联连接写入一个接下一个创建一个闭环。 当LGWR写入在线重做日志时，如果当前日志文件写满以后，LGWR必须切换到下一个联机重做日志，然后开始写入。 切换log file switch等待事件“log file switch (checkpoint incomplete)”。

Oracle 必须等到 checkpoint position  已经写入 在线日志文件. 当数据库等待 checkpoints时， redo产生将会停止，直到log switch完成. 此刻Oracle所有会话会 被等待, 直到checkpoint  完成. 这也是为什么当前会话稳定的假象，因为非sysdba用户已经无法登录。 当checkpoint事件发生时DBWn会将脏块写入到磁盘中，同时数据文件和控制文件的文件头也会被更新以记录checkpoint信息。  When a checkpoint occurs, Oracle must update the headers of all datafiles to record the details of the checkpoint. This is done by the CKPT process. The CKPT process does not write blocks to disk, DBWn (DB Writer Process) always performs that task.  更多信息参考Master Note: Overview of Database Checkpoints (文档 ID 1490838.1)

不能日志切换了，那下一步可以检查:是否为归档模式? 归档路径，空间使用率，日志当前状态，日志大小，日志切换频率，IO情况…

SQL> select group#,thread#,SEQUENCE#  ,STATUS,FIRST_CHANGE# from v$log;

         GROUP#         THREAD#       SEQUENCE# STATUS             FIRST_CHANGE#
--------------- --------------- --------------- ---------------- ---------------
              1               1           10821 ACTIVE               29794542995
              2               1           10822 ACTIVE               29795882841
              3               1           10823 ACTIVE               29797766437
              4               1           10824 CURRENT              29798824118
              5               1           10820 ACTIVE               29792473826
             11               2           13634 INACTIVE             29794540448
             12               2           13635 INACTIVE             29796006289
             13               2           13636 INACTIVE             29798244262
             14               2           13637 INACTIVE             29799746620
             15               2           13638 CURRENT              29801862030

set lines 120; 
set pages 999; 
SELECT 
to_char(first_time,'YYYY-MON-DD') day,
to_char(sum(decode(to_char(first_time,'HH24'),'00',1,0)),'99') "00",
to_char(sum(decode(to_char(first_time,'HH24'),'01',1,0)),'99') "01",
to_char(sum(decode(to_char(first_time,'HH24'),'02',1,0)),'99') "02",
to_char(sum(decode(to_char(first_time,'HH24'),'03',1,0)),'99') "03",
to_char(sum(decode(to_char(first_time,'HH24'),'04',1,0)),'99') "04",
to_char(sum(decode(to_char(first_time,'HH24'),'05',1,0)),'99') "05",
to_char(sum(decode(to_char(first_time,'HH24'),'06',1,0)),'99') "06",
to_char(sum(decode(to_char(first_time,'HH24'),'07',1,0)),'99') "07",
to_char(sum(decode(to_char(first_time,'HH24'),'08',1,0)),'99') "0",
to_char(sum(decode(to_char(first_time,'HH24'),'09',1,0)),'99') "09",
to_char(sum(decode(to_char(first_time,'HH24'),'10',1,0)),'99') "10",
to_char(sum(decode(to_char(first_time,'HH24'),'11',1,0)),'99') "11",
to_char(sum(decode(to_char(first_time,'HH24'),'12',1,0)),'99') "12",
to_char(sum(decode(to_char(first_time,'HH24'),'13',1,0)),'99') "13",
to_char(sum(decode(to_char(first_time,'HH24'),'14',1,0)),'99') "14",
to_char(sum(decode(to_char(first_time,'HH24'),'15',1,0)),'99') "15",
to_char(sum(decode(to_char(first_time,'HH24'),'16',1,0)),'99') "16",
to_char(sum(decode(to_char(first_time,'HH24'),'17',1,0)),'99') "17",
to_char(sum(decode(to_char(first_time,'HH24'),'18',1,0)),'99') "18",
to_char(sum(decode(to_char(first_time,'HH24'),'19',1,0)),'99') "19",
to_char(sum(decode(to_char(first_time,'HH24'),'20',1,0)),'99') "20",
to_char(sum(decode(to_char(first_time,'HH24'),'21',1,0)),'99') "21",
to_char(sum(decode(to_char(first_time,'HH24'),'22',1,0)),'99') "22",
to_char(sum(decode(to_char(first_time,'HH24'),'23',1,0)),'99') "23"
from
v$log_history
WHERE first_time>SYSDATE-7
GROUP by 
to_char(first_time,'YYYY-MON-DD');
       DAY      00  01  02  03  04  05  06  07  08  09  10  11  12  13  14  15  16  17  18  19  20  21  22  23
       -----    --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- --- -- --- --- --- --- ---  
2018-4月 -15	  0	  0	  0	  0	  0	  0	  0	  0	  0	  0	  0	  0	  0	  1	  1	  2	  2	  1	  2	  1	  1	  1	  1	  1
2018-4月 -16	  1	  4	  1	  2	  0	  2	  2	  1	  2	  3	  4	  4	  3	  3	  2	  4	  3	  2	  3	  1	  1	  1	  2	  0
2018-4月 -17	  2	  3	  2	  0	  2	  1	  2	  1	  2	  3	  4	  4	  2	  3	  2	  3	  4	  2	  2	  2	  1	  1	  1	  1
2018-4月 -18	  1	  4	  1	  2	  0	  2	  2	  4	  3	  3	  4	  3	  2	  1	  4	  3	  3	  2	  2	  1	  2	  0	  2	  1
2018-4月 -19	  1	  5	  1	  2	  0	  2	  1	  3	  3	  4	  3	  4	  2	  2	  3	  3	  3	  3	  2	  1	  2	  0	  2	  0
2018-4月 -20	  2	  5	  1	  1	  1	  2	  2	  2	  3	  3	  4	  3	  4	  2	  2	  4	  3	  3	  2	  0	  2	  2	  0	  2
2018-4月 -21	  0	  2	  2	  2	  1	  1	  3	  0	  0	  1	  0	  0	  0	  0	  1	  0	  0	  0	  0	  3	  2	 28	  2	  2
2018-4月 -22	  2	  2	  2	  1	  1	  3	  3	  2	  3	  3	  2	  1	  0	  0	  0	  0	  0	  0	  0	  0	  0	  0	  0	  0

SQL> SELECT EVENT,BLOCKING_SESSION,STATUS FROM V$SESSION WHERE PADDR IN(select ADDR from v$process WHERE PNAME LIKE 'CKP%')
  2  ;

EVENT                                                            BLOCKING_SESSION STATUS
---------------------------------------------------------------- ---------------- --------
rdbms ipc message                                                                 ACTIVE
SQL>SELECT EVENT,BLOCKING_SESSION,STATUS FROM V$SESSION WHERE PADDR IN(select ADDR from v$process WHERE PNAME LIKE 'DBW%')
EVENT                                                            BLOCKING_SESSION STATUS
---------------------------------------------------------------- ---------------- --------
rdbms ipc message                                                                 ACTIVE
rdbms ipc message                                                                 ACTIVE
rdbms ipc message                                                                 ACTIVE
rdbms ipc message                                                                 ACTIVE
rdbms ipc message                                                                 ACTIVE
rdbms ipc message                                                                 ACTIVE
rdbms ipc message                                                                 ACTIVE
...
rdbms ipc message                                                                 ACTIVE
rdbms ipc message                                                                 ACTIVE
rdbms ipc message                                                                 ACTIVE

32 rows selected.

Note:

当前的日志thread 1,也就是实例1的所有log file都是活动状态，无法覆盖reuse,  根据日志组切换频率判断大小和online redo log组数不是瓶颈。手动发起检查点hang 半小时不动，快速恢复手段也是先加上几组日志确保日志可以生成，让当前实例恢复业务，再分析原因。

4. 临时解决办法

SQL> alter database add logfile thread 1 '+DATA' size 2g;

Database altered.

给thread 1(实例1) 临时增加了几组日志后，实例已经可以连接，并且业务恢复正常。

5. 日志收集和hanganalyze 和 systemstate dump，short_stack

OS 无报错，CPU,内存资源空闲， （使用了多路径，也曾怀疑是否有个别存储链路的有IO挂起，但主机无法确认。）
alert log文件无错误日志， 只是曾经有ORA-20的错误； AWR 几个小时已无数据收集； 查看LGWR, DBWR,CKPT Trace 文件；

SQL> oradebug setmypid
Statement processed.
SQL> oradebug unlimit
Statement processed.
SQL> oradebug dump hanganalyze 3;
Statement processed.
SQL> oradebug dump hanganalyze 3;
Statement processed.
SQL> oradebug dump hanganalyze 3;
Statement processed.
SQL> oradebug tracefile_name;

-- Hanganalyze trace 
Chains most likely to have caused the hang:
 [a] Chain 1 Signature: 'rdbms ipc message'<='log file switch (checkpoint incomplete)'
     Chain 1 Signature Hash: 0xfb14fb68
 [b] Chain 2 Signature: 'rdbms ipc message'<='write complete waits'<='buffer busy waits'
     Chain 2 Signature Hash: 0x768e7402
 [c] Chain 3 Signature: 'rdbms ipc message'<='log file switch (checkpoint incomplete)'<='buffer busy waits'
     Chain 3 Signature Hash: 0x4110dae0

同时可以快速做几个进程的short_stack,如CKPT

SQL> oradebug setorapid 60
Oracle pid: 60, Unix process pid: 8931, image: oracle@fwptdb01 (CKPT)
SQL> oradebug short_stack;
ksedsts()+380<-ksdxfstk()+52<-ksdxcb()+3592<-sspuser()+140<-__sighndlr()+12<-call_user_handler()+868<-sigacthandler()+92<-_syscall6()+32<-sskgpwwait()+236<-ksliwat()+1952<-kslwait()+240<-ksarcv()+212<-ksbabs()+764<-ksbrdp()+1720<-opirip()+1680<-opidrv()+748<-sou2o()+88<-opimai_real()+512<-ssthrdmain()+324<-main()+316<-_start()+380

这个问题是这个环境第一次出现，同时对数据库检查确认，AMM 未启用， ASMM 使用，并且SGA TARGET给了500G, buffer cache给了400G, shared pool 96G, 这么大的内存还是比较少， 不过大内存有大内存的麻烦。
并且确认已禁用了Solaris OS DISM.
当时手动多次checkpoint 都无法完成， 并且新加的日志组只能临时缓解还是无法reuse, 于时当晚重启了数据库实例，重启后恢复正常。
怀疑是BUG ，但从已知BUG 未找到。 但后期得知该数据库使用了FLASH BUFFER，flash buffer很少使用，并且BUG较多，与这个案例现象相同的有个bug,
Bug 22083366 CKPT hang blocked by flash buffer in RAC. 但无法确认， 仅记录一下，不过还是建议禁用flash buffer.

总结
因为当前的实例无法checkpoint ，导致在redo log 一直处于active状态，在所有日志组都active后，所有事务会挂起，新会话无法连接。常用的手段是确认日志组足够大；增长日志组；调优checkpoint;确保I/O 性能稳定或使用更快的I/O设备.或确认是否有相关BUG, 针对当前现象, 临时增加日志组争取时间，重启实例释放数据库的所有进程资源恢复。

MariaDB已联合Alibaba, Google 和 Facebook等改进和整合新的特性服务于整个社区，意在提供替代传统数据库的解决方案 。对于MariaDB和Oracle MySQL区别不在本篇范围， 我为什么学习MariaDB而不是Oracle MySQL? 主要是不想看到学的MySQL不久后又闭源了 ^_^!。

MariaDB上周推出了MariaDB TX 3版本， MariaDB TX有一系列组件组成，有MariaDB Server、 MariaDB Cluster、 MariaDB MaxScale、 MariaDB connectors 、MariaDB tools、 MariaDB services。

MariaDB TX 3.0为企业级开源制定标准，企业级开源不同于开源，但是100%支持开源社区， 为客户关键应用提供企业级和可靠的服务支持。 同是做为企业开源的领导者向ORACLE、Microsoft 、IBM发起挑战。查看原文

下面列了一新该版本的新特性：

关于兼容性方面

MariaDB TX 3.0是第一个兼容ORACLE的和企业开源数据库，增添了Oracle兼容性和迁移服务，支持PL/SQL 开发的存储过程，之前可能这类需求只能在其它企业级像DB2中实现，从今天开始可以运行在MariaDB TX。同时还支持ORACLE的数据类型、sequence、trigger、package、cursor和动态执行SQL的语法。

同时支持像ORACLE中闪回查询一样的， 基于时间点的临时查询多系统版本的数据。

存储引擎方面

引入MyROCKS存储引擎，是有FACEBOOK开发，一种空间与写入优化的存储引擎，用于高密集写入的工作负载；引入Spider存储引擎是一种分布式存储引擎，用于为存储写入提升高扩展性和并发性，努力是企业使用单个通用数据库支持各种事务性工作负载。

同时在高可用方面：
• 自动failover
• Transparent query routing 路由
• Multi-master clustering 多活（主）
• Lossless semi-synchronous replication 同步
• Multi-source replication   多源同步

灾难恢复方面引入：
• Backup and restore 备份恢复
• Point-in-time rollback 基于时间点回滚
• Delayed replication  延迟同步

安全方面引入：
• Data obfuscation 数据模糊
• Data masking (full and partial)部分数据屏蔽和完整数据混淆功能
• Pluggable authentication
• Roles
• User resource limits
• Transparent data encryption (TDE)   传输数据加密
• Temporary files
• Tablespaces or tables
• Logs
• Query blocking
• Auditing 审计
• Result limiting

可扩展性方面引入：
• Compression 压缩
o Columns 列级压缩
o Tables: rows or pages 表级压缩
• Consistent reads
• Read-write splitting 读写分离

性能方面
• Fast connection creation
• Thread pool 线程池
• Query result caching 查询缓存
• Bulk insert streams
• NUMA interleave (InnoDB)

Schema
• System versioned tables 多版本查询
• Instant ADD COLUMN 一种快速的增加列方式，不更新每行数据
• Invisible columns 虚拟列
• Check constraints
• Default value functions/expressions
• Multiple triggers per type per table
• Virtual column indexes 虚拟列索引
• Spatial indexes
• Decimal scale of 38

Integration
• Change-data-capture streams 一种CDC方案
•  Tools and services 工具
• Administration
• Monitoring 监控
• Notifications 通知

MariaDB TX组件

可以看出MariaDB 提供了全套的企业级解决方案， 后期再逐个测试其功能，开启新的学习篇章。

Failed Logon Delay

昨天看到一份Oracle 12C R2的AWR top event是”Failed Logon Delay”, 发现12c 在安全方面又有几个参数的变化，简单的记录一下。

这个等待事件常常是因为有程序尝试使用错误的用户密码登录数据库， 如暴力破解程序.

这是一个安全特性用于控制延迟失败的登录，在oracle 11g版本是引入,但是在11g时常因为这个特性带来性能 问题，需要用event 28401 禁用密码延迟认证的特性。 控制认证失败尝试特性是有 sec_max_failed_login_attempts 和sec_protocol_error_further_Action 参数控制，但是在oracle 12c后对于以上参数值有了新的变化， sec_max_failed_login_attempts尝试失败次数(多个用户)11G是10次，在12ck中减少为3， 所以延迟的登录会更多, 这个参数不同于user profile中的失效次数主要是单个用户失败和多个用户失败。 sec_protocol_error_further_Action  这个参数控制失败后的处理方式，在11g时是CONTINUE 也就是可以继续，但是在12c 中默认改变为(DROP, 3)， 为了系统稳定牺牲一个连接。

解决方法就是找错误尝试的主机，修正密码后即可。

查找方法可以使用audit

如 SQL> audit session;   查询aud$

或使用登录trigger(Note:可能会有一点点登录性能影响（<1%），找到源头后可以删除)

CREATE OR REPLACE TRIGGER logon_denied_to_alert
  AFTER servererror ON DATABASE
DECLARE
  message   VARCHAR2(168);
  ip        VARCHAR2(15);
  v_os_user VARCHAR2(80);
  v_module  VARCHAR2(50);
  v_action  VARCHAR2(50);
  v_pid     VARCHAR2(10);
  v_sid     NUMBER;
  v_program VARCHAR2(48);
BEGIN
  IF (ora_is_servererror(1017)) THEN

    -- get ip FOR remote connections :
    IF upper(sys_context('userenv', 'network_protocol')) = 'TCP' THEN
      ip := sys_context('userenv', 'ip_address');
    END IF;

    SELECT sid INTO v_sid FROM sys.v_$mystat WHERE rownum < 2;
    SELECT p.spid, v.program
      INTO v_pid, v_program
      FROM v$process p, v$session v
     WHERE p.addr = v.paddr
       AND v.sid = v_sid;

    v_os_user := sys_context('userenv', 'os_user');
    dbms_application_info.read_module(v_module, v_action);

    message := to_char(SYSDATE, 'YYYYMMDD HH24MISS') ||
               ' logon denied from ' || nvl(ip, 'localhost') || ' OSPid:' ||
               v_pid || ' User:' || v_os_user || ' with ' || v_program || ' – ' ||
               v_module || ' ' || v_action;

    sys.dbms_system.ksdwrt(2, message);

  END IF;
END;
/

_sys_logon_delay

另外对于12c中引入的对于SYS用户的尝试失败登录后的延迟是有参数新的参数”_sys_logon_delay“控制的，默认为1秒，加大参数可以防止非法尝试，配置值为0 可以禁用该特性。

MORE

SEC_MAX_FAILED LOGIN_ATTEMPTS is a new feature in Oracle 11g. It specifies the number of authentication attempts that can be made by a client on a connection to the server process. This parameter target against brute force attacks. An intruder could start a server process first and then try to establish connection by guessing/Program generated password. Using that parameter it is possible to limit the number of failed login attempts. After the specified number of authentication attempts fails, the database processes drop the connection. This initialization parameter is designed to stop intruder from attacking application, it does not apply to valid user. It is not a dynamic parameter.

SEC_MAX_FAILED_LOGIN_ATTEMPTS only works application uses OCI Program. It does not work in sqlplus so a user can try unlimited time with guessing password from sqlplus even though the parameter is set but usually intruder attack using OCI program.

前几日朋友有套数据库升级到oracle 12C r2后出现了ORA-4031, 是一套on Linux x86 64-bit 2节点的RAC数据库，Hard parse并不高，并且没有使用AMM, ASMM。SGA分配80G， shared pool固定在13Gb.

— nodel 1 db alert log file

2018-05-29T13:44:51.107475+08:00
Thread 1 advanced to log sequence 6680 (LGWR switch)
  Current log# 14 seq# 6680 mem# 0: +SSDDG1/anbob/ONLINELOG/group_14.260.944852559
2018-05-29T13:57:28.332831+08:00
Errors in file /oracle/app/diag/rdbms/anbob/anbob1/trace/anbob1_m001_38524.trc  (incident=1804178):
ORA-04031: unable to allocate 640 bytes of shared memory ("shared pool"," SELECT source,        (case...","kkqctdrvTD: co","ckydef:kkqcscpcky")
2018-05-29T13:57:28.332861+08:00
Errors in file /oracle/app/diag/rdbms/anbob/anbob1/trace/anbob1_ora_38534.trc  (incident=1804506):
ORA-04031: unable to allocate 40 bytes of shared memory ("shared pool","select con#,obj#,rcon#,enabl...","SQLA","tmp")
Incident details in: /oracle/app/diag/rdbms/anbob/anbob1/incident/incdir_1804506/anbob1_ora_38534_i1804506.trc
Incident details in: /oracle/app/diag/rdbms/anbob/anbob1/incident/incdir_1804178/anbob1_m001_38524_i1804178.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /oracle/app/diag/rdbms/anbob/anbob1/trace/anbob1_m001_38524.trc  (incident=1804179):
ORA-04031: unable to allocate 4152 bytes of shared memory ("shared pool"," SELECT source,        (case...","9003.kgght","9003.kgght")
ORA-04031: unable to allocate 640 bytes of shared memory ("shared pool"," SELECT source,        (case...","kkqctdrvTD: co","ckydef:kkqcscpcky")
Incident details in: /oracle/app/diag/rdbms/anbob/anbob1/incident/incdir_1804179/anbob1_m001_38524_i1804179.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2018-05-29T13:57:28.588932+08:00
ORA-04031 heap dump being written to trace file /oracle/app/diag/rdbms/anbob/anbob1/incident/incdir_1804506/anbob1_ora_38534_i1804506.trc
2018-05-29T13:57:28.971700+08:00
Errors in file /oracle/app/diag/rdbms/anbob/anbob1/trace/anbob1_ora_36389.trc  (incident=1804658):
ORA-04031: unable to allocate 424 bytes of shared memory ("shared pool","EXEC_OPR_POS","KGLS^62f5cb39","KGLS MEM BLOCK")
Incident details in: /oracle/app/diag/rdbms/anbob/anbob1/incident/incdir_1804658/anbob1_ora_36389_i1804658.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2018-05-29T13:57:29.086756+08:00
Errors in file /oracle/app/diag/rdbms/anbob/anbob1/trace/anbob1_ora_75557.trc  (incident=1800986):
ORA-04031: unable to allocate 40 bytes of shared memory ("shared pool","select * from (select id,msi...","SQLA","tmp")
Incident details in: /oracle/app/diag/rdbms/anbob/anbob1/incident/incdir_1800986/anbob1_ora_75557_i1800986.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

M001进程在申请”shared pool”遇到ora-4031, 当然问题时间段的AWR没有生成， 当时建议做heapdump如下

SQL> connect / as sysdba
SQL> alter session set events 'immediate trace name heapdump level 536870914';

OR

sqlplus "/ as sysdba"
oradebug setmypid
oradebug unlimit
oradebug dump heapdump 536870914
oradebug tracefile_name
exit

让现场的收集了部分trace file, 并没有什么异常信息，这时可以看问题时间点前最近的AWR，找”SGA breakdown difference”部份，初步判断异常变化。

Note:
“gc index split transaction” 内存区使用超过了5Gb， 这在以前的版本中是很少见的. 并且继续确认了几份AWR，该heap 内存区时一直增长的， 如果并且如果手动从X$KSMSS 确认当前的”gc index split transaction”也是在持续增长。

gc index split transaction

gc index split transaction 是用于Oracle 12cr2 引入的新特性”fast index split wait”， 该特性是在KCL layer实现，用于RAC的Global Cache，使用它表示一个进程index splits等待远程或本地进程， 基本上每个session在连接时都会分配一组buffer. 所以这个现象只存在于RAC环境。并且配置_gc_fast_index_split_wait=0没有什么作用。

似乎是因为使用了kclxidinit() used kghalp (permanent memory) 分配的是perm trunk，而不是kghalf (freeable memory)

解决方案

Mos中查找了一下，比较符合bug 27163928.

重启可以临时解决问题，但是会逐渐再次因为fast index split wait增长导致ora-4031.

or

安装one-off patch 27163928

or

升级到oracle 18.1

这是几天前遇到的一个案例，在这个案例又有太多的巧合， 虽然最后也未从SR那里明确BUG，并且因版本不在支持的服务期不能再深究，但是过程还是值得一看，环境是11.2.0.3.7    2-Nodes RAC on HPUX 11.31. 当时是大量的应用会话是”cursor: pin s wait on x”和”row cache lock” 等待， 而最终的堵塞者又正在执行一个DDL triiger 中的insert语句， 为了尽快恢复业务，我们KILL了堵塞者，业务恢复。

先看一下当前的堵塞chains大况，（这里使用的是TanelPoder的脚本包）。

SQL> @ash_wait_chains username||':'||event2 session_type='FOREGROUND' sysdate-2/24 sysdate-1/24

-- Display ASH Wait Chain Signatures script v0.2 BETA by Tanel Poder ( http://blog.tanelpoder.com )

%This     SECONDS        AAS WAIT_CHAIN
------ ---------- ---------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  55%      253828       70.5 -> ICDMAIN:cursor: pin S  -> ICDMAIN:library cache lock
  13%       61698       17.1 -> ICDMAIN:library cache lock  -> ICDMAIN:row cache lock  -> ICDMAIN:row cache lock
   3%       13468        3.7 -> NETRIXDB2:cursor: pin S wait on X  -> NETRIXDB:library cache lock  -> NETRIXDB:row cache lock  -> ICDMAIN:row cache lock
   3%       13468        3.7 -> NETRIXDB4:cursor: pin S wait on X  -> NETRIXDB:library cache lock  -> NETRIXDB:row cache lock  -> ICDMAIN:row cache lock
   3%       13468        3.7 -> NETRIXDB3:cursor: pin S wait on X  -> NETRIXDB:library cache lock  -> NETRIXDB:row cache lock  -> ICDMAIN:row cache lock
   2%       10602        2.9 -> ICDCCT:row cache lock  -> ICDMAIN:row cache lock
   2%       10101        2.8 -> NETRIXDB:cursor: pin S wait on X  -> NETRIXDB:library cache lock  -> NETRIXDB:row cache lock  -> ICDMAIN:row cache lock
   2%        7068          2 -> ICDCCT:library cache lock  -> ICDCCT:row cache lock  -> ICDMAIN:row cache lock
   2%        7068          2 -> ICDNGOC:row cache lock  -> ICDMAIN:row cache lock
   1%        5430        1.5 -> ICDPUB:ON CPU
   1%        3534          1 -> NETRIXDB2:library cache lock  -> NETRIXDB2:row cache lock  -> ICDMAIN:row cache lock
   1%        3534          1 -> NETRIXDB:cursor: pin S wait on X  -> NETRIXDB2:library cache lock  -> NETRIXDB2:row cache lock  -> ICDMAIN:row cache lock
   1%        3534          1 -> NETRIXDB3:cursor: pin S wait on X  -> NETRIXDB2:library cache lock  -> NETRIXDB2:row cache lock  -> ICDMAIN:row cache lock
   1%        3534          1 -> ICDCCT:library cache lock
   1%        3534          1 -> NETRIXDB4:cursor: pin S wait on X  -> NETRIXDB2:library cache lock  -> NETRIXDB2:row cache lock  -> ICDMAIN:row cache lock
   1%        3534          1 -> ICDMAIN:row cache lock
   1%        3534          1 -> NETRIXDB2:row cache lock  -> ICDMAIN:row cache lock
   1%        3534          1 -> NETRIXDB3:row cache lock  -> ICDMAIN:row cache lock
   1%        3534          1 -> ICDSR:library cache lock  -> ICDCCT:row cache lock  -> ICDMAIN:row cache lock
   1%        3533          1 -> ICDKBS:row cache lock  -> ICDMAIN:row cache lock
   1%        3398         .9 -> NETRIXDB4:row cache lock  -> ICDMAIN:row cache lock
   1%        3367         .9 -> NETRIXDB:row cache lock  -> ICDMAIN:row cache lock
   1%        3367         .9 -> NETRIXDB:library cache lock  -> NETRIXDB:row cache lock  -> ICDMAIN:row cache lock
   1%        2938         .8 -> ICDMAIN:row cache lock  -> ICDMAIN:row cache lock
   0%        1861         .5 -> PATROLDB:db file sequential read
   0%        1732         .5 -> ICDSR:row cache lock
   0%        1729         .5 -> ICDMAIN:cursor: pin S wait on X  -> ICDMAIN:library cache lock
   0%        1564         .4 -> PATROLDB:ON CPU
   0%        1521         .4 -> YYJK:ON CPU
   0%        1449         .4 -> ICDCCT:row cache lock

30 rows selected.

Note:
主要问题还是发生在SQL解析相关，最终堵塞是library cache lock 和row cache lock.   这是读的最近2小时的v$active_session_history, 也可以使用dash_wait_chains.sql 包， 当然自己可以灵活的从dba_hist_active_sess_history取10秒快照的活动session堵塞情况。

SELECT
    TO_CHAR(sample_time,'yyyymmdd hh24:mi') etime,
    event,
    program,
    session_state,
    blocking_inst_id,
    blocking_session_status,
    blocking_session,
    machine,
    session_id,
    session_serial#,
    user_id,
    sql_id,
    top_level_sql_id,
    sql_exec_start,
    p1text,
    p1
FROM
    dba_hist_active_sess_history
WHERE
    sample_time BETWEEN TO_DATE('20180718 20:00','yyyymmdd hh24:mi') AND TO_DATE('20180719 00:40','yyyymmdd hh24:mi')

Note:
dba_hist_active_sess_history确实对于历史性能问题的分配提供了大量的信息，不在本篇描述范围，建议顺藤摸瓜找到问题最早的时间点和阻塞链条的终端。

会话阻塞链条

–############ block chains  从上到下的堵塞关系 #############

SQL TEXT:

67r5ngxtfqa4k 是业务的insert  vlaues

5cwvjc63qnwnq 是 INSERT INTO DDL_EVENTS SELECT :B1 , SYSDATE, ORA_LOGIN_USER, ORA_DICT_OBJ_NAME, ORA_DICT_OBJ_OWNER, ORA_DICT_OBJ_TYPE, ORA_SYSEVENT, MACHINE, PROGRAM, OSUSER FROM SYS.DUAL, SYS.V_$SESSION WHERE SYS_CONTEXT(‘USERENV’,’SESSIONID’ ) = AUDSID(+)

cxs4xwrsst0ys 是Procedure  P_SCECurrencyLog

85xgnfc9wuf4m是procedure P_DAYLOG_TRUNCKEYTRACE

9w7203h1f4r43 是procedure P_Rpt_HourJob

SQL堵塞链条

Note:

现在分析当前的情况是大量的应用在执行一个存储中的insert时无法解析，被1节点job 002进程执行的procedure 堵塞， 而job 002又被节点1的9074会话 job 000进程堵塞， job 000进程又被节点2的6423会话job 001进程堵塞，而节点2的job 001又被节点1的job 000进程堵塞，现在是因为会话1：9074和2：6423互相堵塞死锁，并且都在等待row cache lock，而且当前broker会话的sql_id内容也不是top_level_sql_id的内容。后来分析了2个JOB， 发现是2个JOB分布在2个不同的用户下， 执行了不同的存储过程，存储过程中确实有Truncate Table，但是不同的表。 因为有DDL 操作所以才递归的执行了DDL Trigger中的SQL INSERT INTO DDL_EVENTS, 正是当前会话的SQL_ID.

当时CPU 资源空闲，当前的数据库确实是ASMM，但是问题时间段SGA没有自动调整，而且DDL trigger及相关的表最近都没有DDL 变动，并且该SQL最后一次解析是半年前，这个触发器已经上了1年的时间，虽然DDL trigger性能不如Audit 但是当前的数据库并未开启，同时领导有想知道是谁动了某些表时DDL trigger还是能提供依据。并且DDL TRIGGER中只是一条insert，虽然其中有sequence调用, 但sequence是先取nextval后insert, 并且根据row cache lock 的p1 是dc_objects， 而不是dc_sequences，基本上排除了sequence问题。现在需要找出在等待哪个row cache.

DB ALERT LOG

# node 1 db alert log

2018-07-19 04:23:31.138000 +08:00
Thread 1 advanced to log sequence 558160 (LGWR switch)
  Current log# 4 seq# 558160 mem# 0: /dev/ty_oravg02/rty_redo04
2018-07-19 04:23:36.273000 +08:00
Archived Log entry 757912 added for thread 1 sequence 558159 ID 0x8ab9ecdf dest 1:
2018-07-19 04:23:56.163000 +08:00
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=1400
System State dumped to trace file /oracle/app/oracle/diag/rdbms/tyjc/tyjc1/trace/tyjc1_j000_19858.trc
2018-07-19 04:24:19.884000 +08:00
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=850
2018-07-19 04:25:10.395000 +08:00
Thread 1 advanced to log sequence 558161 (LGWR switch)

2018-07-19 07:17:40.669000 +08:00
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=1327
System State dumped to trace file /oracle/app/oracle/diag/rdbms/tyjc/tyjc1/trace/tyjc1_ora_16800.trc

# node 2 db alert log

Thu Jul 19 04:20:17 2018
Archived Log entry 757909 added for thread 2 sequence 199532 ID 0x8ab9ecdf dest 1:
Thu Jul 19 04:24:04 2018
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=177
System State dumped to trace file /oracle/app/oracle/diag/rdbms/tyjc/tyjc2/trace/tyjc2_j001_18176.trc
Thu Jul 19 04:25:12 2018
Thread 2 advanced to log sequence 199534 (LGWR switch)
  Current log# 11 seq# 199534 mem# 0: /dev/ty_oravg02/rty_redo11
Thu Jul 19 04:25:14 2018
Archived Log entry 757913 added for thread 2 sequence 199533 ID 0x8ab9ecdf dest 1:
Thu Jul 19 04:30:12 2018
Thread 2 advanced to log sequence 199535 (LGWR switch)
  Current log# 10 seq# 199535 mem# 0: /dev/ty_oravg02/rty_redo10
Thu Jul 19 04:30:14 2018
Archived Log entry 757917 added for thread 2 sequence 199534 ID 0x8ab9ecdf dest 1:
Thu Jul 19 04:34:31 2018
Restarting dead background process DIA0   <<<<<<<<<
Thu Jul 19 04:34:31 2018
DIA0 started with pid=10, OS id=2512 

Thu Jul 19 05:13:15 2018
Restarting dead background process DIA0
Thu Jul 19 05:13:15 2018
DIA0 started with pid=10, OS id=15632 

Thu Jul 19 05:27:40 2018
Restarting dead background process DIA0
Thu Jul 19 05:27:40 2018
DIA0 started with pid=10, OS id=20443 
Thu Jul 19 05:28:52 2018
Thread 2 advanced to log sequence 199547 (LGWR switch)
  Current log# 10 seq# 199547 mem# 0: /dev/ty_oravg02/rty_redo10
...
Thu Jul 19 05:33:49 2018
Archived Log entry 757969 added for thread 2 sequence 199547 ID 0x8ab9ecdf dest 1:
Thu Jul 19 05:35:26 2018
Restarting dead background process DIA0
Thu Jul 19 05:35:26 2018
DIA0 started with pid=10, OS id=23435 

什么是row cache lock

The shared pool contains a cache of rows from the data dictionary that helps reduce physical I/O on the data dictionary tables. The rowcache lock is used primarily to serialize changes to the data dictionary and is waited on when a lock on a data dictionary cache is required. Waits on this event usually indicate some form of DDL occuring, or possibly recursive operations such as storage management and incrementing sequence numbers.

When DDLs execute, they must acquire locks on the row cache in order to access and change the Data Dictionary information. Once the locks are taken then they can be allowed to modify individual rows in the data dictionary.

The dictionary cache is known as row cache as it keeps the information at row level opposed to the buffer cache that keeps the information at block level. The locks, which protect the definition of the data dictionary objects, are called as row cache locks. Normally DDL statements require row cache locks and the session will wait for the row cache lock to lock the data dictionary information.

This wait is not a RAC specific wait. It is applicable in single instance also, but has a bigger impact in the RAC environment since the library cache and row cache are globally coordinated.

• • The Row Cache:

–Stores data dictionary rows.

–Is part of the shared pool

–Reduces the physical I/O to the SYSTEM tablespace on disk

–Uses Row Cache Locks to support fine-grain locking of data dictionary rows

• • Lock implementation:

–The cached row acts as the resource structure.

–Lock structures are allocated dynamically from the shared pool.

–Types QA…QZ.

• • Row cache lock  event:

-Wait duration is 3 seconds.

-The wait parameters are:

–P1: cache id (cache# in V$ROWCACHE)

–P2: mode that is held

–P3: mode that is requested

-The lock request is abandoned after 1000 timeouts.

row cache 是shared pool中的一块内存区域，以row的形式cache着数据库字典，有助于减少数据字典表如tab$, col$的物理访问，而row cache lock是为了序列化的修改数据字典，在需要锁定数据字典时的一种锁等待，这种等待通常是在DDL发生时或一些递归操作和Sequence 序列的增长。根据P1值可以关连v$rowcache和v$rowcache_parent 以cache#找查具体的cache, 本案例的p1值分别为8 和11，都是dc_objects 更多相关可以查看MOS  note.  278316.1 和DSI 405.

分析这个问题需要的数据：

DB ALERT 、ASH   、DASH   、AWR 、 SystemState DUMP   or Hanalyze ,  xx trace file…

从DASH找到了阻塞会话，通常系统自动生成的SSD（systemstate dump）可以分析row cache的对象。

分析SYSTEMSTATE DUMP

从DB ALERT LOG 可以看到系统有在问题时间段内，两个节点都在4占左右最早生成过一次systemstate dump trace.  下面以该trace分析。因为从DASH阻塞会话的sql_exec_start是覆盖这个时间段的，所以这个SSD是可以做为问题开始的依据。 以下trace 内容已经过截取。分析的顺序可以是找堵塞的session \prcesses\ call \row cache obj \sql等，没有固定的标准。

----------------------------------------

 SO: 0xc000000d1b14b988, type: 4, owner: 0xc000000d19dd8368, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
     proc=0xc000000d19dd8368, name=session, file=ksu.h LINE:12624 ID:, pg=0
    (session) sid: 9074 ser: 33801 trans: 0xc000000ce37d6588, creator: 0xc000000d19dd8368
              flags: (0x210041) USR/- flags_idl: (0x1) BSY/-/-/-/-/-   　
			  
			  <<<<<<<<<<<<     0x00000001   user session (as opposed to recursive session)  
			  <<<<<<<<<<<<     0x00000040   whether user session logs on to ORACLE
			  <<<<<<<<<<<<<    0x00010000   SKIP unusable indexes maintenance
			  <<<<<<<<<<<<<    0x00200000   recource checking in job q process enabled
			  
              flags2: (0x40009) -/-/INC
              DID: , short-term DID: 
              txn branch: 0x0000000000000000
              oct: 2, prv: 0, sql: 0xc000000d369275a8, psql: 0xc000000d369275a8, user: 28/XXXX  
    ksuxds FALSE at location: 0
    service name: SYS$USERS
    client details:
      O/S info: user: oracle, term: UNKNOWN, ospid: 19858
      machine: qdty1 program: oracle@qdty1 (J000)
    Current Wait Stack:
      Not in wait; last wait ended 0.043184 sec ago  
    There are 997 sessions blocked by this session.
    Dumping one waiter:
      inst: 2, sid: 474, ser: 15675
      wait event: 'row cache lock'
        p1: 'cache id'=0xb
        p2: 'mode'=0x0
        p3: 'request'=0x3
      row_wait_obj#: 2550657, block#: 472192, row#: 0, file# 460
      min_blocked_time: 0 secs, waiter_cache_ver: 21515
    Wait State:
      fixed_waits=0 flags=0x21 boundary=0x0000000000000000/-1
    Session Wait History:
        elapsed time of 0.043295 sec since last wait
     0: waited for 'ksdxexeotherwait'
        =0x0, =0x0, =0x0
        wait_id=12076 seq_num=12077 snap_id=1
        wait times: snap=0.064887 sec, exc=0.064887 sec, total=0.064887 sec
        wait times: max=30.000000 sec
        wait counts: calls=0 os=0
        occurred after 0.045113 sec of elapsed time
     1: waited for 'ksdxexeotherwait'
        =0x0, =0x0, =0x0
        wait_id=12075 seq_num=12076 snap_id=1
        wait times: snap=0.084154 sec, exc=0.084154 sec, total=0.084154 sec
        wait times: max=30.000000 sec
        wait counts: calls=0 os=0
        occurred after 0.025841 sec of elapsed time
 ...
     7: waited for 'kcbzps'
        =0xc000000cd93ba5f8, =0xc000000d61d53ed8, =0x0
        wait_id=12069 seq_num=12070 snap_id=1
        wait times: snap=0.019558 sec, exc=0.019558 sec, total=0.019558 sec
        wait times: max=0.010000 sec
        wait counts: calls=1 os=1
        occurred after 0.000439 sec of elapsed time
     8: waited for 'kcbzps'
        =0xc000000cd93ba5f8, =0xc000000d61d53ed8, =0x0
        wait_id=12068 seq_num=12069 snap_id=1
        wait times: snap=0.019598 sec, exc=0.019598 sec, total=0.019598 sec
        wait times: max=0.010000 sec
        wait counts: calls=1 os=1
        occurred after 0.000400 sec of elapsed time
     9: waited for 'kcbzps'
        =0xc000000cd93ba5f8, =0xc000000d61d53ed8, =0x0
        wait_id=12067 seq_num=12068 snap_id=1
        wait times: snap=0.019518 sec, exc=0.019518 sec, total=0.019518 sec
        wait times: max=0.010000 sec
        wait counts: calls=1 os=1
        occurred after 0.000462 sec of elapsed time
	----------------------------------------
      SO: 0xc000000ce37d6588, type: 56, owner: 0xc000000d1b14b988, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
       proc=0xc000000d19dd8368, name=transaction, file=ktccts.h LINE:410 ID:, pg=0
      (trans) flg = 0x00000001, flg2 = 0x00480000, flg3 = 0x00000000, prx = 0x0000000000000000, ros = 2147483647, crtses=0xc000000d1b14b988
      flg  = 0x00000001: ALC 
      flg2 = 0x00480000: NIP DDID 
      flg3 = 0x00000000: 
      bsn = 0xc3f bndsn = 0xc3f spn = 0xc42
      efd = 86 rfd = 0 DID:
      file:kdpur.c lineno:672
      parent xid: 0x0000.000.00000000
      env [0xc000000ce37d69a0]: (scn: 0x0000.00000000  xid: 0x0000.000.00000000  uba: 0x00000000.0000.00  statement num=0  parent xid: 0x0000.000.00000000  st-scn: 0x0000.00000000  hi-scn: 0x0000.00000000  ma-scn: 0x0000.00000000  flg: 0x00000000)
      cev: (spc = 0  arsp = 0x0000000000000000  ubkds (ubk:tsn: 0 rdba: 0x00000000 flag:0x0 hdl:0x0000000000000000 addr:0x0000000000000000)  useg tsn: 0 rdba: 0x00000000
            hwm uba: 0x00000000.0000.00  col uba: 0x00000000.0000.00
            num bl: 0 bk list: 0x0)
            cr opc: 0x0 spc: 0 uba: 0x00000000.0000.00
      Begin scn:0x0000.00000000 uba:0x00000000.0000.00 ts:1531915382[07/18/2018 20:03:02]
      Undo blks: 0 recs: 0
      ccbstg: 0x00000000
      (enqueue) released enqueue or enqueue in flux
      lock_flag: 0x0, lock: 0xc000000ce37d6600, res: 0xc000000d639c0850
      own: 0xc000000d1b14b988, sess: 0xc000000d1b14b988, prv: 0xc000000ce37d6610
       xga: 0x0000000000000000, heap: UGA
      tsnl:0xc000000d3faca840 nent:0 nxt:0x0000000000000000
      Trans IMU st: 0 Pool index 65535, Redo pool 0xc000000ce37d6d48, Undo pool 0xc000000ce37d6e30
      Redo pool range [0x9ffffffffd2eb800 0x9ffffffffd2eb800 0x9ffffffffd2f0000]
      Undo pool range [0x9ffffffffd2e7000 0x9ffffffffd2e7000 0x9ffffffffd2eb800]
       chnf control flags 0x0         CHNF hwm uba uba: 0x00000000.0000.00   
---------------------------  
SO: 0xc000000d4919fca0, type: 78, owner: 0xc000000d1a92b3a8, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
       proc=0xc000000cf9c84e10, name=LIBRARY OBJECT LOCK, file=kgl.h LINE:8547 ID:, pg=0

      LibraryObjectLock:  Address=c000000d4919fca0 Handle=c000000d369275a8 Mode=N CanBeBrokenCount=1 Incarnation=1 ExecutionCount=6         
        
        User=c000000d1a92b3a8 Session=c000000d1a92b3a8 ReferenceCount=1 Flags=[0000] SavepointNum=5b3584ac 
      LibraryHandle:  Address=c000000d369275a8 Hash=876a7296 LockMode=N PinMode=0 LoadLockMode=0 Status=VALD 
        ObjectName:  Name=INSERT INTO DDL_EVENTS SELECT :B1 , SYSDATE, ORA_LOGIN_USER, ORA_DICT_OBJ_NAME, ORA_DICT_OBJ_OWNER, ORA_DICT_OBJ_TYPE, ORA_SYSEVENT, MACHINE, PROGRAM, OSUSER FROM SYS.DUAL, SYS.V_$SESSION WHERE SYS_CONTEXT('USERENV','SESSIONID' ) = AUDSID(+) 
          FullHashValue=6727fe5ca5d3747056737161876a7296 Namespace=SQL AREA(00) Type=CURSOR(00) Identifier=2271900310 OwnerIdn=28 
        Statistics:  InvalidationCount=3 ExecutionCount=18225022 LoadCount=16 ActiveLocks=26 TotalLockCount=577225 TotalPinCount=1 
        Counters:  BrokenCount=1 RevocablePointer=1 KeepDependency=2 BucketInUse=17 HandleInUse=17 HandleReferenceCount=0 
        Concurrency:  DependencyMutex=c000000d36927658(0, 8974, 0, 0) Mutex=c000000d369276d8(0, 2730455, 77, 0) 
        Flags=RON/PIN/TIM/PN0/DBN/[10012841] 
        WaitersLists:  
          Lock=c000000d36927638[c000000d36927638,c000000d36927638] 
          Pin=c000000d36927618[c000000d36927618,c000000d36927618] 
          LoadLock=c000000d36927690[c000000d36927690,c000000d36927690] 
        Timestamp:  Current=07-13-2017 04:32:02 
        HandleReference:  Address=c000000d36927838 Handle=0000000000000000 Flags=[00] 
        LibraryObject:  Address=c000000c1fe2d3b8 HeapMask=0000-0001-0001-0000 Flags=EXS[0000] Flags2=[0000] PublicFlags=[0000] 
          ChildTable:  size='16' 
            Child:  id='0' Table=c000000c1fe2e268 Reference=c000000c1fe2dcb0 Handle=c000000d36926d18 
            Child:  id='1' Table=c000000c1fe2e268 Reference=c000000c1fe2df90 Handle=c000000d346fd7e0 
            Child:  id='2' Table=c000000c1fe2e268 Reference=c000000b1156e508 Handle=c000000107388470 
            Child:  id='3' Table=c000000c1fe2e268 Reference=c000000b1156e810 Handle=c000000d323962e8 
        NamespaceDump:  
          Parent Cursor:  sql_id=5cwvjc63qnwnq parent=c000000c1fe2d458 maxchild=4 plk=y ppn=n 
	   

Note:
从当前可以看出blocker session 的等待事件，正在执行的SQL ，及事务的状态及开始时间，正在执行insert into ddl_event, 接下来向上找PROCESSes 和call,以session SO的owner或proc 都可以。

PROCESS 1400: J000
 ----------------------------------------
 SO: 0xc000000d19dd8368, type: 2, owner: 0x0000000000000000, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0xc000000d19dd8368, name=process, file=ksu.h LINE:12616 ID:, pg=0
 (process) Oracle pid:1400, ser:53, calls cur/top: 0xc000000c7e64d6f0/0xc000000c7ea7f9d8 
 flags : (0x0) -
 flags2: (0x1010), flags3: (0x10) 
 intr error: 0, call error: 0, sess error: 0, txn error 0
 intr queue: empty
 ksudlp FALSE at location: 0
 (post info) last post received: 0 0 80
 last post received-location: kji.h LINE:3418 ID:kjata: wake up enqueue owner 
 last process to post me: c000000cf9c7b7e0 1 6
 last post sent: 0 0 26
 last post sent-location: ksa2.h LINE:285 ID:ksasnd
 last process posted by me: c000000d61c888c8 3 6
 (latch info) wait_event=0 bits=0
 Process Group: DEFAULT, pseudo proc: 0xc000000d021268a8
 O/S info: user: oracle, term: UNKNOWN, ospid: 19858 
 OSD pid info: Unix process pid: 19858, image: oracle@qdty1 (J000) 
 Short stack dump: 
ksedsts()+544<-ksdxfstk()+48<-$cold_ksdxdocmdmultex()+10832<-ksdxdocmdmult()+80
<-$cold_ksudmp_proc()+5264<-ksudss()+4176<-kqrigt()+4112<-$cold_kqrLockAndPinPo()+864<-kqrpre1()+7968<-kqrpre()+64<-kkdlobn2()+144<-$cold_kdpurts()
+3440<-kdpursp()+240<-ktsx_purge_bin()+560<-ktspnr_next()+5968<-ktrsexec()+1664<-ktspbmphwm()+2160<-ktspmvhwm()+96<-ktsp_bump_hwm()+464<-ktspgsp_main()
+3264<-kdtgsp()+2112<-kdtgsph()+1440<-kdtFlushBuf()+1040<-qerltcFlushKdtBuffer()+864<-qerltcEndLoadProcessing()+704<-qerltcFetch()+1808<-insexe()
+1584<-opiexe()+16192<-opipls()+4192<-opiodr()+2416<-rpidrus()+432<-skgmstack()+224<-rpidru()+224<-rpiswu2()+1120<-rpidrv()+2736<-psddr0()+496<-psdnal()+1136<-pevm_EXECC()+1312<-pfrinstr_EXECC()+144<-pfrrun_no_tool()+192<-pfrrun()+1728<-plsql_run()+1200<-peicnt()+608<-kkxexe()+1104<-opiexe()
+22416<-opiall0()+2640<-opikpr()+864<-opiodr()+2416<-rpidrus()+432<-skgmstack()+224<-rpidru()+224<-rpiswu2()+1120<-kprball()+2736<-kktextrg()+704<-kkttrex()+4080<-kktexeevt0()+1184<-kktexefrdevt()+608<-kktfrddltrg()+2064<-opiexe()+23168<-opiosq0()+6960<-opiall0()+4384<-opikpr()+864<-opiodr()+241
6<-rpidrus()+432<-skgmstack()+224<-rpidru()+224<-rpiswu2()+1120<-kprball()+2736<-kdpurtab()+1424<-$cold_kdpurts()+3792<-kdpursp()+
240<-ktsx_purge_bin()+560<-ktspnr_next()+5968<-ktrsexec()+1184<-ktspbmphwm()+2160<-ktspmvhwm()+96<-ktsp_bump_hwm()+464<-ktspgsp_main()+3264
<-kdtgsp()+2112<-kdtgsph()+1440<-kdtFlushBuf()+1040<-qerltcFlushKdtBuffer()+864<-qerltcEndLoadProcessing()+704<-qerltcFetch()+1808<-insexe()
+1584<-opiexe()+16192<-opipls()+4192<-opiodr()+2416<-rpidrus()+432<-skgmstack()+224<-rpidru()+224<-rpiswu2()+1120<-rpidrv()+2736<-psddr0()
+496<-psdnal()+1136<-pevm_EXECC()+1312<-pfrinstr_EXECC()+144<-pfrrun_no_tool()+192<-pfrrun()+1728<-plsql_run()+1200<-peicnt()+608<-kkxexe()+1104<-opiexe()+22416<-opiall0()+2640<-opikpr()+864<-opiodr()+2416<-rpidrus()+432<-skgmstack()+224<-rpidru()+224<-rpiswu2()+1120<-kprball()+2736<-kktextrg()
+704<-kkttrex()+4080<-kktexeevt0()+1184<-kktexefrdevt()+608<-kktfrddltrg()+2064<-opiexe()+23168<-opiosq0()+6960<-opipls()+7584
<-opiodr()+2416<-rpidrus()+432<-skgmstack()+224<-rpidru()+224<-rpiswu2()+1120<-rpidrv()+2736<-psddr0()+496<-psdnal()+1136
<-pevm_EXIM()+592<-pfrinstr_EXIM()+80<-pfrrun_no_tool()+192<-pfrrun()+1728<-plsql_run()+1200<-peicnt()+608<-kkxexe()+1104
<-opiexe()+22416<-opiodr()+2416<-rpidrus()+432<-skgmstack()+224<-rpidru()+224<-rpiswu2()+1120<-rpidrv()+2736<-rpiexe()+144
<-$cold_kkjex1e()+18176<-kkjsexe()+1424<-kkjrdp()+2336<-opirip()+1776<-opidrv()+1152<-sou2o()+256<-opimai_real()+352<-ssthrdmain(576<-main()+336<-main_opd_entry()+80

Note:
根据processes的信息能看到当前的进程号，当前的call调用和call stack信息，kqrpre()是申请row cache lock相关函数。接下来顺着call 找对象。

# CALL
 ----------------------------------------
 SO: 0xc000000c7ea7f9d8, type: 3, owner: 0xc000000d19dd8368, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0xc000000d19dd8368, name=call, file=ksu.h LINE:12620 ID:, pg=0
 (call) sess: cur c000000d1b14b988, rec c000000d1b14b988, usr c000000d1b14b988; flg:30 fl2:1; depth:0
 svpt(xcb:0x0000000000000000 sptn:0x2 uba: 0x00000000.0000.00)
 ksudlc FALSE at location: 0
 ----------------------------------------
 SO: 0xc000000c7ea761d8, type: 3, owner: 0xc000000c7ea7f9d8, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0xc000000d19dd8368, name=call, file=ksu.h LINE:12620 ID:, pg=0
 (call) sess: cur c000000d1b14b988, rec c000000d1b14b988, usr c000000d1b14b988; flg:8 fl2:1; depth:1
 svpt(xcb:0x0000000000000000 sptn:0x5f uba: 0x00000000.0000.00)
 ksudlc FALSE at location: 0
 ----------------------------------------
 SO: 0xc000000c7ea81108, type: 3, owner: 0xc000000c7ea761d8, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0xc000000d19dd8368, name=call, file=ksu.h LINE:12620 ID:, pg=0
 (call) sess: cur c000000d1b14b988, rec c000000d1b14b988, usr c000000d1b14b988; flg:8 fl2:1; depth:2
 svpt(xcb:0x0000000000000000 sptn:0xaf uba: 0x00000000.0000.00)
 ksudlc FALSE at location: 0
 ----------------------------------------
 SO: 0xc000000c7ea86098, type: 3, owner: 0xc000000c7ea81108, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0xc000000d19dd8368, name=call, file=ksu.h LINE:12620 ID:, pg=0
 (call) sess: cur c000000d1b14b988, rec c000000d1b14b988, usr c000000d1b14b988; flg:8 fl2:1; depth:3
 svpt(xcb:0xc000000cefa86040 sptn:0x20a uba: 0x00000000.0000.00)
 ksudlc FALSE at location: 0
 ----------------------------------------
 SO: 0xc000000c7ea88360, type: 3, owner: 0xc000000c7ea86098, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0xc000000d19dd8368, name=call, file=ksu.h LINE:12620 ID:, pg=0
 (call) sess: cur c000000d1b14b988, rec 0, usr c000000d1b14b988; flg:0 fl2:1; depth:4
 svpt(xcb:0xc000000cefa86040 sptn:0x214 uba: 0x00000000.0000.00)
 ksudlc FALSE at location: 0
 ----------------------------------------
 SO: 0xc000000c7ea8c318, type: 3, owner: 0xc000000c7ea88360, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0xc000000d19dd8368, name=call, file=ksu.h LINE:12620 ID:, pg=0
 (call) sess: cur c000000d630cbf10, rec c000000d630cbf10, usr c000000d1b14b988; flg:8 fl2:1; depth:5
 svpt(xcb:0xc000000cefa86040 sptn:0x219 uba: 0x00000000.0000.00)
 ksudlc FALSE at location: 0
 ----------------------------------------
 SO: 0xc000000c7ea8efe0, type: 3, owner: 0xc000000c7ea8c318, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0xc000000d19dd8368, name=call, file=ksu.h LINE:12620 ID:, pg=0
 (call) sess: cur c000000d630cbf10, rec c000000d1b14b988, usr c000000d1b14b988; flg:8 fl2:1; depth:6
 svpt(xcb:0xc000000ce37bfb48 sptn:0x24f uba: 0x00000000.0000.00)
 ksudlc FALSE at location: 0
 ----------------------------------------
 SO: 0xc000000c7c51d938, type: 3, owner: 0xc000000c7ea8efe0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0xc000000d19dd8368, name=call, file=ksu.h LINE:12620 ID:, pg=0
 (call) sess: cur c000000d1b14b988, rec c000000d1b14b988, usr c000000d1b14b988; flg:8 fl2:1; depth:7
 svpt(xcb:0xc000000ce37bfb48 sptn:0xbf5 uba: 0x00000000.0000.00)
 ksudlc FALSE at location: 0
 ----------------------------------------
 SO: 0xc000000c7e64d6f0, type: 3, owner: 0xc000000c7c51d938, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0xc000000d19dd8368, name=call, file=ksu.h LINE:12620 ID:, pg=0
 (call) sess: cur c000000d1b14b988, rec 0, usr c000000d1b14b988; flg:0 fl2:1; depth:8 <<<<<<<<<<<<< DEPTH 8说明是递归了第9层
 svpt(xcb:0xc000000ce37bfb48 sptn:0xbfd uba: 0x00000000.0000.00)
 ksudlc FALSE at location: 0
 ----------------------------------------
 SO: 0xc000000d54226898, type: 75, owner: 0xc000000c7e64d6f0, flag: INIT/-/-/0x00 if: 0x1 c: 0x1
 proc=0xc000000d19dd8368, name=row cache enqueues, file=kqr.h LINE:2064 ID:, pg=0
 row cache enqueue: count=1 session=c000000d1b14b988 object=c000000d4d1f35b0, request=S
 savepoint=0xc42
 row cache parent object: address=c000000d4d1f35b0 cid=11(dc_objects) 
 hash=cde9d4ee typ=21 transaction=0000000000000000 flags=00000000 
 objectno=26440423 ownerid=36 nsp=1 
 name=BIN$bARtuAoEVFfgVEgPz2hlMg==$0 own=c000000d4d1f3678[c000000d4d1f3678,c000000d4d1f3678] wat=c000000d4d1f3688[c000000d54226940,c000000d54226940] mode=N
 status=-/-/-/-/-/-/-/-/-
 request=S release=FALSE flags=a 
 instance lock=QI cde9d4ee 6e02795b 
 set=0, complete=FALSE 
 data=
 00000024 001e4249 4e246241 52747541 6f455646 66675645 67507a32 686c4d67 
 3d3d2430 01000000 00000000 00000000 00000000 00000000 00000000 00000000 
 ......

Note:
我们找到了session正在执行递归了9层的call调用，在在申请S模式的row cache enqueues, 对象号是26440423，所有者ID是36，名为BIN$bARtuAoEVFfgVEgPz2hlMg==$0，没错！回收站对象，相信你应该想到了为什么会递归到它，申请并且还没有成功，下一步看是哪个会持有这个对象锁。以name或objectno为关键字搜索。其实通过hash查找可以LCK进程正在转换实例间该资源锁。

     ----------------------------------------
      SO: 0xc000000d4d1f3860, type: 29, owner: 0xc000000d1b9e61f0, flag: INIT/-/-/0x00 if: 0x1 c: 0x1
       proc=0xc000000d61c888c8, name=ges enqueues, file=kjl.h LINE:587 ID:, pg=0
      ----------enqueue c000000d4d1f3860------------------------
      lock version     : 1
      Owner inst       : 1
      grant_level      : KJUSERNL
      req_level        : KJUSERPR
      bast_level       : KJUSERNL
      notify_func      : 0000000000000000
      resp             : c000000d36175718
      procp            : c000000d1b9e61f0
      pid              : 5156
      proc version     : 0
      oprocp           : 0000000000000000
      opid             : 5156
      group lock owner : 0000000000000000
      xid              : 0000-0000-00000000
      dd_time          : 0.0 secs
      dd_count         : 0
      timeout          : 172.0 secs
      On_timer_q?      : Y
      On_dd_q?         : N
      lock_state       : OPENING CONVERTING 
      ast_flag         : 0x0
      Open Options     :  KJUSERPROCESS_OWNED KJUSERCLIENTLOCK
      Convert options  : KJUSERNODEADLOCKWAIT KJUSERNODEADLOCKBLOCK KJUSERTRCCANCEL 
      History          : 0x9a
      Msg_Seq          : 0x0
      res_seq          : 998
      valblk           : 0x00000000000000000000000000000000 .
      ----------resource c000000d36175718----------------------
      resname       : [0xcde9d4ee][0x6e02795b],[QI][ext 0x0,0x0]
      hash mask     : x3
      Local inst    : 1
      dir_inst      : 1
      master_inst   : 1
      hv idx        : 40
      hv last r.inc : 8
      current inc   : 8
      hv status     : 0
      hv master     : 0
      open options  : 
      grant_bits    : KJUSERNL KJUSEREX 
      grant mode    : KJUSERNL  KJUSERCR  KJUSERCW  KJUSERPR  KJUSERPW  KJUSEREX
      count         : 1         0         0         0         0         1
      val_state     : KJUSERVS_NOVALUE
      valblk        : 0xc000000000667d50c00000000050ba40 .f}PP@
      access_inst   : 2
      vbreq_state   : 0
      state         : x0
      resp          : c000000d36175718
      On Scan_q?    : N
      Total accesses: 506837
      Imm.  accesses: 5492
      Granted_locks : 1 
      Cvting_locks  : 1 
value_block:  c0 00 00 00 00 66 7d 50 c0 00 00 00 00 50 ba 40
      GRANTED_Q :
      lp c000000c84c5ba58 gl KJUSEREX rp c000000d36175718 [0xcde9d4ee][0x6e02795b],[QI][ext 0x0,0x0]
        master 1 owner 2  bast 0 rseq 612 mseq 0x10055 history 0xed8ded8d
        open opt  KJUSERNO_XID 
      CONVERT_Q: 
      lp c000000d4d1f3860 gl KJUSERNL rl KJUSERPR rp c000000d36175718 [0xcde9d4ee][0x6e02795b],[QI][ext 0x0,0x0]
        master 1 pid 5156 bast 0 rseq 998 mseq 0 history 0x9a
        convert opt KJUSERNODEADLOCKWAIT KJUSERNODEADLOCKBLOCK KJUSERTRCCANCEL  

PROCESS 29: LCK0
  ----------------------------------------
  SO: 0xc000000d61c888c8, type: 2, owner: 0x0000000000000000, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
   proc=0xc000000d61c888c8, name=process, file=ksu.h LINE:12616 ID:, pg=0
  (process) Oracle pid:29, ser:3, calls cur/top: 0xc000000cfb52c0b8/0xc000000cfb52c0b8
            flags : (0x6) SYSTEM
            flags2: (0x0),  flags3: (0x10) 
            intr error: 0, call error: 0, sess error: 0, txn error 0
            intr queue: empty
    ksudlp FALSE at location: 0
  (post info) last post received: 0 0 26
              last post received-location: ksa2.h LINE:285 ID:ksasnd
              last process to post me: c000000d09ccb820 28 0
              last post sent: 0 0 240
              last post sent-location: kqr.h LINE:2233 ID:kqrbtm
              last process posted by me: c000000d09ccb820 28 0
    (latch info) wait_event=0 bits=0
    Process Group: DEFAULT, pseudo proc: 0xc000000d021268a8
    O/S info: user: oracle, term: UNKNOWN, ospid: 5156 
    OSD pid info: Unix process pid: 5156, image: oracle@qdty1 (LCK0)
    Short stack dump: 
ksedsts()+544<-ksdxfstk()+48<-ksdxcb()+3216<-sspuser()+688<-<-_poll_sys()+48<-_poll()+224<-ssskgxp_poll()+208<-sskgxp_selectex()+1872<-skgxpiwait()+9424<-skgxpwaiti()+976<-skgxpwait()+416<-ksxpwait()+2880<-$cold_ksliwat()+2288<-kslwaitctx()+304<-kslwait()+192<-ksarcv()+640<-ksbabs()+752<-kclabs()+400<-ksbrdp()+2736<-opirip()+1296<-opidrv()+1152<-sou2o()+256<-opimai_real()+352<-ssthrdmain()+576<-main()+336<-main_opd_entry()+80

Note:
从字面看应该是LCK已经发现了deadlock（猜测）， 下面我们去同时间点另一个节点生成的systemstate dump查看刚才的BIN$XXX对象锁.

      ----------------------------------------
        SO: 0xc000000d4bef1a70, type: 75, owner: 0xc000000cd2882160, flag: INIT/-/-/0x00 if: 0x1 c: 0x1
         proc=0xc000000d11c9f608, name=row cache enqueues, file=kqr.h LINE:2064 ID:, pg=0
        row cache enqueue: count=8 session=c000000cfac55ee0 object=c000000d42c5f400, mode=X 
        savepoint=0x60b
        row cache parent object: address=c000000d42c5f400 cid=8(dc_objects)
        hash=cde9d4ee typ=21 transaction=c000000cd2882160 flags=00002031
        objectno=26440423 ownerid=36 nsp=1
        name=BIN$bARtuAoEVFfgVEgPz2hlMg==$0        own=c000000d42c5f4c8[c000000d4bef1b18,c000000d4bef1b18] wat=c000000d42c5f4d8[c000000d42c5f4d8,c000000d42c5f4d8] mode=X
        status=EMPTY/DELETE/-/-/-/-/NEED_INV/-/-
        request=N release=TRUE flags=8    
        instance lock=QI cde9d4ee 6e02795b
        set=0, complete=FALSE     
        data=
...

----------------------------------------
        SO: 0xc000000d4d9b18f0, type: 79, owner: 0xc000000cd2882160, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
         proc=0xc000000d11c9f608, name=LIBRARY OBJECT PIN, file=kgl.h LINE:8551 ID:, pg=0

        LibraryObjectPin: Address=c000000d4d9b18f0 Handle=c000000a179cf0e8 Mode=X ClusterLock=c000000d66961b08 User=c000000cfac55ee0 Session=c000000cfac5be40 Count=1 Mask=0709 Flags=[2000] SavepointNum=0x525 
        LibraryHandle:  Address=c000000a179cf0e8 Hash=972dc3a9 LockMode=X PinMode=X LoadLockMode=0 Status=VALD 
          ObjectName:  Name=ICDCCT.BIN$bARtuAoEVFfgVEgPz2hlMg==$0   
            FullHashValue=30205d0384d4db1253717722972dc3a9 Namespace=TABLE/PROCEDURE(01) Type=TABLE(02) Identifier=26440423 OwnerIdn=36 
          Statistics:  InvalidationCount=0 ExecutionCount=0 LoadCount=4 ActiveLocks=1 TotalLockCount=20 TotalPinCount=28 
          Counters:  BrokenCount=1030 RevocablePointer=1 KeepDependency=0 BucketInUse=1387 HandleInUse=1387 HandleReferenceCount=0 
          Concurrency:  DependencyMutex=c000000a179cf198(0, 0, 0, 0) Mutex=c000000a179cf218(0, 1754, 0, 0) 
          Flags=PIN/TIM/PTM/FUL/FUP/[0c005881] 
          WaitersLists:  
            Lock=c000000a179cf178[c000000a179cf178,c000000a179cf178] 
            Pin=c000000a179cf158[c000000a179cf158,c000000a179cf158] 
            LoadLock=c000000a179cf1d0[c000000a179cf1d0,c000000a179cf1d0] 
          Timestamp:  Current=NULL Previous=05-13-2018 00:06:01 
          HandleReference:  Address=c000000a179cf2c8 Handle=c000000d4ed70c70 Flags=OWN[200] 
          LockInstance:  id='LB30205d0384d4db12' GlobalEnqueue=0000000000000000 ReleaseCount=0 
          PinInstance:  id='NB30205d0384d4db12' GlobalEnqueue=0000000000000000 
          LibraryObject:  Address=c000000b75ac2a58 HeapMask=0000-0000-0000-0000 Flags=NEX/DRP[0040] Flags2=[0000] PublicFlags=[0040] 
            DataBlocks:  
              Block:  #='0' name=KGLH0^972dc3a9 pins=0 Change=NONE   
                Heap=c000000a17212e68 Pointer=c000000b75ac2af8 Extent=c000000b75ac29d8 Flags=I/-/P/A/-/- 
                FreedLocation=0 Alloc=1.554688 Size=3.976562 LoadTime=61784030550 

# search 0xc000000cd2882160		  
		  	  
	      ----------------------------------------
      SO: 0xc000000cd2882160, type: 56, owner: 0xc000000cfac55ee0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
       proc=0xc000000d11c9f608, name=transaction, file=ktccts.h LINE:410 ID:, pg=0
      (trans) flg = 0x08001603, flg2 = 0x00482000, flg3 = 0x00000000, prx = 0x0000000000000000, ros = 2147483647, crtses=0xc000000cfac55ee0
      flg  = 0x08001603: ALC TRN VUS VID USN DRT 
      flg2 = 0x00482000: CCP NIP DDID 
      flg3 = 0x00000000: 
      bsn = 0x496 bndsn = 0x638 spn = 0x67d
      efd = 50 rfd = 0 DID:
      file:kdpur.c lineno:672
      parent xid: 0x0000.000.00000000
      env [0xc000000cd2882578]: (scn: 0x0e70.641f08e3  xid: 0x3033.012.0013a30b  uba: 0x00000000.0000.00  statement num=0  parent xid: 0x0000.000.00000000  st-scn: 0x0000.00000000  hi-scn: 0x0000.00000000  ma-scn: 0x0000.00000000  flg: 0x00000000)
      cev: (spc = 3622  arsp = 0xc000000cec00e7f8  ubkds (ubk:tsn: 4 rdba: 0x2e008be3 flag:0x8 hdl:0x0000000000000000 addr:0x0000000000000000)  useg tsn: 4 rdba: 0x01019ffd
            hwm uba: 0x2e008be3.1e70.5a  col uba: 0x00000000.0000.00
            num bl: 0 bk list: 0x0)
            cr opc: 0x0 spc: 3622 uba: 0x2e008be3.1e70.5a
      Begin scn:0x0e70.641f08e3 uba:0x2e008be3.1e70.5a ts:1531915382[07/18/2018 20:03:02]
      Undo blks: 1 recs: 1
      ccbstg: 0x00000000
      (enqueue) TX-30330012-0013A30B	DID: 0002-00B1-00003E22
      lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  res_flag: 0x47
      mode: X, lock_flag: 0x0, lock: 0xc000000cd28821d8, res: 0xc000000d23958a18
      own: 0xc000000cfac55ee0, sess: 0xc000000cfac55ee0, proc: 0xc000000d11c9f608, prv: 0xc000000d23958a28


# search 0xc000000cfac55ee0

   ----------------------------------------
    SO: 0xc000000cfac55ee0, type: 4, owner: 0xc000000d11c9f608, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
     proc=0xc000000d11c9f608, name=session, file=ksu.h LINE:12624 ID:, pg=0
    (session) sid: 6423 ser: 45747 trans: 0xc000000cd2882b70, creator: 0xc000000d11c9f608
              flags: (0x210041) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
              flags2: (0x40009) -/-/INC
              DID: , short-term DID: 
              txn branch: 0x0000000000000000
              oct: 2, prv: 0, sql: 0xc000000a46b81718, psql: 0xc000000a46b81718, user: 28/xxxx
    ksuxds FALSE at location: 0
    service name: SYS$USERS
    client details:
      O/S info: user: oracle, term: UNKNOWN, ospid: 18176
      machine: qdty2 program: oracle@qdty2 (J001)
    Current Wait Stack:
      Not in wait; last wait ended 0.029569 sec ago 
    Wait State:
      fixed_waits=0 flags=0x21 boundary=0x0000000000000000/-1
    Session Wait History:
        elapsed time of 0.029602 sec since last wait
     0: waited for 'ksdxexeotherwait'
        =0x0, =0x0, =0x0
        wait_id=10573 seq_num=10574 snap_id=1
        wait times: snap=0.062345 sec, exc=0.062345 sec, total=0.062345 sec
        wait times: max=30.000000 sec
        wait counts: calls=0 os=0
        occurred after 0.007576 sec of elapsed time

# search 		 0xc000000d11c9f608

PROCESS 177: J001
  ----------------------------------------
  SO: 0xc000000d11c9f608, type: 2, owner: 0x0000000000000000, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
   proc=0xc000000d11c9f608, name=process, file=ksu.h LINE:12616 ID:, pg=0
  (process) Oracle pid:177, ser:17, calls cur/top: 0xc00000015c404d38/0xc000000c94cb9308
            flags : (0x0) -
            flags2: (0x1010),  flags3: (0x10) 
            intr error: 0, call error: 0, sess error: 0, txn error 0
            intr queue: empty
    ksudlp FALSE at location: 0
  (post info) last post received: 0 0 241
              last post received-location: kqr.h LINE:2237 ID:kqrbgl: compatible mode
              last process to post me: c000000d61c888c8 1 6
              last post sent: 0 0 26
              last post sent-location: ksa2.h LINE:285 ID:ksasnd
              last process posted by me: c000000d61c888c8 1 6
    (latch info) wait_event=0 bits=0
    Process Group: DEFAULT, pseudo proc: 0xc000000d021268a8
    O/S info: user: oracle, term: UNKNOWN, ospid: 18176 
    OSD pid info: Unix process pid: 18176, image: oracle@qdty2 (J001)
    Short stack dump: 
ksedsts()+544<-ksdxfstk()+48<-$cold_ksdxdocmdmultex()+10832<-ksdxdocmdmult()+80
<-$cold_ksudmp_proc()+5264<-ksudss()+4176<-kqrigt()+4112<-$cold_kqrLockAndPinPo()
+864<-kqrpre1()+7968<-kqrpre()+64<-kkdlobn2()+144<-$cold_kdpurts()+3440<-kdpursp()

Note:
可以看到1:9074 session的j000申请的BIN$bARtuAoEVFfgVEgPz2hlMg==$0 row cache enqueue正在被2:6423 session的j001进程以X模式占有，未且还没有释放，所以另1：9074就审请不成功,该事务同样也是8：03时间点的JOB进程调起。下面可以看j001 在执行什么sql, 什么call.

     ----------------------------------------
	  
	# search SQL c000000a46b81718
	  
	   SO: 0xc000000a26f226f8, type: 78, owner: 0xc000000d1a41db48, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
       proc=0xc000000d61c92fa8, name=LIBRARY OBJECT LOCK, file=kgl.h LINE:8547 ID:, pg=0

      LibraryObjectLock:  Address=c000000a26f226f8 Handle=c000000a46b81718 Mode=N CanBeBrokenCount=1 Incarnation=1 ExecutionCount=1         
        
        User=c000000d1a41db48 Session=c000000d1a41db48 ReferenceCount=1 Flags=[0000] SavepointNum=5b4f6ed5 
      LibraryHandle:  Address=c000000a46b81718 Hash=876a7296 LockMode=N PinMode=0 LoadLockMode=0 Status=VALD 
        ObjectName:  Name=INSERT INTO DDL_EVENTS SELECT :B1 , SYSDATE, ORA_LOGIN_USER, ORA_DICT_OBJ_NAME, ORA_DICT_OBJ_OWNER, ORA_DICT_OBJ_TYPE, ORA_SYSEVENT,
 MACHINE, PROGRAM, OSUSER FROM SYS.DUAL, SYS.V_$SESSION WHERE SYS_CONTEXT('USERENV','SESSIONID' ) = AUDSID(+) 
          FullHashValue=6727fe5ca5d3747056737161876a7296 Namespace=SQL AREA(00) Type=CURSOR(00) Identifier=2271900310 OwnerIdn=28 
        Statistics:  InvalidationCount=1 ExecutionCount=549887 LoadCount=6 ActiveLocks=6 TotalLockCount=516666 TotalPinCount=1 
        Counters:  BrokenCount=1 RevocablePointer=1 KeepDependency=1 BucketInUse=3 HandleInUse=3 HandleReferenceCount=0 
        Concurrency:  DependencyMutex=c000000a46b817c8(0, 6215, 0, 0) Mutex=c000000a46b81848(0, 2212023, 10, 0) 
        Flags=RON/PIN/TIM/PN0/DBN/[10012841] 
        WaitersLists:  
          Lock=c000000a46b817a8[c000000a46b817a8,c000000a46b817a8] 
          Pin=c000000a46b81788[c000000a46b81788,c000000a46b81788] 
          LoadLock=c000000a46b81800[c000000a46b81800,c000000a46b81800] 
        Timestamp:  Current=11-02-2017 17:12:54 
        HandleReference:  Address=c000000a46b819a8 Handle=0000000000000000 Flags=[00] 
        LibraryObject:  Address=c00000099c3b8860 HeapMask=0000-0001-0001-0000 Flags=EXS[0000] Flags2=[0000] PublicFlags=[0000] 
          ChildTable:  size='16' 
            Child:  id='0' Table=c00000099c3b9710 Reference=c00000099c3b9168 Handle=c000000a45aa83e8 
            Child:  id='1' Table=c00000099c3b9710 Reference=c00000099c3b9448 Handle=c000000d28d500b0 
            Child:  id='2' Table=c00000099c3b9710 Reference=c00000099e421e40 Handle=c000000a464b13e8 
        NamespaceDump:  
          Parent Cursor:  sql_id=5cwvjc63qnwnq parent=c00000099c3b8900 maxchild=3 plk=y ppn=n 
  
 # search CALL 0xc00000015c404d38
 
 ----------------------------------------
 SO: 0xc000000c94cb9308, type: 3, owner: 0xc000000d11c9f608, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0xc000000d11c9f608, name=call, file=ksu.h LINE:12620 ID:, pg=0
 (call) sess: cur c000000cfac55ee0, rec c000000cfac55ee0, usr c000000cfac55ee0; flg:30 fl2:1; depth:0
 svpt(xcb:0x0000000000000000 sptn:0x2 uba: 0x00000000.0000.00)
 ksudlc FALSE at location: 0
 ----------------------------------------
 SO: 0xc000000c949106b0, type: 3, owner: 0xc000000c94cb9308, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0xc000000d11c9f608, name=call, file=ksu.h LINE:12620 ID:, pg=0
 (call) sess: cur c000000cfac55ee0, rec c000000cfac55ee0, usr c000000cfac55ee0; flg:8 fl2:1; depth:1
 svpt(xcb:0x0000000000000000 sptn:0x8b uba: 0x00000000.0000.00)
 ksudlc FALSE at location: 0
 ----------------------------------------
 SO: 0xc000000c94db2590, type: 3, owner: 0xc000000c949106b0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0xc000000d11c9f608, name=call, file=ksu.h LINE:12620 ID:, pg=0
 (call) sess: cur c000000cfac55ee0, rec c000000cfac55ee0, usr c000000cfac55ee0; flg:8 fl2:1; depth:2
 svpt(xcb:0x0000000000000000 sptn:0xec uba: 0x00000000.0000.00)
 ksudlc FALSE at location: 0
 ----------------------------------------
 SO: 0xc000000c95de4fd0, type: 3, owner: 0xc000000c94db2590, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0xc000000d11c9f608, name=call, file=ksu.h LINE:12620 ID:, pg=0
 (call) sess: cur c000000cfac55ee0, rec c000000cfac55ee0, usr c000000cfac55ee0; flg:8 fl2:1; depth:3
 svpt(xcb:0xc000000cd2ad4428 sptn:0x252 uba: 0x00000000.0000.00)
 ksudlc FALSE at location: 0
 ----------------------------------------
 SO: 0xc00000015c3aa170, type: 3, owner: 0xc000000c95de4fd0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0xc000000d11c9f608, name=call, file=ksu.h LINE:12620 ID:, pg=0
 (call) sess: cur c000000cfac55ee0, rec 0, usr c000000cfac55ee0; flg:0 fl2:1; depth:4
 svpt(xcb:0xc000000cd2ad4428 sptn:0x25c uba: 0x00000000.0000.00)
 ksudlc FALSE at location: 0
 ----------------------------------------
 SO: 0xc000000c94f16200, type: 3, owner: 0xc00000015c3aa170, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0xc000000d11c9f608, name=call, file=ksu.h LINE:12620 ID:, pg=0
 (call) sess: cur c000000cfac5be40, rec c000000cfac5be40, usr c000000cfac55ee0; flg:8 fl2:1; depth:5
 svpt(xcb:0xc000000cd2ad4428 sptn:0x261 uba: 0x00000000.0000.00)
 ksudlc FALSE at location: 0
 ----------------------------------------
 SO: 0xc00000015c486660, type: 3, owner: 0xc000000c94f16200, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0xc000000d11c9f608, name=call, file=ksu.h LINE:12620 ID:, pg=0
 (call) sess: cur c000000cfac5be40, rec c000000cfac55ee0, usr c000000cfac55ee0; flg:8 fl2:1; depth:6
 svpt(xcb:0xc000000cd2882160 sptn:0x49d uba: 0x00000000.0000.00)
 ksudlc FALSE at location: 0
 ----------------------------------------
 SO: 0xc00000015c486f18, type: 3, owner: 0xc00000015c486660, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0xc000000d11c9f608, name=call, file=ksu.h LINE:12620 ID:, pg=0
 (call) sess: cur c000000cfac55ee0, rec c000000cfac55ee0, usr c000000cfac55ee0; flg:8 fl2:1; depth:7
 svpt(xcb:0xc000000cd2882160 sptn:0x63d uba: 0x00000000.0000.00)
 ksudlc FALSE at location: 0
 ----------------------------------------
 SO: 0xc00000015c404d38, type: 3, owner: 0xc00000015c486f18, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0xc000000d11c9f608, name=call, file=ksu.h LINE:12620 ID:, pg=0
 (call) sess: cur c000000cfac55ee0, rec 0, usr c000000cfac55ee0; flg:0 fl2:1; depth:8
 svpt(xcb:0xc000000cd2882160 sptn:0x645 uba: 0x00000000.0000.00)
 ksudlc FALSE at location: 0
 ----------------------------------------
 SO: 0xc000000a27156670, type: 75, owner: 0xc00000015c404d38, flag: INIT/-/-/0x00 if: 0x1 c: 0x1
 proc=0xc000000d11c9f608, name=row cache enqueues, file=kqr.h LINE:2064 ID:, pg=0
 row cache enqueue: count=1 session=c000000cfac55ee0 object=c000000d43dbeec8, request=S
 savepoint=0x681
 row cache parent object: address=c000000d43dbeec8 cid=11(dc_objects)
 hash=a92c05bd typ=21 transaction=0000000000000000 flags=00000000
 objectno=26440455 ownerid=38 nsp=1
 name=BIN$bARtuAoBVF3gVEgPz2hlMg==$0 own=c000000d43dbef90[c000000d43dbef90,c000000d43dbef90] wat=c000000d43dbefa0[c000000a27156718,c000000d4acf2fa0] mode=N
 status=-/-/-/-/-/-/-/-/-
 request=S release=FALSE flags=a  
 instance lock=QI a92c05bd 955d5c35  
 set=0, complete=FALSE
 data=
 00000026 001e4249 4e246241 52747541 6f425646 33675645 67507a32 686c4d67

Note:
节点2的6423会话J001正在以S模式申请BIN$bARtuAoBVF3gVEgPz2hlMg==$0的row cache enqueuue. 再继续以它为关键字查。

# node1 SSD TRACE search BIN$bARtuAoBVF3gVEgPz2hlMg==$0
      ----------------------------------------
        SO: 0xc000000d51d61560, type: 75, owner: 0xc000000ce37bfb48, flag: INIT/-/-/0x00 if: 0x1 c: 0x1
         proc=0xc000000d19dd8368, name=row cache enqueues, file=kqr.h LINE:2064 ID:, pg=0
        row cache enqueue: count=9 session=c000000d1b14b988 object=c000000d4a503ca8, mode=X
        savepoint=0xbbe
        row cache parent object: address=c000000d4a503ca8 cid=8(dc_objects)
        hash=a92c05bd typ=21 transaction=c000000ce37bfb48 flags=00002031
        objectno=26440455 ownerid=38 nsp=1
        name=BIN$bARtuAoBVF3gVEgPz2hlMg==$0        own=c000000d4a503d70[c000000d51d61608,c000000d51d61608] wat=c000000d4a503d80[c000000d3ace4558,c000000d3a862708] mode=X
        status=EMPTY/DELETE/-/-/-/-/NEED_INV/-/-
        request=N release=TRUE flags=8
        instance lock=QI a92c05bd 955d5c35
        set=0, complete=FALSE
        data=

Note:
根据proc=0xc000000d19dd8368可以找到进程就是1节点刚才的9074会话的J000进程，同时它又再以mode=X 模式占有。到这里就可以看到这个死锁情况：
1节点9074等待2：节点6423：
持有mode=X row cache enqueue(obj# 26440455 ICDSR.BIN$bARtuAoBVF3gVEgPz2hlMg==$0) ,正在以mode=S请求row cache enqueue(obj# 26440423 ICDCCT.BIN$bARtuAoEVFfgVEgPz2hlMg==$0)
2节点6423等待1节点9074：
持有row cache enqueue(obj# 26440423 ICDCCT.BIN$bARtuAoEVFfgVEgPz2hlMg==$0) mode=X，正在以mode=S请求row cache enqueue(obj# 26440455 ICDSR.BIN$bARtuAoBVF3gVEgPz2hlMg==$0)

把我们的发现反馈给Oracle SR后，Oracle方面认为目前版本已经不能再新开BUG,只在已知BUG中发现2个相似但又不同的BUG:

Bug 13387978 – Sessions running TRUNCATE causing a deadlock, even if the constraints are disabled ( Doc ID 13387978.8 )
Bug 12351027 – Redefinition causing deadlock between “row cache lock” and “library cache lock” ( Doc ID 12351027.8 )

上面的BIN$XX对象是kqrLockAndPinPo()函数调用，所以以ksliwat()+3328<-kslwaitctx()+304<-kqrigt()+1664<-$cold_kqrLockAndPinPo()+864<-kqrpre1()+1824<-kqrpre CALL stack 查询又和2319957.1 描述有点像，但是11204 版本。 至于是哪个BUG现在没法确定。

但是为什么会有drop table BIN$xx purge的操作呢？ 当前想到的也只有空间紧张问题。

BUG也是在特定的场景触发，我们查到这里已经有几种方法可以尝试绕开这个局面，来规避这个未知BUG。

如：

• 减少多会话的DDL操作
• 手动清空回收站
• 为表空间增长可用空间

其它这个系统从我们DDL trigger捕获的操作，有的表1小时能truncate 1万5千多次，以建议看是否可以用全局临时表，但不清楚为什么会有这样的设计。

自动清理回收站

当发现有空间紧张时会自动释放回收站里的对象， 使用的是FIFO先进先出的算法， 以下情况会回收再利用表空间的对象

1. o A user creates a new table or adds data that causes their quota to be exceeded.
   o The tablespace needs to extend its file size to accommodate create/insert operations.

sometimes the tablespace is have big available space, but fragmentation can also occur.

空间的申请优先级

1. 同表空间非回收站的可用空间
2. 同表空间回收站的对象再利用
3. 自动扩展数据文件（如果可以）

模拟方法

1. create table a table  on users tablespace and check table size
2. create a tablespace size same as #1 table size
3. move #1 table to #2 new tablespace
4. drop table #1 in recyclebin
5. enable 10046 trace
6. create a new table on #2 tablespace
7. view 10046 trace contain “drop table BIN$xxx  purge” recursive  sql

 为什么hangmanager 没有自动解锁

Hang Manager是从10.2被引进的,用于是检测和处理hang问题。随着版本的增加功能也不断完善增强。在当前的数据库版本中HM在发现hang或死锁后尝试启用dia0进程去kill 堵塞者的会话和进程。

# dia0 trace file

Fail to allocate buffer due to incompletion of previous mcpy
mbal: buffer allocation to node 0 failed
kjzh_slave_workq_chk: can't allocate KJZG buffer, not-ini
TotalCPUTm:26840 ms TotalRunTm:432030 ms %CPU: 6.21%
DIAG detaches from KGXGN service
error 32737 detected in background process
OPIRIP: Uncaught error 447. Error stack:
ORA-00447: fatal error in background process
ORA-32737: Hang analysis aborted due to failed memory allocation

Note:
这个版本又命中了一个BUG,且11.2.0.3.8高一个小版本就已修复。

Bug 12744759 DIA0 process fails with “Fail to allocate buffer due to incompletion of previous mcpy”

Summary:
两个同时启动的JOB， truncate 了不同的对象DDL， 递归触发了DDL trigger的审计操作，在insert DDL日志表时，遇到了回收站空间再利用，再次递归触发了drop table BIN$ purge, 又属于DDL操作, 并且申请的回收站对象的row cache enqueue时，在2个跨实例会话互相等待对方持有对象lock造成死锁，死锁原因已是未知BUG, 同时HM进程的diag 进程又遇到了另一个bug 而反复重启，无法分析处理hang进程。因2个JOB的死锁后期又出现了其它进程在SQL解析的级联堵塞问题。手动解锁阻塞进程即可恢复。

前几日朋友一套CRS停不下来，RAC版本是11.2.0.4, 停在关闭ASM阶段， 手动尝试了停资源也不成功，使用-f选项一样失败， 后来发现在这还存在一个bug，简单记录一下。

crsctl stop crs

失败提示crs-2675 stop of ‘ ora.asm’ on xx failed.

crs-5022 stop of resource ‘ora.crsd’ filed: current state is “UNKNOWN”

使用crsctl stop crs -f 同样失败，查看crs res 状态

显示ora.asm状态为UNKNOWN,

# oraagent_grid trace

2018-07-25 07:27:23.706: [ora.DGSYS.dg][2585]{1:7667:25576} [check] DgpAgent::initOcrDgpSet exit }
2018-07-25 07:27:23.707: [ora.DGSYS.dg][2585]{1:7667:25576} [check] DgpAgent::inUseByOcr - OCR is on diskgroup DGSYS
2018-07-25 07:27:23.707: [ora.DGSYS.dg][2585]{1:7667:25576} [check] DgpAgent::runCheck: OCR dg returning OFFLINE 
2018-07-25 07:27:23.707: [    AGFW][2057]{1:7667:25576} ora.DGSYS.dg kfora1 1 state changed from: STOPPING to: PLANNED_OFFLINE
2018-07-25 07:27:23.708: [    AGFW][2057]{1:7667:25576} Agent sending last reply for: RESOURCE_STOP[ora.DGSYS.dg kfora1 1] ID 4099:5086977
2018-07-25 07:27:24.677: [ora.DGDATA.dg][1046]{1:7667:25576} [stop] DgpAgent::stop reason is dependency, original reason is system
2018-07-25 07:27:24.682: [ora.DGDATA.dg][1046]{1:7667:25576} [stop] ORA-15032: not all alterations performed
crs: diskgroup "DGDATA" does not exist or is not mounted

2018-07-25 07:27:24.682: [ora.DGDATA.dg][1046]{1:7667:25576} [stop] DgpAgent::stop: error 15032
2018-07-25 07:27:24.682: [ora.DGDATA.dg][1046]{1:7667:25576} [stop] DgpAgent::stop: ignoring err diskgroup is dismounted
2018-07-25 07:27:24.682: [ora.DGDATA.dg][1046]{1:7667:25576} [stop] DgpAgent::stop reason is dependency, original reason is system
2018-07-25 07:27:24.683: [ora.DGDATA.dg][1046]{1:7667:25576} [stop] DgpAgent::getConnxn connected
2018-07-25 07:27:24.684: [ora.DGDATA.dg][1046]{1:7667:25576} [stop] ORA-15032: not all alterations performed
ORA-15001: diskgroup "DGDATA" does not exist or is not mounted

2018-07-25 07:27:24.685: [ora.DGDATA.dg][1046]{1:7667:25576} [stop] DgpAgent::stop: error 15032
2018-07-25 07:27:24.685: [ora.DGDATA.dg][1046]{1:7667:25576} [stop] DgpAgent::stop: ignoring err diskgroup is dismounted
2018-07-25 07:27:24.685: [ora.DGDATA.dg][1046]{1:7667:25576} [stop] DgpAgent::stop reason is dependency, original reason is system
2018-07-25 07:27:24.686: [ora.DGDATA.dg][1046]{1:7667:25576} [stop] DgpAgent::getConnxn connected
2018-07-25 07:27:24.687: [ora.DGDATA.dg][1046]{1:7667:25576} [stop] ORA-15032: not all alterations performed
ORA-15001: diskgroup "DGDATA" does not exist or is not mounted

# ASM alert log

SQL> ALTER DISKGROUP DGSYS DISMOUNT  /* asm agent *//* {1:7667:25576} */ 
Wed Jul 25 07:27:23 2018
NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 2
Wed Jul 25 07:27:23 2018
ORA-15032: not all alterations performed
ORA-15027: active use of diskgroup "DGSYS" precludes its dismount
ERROR: ALTER DISKGROUP DGSYS DISMOUNT  /* asm agent *//* {1:7667:25576} */
Wed Jul 25 07:27:23 2018
SQL> ALTER DISKGROUP DGDATA DISMOUNT  /* asm agent *//* {1:7667:25576} */ 
NOTE: cache dismounting (clean) group 1/0x93932725 (DGDATA) 
NOTE: messaging CKPT to quiesce pins Unix process pid: 1835436, image: oracle@kfora1 (TNS V1-V3)
Wed Jul 25 07:27:23 2018
NOTE: LGWR doing clean dismount of group 1 (DGDATA)
NOTE: LGWR closing thread 2 of diskgroup 1 (DGDATA) at ABA 37.6567
NOTE: LGWR released thread recovery enqueue
Wed Jul 25 07:27:24 2018
kjbdomdet send to inst 2
detach from dom 1, sending detach message to inst 2
Wed Jul 25 07:27:24 2018
NOTE: detached from domain 1
NOTE: cache dismounted group 1/0x93932725 (DGDATA) 
Wed Jul 25 07:27:24 2018
GMON dismounting group 1 at 8 for pid 29, osid 1835436
NOTE: Disk DGDATA_0000 in mode 0x7f marked for de-assignment
NOTE: Disk DGDATA_0001 in mode 0x7f marked for de-assignment
NOTE: Disk DGDATA_0002 in mode 0x7f marked for de-assignment
NOTE: Disk DGDATA_0003 in mode 0x7f marked for de-assignment
SUCCESS: diskgroup DGDATA was dismounted
NOTE: cache deleting context for group DGDATA 1/0x93932725
SUCCESS: ALTER DISKGROUP DGDATA DISMOUNT  /* asm agent *//* {1:7667:25576} */
SQL> ALTER DISKGROUP DGDATA DISMOUNT  /* asm agent *//* {1:7667:25576} */ 
ORA-15032: not all alterations performed
ORA-15001: diskgroup "DGDATA" does not exist or is not mounted
ERROR: ALTER DISKGROUP DGDATA DISMOUNT  /* asm agent *//* {1:7667:25576} */
SQL> ALTER DISKGROUP DGDATA DISMOUNT  /* asm agent *//* {1:7667:25576} */ 
ORA-15032: not all alterations performed
ORA-15001: diskgroup "DGDATA" does not exist or is not mounted
ERROR: ALTER DISKGROUP DGDATA DISMOUNT  /* asm agent *//* {1:7667:25576} */
SQL> ALTER DISKGROUP DGDATA DISMOUNT  /* asm agent *//* {1:7667:25576} */ 
ORA-15032: not all alterations performed
ORA-15001: diskgroup "DGDATA" does not exist or is not mounted
ERROR: ALTER DISKGROUP DGDATA DISMOUNT  /* asm agent *//* {1:7667:25576} */
SQL> ALTER DISKGROUP DGDATA DISMOUNT  /* asm agent *//* {1:7667:25576} */ 
ORA-15032: not all alterations performed
ORA-15001: diskgroup "DGDATA" does not exist or is not mounted
ERROR: ALTER DISKGROUP DGDATA DISMOUNT  /* asm agent *//* {1:7667:25576} */
SQL> ALTER DISKGROUP DGDATA DISMOUNT  /* asm agent *//* {1:7667:25576} */ 
ORA-15032: not all alterations performed
ORA-15001: diskgroup "DGDATA" does not exist or is not mounted
ERROR: ALTER DISKGROUP DGDATA DISMOUNT  /* asm agent *//* {1:7667:25576} */

Note:

从asm alert log看dgdata 磁盘组一共有6次尝试dismount, 第1次已经成功，后5次都是失败，所以报出了ora-15001错误。

原因：

当前版本是11.2.0.4.4  ，MOS中发现该版本存在一个BUG,在11.2.0.4.8中修复。

The issue was investigated in:

Bug 17816316 – ORA-15001 IN ASM ALERT LOG WHEN STOPPING CRS IN 11.2.0.4 RAC

Which is closed as duplicate of:

Bug 16798862 – FAIL TO START DB SERVICE BECAUSE OF PLS-553

The fix is included in GIPSU 11.2.0.4.8, apply latest GI PSU to fix the problem, or ignore the messages if patch can’t be applied immediately.

可以尝试KILL 进程，但是后来客户重启了OS，启动正常。

here is all of the Oracle 18c 18.3 on-premises hidden undocumented parameters:

SQL> select n.ksppinm p_name, c.ksppstvl p_value
2 from sys.x$ksppi n, sys.x$ksppcv c
3 where n.indx=c.indx
4 and lower(n.ksppinm) like lower(‘\_%’) escape ‘\’;

P_NAME                                             P_VALUE
-------------------------------------------------- --------------------------------------------------
_appqos_qt                                         10
_appqos_po_multiplier                              1000
_appqos_cdb_setting                                0
_ior_serialize_fault                               0
_shutdown_completion_timeout_mins                  60
_inject_startup_fault                              0
_wait_outlier_detection_enable                     OFF
_wait_outlier_event_names
_wait_outlier_lambda_x1000                         1500
_wait_outlier_dump_flags                           0
_wait_outlier_min_waits                            10
_wait_outlier_num_outliers                         600
_spin_count                                        2000
_latch_miss_stat_sid                               0
_max_sleep_holding_latch                           4
_max_exponential_sleep                             0
_other_wait_threshold                              0
_other_wait_event_exclusion
_use_vector_post                                   TRUE
_latch_class_0
_latch_class_1
_latch_class_2
_latch_class_3
_latch_class_4
_latch_class_5
_latch_class_6
_latch_class_7
_latch_classes
_ultrafast_latch_statistics                        TRUE
_enable_reliable_latch_waits                       TRUE
_latch_wait_list_pri_sleep_secs                    1
_wait_breakup_time_csecs                           300
_wait_breakup_threshold_csecs                      600
_disable_wait_state
_wait_tracker_num_intervals                        0
_wait_tracker_interval_secs                        10
_wait_samples_max_time_secs                        120
_wait_samples_max_sections                         40
_wait_yield_mode                                   yield
_wait_yield_hp_mode                                yield
_wait_yield_sleep_time_msecs                       1
_wait_yield_sleep_freq                             100
_wait_yield_yield_freq                             20
_post_wait_queues_num_per_class
_super_shared_enable                               FALSE
_super_shared_conversion_threshold                 300
_super_shared_max_exclusive_conversions            4
__wait_test_param                                  0
__latch_test_param                                 0
_latch_pause_mode                                  static
_latch_pause_count                                 0
_min_sys_process                                   2
_session_idle_bit_latches                          0
_ksu_diag_kill_time                                5
_ksuitm_dont_kill_dumper                           FALSE
_disable_image_check                               FALSE
_num_longop_child_latches                          2
_longops_enabled                                   TRUE
_test_ksusigskip                                   5
_disable_kcbhxor_osd                               FALSE
_disable_kgghshcrc32_osd                           FALSE
_disable_system_state                              4294967294
_system_state_runtime_limit                        4294967294
_system_state_cputime_limit                        4294967294
_disable_system_state_wait_samples                 FALSE
_session_wait_history                              10
_pkt_enable                                        FALSE
_pkt_start                                         FALSE
_pkt_pmon_interval                                 50
_collapse_wait_history                             FALSE
_short_stack_timeout_ms                            30000
_sga_early_trace                                   0
_kill_session_dump                                 FALSE
_logout_storm_rate                                 0
_logout_storm_timeout                              5
_logout_storm_retrycnt                             600
_ksuitm_addon_trccmd
_timeout_actions_enabled                           TRUE
_idle_session_kill_enabled                         TRUE
_session_allocation_latches                        2
_suspended_session_timeout                         300
_wait_for_busy_session                             10
_process_allocation_slots                          100
_show_login_pdb_sessions                           FALSE
_disable_highres_ticks                             FALSE
_disable_os_time_page                              FALSE
_timer_precision                                   10
_dbrm_quantum
_highres_drift_allowed_sec                         1
_lowres_drift_allowed_sec                          5
_vktm_assert_thresh                                30
_iorm_tout                                         1000
__oracle_base                                      /oracle
_rman_roundrobin_placement
_process_heartbeat_range                           3
_max_shutdown_abort_secs                           10
_single_process                                    FALSE
_disable_cpu_check                                 FALSE
_available_core_count                              0
_cpu_eff_thread_multiplier
_dbg_proc_startup                                  FALSE
_enqueue_deadlock_time_sec                         5
_enqueue_sync_retry_attempts                       15
_enqueue_sync_sim_mem_error                        FALSE
_number_cached_attributes                          10
_number_cached_group_memberships                   2048
_number_group_memberships_per_cache_line           6
_ksim_time_monitor                                 auto
_group_membership_bucket_scan_timeout              5
_group_membership_bucket_batch_size                8
_group_membership_entries_per_bucket               100
_kss_quiet                                         FALSE
_kss_callstack_type
_oradebug_force                                    FALSE
_ksdxdocmd_default_timeout_ms                      30000
_ksdxdocmd_enabled                                 TRUE
_ksdx_charset_ratio                                0
_oradebug_cmds_at_startup
_disable_oradebug_commands                         none
_watchpoint_on                                     FALSE
_ksdxw_num_sgw                                     10
_ksdxw_num_pgw                                     10
_ksdxw_stack_depth                                 4
_ksdxw_stack_readable                              FALSE
_ksdxw_cini_flg                                    0
_ksdxw_nbufs                                       1000
_hw_watchpoint_on                                  TRUE
_enable_shared_pool_durations                      TRUE
_NUMA_pool_size                                    Not specified
_enable_NUMA_optimization                          FALSE
_enable_NUMA_support                               FALSE
_enable_NUMA_interleave                            TRUE
_touch_sga_pages_during_allocation                 FALSE
_max_largepage_alloc_time_secs                     10
_use_hugetlbfs_for_SGA                             FALSE
_hugetlbfs_mount_point_for_sga
_dump_10261_level                                  0
_numa_shift_enabled                                TRUE
_numa_shift_value                                  0
_freeze_kgh_timestamp                              FALSE
_use_ism                                           TRUE
_NUMA_instance_mapping                             Not specified
_simulator_upper_bound_multiple                    2
_simulator_pin_inval_maxcnt                        16
_simulator_lru_rebalance_thresh                    10240
_simulator_lru_rebalance_sizthr                    5
_simulator_bucket_mindelta                         8192
_simulator_lru_scan_count                          8
_simulator_internal_bound                          10
_simulator_reserved_obj_count                      1024
_simulator_reserved_heap_count                     4096
_simulator_sampling_factor                         2
_realfree_heap_max_size                            32768
_realfree_heap_pagesize                            65536
_realfree_pq_heap_pagesize                         65536
_realfree_heap_mode                                0
_use_realfree_heap                                 TRUE
_pga_large_extent_size                             1048576
_uga_cga_large_extent_size                         262144
_total_large_extent_memory                         0
_use_ism_for_pga                                   TRUE
_private_memory_address
_kgh_restricted_trace                              0
_kgh_restricted_subheaps                           180
_ksm_shared_pool_stats_num_pdb                     8
_ksm_shared_pool_stats_minsz                       524288
_ksm_sp_rcr_hits                                   10
_mem_annotation_sh_lev                             0
_mem_annotation_pr_lev                             0
_mem_annotation_scale                              1
_mem_annotation_store                              FALSE
_4031_dump_bitvec                                  67194879
_4031_max_dumps                                    100
_4031_dump_interval                                300
_4031_sga_dump_interval                            3600
_4031_sga_max_dumps                                10
_4030_dump_bitvec                                  4095
_numa_trace_level                                  0
_mem_std_extent_size                               4096
_kgsb_threshold_size                               16777216
_endprot_chunk_comment                             chk 10235 dflt
_endprot_heap_comment                              hp 10235 dflt
_endprot_subheaps                                  TRUE
_alloc_perm_as_free                                FALSE
_ksm_pre_sga_init_notif_delay_secs                 0
_ksm_post_sga_init_notif_delay_secs                0
_defer_sga_enabled                                 FALSE
_defer_sga_min_total_defer_segs_sz                 107374182400
_defer_sga_alloc_chunk_size                        67108864
_defer_sga_min_spsz_at_startup                     53687091200
_defer_sga_test_alloc_intv                         0
_sga_alloc_slaves_term_timeout_secs                120
_ksmlsaf                                           0
_pga_limit_target_perc                             200
_pga_limit_watch_perc                              50
_pga_limit_time_to_interrupt                       2
_pga_limit_interrupt_smaller                       FALSE
_pga_limit_time_until_idle                         15
_pga_limit_time_until_killed                       15
_pga_limit_use_immediate_kill                      TRUE
_pga_limit_dump_summary                            FALSE
_pga_limit_watch_size                              104857600
_pga_limit_min_req_size                            4194304
_pga_limit_check_wait_time                         1000000
_pga_limit_simulated_physmem_size                  0
_pga_limit_physmem_perc                            90
_max_physmem_perc_sga                              80
_max_physmem_perc_mmt                              90
_pga_limit_per_process_minimum                     3145728
_pga_auto_snapshot_threshold                       524288000
_pga_auto_snapshot_percentage                      20
_pga_detail_combine_auto                           FALSE
_memory_adi_enabled                                TRUE
_memory_adi_precise_errors                         FALSE
_sga_heap_chunk_alignment                          FALSE
_sga_heap_chunk_alignment_disabled                 FALSE
_memory_adi_bytes_per_alloc                        0
_memory_adi_extend                                 FALSE
_memory_adi_heap_mask                              610
_memory_adi_module_mask                            0
_NUMA_bind_mode                                    default
_pdb_vm_max_size
__sga_current_size                                 0
__shared_pool_size                                 230686720
__large_pool_size                                  8388608
__java_pool_size                                   4194304
__streams_pool_size                                0
_large_pool_min_alloc                              65536
_shared_pool_reserved_pct                          5
_shared_pool_reserved_min_alloc                    4400
_kghdsidx_count                                    1
_pga_limit_tracing                                 0
_force_java_pool_zero                              FALSE
_pga_aggregate_xmem_limit                          0
_test_param_1                                      25
_test_param_2
_test_param_3
_test_param_4
_test_param_5                                      25
_test_param_6                                      0
_test_param_7                                      100, 1100, 2100, 3100, 4100
_test_param_8                                      20
_test_param_9
_test_param_9_1                                    0
_test_param_9_2                                    0_startup
_test_param_9_3                                    1_startup, 2_startup, 3_startup
_test_param_10
_test_param_10_bi                                  10
_test_param_10_i
_test_param_11_base                                3
_test_param_11_dep                                 0_STARTUP, 1_STARTUP, 2_STARTUP
_test_param_12                                     0
_disable_instance_params_check                     FALSE
_parameter_table_block_size                        2048
_high_priority_processes                           LMS*|LM1*|LM2*|LM3*|LM4*|LM5*|LM6*|LM7*|LM8*|LM9*
_highest_priority_processes                        VKTM
_os_sched_high_priority                            1
_os_sched_highest_priority                         1
_ksb_restart_clean_time                            30000
_ksb_restart_policy_times                          0, 60, 120, 240
_bg_spawn_diag_opts                                0
_static_backgrounds
_mpmt_enabled_backgrounds                          *
_background_process_opts                           0
_ksd_test_param                                    999
_kse_die_timeout                                   60000
_stack_guard_level                                 0
_kse_pc_table_size                                 256
_kse_signature_entries                             0
_kse_signature_limit                               7
_kse_snap_ring_size                                0
_kse_snap_ring_record_stack                        FALSE
_kse_snap_ring_suppress                            942 1403
_kse_snap_ring_disable                             FALSE
_kse_trace_int_msg_clear                           FALSE
_system_api_interception_debug                     FALSE
_kse_ssnt                                          FALSE
_kse_alt_stack_sig_syms                            10
_messages                                          600
_enqueue_locks                                     5620
_enqueue_resources                                 2296
_enqueue_hash                                      979
_enqueue_debug_multi_instance                      FALSE
_enqueue_hash_chain_latches                        2
_enqueue_deadlock_scan_secs                        0
_enqueue_deadlock_detect_all_global_locks          FALSE
_enqueue_paranoia_mode_enabled                     FALSE
_ksi_trace
_ksi_trace_bucket                                  PRIVATE
_ksi_trace_bucket_size                             LCK1:1048576-REST:8192
_ksi_clientlocks_enabled                           TRUE
_ksi_pdb_checks                                    TRUE
_trace_processes                                   ALL
_trace_events
_trace_buffers                                     ALL:256
_trace_dump_static_only                            FALSE
_trace_dump_all_procs                              FALSE
_trace_dump_cur_proc_only                          FALSE
_trace_dump_client_buckets                         TRUE
_cdmp_diagnostic_level                             2
_nchar_imp_cnv                                     TRUE
_nls_parameter_sync_enabled                        TRUE
_ioslave_issue_count                               500
_ioslave_batch_count                               1
_io_slaves_disabled                                FALSE
_lgwr_io_slaves                                    0
_arch_io_slaves                                    0
_backup_disk_io_slaves                             0
_fg_iorm_slaves                                    1
_backup_io_pool_size                               1048576
_disable_file_locks                                FALSE
_ksfd_verify_write                                 FALSE
_disable_odm                                       FALSE
_enable_fast_file_zero                             TRUE
_enable_kernel_io_outliers                         FALSE
_enable_list_io                                    FALSE
_dnfs_rdma_max                                     1048576
_dnfs_rdma_min                                     8192
_dnfs_rdma_enable                                  rman
_omf                                               enabled
_aiowait_timeouts                                  100
_io_shared_pool_size                               4194304
_max_io_size                                       1048576
_io_statistics                                     TRUE
_disk_sector_size_override                         FALSE
_simulate_disk_sectorsize                          0
_simulate_logical_sectorsize                       0
_enable_asyncvio                                   FALSE
_iocalibrate_max_ios                               0
_iocalibrate_init_ios                              2
_io_outlier_threshold                              500
_io_internal_test                                  0
_lgwr_io_outlier                                   0
_asm_access                                        auto
_io_osd_param                                      1
_fob_dgaalloc                                      TRUE
_fob_ospshare                                      TRUE
_ksfd_fob_pct                                      0
_disksize_binary_search                            FALSE
_instant_file_create                               FALSE
_db_file_direct_io_count                           1048576
_cell_fast_file_create                             TRUE
_cell_fast_file_restore                            TRUE
_file_size_increase_increment                      67108864
_ofs_write_buffer_size                             1048576
_reset_maxcap_history                              10
_use_dynamic_shares                                1
_dynamic_share_range_factor                        2
_cpu_util_adj_force                                0
_cpu_util_adj_target                               0
_low_server_threshold                              0
_high_threshold_delta                              65535
_min_lwt_lt                                        24
_max_lwt_cpu_ratio                                 2
_active_session_idle_limit                         5
_active_session_legacy_behavior                    FALSE
_resource_manager_plan
_vkrm_schedule_interval                            10
_dbrm_dynamic_threshold                            989922280
_resource_manager_always_off                       FALSE
_io_resource_manager_always_on                     FALSE
_max_small_io                                      0
_max_large_io                                      0
_auto_assign_cg_for_sessions                       FALSE
_rm_numa_simulation_pgs                            0
_rm_numa_simulation_cpus                           0
_rm_numa_sched_enable                              TRUE
_dbrm_runchk                                       32769000
_dbrm_short_wait_us                                300
_dbrm_num_runnable_list                            0
_db_check_cell_hints                               FALSE
_pqq_enabled                                       TRUE
_rm_force_caging                                   FALSE
_min_sys_percentage
_min_autotask_percentage
_db_cache_max_sz                                   0
_shared_pool_max_sz                                0
_rm_exadata_pdb_cpu_cnt                            FALSE
_rm_exadata_partition_fc                           FALSE
_rm_exadata_pdb_cpu_cnt_mult                       2
_pqq_debug_txn_act                                 FALSE
_ksr_unit_test_processes                           0
_ksv_spawn_control_all                             FALSE
_ksv_max_spawn_fail_limit                          5
_ksv_pool_wait_timeout                             600
_ksv_pool_hang_kill_to                             0
_ksvppktmode                                       0
_ksv_static_flags1                                 0
_ksv_dynamic_flags1                                0
_ksv_slave_exit_timeout                            120
_first_spare_parameter
_second_spare_parameter
_third_spare_parameter
_fourth_spare_parameter
_fifth_spare_parameter
_sixth_spare_parameter
_seventh_spare_parameter
_eighth_spare_parameter
_ninth_spare_parameter
_tenth_spare_parameter
_eleventh_spare_parameter
_twelfth_spare_parameter
_thirteenth_spare_parameter
_fourteenth_spare_parameter
_fifteenth_spare_parameter
_sixteenth_spare_parameter
_seventeenth_spare_parameter
_eighteenth_spare_parameter
_nineteenth_spare_parameter
_twentieth_spare_parameter
_twenty-first_spare_parameter
_twenty-second_spare_parameter
_twenty-third_spare_parameter
_twenty-fourth_spare_parameter
_twenty-fifth_spare_parameter
_twenty-sixth_spare_parameter
_twenty-seventh_spare_parameter
_twenty-eighth_spare_parameter
_twenty-ninth_spare_parameter
_thirtieth_spare_parameter
_thirty-first_spare_parameter
_thirty-second_spare_parameter
_thirty-third_spare_parameter
_thirty-fourth_spare_parameter
_thirty-fifth_spare_parameter
_thirty-sixth_spare_parameter
_thirty-seventh_spare_parameter
_thirty-eighth_spare_parameter
_thirty-ninth_spare_parameter
_fortieth_spare_parameter
_forty-first_spare_parameter
_forty-second_spare_parameter
_forty-third_spare_parameter
_forty-fourth_spare_parameter
_forty-fifth_spare_parameter
_forty-sixth_spare_parameter
_forty-seventh_spare_parameter
_forty-eighth_spare_parameter
_forty-ninth_spare_parameter
_fiftieth_spare_parameter
_fifty-first_spare_parameter
_fifty-second_spare_parameter
_fifty-third_spare_parameter
_fifty-fourth_spare_parameter
_fifty-fifth_spare_parameter
_fifty-sixth_spare_parameter
_fifty-seventh_spare_parameter
_fifty-eighth_spare_parameter
_fifty-ninth_spare_parameter
_sixtieth_spare_parameter
_sixty-first_spare_parameter
_sixty-second_spare_parameter
_sixty-third_spare_parameter
_sixty-fourth_spare_parameter
_sixty-fifth_spare_parameter
_sixty-sixth_spare_parameter
_sixty-seventh_spare_parameter
_sixty-eighth_spare_parameter
_sixty-ninth_spare_parameter
_seventieth_spare_parameter
_seventy-first_spare_parameter
_seventy-second_spare_parameter
_seventy-third_spare_parameter
_seventy-fourth_spare_parameter
_seventy-fifth_spare_parameter
_seventy-sixth_spare_parameter
_seventy-seventh_spare_parameter
_seventy-eighth_spare_parameter
_seventy-ninth_spare_parameter
_eightieth_spare_parameter
_eighty-first_spare_parameter
_eighty-second_spare_parameter
_eighty-third_spare_parameter
_eighty-fourth_spare_parameter
_eighty-fifth_spare_parameter
_eighty-sixth_spare_parameter
_eighty-seventh_spare_parameter
_eighty-eighth_spare_parameter
_eighty-ninth_spare_parameter
_ninetieth_spare_parameter
_ninety-first_spare_parameter
_ninety-second_spare_parameter
_ninety-third_spare_parameter
_ninety-fourth_spare_parameter
_ninety-fifth_spare_parameter
_ninety-sixth_spare_parameter
_ninety-seventh_spare_parameter
_ninety-eighth_spare_parameter
_ninety-ninth_spare_parameter
_one-hundredth_spare_parameter
_one-hundred-and-first_spare_parameter
_one-hundred-and-second_spare_parameter
_one-hundred-and-third_spare_parameter
_one-hundred-and-fourth_spare_parameter
_one-hundred-and-fifth_spare_parameter
_one-hundred-and-sixth_spare_parameter
_one-hundred-and-seventh_spare_parameter
_one-hundred-and-eighth_spare_parameter
_one-hundred-and-ninth_spare_parameter
_one-hundred-and-tenth_spare_parameter
_one-hundred-and-eleventh_spare_parameter
_one-hundred-and-twelfth_spare_parameter
_one-hundred-and-thirteenth_spare_parameter
_one-hundred-and-fourteenth_spare_parameter
_one-hundred-and-fifteenth_spare_parameter
_one-hundred-and-sixteenth_spare_parameter
_one-hundred-and-seventeenth_spare_parameter
_one-hundred-and-eighteenth_spare_parameter
_one-hundred-and-nineteenth_spare_parameter
_one-hundred-and-twentieth_spare_parameter
_one-hundred-and-twenty-first_spare_parameter
_one-hundred-and-twenty-second_spare_parameter
_one-hundred-and-twenty-third_spare_parameter
_one-hundred-and-twenty-fourth_spare_parameter
_one-hundred-and-twenty-fifth_spare_parameter
_one-hundred-and-twenty-sixth_spare_parameter
_one-hundred-and-twenty-seventh_spare_parameter
_one-hundred-and-twenty-eighth_spare_parameter
_one-hundred-and-twenty-ninth_spare_parameter
_one-hundred-and-thirtieth_spare_parameter
_one-hundred-and-thirty-first_spare_parameter
_one-hundred-and-thirty-second_spare_parameter
_one-hundred-and-thirty-third_spare_parameter
_one-hundred-and-thirty-fourth_spare_parameter
_one-hundred-and-thirty-fifth_spare_parameter
_one-hundred-and-thirty-sixth_spare_parameter
_one-hundred-and-thirty-seventh_spare_parameter
_one-hundred-and-thirty-eighth_spare_parameter
_one-hundred-and-thirty-ninth_spare_parameter
_one-hundred-and-fortieth_spare_parameter
_one-hundred-and-forty-first_spare_parameter       FALSE
_one-hundred-and-forty-second_spare_parameter      FALSE
_one-hundred-and-forty-third_spare_parameter       FALSE
_one-hundred-and-forty-fourth_spare_parameter      FALSE
_one-hundred-and-forty-fifth_spare_parameter       FALSE
_one-hundred-and-forty-sixth_spare_parameter       FALSE
_one-hundred-and-forty-seventh_spare_parameter     FALSE
_one-hundred-and-forty-eighth_spare_parameter      FALSE
_one-hundred-and-forty-ninth_spare_parameter       FALSE
_one-hundred-and-fiftieth_spare_parameter          FALSE
_one-hundred-and-fifty-first_spare_parameter       FALSE
_one-hundred-and-fifty-second_spare_parameter      FALSE
_one-hundred-and-fifty-third_spare_parameter       FALSE
_one-hundred-and-fifty-fourth_spare_parameter      FALSE
_one-hundred-and-fifty-fifth_spare_parameter       FALSE
_one-hundred-and-fifty-sixth_spare_parameter       FALSE
_one-hundred-and-fifty-seventh_spare_parameter     FALSE
_one-hundred-and-fifty-eighth_spare_parameter      FALSE
_one-hundred-and-fifty-ninth_spare_parameter       FALSE
_one-hundred-and-sixtieth_spare_parameter          FALSE
_one-hundred-and-sixty-first_spare_parameter       FALSE
_one-hundred-and-sixty-second_spare_parameter      FALSE
_one-hundred-and-sixty-third_spare_parameter       FALSE
_one-hundred-and-sixty-fourth_spare_parameter      FALSE
_one-hundred-and-sixty-fifth_spare_parameter       FALSE
_one-hundred-and-sixty-sixth_spare_parameter       FALSE
_one-hundred-and-sixty-seventh_spare_parameter     FALSE
_one-hundred-and-sixty-eighth_spare_parameter      FALSE
_one-hundred-and-sixty-ninth_spare_parameter       FALSE
_one-hundred-and-seventieth_spare_parameter        FALSE
_one-hundred-and-seventy-first_spare_parameter     FALSE
_one-hundred-and-seventy-second_spare_parameter    FALSE
_one-hundred-and-seventy-third_spare_parameter     FALSE
_one-hundred-and-seventy-fourth_spare_parameter    FALSE
_one-hundred-and-seventy-fifth_spare_parameter     FALSE
_one-hundred-and-seventy-sixth_spare_parameter     FALSE
_one-hundred-and-seventy-seventh_spare_parameter   FALSE
_one-hundred-and-seventy-eighth_spare_parameter    FALSE
_one-hundred-and-seventy-ninth_spare_parameter     FALSE
_one-hundred-and-eightieth_spare_parameter         FALSE
_one-hundred-and-eighty-first_spare_parameter      FALSE
_one-hundred-and-eighty-second_spare_parameter     FALSE
_one-hundred-and-eighty-third_spare_parameter      FALSE
_one-hundred-and-eighty-fourth_spare_parameter     FALSE
_one-hundred-and-eighty-fifth_spare_parameter      FALSE
_one-hundred-and-eighty-sixth_spare_parameter      FALSE
_one-hundred-and-eighty-seventh_spare_parameter    FALSE
_one-hundred-and-eighty-eighth_spare_parameter     FALSE
_one-hundred-and-eighty-ninth_spare_parameter      FALSE
_one-hundred-and-ninetieth_spare_parameter         FALSE
_one-hundred-and-ninety-first_spare_parameter      FALSE
_one-hundred-and-ninety-second_spare_parameter     FALSE
_one-hundred-and-ninety-third_spare_parameter      FALSE
_one-hundred-and-ninety-fourth_spare_parameter     FALSE
_one-hundred-and-ninety-fifth_spare_parameter      FALSE
_one-hundred-and-ninety-sixth_spare_parameter      FALSE
_one-hundred-and-ninety-seventh_spare_parameter    FALSE
_one-hundred-and-ninety-eighth_spare_parameter     FALSE
_one-hundred-and-ninety-ninth_spare_parameter      FALSE
_two-hundredth_spare_parameter                     FALSE
_two-hundred-and-first_spare_parameter             FALSE
_two-hundred-and-second_spare_parameter            FALSE
_two-hundred-and-third_spare_parameter             FALSE
_two-hundred-and-fourth_spare_parameter            FALSE
_two-hundred-and-fifth_spare_parameter             FALSE
_two-hundred-and-sixth_spare_parameter             FALSE
_two-hundred-and-seventh_spare_parameter           FALSE
_two-hundred-and-eighth_spare_parameter            FALSE
_two-hundred-and-ninth_spare_parameter             FALSE
_two-hundred-and-tenth_spare_parameter             FALSE
_two-hundred-and-eleventh_spare_parameter          FALSE
_two-hundred-and-twelfth_spare_parameter           FALSE
_two-hundred-and-thirteenth_spare_parameter        FALSE
_two-hundred-and-fourteenth_spare_parameter        FALSE
_two-hundred-and-fifteenth_spare_parameter         FALSE
_two-hundred-and-sixteenth_spare_parameter         FALSE
_two-hundred-and-seventeenth_spare_parameter       FALSE
_two-hundred-and-eighteenth_spare_parameter        FALSE
_two-hundred-and-nineteenth_spare_parameter        FALSE
_two-hundred-and-twentieth_spare_parameter         FALSE
_ksipc_mode                                        0
_inet_cluster_interconnects
_ksipc_loopback_ips
_ksipc_service_level
_ksipc_window_size
_ksipc_common_sl
_ksipc_service_mask                                1
_ksipc_heap_extent                                 1048576
_ksipc_mga_segment_size                            268435456
_ksipc_group_sz                                    1024
_ksipcsnsrv
_ksipc_trace_bucket                                PRIVATE
_ksipc_trace_bucket_size                           IPC0:1048576-REST:8192
_ksipc_libipc_path
_ksipc_wait_flags                                  0
_ksipc_spare_param1                                0
_ksipc_spare_param2
_ksipc_spare_param3                                0
_ksipc_spare_param4                                0
_ksipc_spare_param5                                0
_ksipc_skgxp_library_path
_ksipc_skgxp_compat_library_path
_ksipc_ipclw_library_path
_ksxp_send_timeout                                 300
_ksxp_ping_enable                                  TRUE
_ksxp_ping_polling_time                            0
_ksxp_disable_dynamic_loading                      FALSE
_ksxp_disable_rolling_migration                    FALSE
_skgxp_udp_use_tcb                                 TRUE
_ksxp_disable_ipc_stats                            FALSE
_ksxp_max_stats_bkts                               0
_ksxp_init_stats_bkts                              0
_ksxp_stats_mem_lmt                                0
_rm_cluster_interconnects
_ksxp_disable_clss                                 0
_ksxp_dump_timeout                                 20
_ksxp_diagmode                                     OFF
_skgxp_reaping                                     1000
_ksxp_reaping                                      50
_ksxp_wait_flags                                   0
_ksxp_control_flags                                0
_skgxp_udp_hiwat_warn                              1000
_skgxp_udp_ach_reaping_time                        120
_skgxp_udp_timed_wait_seconds                      5
_skgxp_udp_timed_wait_buffering                    1024
_skgxp_udp_keep_alive_ping_timer_secs              300
_disable_duplex_link                               TRUE
_diag_diagnostics                                  TRUE
_disable_interface_checking                        FALSE
_skgxp_udp_interface_detection_time_secs           60
_skgxp_udp_lmp_on                                  FALSE
_skgxp_udp_lmp_mtusize                             0
_skgxp_udp_enable_dynamic_credit_mgmt              0
_skgxp_udp_ack_delay                               0
_skgxp_gen_ant_ping_misscount                      3
_skgxp_gen_rpc_no_path_check_in_sec                5
_skgxp_gen_rpc_timeout_in_sec                      300
_skgxp_gen_ant_off_rpc_timeout_in_sec              30
_skgxp_min_zcpy_len                                0
_skgxp_min_rpc_rcv_zcpy_len                        0
_skgxp_zcpy_flags                                  0
_skgxp_ctx_flags1                                  0
_skgxp_ctx_flags1mask                              0
_skgxp_dynamic_protocol                            0
_skgxp_inets
_ksxp_skgxp_inets
_skgxp_ant_options
_ksxp_skgxp_ant_options
_skgxp_rgn_ports                                   0
_ksxp_skgxp_ctx_flags1                             0
_ksxp_skgxp_ctx_flags1mask                         0
_ksxp_skgxp_dynamic_protocol                       4096
_ksxp_skgxp_rgn_ports                              0
_ksxp_dynamic_skgxp_param
_ksxp_if_config                                    0
_ksxp_exa_ip_config                                0
_ksxp_ksmsq_ip_config                              0
_ksxp_compat_flags                                 0
_ksxp_validate_cnh_life_cycle                      0
_skgxp_spare_param1
_ksxp_skgxp_spare_param1
_skgxp_spare_param2
_ksxp_skgxp_spare_param2
_skgxp_spare_param3
_ksxp_skgxp_spare_param3
_skgxp_spare_param4
_ksxp_xrc_cm_port
_skgxp_spare_param5
_skgxpg_last_parameter                             27
_ksxp_skgxpg_last_parameter                        27
_ksxp_testing                                      0
_ksxp_reporting_process                            LMD0
_ksxp_unit_test_byte_transformation                FALSE
_ksxp_ipclw_enabled                                1
_ipc_config_opts_stat                              0
_ipc_config_opts_dyn                               0
_ksipc_efchecks                                    3
_ksxp_save_sockets                                 0
_ksxp_save_sockets_pattern                         *
_ksxp_ipclw_ptswinsz                               16
_ksmd_protect_mode                                 off
_ksmd_trace                                        0
_ksmg_granule_size                                 4194304
_filemap_dir
_object_statistics                                 TRUE
_object_stats_max_entries                          3072
_datapump_conv_pga_lim                             0
_datapump_metadata_buffer_size                     131072
_datapump_tabledata_buffer_size                    262144
_datapump_compressbas_buffer_size                  0
_datapump_gather_stats_on_load                     FALSE
__dg_broker_service_names
_enable_rlb                                        TRUE
_enable_midtier_affinity                           TRUE
_midtier_affinity_cluswait_prc_threshold           6
_midtier_affinity_goodness_threshold               2000
_service_cleanup_timeout                           30
_disable_modsvc_refresh
_max_services                                      8200
_kswsas_ht_log_size                                7
_ksws_java_patching                                0
_kswsas_db_svc_mapping
_kswsas_num_jp_slaves
_kswsas_drain_kill_batch_size                      5
_disable_health_check                              FALSE
_mpmt_enabled                                      FALSE
_mpmt_prefer_enabled                               TRUE
_mpmt_single_process_instance                      FALSE
_mpmt_procs_per_osp                                100
_mpmt_use_fthread                                  FALSE
_mpmt_fthread_mode                                 0
_prespawn_enabled                                  FALSE
_prespawn_min_count                                10
_prespawn_batch_count                              20
_prespawn_fg_init_count                            0
_prespawn_bg_init_count                            0
_proc_grp_enabled                                  3
_proc_grp_numa_map
_spawn_diag_thresh_secs                            30
_spawn_diag_opts                                   0
_sched_delay_sample_interval_ms                    1000
_sched_delay_max_samples                           4
_sched_delay_sample_collection_thresh_ms           200
_sched_delay_measurement_sleep_us                  1000
_sched_delay_os_tick_granularity_us                16000
_min_time_between_psp0_diag_secs                   300
_NUMA_float_spawner                                FALSE
_accept_versions
_disable_rolling_patch                             0
_hang_analysis_num_call_stacks                     3
_local_hang_analysis_interval_secs                 3
_deadlock_resolution_level                         1
_deadlock_resolution_incidents_enabled             TRUE
_deadlock_resolution_incidents_always              FALSE
_deadlock_resolution_min_wait_timeout_secs         60
_deadlock_resolution_signal_process_thresh_secs    60
_heur_deadlock_resolution_secs                     0
_deadlock_diagnostic_level                         2
_deadlock_record_to_alert_log                      TRUE
_blocking_sess_graph_cache_size
_diag_proc_enabled                                 TRUE
_diag_proc_stack_capture_type                      1
_diag_proc_max_time_ms                             30000
_hang_msg_checksum_enabled                         TRUE
_kspol_tac_timeout                                 5
_disable_12751                                     FALSE
_diskmon_pipe_name
_dskm_health_check_cnt                             20
_dskm_single_instance                              FALSE
_exadata_feature_on                                FALSE
_ksmb_debug                                        0
_pmon_enable_dead_blkrs                            TRUE
_pmon_dead_blkrs_scan_rate_secs                    3
_pmon_dead_blkrs_alive_chk_rate_secs               3
_pmon_dead_blkrs_max_cleanup_attempts              5
_pmon_dead_blkrs_max_blkrs                         50
_pmon_cleanup_max_dfs                              100
_pmon_max_consec_posts                             5
_dead_process_scan_interval                        60
_main_dead_process_scan_interval                   0
_pmon_idle_scan_interval                           60
_cleanup_timeout                                   150
_cleanup_timeout_flags                             0
_pmon_min_slaves                                   0
_pmon_max_slaves                                   10
_pmon_slaves_arr_size                              150
_pmon_trees_per_slave                              10
_pmon_new_slave_secs                               30
_pmon_slave_stop_secs                              60
_pmon_incr_slaves                                  3
_pmon_exitnfy_enabled                              FALSE
_use_platform_compression_lib                      TRUE
_use_platform_encryption_lib                       TRUE
_use_hybrid_encryption_mode                        FALSE
_xengem_diagmode                                   OFF
_xengem_devname                                    DEFAULT
_xengem_enabled                                    TRUE
_ksmsq_hintmaxinst                                 1024
_ksmsq_hintmaxproc                                 300
_lthread_cleanup_intv_secs                         1
_lthread_enabled                                   TRUE
_lthread_debug                                     FALSE
_lthread_max_spawn_time_csecs                      12000
_lthread_spawn_check_intv_ms                       10
_lthread_clnup_pmon_softkill_wait_secs             1
_lthread_clnup_spawner_sk_wait_secs                30
_lthread_clnup_hk_wait_secs                        3
_lthread_step_debugging                            FALSE
_lthread_idle_timeout_secs                         600
_maxrpop_files_inst                                4
_maxrpop_instances                                 8
_use_large_pages_for_mga                           FALSE
_use_fallocate_for_mga                             FALSE
_mga_large_page_path
_ksrma_enabled                                     off
_ksrma_gsm_base_addr
_gsm_pool_size                                     134217728
_force_cloud_on                                    FALSE
_cloud_service_sim                                 0
_quarantine_enabled                                TRUE
_quarantine_max_objs                               10000
_quarantine_max_mem                                1
_quarantine_per_hour                               60
_test_xmem_alloc_size                              0
_use_platform_hash_function                        TRUE
_capability_simulate
_diag_daemon                                       TRUE
_dump_system_state_scope                           local
_dump_trace_scope                                  global
_dump_interval_limit                               120
_dump_max_limit                                    5
_diag_dump_timeout                                 30
_diag_dump_request_debug_level                     1
_diag_crashdump_level                              10
_full_diag_on_rim                                  FALSE
_diag_large_packets                                TRUE
_hang_detection_enabled                            TRUE
_hang_deadlock_resolution_enabled                  TRUE
_hang_cross_boundary_hang_detection_enabled        TRUE
_hang_cross_cluster_hang_detection_enabled         TRUE
_hang_asm_hang_resolution_enabled                  FALSE
_hang_trace_interval                               32
_hang_root_ha_phase_trigger_time                   300
_hang_hung_session_ewarn_percent                   34
_hang_max_session_hang_time                        96
_hang_hang_blocked_session_delta_percent_threshold 20
_hang_blocked_session_percent_threshold            10
_hang_resolution_scope                             OFF
_hang_allow_resolution_on_single_nonrac            FALSE
_hang_resolution_policy                            HIGH
_hang_resolution_confidence_promotion              FALSE
_hang_resolution_global_hang_confidence_promotion  FALSE
_hang_resolution_allow_archiving_issue_termination TRUE
_hang_resolution_promote_process_termination       TRUE
_hang_promote_process_termination_interval         70
_hang_resolution_percent_hung_sessions_threshold   300
_hang_signature_list_match_output_frequency        10
_hang_hang_analyze_output_hang_chains              TRUE
_hang_short_stacks_output_enabled                  TRUE
_hm_analysis_oradebug_sys_dump_level               0
_global_hang_analysis_interval_secs                10
_hang_verification_interval                        46
_hang_intersecting_chains_scanning_enabled         TRUE
_hang_log_verified_hangs_to_alert                  FALSE
_hang_log_io_hung_sessions_to_alert                FALSE
_hang_log_important_hangs_to_alert                 TRUE
_hang_appl_issue_session_threshold                 0
_hang_ignored_hangs_interval                       300
_hang_ignored_hang_count                           1
_hang_ignore_hngmtrc_interval                      150
_hang_monitor_archiving_related_hang_interval      300
_hang_hiload_promoted_ignored_hang_count           2
_hang_delay_resolution_for_libcache                TRUE
_hang_terminate_session_replay_enabled             TRUE
_hang_long_wait_time_threshold                     0
_hang_lws_file_count                               5
_hang_lws_file_space_limit                         100000000
_hang_base_file_count                              5
_hang_base_file_space_limit                        100000000
_hang_running_in_lrg                               FALSE
_hang_bool_spare1                                  TRUE
_hang_bool_spare2                                  TRUE
_hang_int_spare1                                   0
_hang_int_spare2                                   0
_hang_hiprior_session_attribute_list
_hang_enable_processstate                          TRUE
_diag_xm_enabled                                   FALSE
_hm_xm_enabled                                     TRUE
_trace_navigation_scope                            global
_max_protocol_support                              10000
_lm_check_ges_resource                             0
_lm_hash_control                                   1
_lm_lms                                            0
_lm_dynamic_lms                                    FALSE
_lm_max_lms                                        0
_ges_server_processes                              1
_lm_activate_lms_threshold                         100
_lm_lmd_waittime                                   8
_lm_lms_waittime                                   3
_lm_procs                                          1088
_lm_lms_priority_dynamic                           TRUE
_lm_lms_rt_threshold
_lm_lms_opt_priority                               TRUE
_lm_lms_priority_check_frequency                   60000
_lm_db_rank                                        6
_lm_db_ranks
_lm_ress                                           6000
_lm_locks                                          12000
_lm_master_weight                                  1
_active_instance_count
_active_standby_fast_reconfiguration               TRUE
_lm_enq_rcfg                                       TRUE
_lm_asm_enq_hashing                                TRUE
_lm_xids                                           1196
_lm_res_part                                       128
_lm_drm_window                                     0
_lm_drm_max_requests                               100
_lm_drm_xlatch                                     0
_lm_drm_disable                                    0
_lm_drm_disable_kjfc                               FALSE
_lm_contiguous_res_count                           128
_lm_num_pt_buckets                                 8192
_lm_num_pt_latches                                 128
_lm_node_join_opt                                  FALSE
_lm_no_sync                                        TRUE
_lm_non_fault_tolerant                             FALSE
_lm_cache_res_cleanup                              25
_lm_cache_allocated_res_ratio                      50
_lm_cache_res_skip_cleanup                         20
_lm_cache_res_cleanup_tries                        10
_lm_cache_res_type                                 TMHWHVDI
_lm_cache_enqueue
_lm_reloc_use_mhint                                FALSE
_lm_cache_lvl0_cleanup                             0
_lm_cache_res_options                              0
_ogms_home
_lm_sync_timeout                                   163
_lm_ticket_active_sendback                         50
_lm_rcfg_timeout                                   489
_lm_rcfg_kjcdump_time                              60
_lm_enq_lock_freelist
_lm_enqueue_freelist                               3
_lm_dd_interval                                    10
_lm_dd_scan_interval                               5
_lm_dd_search_cnt                                  3
_lm_dd_max_search_time                             180
_lm_dd_maxdump                                     50
_lm_dd_ignore_nodd                                 FALSE
_lm_enqueue_blocker_dump_timeout                   120
_lm_enqueue_blocker_dump_timeout_cnt               30
_lm_enqueue_blocker_kill_timeout                   0
_dlmtrace
_lm_tx_delta                                       16
_lm_proc_freeze_timeout                            70
_lm_deferred_msg_timeout                           163
_lm_use_new_defmsgtmo_action                       TRUE
_lm_dump_null_lock                                 FALSE
_lm_file_affinity
_lm_file_read_mostly
_lm_enable_aff_benefit_stats                       FALSE
_lm_num_bnft_stats_buckets                         1
_lm_drm_banned_objs
_lm_drm_max_banned_objs                            235
_lm_share_lock_opt                                 TRUE
_lm_share_lock_pdbisolation                        TRUE
_lm_enq_iso_enabled                                TRUE
_lm_res_hash_bucket                                0
_lm_res_tm_hash_bucket                             0
_ges_diagnostics                                   TRUE
_fair_remote_cvt                                   FALSE
_lm_rcvr_hang_check_frequency                      20
_lm_rcvr_hang_allow_time                           70
_lm_rcvr_hang_kill                                 TRUE
_lm_rcvr_hang_check_system_load                    TRUE
_lm_rcvr_hang_systemstate_dump_level               0
_lm_rcvr_hang_cfio_kill                            FALSE
_lm_lmon_nowait_latch                              TRUE
_ges_dd_debug                                      1
_lm_global_posts                                   TRUE
_rcfg_parallel_replay                              TRUE
_parallel_replay_msg_limit                         4000
_rcfg_parallel_fixwrite                            TRUE
_parallel_fixwrite_bucket                          1000
_rcfg_parallel_verify                              FALSE
_rcfg_disable_verify                               TRUE
_drm_parallel_freeze                               TRUE
_dump_rcvr_ipc                                     TRUE

P_NAME                                             P_VALUE
-------------------------------------------------- --------------------------------------------------
_ges_health_check                                  0
_kill_enqueue_blocker                              2
_lm_psrcfg                                         TRUE
_lm_single_inst_affinity_lock                      FALSE
_lm_preregister_css_restype                        CF
_inquiry_retry_interval                            3
_lm_drm_object_scan                                TRUE
_lm_spare_threads                                  0
_lm_spare_undo                                     0
_lm_rcvinst                                        TRUE
_lm_use_gcr                                        TRUE
_lm_use_tx_tsn                                     TRUE
_lm_local_hp_enq                                   TRUE
_lm_broadcast_res                                  enable_broadcast
_gcs_testing                                       0
_gcs_pkey_history                                  4000
_lm_drm_we_size                                    2000
_lm_drm_we_interval                                60
_read_mostly_instance                              FALSE
_read_mostly_instance_qa_control                   0
_read_mostly_slave_timeout                         20
_read_only_slave_timeout                           30
_read_mostly_enable_logon                          FALSE
_ges_designated_master                             TRUE
_lm_big_cluster_optimizations                      TRUE
_ges_lmd_mapping                                   *
_ges_hash_groups                                   *
_ges_default_lmds                                  *
_ges_nres_divide                                   0
_lm_lms_spin                                       FALSE
_lm_freeze_kill_time                               30
_lm_no_lh_check                                    FALSE
_lm_drmopt12                                       56
_lm_drm_filters                                    3
_lm_drm_duration_limit
_lm_drm_duration_limit_type                        2
_lm_fdrm_stats                                     FALSE
_lm_drm_filter_history_window
_lm_drm_filter_history_window_type                 2
_lm_lhupd_interval                                 5
_lm_high_load_threshold                            5
_lm_high_load_sysload_percentage                   90
_lm_low_load_percentage                            75
_lm_drm_hiload_percentage                          200
_lm_drm_lowload_percentage                         200
_lm_drm_min_interval                               600
_lm_drm_batch_time                                 10
_lm_inherited_max_requests                         100
_lm_adrm_options                                   3
_lm_adrm_scan_timeout                              50
_lm_adrm_interval                                  300
_lm_adrm_time_out                                  120
_lm_enqueue_timeout                                360
_lm_resend_open_convert_timeout                    30
_lm_process_lock_q_scan_limit                      100
_ges_direct_free                                   FALSE
_ges_resource_memory_opt                           4
_ges_gather_res_reuse_stats                        FALSE
_ges_direct_free_res_type
_lm_domain_hash_buckets                            64
_ges_fggl                                          TRUE
_ges_freeable_res_chunk_free                       FALSE
_ges_freeable_res_chunk_free_interval              180
_lm_share_lock_restype
_lm_nonisolated_restype                            TOTTUL
_ges_dump_open_locks                               FALSE
_lm_exadata_fence_type                             TRUE
_rond_test_mode                                    0
_lm_pdb_domains_enable                             7
_lm_singleton_pdb_opt                              TRUE
_lm_uid_default_lookup_value                       TRUE
_lm_enable_translation                             FALSE
_lm_chk_inv_domenq_ops                             TRUE
_lm_rm_object_bypass                               TRUE
_cluster_instances_number                          4
_reconfiguration_opt_level                         13
_lm_lazy_domain_timeout                            600
_enqueue_scan_interval                             0
_parallel_lmd_reconfig                             1
_rac_dbtype_reset                                  FALSE
_lm_recovery_set                                   1
_reader_farm_isolation_enable                      FALSE
_reader_farm_isolation_time_threshold              200
_lm_wait_for_hub_rcv_timeout                       300000
_lm_pdb_wait_all_gone                              FALSE
_adg_distributed_lockmaster                        FALSE
_ges_mseq_demo                                     0
_lm_better_ddvictim                                TRUE
_kjdd_call_stack_dump_enabled                      FALSE
_kjdd_wfg_dump_cntrl                               0
_dd_validate_remote_locks                          TRUE
_omni_enqueue_enable                               1
_dlm_stats_collect                                 1
_dlm_stats_collect_mode                            4
_dlm_stats_collect_slot_interval                   60
_dlm_stats_collect_du_limit                        3000
_lm_msg_batch_size                                 0
_lm_tickets                                        1000
_lm_ticket_min                                     50
_lm_msg_cleanup_interval                           3000
_lm_throttle_time_interval                         60
_lm_idle_connection_check                          TRUE
_lm_idle_connection_load_check                     TRUE
_lm_idle_connection_check_interval                 140
_lm_idle_connection_kill                           TRUE
_lm_idle_connection_kill_max_skips                 1
_lm_idle_connection_instance_check_callout         TRUE
_lm_idle_connection_quorum_threshold               50
_lm_idle_connection_action                         kill
_lm_send_mode                                      auto
_lm_postevent_buffer_size                          256
_lm_msg_pool_dump_threshold                        20000
_lm_msg_pool_user_callstack_dump                   FALSE
_lm_num_msg_pools_per_type                         1
_lm_ticket_check_inject                            FALSE
_lm_kill_fg_on_timeout                             TRUE
_lm_idle_connection_max_ignore_kill_count          2
_lm_send_queue_length                              5000
_lm_send_queue_batching                            TRUE
_lm_process_batching                               TRUE
_lm_sq_batch_factor                                2
_lm_sq_batch_type                                  auto
_lm_sq_batch_waittick                              3
_lm_sendproxy_reserve                              25
_lm_checksum_batch_msg                             0
_lm_batch_compression_threshold                    0
_lm_compression_scheme                             zlib
_lm_validate_pbatch                                FALSE
_lm_watchpoint_maximum                             3
_lm_watchpoint_timeout                             3600
_lm_free_queue_threshold                           0
_lm_mp_avail_queue_threshold                       0
_lm_mp_bulk_mbuf_free                              FALSE
_lm_wait_pending_send_queue                        TRUE
_lm_hashtable_bkt_low                              3
_lm_hashtable_bkt_high                             5
_lm_hashtable_bkt_thr                              70
_lm_comm_tkts_adaptive                             TRUE
_lm_comm_tkts_calc_period_length                   1000
_lm_comm_tkts_max_periods                          10
_lm_comm_tkts_min_increase_wait                    10
_lm_comm_tkts_min_decrease_wait                    120
_lm_comm_tkts_nullreq_threshold                    1
_lm_comm_tkts_mult_factor                          1
_lm_comm_tkts_sub_factor                           10
_lm_comm_tkts_max_add                              5
_lm_comm_msgq_fixed_buffers                        FALSE
_lm_comm_msgq_copy_buffers                         FALSE
_lm_comm_msgq_bufr_multiple                        8
_lm_comm_channel                                   msgq
_lm_comm_msgq_busywait                             0
_lm_comm_reap_count                                1
_lm_comm_slow_op_stat_dump_threshold               5
_lm_comm_slow_op_loop_threshold                    15
_lm_comm_rcv_msg_history_slots                     50
_lm_tkt_leak_check_count                           5
_lm_tkt_leak_check_seconds                         300
_lm_gl_hash_scheme                                 1
_abort_recovery_on_join                            FALSE
_send_ast_to_foreground                            AUTO
_reliable_block_sends                              TRUE
_blocks_per_cache_server                           16
_object_reuse_bast                                 2
_delta_push_share_blockers                         0
_cluster_flash_cache_slave_file
_cluster_flash_cache_slave_size                    0
__persistent_cl2_slave_size                        0
_gcs_trace_bucket                                  FALSE
_gcs_trace_bucket_size                             LMS:1048576-RMV:1048576-CRV:1048576-DBW:524288
_lms_rollbacks                                     1000
_gcs_enable_private_iterator                       TRUE
_rf_blocks_per_entity                              5
_gcs_cluster_flash_cache_persistency               FALSE
_gcs_track_reliable_block_sends                    FALSE
_send_close_with_block                             TRUE
_gcs_fast_reconfig                                 TRUE
_cr_grant_global_role                              TRUE
_cr_grant_local_role                               AUTO
_cr_grant_only                                     FALSE
_skip_assume_msg                                   TRUE
_gcs_resources
_gcs_latches                                       128
_gcs_process_in_recovery                           TRUE
_scatter_gcs_resources                             FALSE
_gcs_res_per_bucket                                4
_gcs_res_hash_buckets
_gcs_reserved_resources                            400
_gcs_cr_master_ping_remote                         TRUE
_cr_multiblock_grant_only                          FALSE
_gcs_shadow_locks
_scatter_gcs_shadows                               FALSE
_gcs_disable_remote_handles                        FALSE
_gcs_disable_skip_close_remastering                FALSE
_gcs_min_slaves                                    0
_gcs_min_cr_slaves                                 0
_gcs_dynamic_slaves                                TRUE
_gcs_reserved_shadows                              400
_gcs_crslave_longq_cnt                             2000
_gcs_crslave_longq_us                              1000
_gcs_crslave_check_time                            10
_gcs_recoverable_asserts                           1
_gcs_disable_imc_preallocation                     FALSE
_side_channel_batch_size                           240
_side_channel_batch_timeout                        6
_side_channel_batch_timeout_ms                     500
_broadcast_scn_wait_timeout                        10
_broadcast_scn_mode                                1
_hb_redo_msg_interval                              100
_master_direct_sends                               63
_read_mostly_lock_mitigation                       TRUE
_nameservice_consistency_check                     TRUE
_nameservice_request_batching                      TRUE
_cgs_send_timeout                                  300
_cgs_ticket_sendback                               50
_cgs_msg_batch_size                                4096
_cgs_msg_batching                                  TRUE
_cgs_comm_readiness_check                          1
_imr_active                                        TRUE
_imr_max_reconfig_delay                            75
_imr_splitbrain_res_wait                           0
_imr_disk_voting_interval                          3
_imr_systemload_check                              TRUE
_imr_device_type                                   controlfile
_imr_highload_threshold
_imr_evicted_member_kill                           TRUE
_imr_evicted_member_kill_wait                      20
_imr_avoid_double_voting                           TRUE
_imr_diskvote_implementation                       auto
_imr_extra_reconfig_wait                           10
_imr_controlfile_access_wait_time                  10
_imr_rr_holder_kill_time                           300
_imr_check_css_incarnation_number                  TRUE
_imr_rim_mount_device                              FALSE
_imr_largest_hub_membership                        TRUE
_imr_non_blocking_device_driver                    TRUE
_imr_dd_slave_wait_time                            30
_imr_mount_retry_wait_time                         20
_imr_remount_retry_wait_time                       60
_cluster_library                                   clss
_cgs_reconfig_timeout                              0
_cgs_node_kill_escalation                          TRUE
_cgs_node_kill_escalation_wait                     0
_cgs_zombie_member_kill_wait                       20
_cgs_reconfig_extra_wait                           3
_cgs_health_check_in_reconfig                      FALSE
_cgs_memberkill_from_rim_instance                  FALSE
_cgs_os_level_connection_check                     1
_cgs_os_level_connection_reqno                     0
_cgs_os_level_connection_pthreadno                 0
_cgs_combine_css_events                            7
_cgs_publish_netinfo_collect_event_haip            FALSE
_cgs_publish_netinfo_collect_event_chm             TRUE
_cgs_tickets                                       1000
_cgs_dball_group_registration                      local
_cgs_dbgroup_poll_time                             600
_cgs_allgroup_poll_time                            20000
_cgs_big_group_enabled                             FALSE
_cgs_support_rim_disc                              TRUE
_ipc_switch_reconfig_needed                        FALSE
_lm_dynamic_load                                   TRUE
_lm_rm_slaves                                      1
_lm_timed_statistics_level                         0
_shutdown_sync_enable                              TRUE
_lm_lms_no_yield                                   FALSE
_lm_max_lms_block_time                             5
_lm_sndq_flush_int                                 5
_lm_watchpoint_kjmddp
_rmv_dynamic_priority                              TRUE
_lm_use_us_timer                                   FALSE
_notify_crs                                        FALSE
_crs_2phase                                        bypass
_kill_diagnostics_timeout                          60
_ges_diagnostics_asm_dump_level                    11
_ges_num_blockers_to_kill                          1
_ges_vbfreelists                                   0
_lm_hb_callstack_collect_time                      5
_lm_hb_callstack_collect_time_long                 60
_lm_hb_disable_check_list                          none
_lm_hb_enable_acl_check                            TRUE
_lm_hb_acceptable_hang_condition                   default
_lm_hb_maximum_hang_report_count                   20
_lm_hb_exponential_hang_time_factor                2
_lm_hb_cfio_timeout                                70
_lm_hb_per_proc_timeout                            default
_lm_hb_per_proc_behavior                           default
_lm_hb_timeout_extension
_lm_hwc_disabled                                   FALSE
_lmhb_procstate_dump_runtime_limit                 60
_lmhb_procstate_dump_cputime_limit                 60
_hang_statistics_collection_interval               15
_hang_statistics_collection_ma_alpha               30
_hang_statistics_high_io_percentage_threshold      25
_gcr_enable_high_cpu_kill                          FALSE
_gcr_enable_high_memory_kill                       FALSE
_gcr_enable_high_cpu_rm                            TRUE
_gcr_enable_high_cpu_rt                            TRUE
_gcr_high_cpu_threshold                            10
_gcr_high_memory_threshold                         10
_gcr_min_free_memory_hard_limit                    1073741824
_gcr_cpu_min_hard_limit                            2560
_gcr_cpu_min_free                                  10
_gcr_mem_min_free                                  10
_gcr_use_css                                       TRUE
_gcr_css_use_2group_lock                           TRUE
_gcr_css_group_try_lock_delay
_gcr_css_group_query_boost
_gcr_css_group_update_interval
_gcr_css_group_update2_interval
_gcr_css_group_large
_gcr_tick
_gcr_enable_statistical_cpu_check                  TRUE
_gcr_enable_new_drm_check                          FALSE
_gcr_max_rt_procs
_gcr_enable_kill_inst_diags                        FALSE
_increase_lms_process                              0
_size_of_log_table                                 30
_size_of_status_table                              100
_gcr_dump_cpu_consumers                            FALSE
_gcr_cpu_consumer_dump_level                       0
_kjac_force_outcome_current_session                FALSE
_lm_rac_spare_p1                                   0
_lm_rac_spare_p2                                   0
_lm_rac_spare_p3                                   0
_lm_rac_spare_p4                                   0
_lm_rac_spare_p5                                   0
_lm_rac_spare_p6
_lm_rac_spare_p7
_lm_rac_spare_p8
_lm_rac_spare_p9
_lm_rac_spare_p10
_lm_rac_spare_dp1                                  0
_lm_rac_spare_dp2                                  0
_lm_rac_spare_dp3                                  0
_lm_rac_spare_dp4                                  0
_lm_rac_spare_dp5                                  0
_lm_rac_spare_dp6
_lm_rac_spare_dp7
_lm_rac_spare_dp8
_lm_rac_spare_dp9
_lm_rac_spare_dp10
_ka_enabled                                        FALSE
_ka_mode                                           0
_ka_locks_per_sector                               4
_ka_msg_reap_count                                 40
_ka_compatibility_requirement                      all
_ka_allow_reenable                                 FALSE
_ka_pbatch_messages                                TRUE
_ka_doorbell                                       0
_ka_scn_enabled                                    FALSE
_ka_scn_accel_shrmem                               TRUE
_ka_scn_use_ka_msgs                                TRUE
_ka_grant_policy                                   AUTO
_ka_msg_wait_count                                 20
_ka_max_wait_delay                                 1000
_kjlton                                            FALSE
_kjltmaxgt                                         1000
_kjltmaxht                                         1000
_ipddb_enable                                      FALSE
_gcs_cluster_flash_cache_mode                      0
_gcs_flash_cache_mode                              0
__sga_target                                       578813952
_disable_streams_pool_auto_tuning                  FALSE
_memory_management_tracing                         0
_memory_mgmt_immreq_timeout                        150
_memory_checkinuse_timeintv                        30
_memory_mgmt_fail_immreq                           FALSE
_memory_sanity_check                               0
_init_granule_interval                             10
_shared_pool_max_size                              0
_shared_pool_minsize_on                            FALSE
_streams_pool_max_size                             0
_simulate_mem_transfer                             FALSE
_memory_nocancel_defsgareq                         FALSE
_memory_imm_mode_without_autosga                   TRUE
_max_defer_gran_xfer_atonce                        10
_asm_allow_small_memory_target                     FALSE
_memory_broker_stat_interval                       30
_automemory_broker_interval                        3
_memory_broker_shrink_heaps                        15
_memory_broker_shrink_java_heaps                   900
_memory_broker_shrink_streams_pool                 900
_memory_broker_shrink_timeout                      60000000
_memory_broker_log_stat_entries                    5
_memory_initial_sga_split_perc                     60
_memory_broker_marginal_utility_sp                 7
_memory_broker_marginal_utility_bc                 12
_dump_scn_increment_stack
_big_scn_test_mode                                 2
_controlfile_enqueue_timeout                       900
_controlfile_enqueue_holding_time                  120
_controlfile_enqueue_holding_time_tracking_size    0
_controlfile_update_check                          OFF
_controlfile_enqueue_dump                          FALSE
_controlfile_block_size                            0
_controlfile_section_init_size
_controlfile_section_max_expand
_kill_controlfile_enqueue_blocker                  TRUE
_controlfile_backup_copy_check                     TRUE
_controlfile_cell_flash_caching                    3
_controlfile_split_brain_check                     TRUE
_controlfile_auto_convert_behaviour                AUTO_CONVERT
_controlfile_verify_on_mount                       FALSE
_db_block_buffers                                  35136
_db_block_cache_protect                            false
_db_block_cache_protect_internal                   0
_dbwr_tracing                                      0
_tsenc_tracing                                     0
_disable_multiple_block_sizes                      FALSE
_db_fast_obj_truncate                              TRUE
_db_fast_obj_ckpt                                  TRUE
_enable_obj_queues                                 TRUE
_db_obj_enable_ksr                                 TRUE
_small_table_threshold                             702
_very_large_object_threshold                       500
_dbwr_async_io                                     TRUE
_recovery_percentage                               50
_db_lost_write_checking                            2
_db_lost_write_tracing                             FALSE
__db_cache_size                                    318767104
_db_block_numa                                     1
__shared_io_pool_size                              0
_shared_io_pool_size                               0
_db_hot_block_tracking                             FALSE
_db_block_bad_write_check                          FALSE
_db_full_db_cache_diff_pct                         5
_db_cache_xmem_size                                0
__db_cache_xmem_size_metadata                      0
_db_keep_cache_xmem_size                           0
__db_keep_cache_xmem_size_metadata                 0
_db_recycle_cache_xmem_size                        0
__db_recycle_cache_xmem_size_metadata              0
_db_2k_cache_xmem_size                             0
__db_2k_cache_xmem_size_metadata                   0
_db_4k_cache_xmem_size                             0
__db_4k_cache_xmem_size_metadata                   0
_db_8k_cache_xmem_size                             0
__db_8k_cache_xmem_size_metadata                   0
_db_16k_cache_xmem_size                            0
__db_16k_cache_xmem_size_metadata                  0
_db_32k_cache_xmem_size                            0
__db_32k_cache_xmem_size_metadata                  0
_memoptimize_xmem_pool_size                        0
__memoptimize_xmem_pool_size_metadata              0
_disable_system_tablespaces_enc_enforcement        FALSE
_cell_object_expiration_hours                      24
_db_block_lru_latches                              18
_db_percpu_create_cachesize                        2
_db_initial_cachesize_create_mb                    256
_db_num_evict_waitevents                           64
_db_todefer_cache_create                           TRUE
_db_minimum_auxsize_percent                        10
_db_imoltp_hashidx_force_nonctg                    0
_db_block_known_clean_pct                          2
_db_block_max_scan_pct                             40
_db_large_dirty_queue                              25
_db_writer_max_writes                              0
_db_writer_chunk_writes                            0
_db_block_med_priority_batch_size                  0
_db_block_hi_priority_batch_size                   0
_db_writer_histogram_statistics                    FALSE
_dbwr_scan_interval                                300
_db_writer_flush_imu                               TRUE
_db_writer_coalesce_write_limit                    131072
_db_writer_coalesce_encrypted_buffers              TRUE
_db_writer_coalesce_area_size                      3014656
_db_writer_nomemcopy_coalesce                      FALSE
_selftune_checkpoint_write_pct                     3
_db_writer_verify_writes                           FALSE
_dbwr_stall_write_detection_interval               900
_dbwr_flashlock_shrink_limit                       0
_enable_dbwr_auto_tracing                          0
_dbwr_nowrite_assert_interval                      7200
_dbwr_nwp_assert_interval                          1800
_remessage_dbwrs                                   0
_db_block_prefetch_quota                           10
_db_block_prefetch_wasted_threshold_perc           2
_db_block_prefetch_limit                           0
_db_block_prefetch_override                        0
_rbr_ckpt_tracing                                  0
_db_cache_pre_warm                                 TRUE
_db_percent_hot_default                            50
_db_percent_hot_keep                               0
_db_percent_hot_recycle                            0
_db_aging_hot_criteria                             2
_db_xmem_hot_switch_criteria                       4
_db_aging_stay_count                               0
_db_aging_cool_count                               1
_db_aging_touch_time                               3
_db_aging_freeze_cr                                FALSE
_db_block_hash_buckets                             131072
_db_block_hash_latches                             4096
_db_blocks_per_hash_latch
_db_handles_cached                                 10
_db_handles                                        3000
_wait_for_sync                                     TRUE
_db_block_vlm_check                                FALSE
_db_block_vlm_leak_threshold                       4
_db_block_trace_protect                            FALSE
_db_flash_cache_slow_io_adjustment_interval        3600
_db_cache_miss_check_les                           FALSE
_db_block_max_cr_dba                               6
_write_clones                                      3
_db_cache_silicon_secured_memory                   TRUE
_check_block_after_checksum                        TRUE
_trace_pin_time                                    FALSE
_pin_time_statistics                               FALSE
_db_fast_obj_check                                 FALSE
_db_block_temp_redo                                FALSE
_db_block_adjchk_level                             0
_db_block_adjcheck                                 TRUE
_db_required_percent_fairshare_usage               10
_db_block_check_objtyp                             TRUE
_db_block_do_full_mbreads                          FALSE
_shared_iop_max_size                               536870912
_shared_io_pool_buf_size                           1048576
_siop_flashback_scandepth                          20
_shared_io_pool_debug_trc                          0
_shared_io_set_value                               FALSE
_trace_buffer_wait_timeouts                        0
_buffer_busy_wait_timeout                          100
_influx_scn_wait_timeout                           1
_influx_scn_waits                                  100000
_db_cache_crx_check                                FALSE
_db_cache_mman_latch_check                         FALSE
_db_block_cache_history                            0
_db_block_cache_history_level                      2
_db_block_cache_history_lru                        FALSE
_db_flash_cache_encryption                         FALSE
_db_xmem_cache_encryption                          TRUE
_db_block_header_guard_level                       0
_db_block_table_scan_buffer_size                   4194304
_db_cache_process_cr_pin_max                       2147483647
_db_block_corruption_recovery_threshold            5
_db_block_chunkify_ncmbr                           FALSE
_db_cache_wait_debug                               0
_db_full_caching                                   TRUE
_fastpin_enable                                    1
_bps_sanity_checks                                 FALSE
_db_prefetch_histogram_statistics                  FALSE
_flush_undo_after_tx_recovery                      TRUE
_enable_buffer_header_history                      TRUE
_db_block_iterations_for_rm                        2000
_db_block_scandepth_for_rm                         20
_spare_test_parameter                              0
_db_2k_flash_cache_file
_db_2k_flash_cache_size                            0
_db_4k_flash_cache_file
_db_4k_flash_cache_size                            0
_db_8k_flash_cache_file
_db_8k_flash_cache_size                            0
_db_16k_flash_cache_file
_db_16k_flash_cache_size                           0
_db_32k_flash_cache_file
_db_32k_flash_cache_size                           0
_db_flash_cache_keep_limit                         80
_db_flash_cache_write_limit                        1
_db_flash_cache_force_replenish_limit              8
_db_l2_tracing                                     0
_db_flash_cache_max_read_retry                     3
_db_flash_cache_max_latency                        400
_db_flash_cache_max_slow_io                        3
_db_flash_cache_max_outstanding_writes             32
_db_flash_cache_disable_write_batchsize            4096
_db_dump_from_disk_and_efc                         0
_numa_buffer_cache_stats                           0
_switch_current_scan_scn                           TRUE
_cleanout_shrcur_buffers                           TRUE
_avoid_scn_growth                                  1
_fast_cr_clone                                     TRUE
_clone_during_split                                TRUE
_db_cache_block_read_stack_trace                   0
_child_read_ahead_dba_check                        FALSE
_data_warehousing_scan_buffers                     TRUE
_data_warehousing_scan_flash_buffers               FALSE
_data_warehousing_serial_scan                      TRUE
_db_dw_scan_obj_cooling_policy                     CACHE_SIZE
_db_dw_scan_obj_cooling_interval                   100
_db_dw_scan_obj_cooling_factor                     500
_db_dw_scan_obj_warming_increment                  1000
_db_dw_scan_adaptive_cooling                       FALSE
_db_dw_scan_max_shadow_count                       5
_pq_numa_working_set_affinity                      TRUE
_db_bt_cache_only_readmostly_obj_on_roi            FALSE
_auto_bmr_noretry_window                           1800
_sga_clear_dump                                    FALSE
_clear_buffer_before_reuse                         FALSE
_disable_adp_adj_buf_check                         FALSE
_db_cache_block_read_stack_trace_where1            0
_db_cache_block_read_stack_trace_where2            0
_db_cache_block_read_stack_trace_where3            0
_override_datafile_encrypt_check                   FALSE
_disable_data_block_check_after_decrypt            FALSE
_assert_encrypted_tablespace_blocks                TRUE
_verify_encrypted_tablespace_keys                  TRUE
_inject_simulated_error_period                     131072
_db_cache_advice_sample_factor                     4
_db_cache_advice_max_size_factor                   2
_db_cache_advice_sanity_check                      FALSE
_db_cache_advice_hash_latch_multiple               16
_db_mttr_advice                                    ON
_db_mttr_sim_target
_db_mttr_sample_factor                             64
_db_mttr_partitions                                0
_db_mttr_sim_trace_size                            256
_db_mttr_trace_to_alert                            FALSE
_obj_ckpt_tracing                                  0
_immediate_commit_propagation                      TRUE
_external_scn_rejection_threshold_hours            24
_c3_external_scn_rejection_threshold_hours         4464
_max_reasonable_scn_rate                           32768
_external_scn_logging_threshold_seconds            86400
_external_scn_rejection_delta_threshold_minutes    0
_low_scn_headroom_warning_threshold_days           90
_high_intrinsic_scn_growth_alert                   1440
_redo_compatibility_check                          FALSE
_db_block_check_for_debug                          FALSE
_db_always_check_system_ts                         TRUE
_skip_unconverted_change_vector                    FALSE
_log_checkpoint_recovery_check                     0
_two_pass                                          TRUE
_recovery_verify_writes                            FALSE
_disable_recovery_read_skip                        FALSE
_buddy_instance                                    1
_buddy_instance_start_rba_timeout                  9
_buddy_instance_num_read_buffers                   4
_buddy_instance_scan_phase_threshold               3
_instance_recovery_bloom_filter_size               83886080
_instance_recovery_bloom_filter_fprate             0
_fatalprocess_redo_dump_time_limit                 0
_nonfatalprocess_redo_dump_time_limit              3600
_redo_transport_stream_writes                      TRUE
_redo_transport_stream_test                        TRUE
_redo_transport_vio_size_req                       1048576
_arch_sim_mode                                     0
_dg_corrupt_redo_log                               0
_local_arc_assert_on_wait                          FALSE
_defer_eor_orl_arch_for_so                         TRUE
_enable_ffw                                        TRUE
_log_archive_trace_pids
_redo_transport_sanity_check                       0
_fix_fuzzy_scn                                     FALSE
_dg_cf_check_timer                                 15
_bypass_srl_for_so_eor                             FALSE
_rta_sync_wait_timeout                             10
_log_archive_prot_auto_demote                      FALSE
_serialize_lgwr_sync_io                            FALSE
_mgd_rcv_handle_orphan_datafiles                   FALSE
_real_time_apply_sim                               0
_defer_rcv_during_sw_to_sby                        FALSE
_skip_trstamp_check                                TRUE
_show_mgd_recovery_state                           FALSE
_log_archive_buffers                               10
_redo_transport_compress_all                       TRUE
_rtc_infeasible_threshold                          25
_disable_thread_snapshot                           TRUE
_redo_log_record_life                              168
_redo_log_debug_config                             0
_redo_log_check_backup                             10
_log_archive_avoid_memcpy                          TRUE
_async_rta_broadcast                               FALSE
_rtabnd_update_offload                             TRUE
_redo_transport_async_mode                         0
_log_archive_strong_auth                           TRUE
_redo_transport_max_kbytes_sec                     0
_redo_transport_min_kbytes_sec                     10
_redo_transport_catch_up_bandwidth_percentage      50
_redo_transport_resource_management                FALSE
_lgwr_ns_nl_min                                    500
_lgwr_ns_nl_max                                    1000
_redo_transport_stall_time                         360
_redo_transport_stall_time_long                    3600
_max_lns_shutdown_archival_time                    30
_ta_lns_wait_for_arch_log                          20
_lgwr_max_ns_wt                                    1
_ns_max_flush_wt                                   1
_ns_max_send_delay                                 15
_lgwr_ta_sim_err                                   0
_lgwr_ns_sim_err                                   0
_log_archive_network_redo_size                     10
_switchover_to_standby_switch_log                  FALSE
_max_async_wait_for_catch_up                       20
_redo_transport_evict_threshold                    10
_num_rlslaves                                      4
_lgwr_delay_write                                  FALSE
_enable_redo_global_post                           FALSE
_cache_orl_during_open                             ALL
_image_redo_gen_delay                              0
_simulate_dax_storage                              DISABLE
_log_silicon_secured_memory                        TRUE
_target_log_write_size                             0
_target_log_write_size_timeout                     0
_target_log_write_size_spinwait                    0
_target_log_write_size_percent_for_poke            100
_log_switch_timeout                                0
_log_buffers_debug                                 FALSE
_log_buffers_corrupt                               FALSE
_log_simultaneous_copies                           4
_use_single_log_writer                             ADAPTIVE
_max_outstanding_log_writes                        2
_log_write_worker_task_prefetch                    EARLY
_log_write_worker_task_spinwait                    0
_log_write_worker_post_spinwait                    0
_adaptive_scalable_log_writer_enable_worker_thresh 200
old

_adaptive_scalable_log_writer_disable_worker_thres 50
hold

_adaptive_scalable_log_writer_sampling_count       128
_adaptive_scalable_log_writer_sampling_time        3
_adaptive_scalable_log_writer_enable_worker_aging  999900
_log_parallelism_max                               2
_log_parallelism_dynamic                           TRUE
_max_log_write_parallelism                         1
_serial_log_write_worker_io                        FALSE
_parallel_log_write_cleanup                        FALSE
_log_private_parallelism_mul                       10
_log_private_mul                                   5
_lightweight_hdrs                                  TRUE
_redo_read_from_memory                             TRUE
_desired_readmem_rate                              90
_log_read_buffer_size                              8
_validate_readmem_redo                             HEADER_ONLY
_mirror_redo_buffers                               FALSE
_fg_log_checksum                                   TRUE
_verify_fg_log_checksum                            FALSE
_disable_logging                                   FALSE
_simulated_log_write_usecs                         0
_log_buffer_coalesce                               FALSE
_redo_write_coalesce_all_threshold                 0
_redo_write_coalesce_slave_threshold               0
_redo_write_sync_single_io                         TRUE
_fg_sync_sleep_usecs                               0
_fg_fast_sync_sleep_usecs                          50
_fg_fast_sync_spin_usecs                           300
_log_file_sync_timeout                             10
_use_adaptive_log_file_sync                        TRUE
_adaptive_log_file_sync_use_polling_threshold      200
_adaptive_log_file_sync_use_postwait_threshold     50
_adaptive_log_file_sync_use_postwait_threshold_agi 1001
ng

_adaptive_log_file_sync_sampling_count             128
_adaptive_log_file_sync_sampling_time              3
_adaptive_log_file_sync_sched_delay_window         60
_adaptive_log_file_sync_poll_aggressiveness        0
_adaptive_log_file_sync_high_switch_freq_threshold 3
_max_pending_scn_bcasts                            9
_lgwr_posts_for_pending_bcasts                     FALSE
_long_log_write_warning_threshold                  500
_long_bcast_ack_warning_threshold                  500
_enable_flash_logging                              TRUE
_check_pdbid_in_redo                               FALSE
_log_write_info_size                               4096
_cache_fusion_pipelined_updates                    TRUE
_rac_propagate_last_rba                            TRUE
_log_writer_worker_dlm_hearbeat_update_freq        5000
_hb_redo_interval                                  1000
_db_file_exec_read_count                           74
_db_file_optimizer_read_count                      8
_db_file_noncontig_mblock_read_count               11
_db_recovery_temporal_file_dest
_standby_switchover_timeout                        120
_hard_protection                                   FALSE
_datafile_cow                                      FALSE
_max_filestat_tries                                10
_disable_file_resize_logging                       FALSE
_file_set_enqueue_timeout                          1200
_file_offline_sync_timeout                         900
_allow_error_simulation                            FALSE
_roi_first_open_wait                               120
_db_file_format_io_buffers                         4
_disable_read_only_open_dict_check                 FALSE
_standby_flush_mode                                SLFLUSH
_causal_standby_wait_timeout                       20
_online_file_conversion_batchsize                  16
_gc_undo_affinity                                  TRUE
_gc_disable_s_lock_brr_ping_check                  TRUE
_gc_policy_time                                    20
_gc_policy_rm_dirty_percent                        1
_gc_policy_minimum                                 1500
_gc_affinity_ratio                                 50
_gc_transfer_ratio                                 75
_gc_anti_lock_ratio                                66
_recovery_asserts                                  FALSE
_gc_vector_read                                    TRUE
_gc_element_percent                                105
_gc_latches                                        32
_gc_keep_recovery_buffers                          TRUE
_gc_keep_undo_recovery_buffers                     TRUE
_recovery_read_limit                               1024
_gc_dump_remote_lock                               TRUE
_gc_drm_windows                                    8
_gc_defer_time                                     0
_gc_defer_ping_index_only                          TRUE
_gc_delay_ping                                     TRUE
_kcl_debug                                         TRUE
_kcl_index_split                                   TRUE
_fairness_threshold                                2
_gc_interconnect_checksum                          FALSE
_cr_server_log_flush                               TRUE
_gc_log_flush                                      TRUE
_async_recovery_reads                              TRUE
_async_recovery_claims                             TRUE
_bwr_for_flushed_pi                                TRUE
_send_requests_to_pi                               TRUE
_kcl_conservative_log_flush                        FALSE
_gc_check_bscn                                     TRUE
_gc_global_lru                                     AUTO
_gc_global_lru_touch_count                         5
_gc_global_lru_touch_time                          60
_gc_statistics                                     TRUE
_gc_coalesce_recovery_reads                        TRUE
_gc_global_checkpoint_scn                          TRUE
_gc_undo_block_disk_reads                          TRUE
_gc_affinity_locking                               TRUE
_gc_affinity_locks                                 TRUE
_gc_read_mostly_locking                            TRUE
_gc_persistent_read_mostly                         TRUE
_gc_bypass_readers                                 TRUE
_gc_max_downcvt                                    1024
_gc_fusion_compression                             1024
_gc_down_convert_after_keep                        TRUE
_gc_delta_push_objects                             0
_gc_flush_during_affinity                          TRUE
_gc_cr_server_read_wait                            TRUE
_gc_read_mostly_flush_check                        FALSE
_gc_override_force_cr                              TRUE
_gc_bg_merge                                       TRUE
_gc_fg_merge                                       TRUE
_gc_first_dirty_merge                              TRUE
_gc_no_fairness_for_clones                         TRUE
_gc_sanity_check_cr_buffers                        FALSE
_gc_save_cleanout                                  TRUE
_gc_temp_affinity                                  FALSE
_gc_fg_spin_time                                   0
_gc_object_queue_max_length                        0
_gc_async_send                                     TRUE
_gc_async_receive                                  FALSE
_gc_try_to_skip_imc_flush                          TRUE
_gc_msgq_buffers                                   0
_gc_serve_from_flash_cache                         FALSE
_gc_trace_freelist_empty                           FALSE
_gc_skip_undo_disk_read                            TRUE
_gc_blocking_pins                                  FALSE
_gc_trace_blocking_pins                            FALSE
_gc_enable_cr_bypass                               TRUE
_gc_numa_lock_elements                             FALSE
_gc_buckets_per_latch                              0
_gc_integrity_checks                               1
_light_work_rule_debug                             FALSE
_gc_fast_index_split_wait                          0
_gc_max_reg_sz                                     68719476736
_gc_partial_cleanout                               FALSE
_gc_undo_rdma_read                                 FALSE
_gc_lease_time                                     500
_adaptive_direct_read                              TRUE
_adaptive_direct_write                             TRUE
_db_block_align_direct_read                        TRUE
_db_noarch_disble_optim                            FALSE
_db_disable_temp_encryption                        FALSE
_kcbl_assert_reset_slot                            TRUE
_direct_io_slots                                   0
_direct_io_skip_cur_slot_on_error                  TRUE
_direct_io_wslots                                  0
_thread_state_change_timeout_pnp                   1800
_disable_incremental_checkpoints                   FALSE
_disable_thread_internal_disable                   FALSE
_disable_selftune_checkpointing                    FALSE
_selftune_checkpointing_lag                        300
_target_rba_max_lag_percentage                     81
_no_recovery_through_resetlogs                     FALSE
_auto_rekey_during_mrcv                            TRUE
_auto_rcv_pdb_open                                 1
_log_blocks_during_backup                          TRUE
_allow_resetlogs_corruption                        FALSE
_allow_terminal_recovery_corruption                FALSE
_allow_read_only_corruption                        FALSE
_allow_file_1_offline_error_1245                   FALSE
_recovery_skip_cfseq_check                         FALSE
_db_shadow_lost_write_protect                      NOT_SET
_db_lost_write_corrupt_block                       FALSE
_flush_log_buffer_timeout                          0
_flush_log_buffer_force                            FALSE
_switchover_to_standby_option                      OPEN_ALL_IGNORE_SESSIONS
_switchover_timeout                                0
_switchover_through_cascade                        TRUE
_auto_bmr_file_header                              TRUE
_fast_psby_conversion                              TRUE
_adg_instance_recovery                             TRUE
_ac_enable_dscn_in_rac                             FALSE
_ac_strict_SCN_check                               FALSE
_allow_convert_to_standby                          FALSE
_early_flush_delta                                 0
_flush_redo_to_standby                             0
_disable_dict_check_pdb_open                       FALSE
_clear_preserved_buffers                           TRUE
_skip_pdb_recovery_if_keystore_not_open            FALSE
_serial_recovery                                   FALSE
_coord_message_buffer                              0
_multiple_instance_recovery                        FALSE
_backup_kgc_scheme                                 ZLIB
_backup_kgc_bufsz                                  0
_backup_kgc_type                                   0
_backup_kgc_niters                                 0
_backup_lzo_size                                   262144
_backup_kgc_perflevel                              1
_backup_kgc_windowbits                             15
_backup_kgc_memlevel                               8
_backup_kgc_blksiz                                 9
_backup_ksfq_bufmem_max                            268435456
_backup_ksfq_bufsz                                 0
_backup_ksfq_bufcnt                                0
_backup_disk_bufsz                                 0
_backup_disk_bufcnt                                0
_backup_seq_bufsz                                  0
_backup_seq_bufcnt                                 0
_backup_file_bufsz                                 0
_backup_file_bufcnt                                0
_restore_maxopenfiles                              8
_backup_max_gap_size                               4294967294
_backup_min_ct_unused_optim                        2097152
_unused_block_compression                          TRUE
_restore_spfile
_dummy_instance                                    FALSE
_rman_io_priority                                  3
_backup_encrypt_opt_mode                           4294967294
_disable_initial_block_compression                 FALSE
_undo_block_compression                            TRUE
_update_datafile_headers_with_space_information    FALSE
_disable_cell_optimized_backups                    FALSE
_backup_align_write_io                             TRUE
_backup_dynamic_buffers                            TRUE
_backup_automatic_retry                            10
_krb_trace_buffer_size                             131072
_sparse_backing_file                               AVM
_catalog_foreign_restore                           FALSE
_restore_create_directory                          TRUE
_backup_max_wallet_session                         50
_backup_block0                                     default
_restore_block0                                    on
_backup_int_spare1                                 4294967294
_backup_int_spare2                                 4294967294
_backup_int_spare3                                 4294967294
_backup_int_spare4                                 4294967294
_backup_text_spare1
_backup_text_spare2
_backup_text_spare3
_backup_text_spare4
_backup_bool_spare1                                FALSE
_backup_bool_spare2                                FALSE
_backup_bool_spare3                                FALSE
_backup_bool_spare4                                FALSE
_krb_check_osd_block_endianess                     TRUE
_controlfile_autobackup_delay                      300
_deferred_log_dest_is_valid                        TRUE
_log_deletion_policy                               mandatory
_prefered_standby
_auto_bmr                                          enabled
_auto_bmr_req_timeout                              60
_auto_bmr_sess_threshold                           30
_auto_bmr_pub_timeout                              10
_auto_bmr_fc_time                                  60
_auto_bmr_bg_time                                  3600
_auto_bmr_sys_threshold                            100
_auto_bmr_max_rowno                                1024
_krbabr_trace_buffer_size                          131072
_preplugin_backup                                  enabled
_auto_export_preplugin_backup                      TRUE
_backup_appliance_enabled                          FALSE
_ba_container_filesystem_ausize                    4194304
_ba_max_groups                                     0
_ba_max_containers                                 0
_ba_max_seg_bytes                                  4000
_ba_cf_trace_buffer_size                           131072
_ba_timeouts_enabled                               TRUE
_tts_allow_charset_mismatch                        FALSE
_xtts_allow_pre10                                  FALSE
_xtts_set_platform_info                            FALSE
_bypass_xplatform_error                            FALSE
_aux_dfc_keep_time                                 1440
_forced_endian_type                                UNKNOWN
_allow_cross_endian_dictionary                     FALSE
_query_on_physical                                 TRUE
_recoverable_recovery_batch_percent                50
_undo_tbs_slave_percent                            0
_recoverable_recovery_max_influx_buffers           100000
_disable_incremental_recovery_ckpt                 FALSE
_clone_one_pdb_recovery                            FALSE
_incremental_recovery_ckpt_min_batch               500
_time_based_rcv_ckpt_target                        180
_time_based_rcv_hdr_update_interval                180
_multi_instance_pmr                                TRUE
_disable_active_influx_move                        FALSE
_abort_on_mrp_crash                                FALSE
_keep_19907_during_recovery                        FALSE
_defer_log_boundary_ckpt                           TRUE

P_NAME                                             P_VALUE
-------------------------------------------------- --------------------------------------------------
_defer_log_count                                   100
_transient_logical_clear_hold_mrp_bit              FALSE
_log_max_optimize_threads                          128
_reduce_sby_log_scan                               TRUE
_sync_primary_wait_time                            5
_standby_implicit_rcv_timeout                      1
_readable_standby_sync_timeout                     10
_readable_standby_sync_interval                    0
_adg_parselock_timeout                             0
_adg_influx_qscn_gap                               0
_adg_parselock_timeout_sleep                       100
_adg_objectlock_timeout                            0
_adg_objectlock_attempts                           2
_adg_objectlock_maxnum                             1000
_adg_count_beyond_limit                            0
_adg_defer_segstat                                 TRUE
_adg_adt_redirect_catchup_wait_time                12000
_adg_adt_redirect_apply_lag_threshold              12000
_read_only_violation_max_count                     500
_read_only_violation_max_count_per_module          100
_read_only_violation_dump_to_trace                 FALSE
_adg_redirect_upd_to_primary_max_retries           500
_media_recovery_read_batch                         32
_standby_causal_heartbeat_timeout                  2
_use_pdb_parselock                                 TRUE
_snapshot_recovery_enabled                         TRUE
_adg_auto_close_pdb                                TRUE
_bct_public_dba_buffer_size                        0
_bct_initial_private_dba_buffer_size               0
_bct_bitmaps_per_file                              8
_bct_file_block_size                               0
_bct_file_extent_size                              0
_bct_chunk_size                                    0
_bct_crash_reserve_size                            262144
_bct_buffer_allocation_size                        2097152
_bct_buffer_allocation_max                         104857600
_bct_buffer_allocation_min_extents                 1
_bct_fixtab_file
_disable_primary_bitmap_switch                     FALSE
_bct_health_check_interval                         60
_bct_public_dba_buffer_dynresize                   2
_bct_public_dba_buffer_maxsize                     0
_bct_mrp_timeout                                   600
_krc_trace_buffer_size                             131072
_kra_trace_buffer_size                             131072
_kra_cfile_compaction                              TRUE
_validate_flashback_database                       FALSE
_flashback_allow_noarchivelog                      FALSE
_verify_flashback_redo                             TRUE
_flashback_verbose_info                            FALSE
_flashback_dynamic_enable_failure                  0
_flashback_dynamic_enable                          TRUE
_allow_drop_snapshot_standby_grsp                  FALSE
_flashback_marker_cache_size                       328
_flashback_marker_cache_enabled                    TRUE
_flashback_database_test_only                      FALSE
_allow_drop_ts_with_grp                            FALSE
_df_hist_offl_override                             FALSE
_flashback_logfile_enqueue_timeout                 600
_flashback_barrier_interval                        1800
_flashback_standby_barrier_interval                1
_flashback_stby_support_mira                       TRUE
_flashback_standby_check_barrier_MIRA              20
_flashback_standby_check_minpfh_MIRA               15
_flashback_max_standby_sync_span                   300
_flashback_fuzzy_barrier                           TRUE
_disable_kcb_flashback_blocknew_opt                FALSE
_disable_kcbl_flashback_blocknew_opt               FALSE
_disable_flashback_wait_callback                   FALSE
_check_block_new_invariant_for_flashback           FALSE
_allow_compatibility_adv_w_grp                     FALSE
_minimum_db_flashback_retention                    60
_flashback_delete_chunk_MB                         128
_flashback_11_1_block_new_opt                      FALSE
_flashback_marker_for_every_grp                    FALSE
_flashback_log_size                                1000
_flashback_log_min_size                            100
_db_flashback_log_min_size                         16777216
_db_flashback_log_min_total_space                  0
_flashback_copy_latches                            10
_flashback_n_log_per_thread                        128
_flashback_max_n_log_per_thread                    2048
_flashback_generation_buffer_size                  8388608
_allocate_flashback_buffer                         FALSE
_flashback_max_log_size                            0
_flashback_log_io_error_behavior                   0
_flashback_prepare_log                             TRUE
_flashback_size_based_on_redo                      TRUE
_flashback_validate_controlfile                    FALSE
_flashback_format_chunk_mb                         4
_flashback_format_chunk_mb_dwrite                  16
_flashback_log_rac_balance_factor                  10
_flashback_write_max_loop_limit                    10
_flashback_hint_barrier_percent                    20
_drop_flashback_logical_operations_enq             FALSE
_percent_flashback_buf_partial_full                50
_tdb_debug_mode                                    16
_log_read_buffers                                  8
_change_vector_buffers                             1
_change_vector_read_sample_ratio                   0
_kcfis_spawn_debugger                              FALSE
_kcfis_trace_bucket_size                           131072
_kcfis_fault_control                               0
_kcfis_caching_enabled                             TRUE
_kcfis_large_payload_enabled                       FALSE
_kcfis_automem_level                               1
_kcfis_cell_passthru_enabled                       FALSE
_kcfis_cell_passthru_dataonly                      TRUE
_kcfis_dump_corrupt_block                          TRUE
_kcfis_kept_in_cellfc_enabled                      TRUE
_kcfis_nonkept_in_cellfc_enabled                   FALSE
_kcfis_rdbms_blockio_enabled                       FALSE
_kcfis_fast_response_enabled                       TRUE
_kcfis_fast_response_threshold                     1048576
_kcfis_fast_response_initiosize                    2
_kcfis_fast_response_iosizemult                    4
_kcfis_max_out_translations                        5000
_kcfis_max_cached_sessions                         10
_kcfis_storageidx_disabled                         FALSE
_kcfis_disable_platform_decryption                 FALSE
_kcfis_storageidx_diag_mode                        0
_kcfis_storageidx_set_membership_disabled          FALSE
_kcfis_byteswap_opt_disabled                       FALSE
_kcfis_cellcache_disabled                          FALSE
_kcfis_test_control1                               0
_kcfis_stats_level                                 0
_kcfis_io_prefetch_size                            8
_kcfis_block_dump_level                            0
_kcfis_ioreqs_throttle_enabled                     TRUE
_kcfis_qm_prioritize_sys_plan                      TRUE
_kcfis_qm_user_plan_name
_kcfis_control1                                    0
_kcfis_control2                                    0
_kcfis_control3                                    0
_kcfis_control4                                    0
_kcfis_control5                                    0
_kcfis_control6                                    0
_kcfis_oss_io_size                                 0
_kcfis_read_buffer_limit                           0
_cell_file_format_chunk_size                       0
_cell_offload_capabilities_enabled                 1
_kcfis_cell_passthru_fromcpu_enabled               TRUE
_kcfis_celloflsrv_usage_enabled                    TRUE
_kcfis_celloflsrv_passthru_enabled                 FALSE
_kcfis_fastfileinit_disabled                       FALSE
_kcfis_xtgran_prefetch_count                       1
_kcfis_storageidx_xtss_disabled                    FALSE
_kcfis_pmem_enabled                                FALSE
_db_block_prefetch_skip_reading_enabled            TRUE
_db_block_prefetch_fast_longjumps_enabled          TRUE
_db_block_prefetch_private_cache_enabled           TRUE
_datafile_write_errors_crash_instance              TRUE
_datafile_open_errors_crash_instance               TRUE
_siop_perc_of_bc_x100                              625
_db_flashback_iobuf_size                           1
_db_flashback_num_iobuf                            64
_flashback_enable_ra                               TRUE
_nologging_sendbuf_ratio                           90
_nologging_slots                                   20
_nologging_load_slotsz                             1048576
_nologging_block_cleanout_delay                    300
_force_logging_in_upgrade                          TRUE
_nologging_mode_override                           0
_nologging_fetch_slv_wt                            600
_nologging_progress_keep_alive                     10
_nologging_kcnbuf_hash_buckets                     1024
_nologging_kcnbuf_hash_latches                     256
_nologging_standby_hot_buffer_timeout              500
_nologging_standby_cold_buffer_timeout             500
_nbr_recovery_timeout                              60
_nbr_recovery_max_request_size                     8
_nbr_recovery_target_bufs                          0
_nologging_apply_stall_time                        1000
_nologging_standby_fetch_disable                   FALSE
_nologging_standby_refetch_disable                 FALSE
_nologging_standby_dtc_expire                      600
_nologging_sdcl_append_wait                        200
_nologging_txn_cmt_wait                            1500
_max_data_transfer_cache_size                      536870912
_data_transfer_cache_bc_perc_x100                  500
__data_transfer_cache_size                         0
_nologging_fetch_retry_interval_hot                60
_nologging_fetch_retry_interval_cold               1800
_nologging_fetch_demote_count                      10
_nologging_fetch_initial_interval                  2
_max_kcnibr_ranges                                 1048576
_mira_mark_archivelog_timeout                      60
_enable_imc_mira                                   FALSE
_mira_num_receive_buffers                          25
_mira_num_local_buffers                            25
_mira_sender_process                               2
_mira_rcv_max_buffers                              500
_krpm_trace_buffer_size                            262144
_mira_rcv_catchup_buffers                          5
_mira_free_unused_buffers                          TRUE
_dbcomp_msg_ver                                    1
_dbcomp_maxdump                                    100
_shadow_lost_write_found                           118
_shadow_lost_write_tracing                         0
_adg_redirect_flags                                0
_row_locking                                       always
_serializable                                      FALSE
_scn_wait_interface_max_timeout_secs               2147483647
_scn_wait_interface_max_backoff_time_secs          600
_ktb_debug_flags                                   8
_log_committime_block_cleanout                     FALSE
_ktc_latches                                       0
_prescomm                                          FALSE
_recursive_imu_transactions                        FALSE
_ktc_debug                                         0
_enable_minscn_cr                                  TRUE
_txn_control_trace_buf_size                        4096
_ktu_latches                                       0
_transaction_auditing                              TRUE
_smu_error_simulation_site                         0
_smu_error_simulation_type                         0
_imtxn_table_enable                                TRUE
_imtxn_table_max_undo_segs                         1024
_imtxn_table_max_slts_per_seg                      34
_imtxn_table_max_inc_per_slt                       1
_imtxn_table_flags                                 3
_temp_undo_disable_adg                             FALSE
_temp_undo_optimal_extents                         0
_rollback_stopat                                   0
_discrete_transactions_enabled                     FALSE
_in_memory_undo                                    TRUE
_imu_pools                                         3
_branch_tagging                                    TRUE
_row_cr                                            TRUE
_enable_block_level_transaction_recovery           TRUE
_cleanup_rollback_entries                          100
_rollback_segment_initial                          1
_offline_rollback_segments
_corrupted_rollback_segments
_rollback_segment_count                            0
_smu_timeouts
_smu_debug_mode                                    0
_undo_debug_mode                                   0
_tablespaces_per_transaction                       10
_verify_undo_quota                                 FALSE
_in_memory_tbs_search                              TRUE
_highthreshold_undoretention                       4294967294
_undo_autotune                                     TRUE
_collect_undo_stats                                TRUE
_collect_tempundo_stats                            TRUE
_indoubt_pdb_transactions_force_outcome            COMMIT
_imtxnrma_table_enable                             FALSE
_transaction_recovery_servers                      0
_parallel_recovery_stopat                          32767
_resumable_critical_alert                          0
_cr_trc_buf_size                                   8192
_max_cr_rollbacks                                  1000
_disable_txn_alert                                 0
_undo_debug_usage                                  0
_db_change_notification_enable                     TRUE
_minact_timeout                                    180
_disable_flashback_archiver                        0
_disable_fba_qrw                                   0
_disable_fba_wpr                                   0
_flashback_archiver_partition_size                 0
_fbda_busy_percentage                              0
_fbda_inline_percentage                            0
_fbda_debug_mode                                   0
_fbda_debug_assert                                 0
_fbda_global_bscn_lag                              0
_fbda_rac_inactive_limit                           0
_fbda_tcrv_cleanup_lag                             3600
_securefile_log_num_latches                        0
_securefile_log_shared_pool_size                   0
_cli_cachebktalloc                                 100
_ilm_mem_limit                                     10
_ktilmsc_exp                                       600
_ktilm_uga_off                                     FALSE
_inmemory_check_prot_meta                          FALSE
_inmemory_private_journal_quota                    100
_inmemory_txnpjrnl_debug                           0
_inmemory_private_journal_sharedpool_quota         20
_inmemory_private_journal_numbkts                  512
_inmemory_private_journal_numgran                  255
_inmemory_private_journal_maxexts                  5000
_inmemory_journal_scan                             0
_inmemory_journal_format                           1
_inmemory_journal_scan_format                      TRUE
_inmemory_journal_cla_stride                       4
_inmemory_partrowlock_threshold                    10
_inmemory_pin_hist_mode                            16
_inmemory_txn_checksum                             0
_inmemory_buffer_waittime                          100
_inmemory_retention_time                           120
_inmemory_cudrop_timeout                           1000
_inmemory_exclto_timeout                           100
_inmemory_num_hash_latches                         256
_inmemory_strdlxid_timeout                         0
_inmemory_incremental_repopulation                 TRUE
_inmemory_repopulate_optimize                      TRUE
_inmemory_transaction_options                      758
_inmemory_waitinvis_count                          8
_inmemory_lock_for_smucreate                       FALSE
_inmemory_relimcusz_thresh                         25
_inmemory_shared_journal_maxexts                   1024
_inmemory_shared_journal_minexts_repop             256
_inmemory_direct_reads                             1
_inmemory_grpcolinv_buffer_size                    131072
_inmemory_grpcolinv_granularity                    1
_inmemory_crclone_buffer_size                      131072
_inmemory_crclone_threshold_rows                   100
_inmemory_crclone_min_clones                       6
_inmemory_crclone_min_space_percent                50
_inmemory_patch_threshold_blocks                   25
_inmemory_patch_background_blocks                  200
_inmemory_patch_commit_path                        FALSE
_inmemory_smu_patch_options                        7
_inmemory_adg_quiesce_timeout                      2
_inmemory_adg_periodic_sort                        FALSE
_inmemory_adg_batched_flush                        TRUE
_inmemory_adg_parallel_flush                       TRUE
_inmemory_adg_journal_quota                        FALSE
_ktst_rss_min                                      100
_ktst_rss_max                                      1000
_ktst_rss_retry                                    5
_ktst_tscleanup_timeout                            1800
_force_active_temp_detection                       FALSE
_partition_large_extents                           TRUE
_index_partition_large_extents                     FALSE
_force_local_temp                                  FALSE
_prefer_local_temp                                 FALSE
_allocate_creation_order                           FALSE
_log_space_errors                                  TRUE
_assm_low_gsp_threshold                            10000
_assm_high_gsp_threshold                           11024
_simulate_io_wait                                  0
_bump_highwater_mark_count                         0
_assm_segment_extension_percent                    100
_old_extent_scheme                                 FALSE
_trace_temp                                        FALSE
_concurrency_chosen                                10
_no_small_file                                     FALSE
_assm_default                                      TRUE
_enable_hwm_sync                                   TRUE
_hwm_sync_threshold                                10
_enable_check_truncate                             TRUE
_enable_tablespace_alerts                          TRUE
_disable_temp_tablespace_alerts                    FALSE
_disable_undo_tablespace_alerts                    FALSE
_disable_lostwrite_tablespace_alerts               FALSE
_last_allocation_period                            5
_securefiles_forceflush                            FALSE
_securefiles_concurrency_estimate                  12
_securefiles_memory_percentofSGA                   8
_inst_locking_period                               5
_allocation_update_interval                        3
_minimum_blocks_to_shrink                          0
_minimum_extents_to_shrink                         1
_use_best_fit                                      FALSE
_step_down_limit_in_pct                            20
_assm_test_force_rej                               0
_assm_test_force_rej2                              0
_assm_test_reentrant_gsp                           FALSE
_enable_space_preallocation                        3
_ktspsrch_maxsc                                    1024
_ktspsrch_scexp                                    60
_ktspsrch_scchk                                    60
_ktspsrch_maxskip                                  5
_max_sys_next_extent                               0
_assm_force_fetchmeta                              FALSE
_max_fsu_segments                                  1024
_max_fsu_sgapcent                                  10
_max_fsu_exptime                                   7200
_max_fsu_stale_time                                600
_disable_flashback_recyclebin_opt                  TRUE
_max_shrink_obj_stats                              0
_check_ts_threshold                                0
_async_ts_threshold                                1
_enable_securefile_flashback_opt                   FALSE
_disable_def_seg_update                            0
_use_cached_asm_free_space                         FALSE
_ena_storage_lmt                                   DEFAULT
_kttext_warning                                    5
_ktt_tsid_reuse_threshold                          60000
_enable_verbose_gdr                                FALSE
_enable_sysaux_gdr                                 FALSE
_gdr_clear_inactive_only                           FALSE
_gdr_clear_active_only                             FALSE
_drop_tablespace_objects                           0
_smon_internal_errlimit                            100
_smon_undo_seg_rescan_limit                        10
_enable_12g_bft                                    TRUE
_space_align_size                                  1048576
_undotbs_stepdown_pcent                            75
_undotbs_regular_tables                            FALSE
_enable_spacebg                                    TRUE
_max_spacebg_slaves                                1024
_minmax_spacebg_slaves                             8
_min_spacebg_slaves                                2
_max_spacebg_tasks                                 8192
_max_spacebg_msgs_percentage                       50
_ktslj_segext_warning                              10
_ktslj_segext_warning_mb                           0
_ktslj_segext_max_mb                               0
_ktslj_segext_retry                                5
_securefiles_fg_retry                              100
_securefiles_breakreten_retry                      5
_securefiles_bulkclnout                            TRUE
_securefiles_bulkinsert                            FALSE
_securefiles_bgtimeout                             0
_securefiles_spcutl                                FALSE
_spacebg_sync_segblocks                            TRUE
_trace_ktfs                                        FALSE
_trace_ktfs_mem                                    FALSE
_enable_ilm_flush_stats                            TRUE
_enable_ilm_testflush_stats                        FALSE
_disable_12cbigfile                                FALSE
_enable_heatmap_internal                           FALSE
_heatmap_format_1block                             FALSE
_test_hm_extent_map                                FALSE
_ilmset_stat_limit                                 0
_ilmflush_stat_limit                               0
_heatmap_min_maxsize                               0
_ilmstat_memlimit                                  5
_flush_ilm_stats                                   0
_create_stat_segment                               0
_drop_stat_segment                                 0
_print_stat_segment                                0
_print_inmem_heatmap                               0
_blockhm_flush_period                              3600
_inmemory_auto_distribute                          TRUE
_inmemory_autodist_2safe                           FALSE
_inmemory_distribute_timeout                       300
_inmemory_distribute_ondemand_timeout              300
_inmemory_granule_size                             134217728
_inmemory_64k_percent                              30
_inmemory_expressions_area_percent                 10
_inmemory_min_ima_defersize                        0
_inmemory_memprot                                  TRUE
__inmemory_ext_rwarea                              0
_inmemory_ext_rwarea                               0
__inmemory_ext_roarea                              0
_inmemory_ext_roarea                               0
_inmemory_xmem_size                                0
_inmemory_prefer_xmem_memcompress
_inmemory_prefer_xmem_priority
_inmemory_drcancel_cu_percent                      80
_trace_ktds                                        FALSE
_track_space_reuse                                 FALSE
_track_space_reuse_rac                             FALSE
_spacereuse_track_memlimit                         10
_print_inmem_srmap                                 0
_enable_rejection_cache                            TRUE
_assm_space_cache_hb_expire                        3
_assm_l1cleanout_throttle_time                     3
_assm_examination_time_threshold                   60
_assm_examination_blocks_threshold                 8196
_assm_examination_enable_sla                       FALSE
_assm_space_cache_max_segments                     1024
_assm_segment_repair_fg                            0
_assm_segment_repair_bg                            TRUE
_assm_segment_repair_maxblks                       4294967294
_assm_latency_sampling_frequency                   0
_assm_segment_repair_timelimit                     60
_db_row_overlap_checking                           TRUE
_db_index_block_checking                           TRUE
_kdi_avoid_block_checking                          FALSE
_disable_block_checking                            FALSE
_kd_symtab_chk                                     TRUE
_sage_block_checking                               FALSE
_suppress_identifiers_on_dupkey                    FALSE
_compression_compatibility                         18.0.0
_check_column_length                               TRUE
_oltp_compression                                  TRUE
_inplace_update_retry                              TRUE
_widetab_comp_enabled                              TRUE
_partial_comp_enabled                              TRUE
_limit_itls                                        20
_force_hsc_compress                                FALSE
_force_oltp_compress                               FALSE
_force_partial_compress                            FALSE
_force_sys_compress                                TRUE
_force_oltp_update_opt                             TRUE
_force_arch_compress                               0
_compression_advisor                               0
_compression_chain                                 90
_compression_above_cache                           0
_oltp_compress_dbg                                 0
_oltp_compression_gain                             10
_arch_compression                                  TRUE
_arch_comp_dbg_scan                                0
_oltp_comp_dbg_scan                                0
_dbg_scan                                          0
_arch_comp_dec_block_check_dump                    1
_alternate_iot_leaf_block_split_points             TRUE
_kdu_array_depth                                   16
_cell_offload_hybridcolumnar                       TRUE
_disable_implicit_row_movement                     FALSE
_cu_row_locking                                    0
_oltp_spill                                        FALSE
_delete_ghost_data                                 FALSE
_kd_dbg_control                                    0
_block_dump_assert                                 FALSE
_release_insert_threshold                          5
_walk_insert_threshold                             0
_enable_hash_overflow                              FALSE
_kdtgsp_retries                                    1024
_rowlen_for_chaining_threshold                     1000
_minfree_plus                                      0
_disable_hcc_array_insert                          FALSE
_extra_lmn_enabled                                 TRUE
_use_seq_process_cache                             TRUE
_pdb_use_sequence_cache                            TRUE
_sequence_scale_noextend                           FALSE
_sequence_scale_extend                             FALSE
_analyze_comprehensive                             FALSE
_disable_index_block_prefetching                   FALSE
_index_scan_check_skip_corrupt                     FALSE
_index_scan_check_stopkey                          FALSE
_index_max_inc_trans_pct                           20
_index_split_chk_cancel                            5
_advanced_index_compression_trace                  0
_gdr_control_flags                                 0
_index_alert_key_not_found                         FALSE
_reuse_index_loop                                  5
_kdis_reject_limit                                 5
_kdis_reject_level                                 24
_kdis_reject_ops                                   FALSE
_kdis_split_xid_prune                              TRUE
_kdbl_enable_post_allocation                       FALSE
_ldr_io_size                                       262144
_ldr_io_size2                                      1048576
_ldr_pga_lim                                       0
_ldr_tempseg_threshold                             8388608
_lmn_invalidlkr_enabled                            TRUE
_ignore_desc_in_index                              FALSE
_kdic_segarr_sz                                    0
_keep_remote_column_size                           FALSE
_kdli_trace                                        0
_kdli_sort_dbas                                    FALSE
_kdli_safe_callbacks                               TRUE
_kdli_inode_preference                             data
_kdli_reshape                                      FALSE
_kdli_cache_inode                                  TRUE
_kdli_cache_verify                                 FALSE
_kdli_cache_size                                   8
_kdli_memory_protect                               FALSE
_kdli_rci_lobmap_entries                           255
_kdli_readahead_strategy                           contig
_kdli_readahead_limit                              0
_kdli_sio_on                                       TRUE
_kdli_sio_min_read                                 0
_kdli_sio_min_write                                0
_kdli_sio_async                                    TRUE
_kdli_sio_fgio                                     TRUE
_kdli_sio_nbufs                                    8
_kdli_sio_niods                                    8
_kdli_sio_strategy                                 extent
_kdli_sio_fileopen                                 none
_kdli_sio_flush                                    FALSE
_kdli_sio_bps                                      0
_kdli_sio_pga                                      FALSE
_kdli_sio_pga_top                                  FALSE
_kdli_sio_dop                                      2
_kdli_sio_free                                     TRUE
_kdli_sio_backoff                                  FALSE
_kdli_inline_xfm                                   TRUE
_kdli_timer_trc                                    FALSE
_kdli_timer_dmp                                    FALSE
_kdli_cacheable_length                             0
_kdli_small_cache_limit                            32
_kdli_inject_assert                                0
_kdli_inject_crash                                 0
_kdli_space_cache_limit                            2048
_kdli_force_cr                                     TRUE
_kdli_force_cr_meta                                TRUE
_kdli_recent_scn                                   FALSE
_kdli_itree_entries                                0
_kdli_cache_read_threshold                         0
_kdli_cache_write_threshold                        0
_kdli_full_readahead_threshold                     0
_kdli_force_storage                                none
_kdli_allow_corrupt                                TRUE
_kdli_squeeze                                      TRUE
_kdli_buffer_inject                                TRUE
_kdli_inject_batch                                 0
_kdli_flush_injections                             TRUE
_kdli_flush_cache_reads                            FALSE
_kdli_checkpoint_flush                             FALSE
_kdli_delay_flushes                                TRUE
_kdli_dbc                                          none
_kdli_preallocation_pct                            0
_kdli_preallocation_mode                           length
_kdli_inplace_overwrite                            0
_kdli_sio_write_pct                                100
_kdli_sio_fbwrite_pct                              35
_kdli_vll_direct                                   TRUE
_kdli_descn_adj                                    FALSE
_kdli_oneblk                                       FALSE
_kdli_mts_so                                       TRUE
_kdli_ralc_length                                  10485760
_kdli_ralc_rounding                                1048576
_kdli_space_cache_segments                         16
_kdli_icache_entries                               3
_kdli_full_vll                                     FALSE
_kdli_small_append_redo                            TRUE
_kdli_STOP_tsn                                     0
_kdli_STOP_dba                                     0
_kdli_STOP_fsz                                     0
_kdli_STOP_bsz                                     0
_kdli_STOP_nio                                     0
_kdlw_enable_write_gathering                       TRUE
_kdlw_enable_ksi_locking                           FALSE
_kdlwp_flush_threshold                             4194304
_kdlxp_cmp_subunit_size                            262144
_kdlxp_minxfm_size                                 32768
_kdlxp_lobcompress                                 FALSE
_kdlxp_lobcmplevel                                 2
_kdlxp_lobcmprciver                                1
_kdlxp_lobencrypt                                  FALSE
_kdlxp_xfmcache                                    TRUE
_kdlxp_lobcmpadp                                   FALSE
_kdlxp_uncmp                                       FALSE
_kdlxp_mincmplen                                   200
_kdlxp_mincmp                                      20
_sf_default_enabled                                TRUE
_kdlf_read_flag                                    0
_kdlxp_lobdeduplicate                              FALSE
_kdlxp_lobdedupvalidate                            TRUE
_kdlxp_dedup_hash_algo                             SHA1
_kdlxp_dedup_flush_threshold                       8388608
_kdlxp_dedup_prefix_threshold                      1048576
_kdlxp_dedup_inl_pctfree                           5
_kdlxp_dedup_wapp_len                              0
_kdlxp_spare1                                      0
_kdlxp_no_dedup_on_insert                          FALSE
_securefile_timers                                 FALSE
_kdlu_trace_system                                 0
_kdlu_trace_layer                                  0
_kdlu_max_bucket_size                              4194304
_kdlu_max_bucket_size_mts                          131072
_kdz_hcc_track_upd_rids                            FALSE
_kdz_hcc_flags                                     0
_kdz_pcode_flags                                   0
_enable_columnar_cache                             1
_kdz_proj_nrows                                    1024
_kdz_pred_nrows                                    32767
_kdzk_enable_init_trace                            FALSE
_kdzk_load_specialized_library                     3
_kdzk_specialized_library_name
_kdzk_trace_level                                  0
_hpk_project_cost_weighting                        20
_hpk_throughput_range                              5
_hpk_compression_range                             5
_inmemory_analyzer_optimize_for                    0
_kdz_clear_analysis_percent                        60
_dbfs_modify_implicit_fetch                        TRUE
_ILM_FILTER_TIME                                   0
_ILM_FILTER_TIME_LOWER                             0
_ILM_POLICY_NAME                                   FALSE
_inmemory_default_flags                            8459
_inmemory_default_new                              FALSE
_inmemory_default_svc                              0
_inmemory_default_svcname
_inmemory_disable_selective_vircols                FALSE
_inmemory_expression_count                         20
_inmemory_max_expressions_table                    50
_inmemory_expressions_mark_unused                  FALSE
_inmemory_query_scan                               TRUE
_inmemory_scan_override                            FALSE
_inmemory_small_segment_threshold                  65536
_inmemory_query_fetch_by_rowid                     FALSE
_inmemory_pruning                                  ON
_inmemory_subcusize                                512
_inmemory_subcu_histsize                           0
_inmemory_enable_sys                               FALSE
_inmemory_populate_fg                              FALSE
_inmemory_pga_per_server                           536870912
_inmemory_servers_throttle_pgalim_percent          55
_inmemory_populate_wait                            FALSE
_inmemory_populate_wait_max                        600
_inmemory_imco_cycle                               120
_inmemory_disable_alter_imco_cycle                 FALSE
_inmemory_enable_population_verify                 1
_inmemory_log_level                                0
_inmemory_fs_verify                                FALSE
_inmemory_force_fs                                 FALSE
_inmemory_force_fs_tbs                             SYSAUX
_inmemory_force_fs_tbs_size                        1073741824
_inmemory_fs_raise_error                           FALSE
_inmemory_fs_nodml                                 FALSE
_inmemory_fs_enable                                TRUE
_inmemory_fsdw_enable                              TRUE
_inmemory_fsdw_populate_threshold_multiplier       10
_inmemory_faststart_control_flags                  0
_inmemory_fsdw_on_populate                         TRUE
_inmemory_fsdw_penalty_threshold                   20
_inmemory_fsdw_inline                              FALSE
_inmemory_fs_enable_blk_lvl_inv                    TRUE
_inmemory_fsdw_cnt                                 2147483647
_inmemory_fsdw_maxcnt                              10
_inmemory_fsdw_maxmem                              104755200
_inmemory_fsdw_threshold                           300
_inmemory_fsdw_priority_population_percent         100
_inmemory_fsdw_priority_repopulation_percent       90
_inmemory_fsdw_priority_penalty_percent            10
_inmemory_fsdw_schedlrtm                           1
_inmemory_fsdw_scheduler_dedicated                 FALSE
_inmemory_fs_blk_inv_blkcnt
_inmemory_fs_blk_inv_blk_percent                   50
_inmemory_fs_prune_pct                             10
_inmemory_fs_dmlverify_read_ahead_cnt              10
_inmemory_fsdw_task_timeout                        20
_inmemory_fs_tbsmaxsz
_inmemory_enable_stat_alert                        FALSE
_inmemory_imcu_align                               TRUE
_inmemory_max_populate_retry                       3
_inmemory_imcu_target_rows                         0
_inmemory_imcu_target_bytes                        0
_inmemory_imcu_source_extents                      0
_inmemory_imcu_source_blocks                       0
_inmemory_imcu_source_minbytes                     1048576
_inmemory_imcu_populate_minbytes                   5242880
_inmemory_imcu_source_analyze_bytes                134217728
_inmemory_imcu_target_maxrows                      8388608
_inmemory_imcu_source_maxbytes                     536870912
_inmemory_max_queued_tasks                         0
_inmemory_repopulate_threshold_rows
_inmemory_repopulate_threshold_blocks
_inmemory_pct_inv_rows_invalidate_imcu             50
_inmemory_pct_inv_blocks_invalidate_imcu           100
_inmemory_repopulate_threshold_mintime_factor      5
_inmemory_repopulate_threshold_mintime             0
_inmemory_repopulate_threshold_scans               0
_inmemory_repopulate_priority_scale_factor         100
_inmemory_repopulate_invalidate_rate_percent       0
_inmemory_repopulate_priority_threshold_row        160
_inmemory_repopulate_priority_threshold_block      320
_inmemory_repopulate_threshold_rows_percent        5
_inmemory_repopulate_threshold_blocks_percent      10
_inmemory_repopulate_disable                       2
_inmemory_check_protect                            FALSE
_inmemory_checksum                                 FALSE
_inmemory_cu_checksum                              FALSE
_inmemory_validate_fetch                           FALSE
_inmemory_journal_row_logging                      FALSE
_inmemory_prefix_encode_dsbs                       FALSE
_inmemory_vector_encode_override                   FALSE
_inmemory_journal_check                            0
_inmemory_rows_check_interrupt                     1000
_inmemory_enable_readonly                          TRUE
_inmemory_repopulate_flags                         2
_inmemory_hwm_expansion                            0
_inmemory_dbg_scan                                 0
_inmemory_query_check                              0
_inmemory_test_verification                        0
_inmemory_invalidate_cursors                       TRUE
_inmemory_prepopulate_fg                           0
_inmemory_format_compatible                        12.2.0.0.0
_inmemory_prepopulate                              TRUE
_inmemory_trickle_repopulate                       TRUE
_inmemory_trickle_repopulate_threshold_dirty_ratio 0
_inmemory_trickle_repopulate_min_interval          300
_inmemory_trickle_repopulate_fg                    0
_inmemory_max_delta                                5
_inmemory_delta_population                         0
_inmemory_min_delta_blocks                         50
_inmemory_min_delta_rows                           255
_inmemory_scan_invalid_percent                     30
_inmemory_external_table                           TRUE
_inmemory_dynamic_scans                            AUTO
_inmemory_dynamic_scans_batch_size                 500
_inmemory_dynamic_scans_analyze_batch_size         5
_inmemory_dynamic_scan_disable_threshold           10
_inmemory_suppress_vsga_ima                        FALSE
_inmemory_force_non_engineered                     FALSE
_inmemory_hwm_expand_percent                       20
_globaldict_enable                                 2
_globaldict_chunk_minalloc                         FALSE
_globaldict_use_ndv                                TRUE
_globaldict_dbg                                    0
_globaldict_reprobe_limit                          1
_globaldict_chain_limit                            32
_globaldict_check                                  0
_globaldict_analyzer_pct                           100
_globaldict_max_gdsize                             1073741824
_imado_diagtasks_log_period                        5
_imado_diagtasks_purge_period                      30
_imado_sysaux_usage_limit                          90
_imado_disable_bg                                  FALSE
_imado_mem_threshold                               98
_imado_optim_algo                                  GREEDY
_imado_optimize_period                             0
_imado_verification                                0
_imado_evict_sf                                    2
_cellcache_default_flags                           2
_cellcache_default_new                             FALSE
_cellcache_clause_default
_index_load_buf_oltp_sacrifice_pct                 10
_index_load_buf_oltp_under_pct                     85
_index_load_buf_oltp_over_retry                    0
_index_load_last_leaf_comp                         85
_index_load_min_key_count                          10
_index_load_analysis_frequency                     4
_kdizoltp_uncompsentinal_freq                      16
_kdfs_trace                                        1
_kdfs_trace_size                                   131072
_kdfs_timer_dmp                                    FALSE
_kdfs_fix_control                                  0
_worker_threads                                    0
_advanced_index_compression_umem_options           2147483647
_advanced_index_compression_options                0
_advanced_index_compression_cmp_options            0
_advanced_index_compression_tst_options            0
_advanced_index_compression_opt_options            0
_advanced_index_compression_options_value          0
_advanced_index_compression_recmp_cusz             90
_advanced_index_compression_recmp_crsz             10
_advanced_index_compression_recmp_nprg             10
_column_level_stats                                OFF
_column_stats_mem_limit                            10
_column_stats_flush_interval                       60
_column_stats_max_entries_per_stat                 5
_kdkv_trace                                        FALSE
_kdkv_force_fastpath                               FALSE
_kdkv_index_lossy                                  TRUE
_kdkv_index_relocate                               FALSE
_kdkv_fg_populate                                  FALSE
_kdkv_fg_no_memopt                                 FALSE
_kdkv_fg_drop_memopt                               TRUE
_kdkv_fg_repopulate                                FALSE
_kdkv_indexinvalid                                 FALSE
_kdfip_flush_nrows                                 10
_kdfip_flush_rowsz                                 1024
_kdfip_flush_rowtm                                 60
_kdfip_cmap_nbkt                                   16
_kdfip_iga_bufsz                                   65536
_kdfip_iga_maxsz                                   52428800
_kdfip_par_flush                                   TRUE
_kdfip_trace                                       FALSE
_kdfip_debug                                       0
_kdfip_buf_nclatch                                 64
_kdfip_elem_nclatch                                64
_kdfip_bufl_nbkt                                   64
_kdfip_drain_sleeps                                60
_kqr_optimistic_reads                              TRUE
_kqr_enable_conservative_logging                   TRUE
_max_row_cache_dumps                               1000
_row_cache_dump_interval                           0
_row_cache_dump_cooling_period                     10
_row_cache_cursors                                 20
_kqdsn_instance_digits                             2
_kqdsn_cpu_digits                                  3
_kgl_cluster_lock                                  TRUE
_kgl_cluster_pin                                   TRUE
_kgl_cluster_lock_read_mostly                      FALSE
_kgl_latch_count                                   0
_kgl_heap_size                                     4096
_kgl_fixed_extents                                 TRUE
_kgl_bucket_count                                  9
_library_cache_advice                              TRUE
_kglsim_maxmem_percent                             5
_kgl_hash_collision                                FALSE
_kgl_time_to_wait_for_locks                        15
_kgl_large_heap_warning_threshold                  52428800
_kgl_large_heap_assert_threshold                   524288000
_synonym_repoint_tracing                           FALSE
_ignore_fg_deps
_trace_kqlidp                                      FALSE
_force_standard_compile                            FALSE
_disable_fast_validate                             FALSE
_kgl_message_locks                                 256
_kql_clientlocks_enabled                           15
_kgl_debug
_mutex_wait_time                                   1
_mutex_spin_count                                  255
_mutex_wait_scheme                                 2
_kgl_min_cached_so_count                           1
_kgl_hot_object_copies                             0
_kgl_kqr_cap_so_stacks                             0
_invalidate_upon_revoke                            TRUE
_pseudo_bootstrap                                  FALSE
_kgl_iterator_action                               SKIP
_max_library_cache_dumps                           1000
_library_cache_dump_interval                       0
_library_cache_dump_cooling_period                 10
_enable_kqf_purge                                  TRUE
_object_number_cache_size                          5
_enable_Front_End_View_Optimization                1
_max_remote_tool_requests                          10
_cursor_features_enabled                           2
_session_page_extent                               2048
_cursor_runtimeheap_memlimit                       5242880
_kgx_latches                                       1024
_cursor_stats_enabled                              TRUE
_enable_editions_for_users                         FALSE
_disable_actualization_for_grant                   FALSE
_ignore_edition_enabled_for_EV_creation            FALSE
_lock_dc_users_time                                120
_enable_secure_sub_role                            FALSE
_case_sensitive_logon                              TRUE
_VIEW_DICTIONARY_ACCESSIBILITY                     FALSE
_restrict_become_user                              TRUE
_enable_http_digest_generation                     TRUE
_passwordfile_enqueue_timeout                      900
_iat_frequency_short                               FALSE
_dynamic_rls_policies                              TRUE
_virtual_column_access_control                     TRUE
_allow_insert_with_update_check                    FALSE
_session_context_size                              10000
_default_encrypt_alg                               0
_tsenc_obfuscate_key                               BOTH
_db_discard_lost_masterkey                         FALSE
_db_generate_dummy_masterkey                       FALSE
_use_fips_mode                                     FALSE
_heartbeat_period_multiplier                       0
_heartbeat_config                                  NONE
_redo_log_key_reuse_count                          0
_stats_encryption_enabled                          FALSE
_REMOVE_INACTIVE_STANDBY_TDE_MASTER_KEY            FALSE
_standby_newkey_keystore_location
_REMOVE_STDBY_OLD_KEY_AFTER_CHECKPOINT_SCN         TRUE
_ols_cleanup_task                                  TRUE
_sys_logon_delay                                   1
_pbkdf2_sder_count                                 3
_resource_includes_unlimited_tablespace            FALSE
_priv_for_set_current_schema                       FALSE
_xs_cleanup_task                                   TRUE
_xs_logon_grant                                    FALSE
_xds_max_child_cursors                             100
_fusion_security                                   TRUE
_radm_enabled                                      TRUE
_strict_redaction_semantics                        TRUE
_xs_dispatcher_only                                FALSE
_unified_audit_policy_disabled                     FALSE
_unified_audit_flush_threshold                     85
_unified_audit_flush_interval                      3
_asm_audit_sp_param
_disable_asm_audit_feat                            0
_ldap_config_ssl_for_sasl_md5                      TRUE
_ldap_config_force_sync_up                         FALSE
_ldap_reset_user_account_flc                       TRUE
_distributed_recovery_connection_hold_time         200
_reco_sessions_max_percentage                      50
_k2q_latches                                       0
_clusterwide_global_transactions                   TRUE
_autotune_gtx_threshold                            60
_forwarded_2pc_threshold                           10
_disable_autotune_gtx                              FALSE
_autotune_gtx_interval                             5
_autotune_gtx_idle_time                            600
_enable_separable_transactions                     FALSE
_xa_internal_retries                               600
_restrict_db_link_user_entries                     10000
_db_link_sources_tracking                          TRUE
_external_scn_activity_tracking                    TRUE
_pmon_load_constants                               300,192,64,3,10,10,0,0
_dispatcher_rate_ttl
_dispatcher_rate_scale
_enable_shared_server_vector_io                    FALSE
_shared_server_load_balance                        0
_enable_shared_server_sizing                       TRUE
_enable_shared_server_sizing_slope                 FALSE
_shared_server_sizing_coeff                        1000
_shared_servers_performance                        0
_dispatcher_listen_on_vip                          FALSE
_shared_server_num_queues                          2
_kernel_message_network_driver                     FALSE
_dedicated_server_poll_count                       10
_dedicated_server_post_wait_call                   FALSE
_dedicated_server_post_wait                        FALSE
_mpmt_fg_enabled                                   FALSE
_bequeath_via_broker                               FALSE
_cp_num_hash_latches                               1
_connection_broker_host                            localhost
_connection_broker_handout_accept                  FALSE
__reload_lsnr                                      0
_disable_duplicate_service_warning                 FALSE
_pdb_service_on_root_listener                      FALSE
_fast_cursor_reexecute                             FALSE
_cursor_obsolete_threshold                         8192
_disable_cursor_sharing                            FALSE
_cursor_reload_failure_threshold                   0
_cursor_diagnostic_node_agedout_count              100
_max_sql_stmt_length                               0
_monitor_sql_stmt_length                           FALSE
_kks_cached_parse_errors                           0
_kks_obsolete_dump_threshold                       1
_kks_parse_error_warning                           100
_dump_qbc_tree                                     0
_sql_alias_scope                                   TRUE
_plsql_dump_buffer_events
_ncomp_shared_objects_dir
_plsql_anon_block_code_type                        INTERPRETED
_inline_sql_in_plsql                               FALSE
_plsql_native_frame_threshold                      4294967294
_plsql_nvl_optimize                                FALSE
_plsql_icd_arg_dump                                FALSE
_plsql_max_stack_size                              0
_EnableShadowTypes                                 FALSE
_job_queue_interval                                5
_redef_on_statement                                FALSE

P_NAME                                             P_VALUE
-------------------------------------------------- --------------------------------------------------
_duplicated_table_complete_refresh_percent         50
_dbms_sql_security_level                           1
_parallel_default_max_instances                    1
_system_trig_enabled                               TRUE
_AllowMultInsteadofDDLTrigger                      0
_kktAllowInsteadOfDDLTriggeronDDL                  0
_EnableDDLTtriggerTracing                          FALSE
_kkfi_trace                                        FALSE
_lock_ref_constraint_count                         50
_lock_next_constraint_count                        3
_adjust_literal_replacement                        FALSE
_kolfuseslf                                        FALSE
_lock_ref_descendants_count                        50
_kkpogpi_nocpy                                     0
_result_cache_auto_size_threshold                  100
_result_cache_auto_time_threshold                  1000
_result_cache_auto_execution_threshold             1
_result_cache_deterministic_plsql                  FALSE
_max_string_size_bypass                            0
_qkslvc_extended_bind_sz                           1
_qkslvc_47510_test                                 0
_qkslvc_47510_sys                                  1
_STFTranslateDynamicSQL                            FALSE
_bigdata_external_table                            TRUE
_external_table_hive_partition_restricted          TRUE
_qksfgi_disable_action_mask                        0
_qksfgi_disable_oct_mask                           0
_qksfgi_dynamic_partitions
_qksfgi_feature_level                              0
_session_cached_instantiations                     60
_protect_frame_heaps                               FALSE
_frame_cache_time                                  0
_cursor_cache_time                                 900
_aged_out_cursor_cache_time                        300
_kxscio_cap_stacks                                 FALSE
_plsql_share_instantiation                         TRUE
_px_trace                                          none
_xt_trace                                          none
_xt_coverage                                       none
_ku_trace                                          none
_optimizer_trace                                   none
_parallel_server_idle_time                         30000
_parallel_server_sleep_time                        1
_px_send_timeout                                   300
_dynamic_stats_threshold                           30
_px_execution_services_enabled                     TRUE
_parallel_fake_class_pct                           0
_px_load_publish_interval                          200
_px_proactive_slave_alloc_threshold                4
_px_load_balancing_policy                          UNIFORM
_px_dp_array_size                                  32767
_px_parallel_spawn_min_count                       2
_parallel_execution_message_align                  FALSE
_PX_use_large_pool                                 FALSE
_parallel_min_message_pool                         491520
_px_buffer_ttl                                     30
_tq_dump_period                                    0
_affinity_on                                       TRUE
_enable_default_affinity                           0
_px_max_map_val                                    32
_dss_cache_flush                                   FALSE
_dss_cache_flush_threshold                         1
_hash_join_enabled                                 TRUE
_hash_multiblock_io_count                          0
_cursor_db_buffers_pinned                          115
_old_connect_by_enabled                            FALSE
_table_lookup_prefetch_size                        40
_multi_join_key_table_lookup                       TRUE
_table_lookup_prefetch_thresh                      2
_adaptive_fetch_enabled                            TRUE
_query_execution_cache_max_size                    131072
_fast_dual_enabled                                 TRUE
_newsort_enabled                                   TRUE
_newsort_type                                      0
_newsort_ordered_pct                               63
_recursive_with_max_recursion_level                0
_sort_spill_threshold                              0
_queue_buffer_max_dump_len                         65536
_force_hash_join_spill                             FALSE
_nlj_batching_ae_flag                              2
_sort_sync_min_spillsize                           262144
_sort_sync_min_spill_threshold                     90
_sqlexec_cache_aware_hash_join_enabled             TRUE
_sqlexec_encoding_aware_hj_enabled                 TRUE
_sqlexec_sort_uses_xmem                            TRUE
_sqlexec_hash_join_uses_xmem                       FALSE
_sqlexec_encoding_aware_hj_min_compression_ratio   8
_sqlexec_join_group_aware_hj_enabled               TRUE
_sqlexec_join_group_aware_hj_unencoded_rowsets_tol 50
erated

_sqlexec_cache_aware_hash_aggr_enabled             TRUE
_in_memory_cdt                                     LIMITED
_in_memory_ts_only                                 OFF
_in_memory_memory_threshold                        40
_in_memory_cdt_maxpx                               4
_ptt_max_num                                       16
_fix_control
_sdiag_crash                                       NONE
_result_cache_block_size                           1024
_result_cache_copy_block_count                     1
_result_cache_global                               TRUE
_result_cache_per_pdb                              TRUE
_result_cache_timeout                              10
_result_cache_white_list
_result_cache_black_list
_result_cache_latch_free_reads                     ADMIN
_result_cache_auto_time_distance                   300
_result_cache_auto_dml_monitoring_slots            4
_result_cache_auto_dml_monitoring_duration         15
_result_cache_auto_dml_threshold                   16
_result_cache_auto_dml_trend_threshold             20
_parallel_queuing_max_waitingtime
_cell_offload_expressions                          TRUE
_cell_materialize_virtual_columns                  TRUE
_cell_materialize_all_expressions                  FALSE
_cell_offload_sys_context                          TRUE
_gby_vector_aggregation_enabled                    TRUE
_key_vector_max_size                               0
_key_vector_predicate_enabled                      TRUE
_key_vector_predicate_threshold                    0
_key_vector_offload                                predicate
_vector_operations_control                         0
_vector_serialize_temp_threshold                   1000
_always_vector_transformation                      FALSE
_vector_aggregation_max_size                       0
_vector_dense_accum_max                            100
_key_vector_create_pushdown_threshold              20000
_key_vector_alternate_dictionary                   TRUE
_key_vector_force_alternate_dictionary             FALSE
_key_vector_shared_dgk_ht                          TRUE
_key_vector_double_enabled                         TRUE
_key_vector_timestamp_enabled                      TRUE
_ptf_max_rows                                      1024
_qesma_mvob_lru_sz                                 25
_qesma_bo_lru_sz                                   25
_qesmasc_trc                                       0
_ptf_enable_objects                                FALSE
_oss_skgxp_udp_dynamic_credit_mgmt
_oradbg_pathname
_shmprotect                                        0
_crash_domain_on_exception                         0
_online_patch_disable_stack_check                  FALSE
_vendor_lib_loc
_disable_sun_rsm                                   TRUE
_ipc_test_failover                                 0
_ipc_test_mult_nets                                0
_ipc_fail_network                                  0
_enable_event_ports                                TRUE
_event_port_opts
_enable_thr_kill                                   TRUE
_sem_per_semid
_ops_per_semop
_use_futex_ipc                                     FALSE
_posix_spawn_enabled                               TRUE
_datapump_inherit_svcname                          FALSE
_disable_streams_diagnostics                       0
_max_aq_persistent_queue_memory                    10
_no_objects                                        FALSE
_kokli_cache_size                                  128
_kokln_current_read                                FALSE
_domain_index_batch_size                           2000
_domain_index_dml_batch_size                       200
_odci_aggregate_save_space                         FALSE
_odci_index_pmo_rebuild                            FALSE
_nchar_imp_conv                                    TRUE
_insert_enable_hwm_brokered                        TRUE
_pgactx_cap_stacks                                 FALSE
_all_shared_dblinks
_disttxn_for_queries                               TRUE
_close_cached_open_cursors                         FALSE
_disable_savepoint_reset                           FALSE
_init_sql_file                                     ?/rdbms/admin/sql.bsq
_sort_multiblock_read_count                        2
_shrunk_aggs_enabled                               TRUE
_sql_ncg_mode                                      OFF
_cell_storidx_mode                                 EVA
_cell_storidx_minmax_enabled                       TRUE
_deferred_constant_folding_mode                    DEFAULT
_rdbms_internal_fplib_raise_errors                 FALSE
_rdbms_internal_fplib_enabled                      FALSE
_cell_index_scan_enabled                           TRUE
_cell_range_scan_enabled                           TRUE
_cell_offload_virtual_columns                      TRUE
_cell_offload_predicate_reordering_enabled         FALSE
_cell_offload_timezone                             TRUE
_direct_read_decision_statistics_driven            TRUE
_cell_offload_hybrid_processing                    TRUE
_cell_offload_complex_processing                   TRUE
_cell_offload_vector_groupby                       TRUE
_cell_offload_vector_groupby_force                 FALSE
_shrunk_aggs_disable_threshold                     60
_gby_onekey_enabled                                TRUE
_sql_connect_capability_table
_optimizer_mode_force                              TRUE
_explain_rewrite_mode                              FALSE
_query_rewrite_or_error                            FALSE
_sort_elimination_cost_ratio                       0
_grant_secure_role                                 FALSE
_sql_connect_capability_override                   0
_always_anti_join                                  CHOOSE
_optimizer_null_aware_antijoin                     TRUE
_optimizer_partial_join_eval                       TRUE
_partition_view_enabled                            TRUE
_always_star_transformation                        FALSE
_b_tree_bitmap_plans                               TRUE
_column_elimination_off                            FALSE
_cpu_to_io                                         0
_optimizer_extended_cursor_sharing                 UDO
_optimizer_extended_cursor_sharing_rel             SIMPLE
_optimizer_adaptive_cursor_sharing                 TRUE
_optimizer_cost_model                              CHOOSE
_optimizer_undo_cost_change                        18.1.0
_optimizer_system_stats_usage                      TRUE
_optimizer_cache_stats                             FALSE
_new_sort_cost_estimate                            TRUE
_complex_view_merging                              TRUE
_simple_view_merging                               TRUE
_unnest_subquery                                   TRUE
_optimizer_unnest_all_subqueries                   TRUE
_optimizer_unnest_scalar_sq                        TRUE
_eliminate_common_subexpr                          TRUE
_pred_move_around                                  TRUE
_convert_set_to_join                               FALSE
_px_bind_peek_sharing                              TRUE
_px_ual_serial_input                               TRUE
_px_minus_intersect                                TRUE
_remove_aggr_subquery                              TRUE
_distinct_view_unnesting                           FALSE
_optimizer_push_down_distinct                      0
_optimizer_cost_based_transformation               LINEAR
_optimizer_squ_bottomup                            TRUE
_optimizer_cbqt_factor                             50
_push_join_predicate                               TRUE
_push_join_union_view                              TRUE
_push_join_union_view2                             TRUE
_fast_full_scan_enabled                            TRUE
_optimizer_skip_scan_enabled                       TRUE
_optimizer_join_sel_sanity_check                   TRUE
_optim_enhance_nnull_detection                     TRUE
_enable_cscn_caching                               FALSE
_parallel_broadcast_enabled                        TRUE
_px_broadcast_fudge_factor                         100
_px_kxib_tracing                                   0
_px_granule_size                                   1000000
_px_chunk_size                                     1000000
_px_xtgranule_size                                 10000
_xtbigdata_max_buffersize                          10240
_xtbuffer_size                                     0
_px_min_granules_per_slave                         13
_px_split_batches_per_slave                        7
_px_split_use_single_list                          TRUE
_px_split_multi_msg                                TRUE
_px_max_granules_per_slave                         100
_px_no_stealing                                    FALSE
_px_no_granule_sort                                FALSE
_px_proc_constrain                                 TRUE
_px_freelist_latch_divisor                         2
_parallel_adaptive_max_users                       4
_parallel_cluster_cache_policy                     ADAPTIVE
_parallel_load_balancing                           TRUE
_parallel_load_bal_unit                            0
_parallel_slave_acquisition_wait                   1
_px_io_system_bandwidth                            0
_px_io_process_bandwidth                           200
_pdml_slaves_diff_part                             TRUE
_pdml_gim_sampling                                 5000
_pdml_gim_staggered                                FALSE
_px_dynamic_opt                                    TRUE
_px_dynamic_sample_size                            50
_px_rownum_pd                                      TRUE
_predicate_elimination_enabled                     TRUE
_groupby_nopushdown_cut_ratio                      3
_groupby_orderby_combine                           5000
_temp_tran_block_threshold                         100
_temp_tran_cache                                   TRUE
_ordered_semijoin                                  TRUE
_always_semi_join                                  CHOOSE
_ordered_nested_loop                               TRUE
_nested_loop_fudge                                 100
_project_view_columns                              TRUE
_no_or_expansion                                   FALSE
_optimizer_max_permutations                        2000
_system_index_caching                              0
_serial_direct_read                                auto
_disable_datalayer_sampling                        FALSE
_disable_sample_io_optim                           FALSE
_sample_rows_per_block                             4
_ncmb_readahead_enabled                            0
_ncmb_readahead_tracing                            0
_nlj_batching_enabled                              1
_nlj_batching_misses_enabled                       1
_ioq_fanin_multiplier                              2
_index_prefetch_factor                             100
_query_cost_rewrite                                TRUE
_query_rewrite_2                                   TRUE
_query_rewrite_1                                   TRUE
_query_rewrite_fudge                               90
_query_rewrite_expression                          TRUE
_query_rewrite_jgmigrate                           TRUE
_query_rewrite_fpc                                 TRUE
_query_rewrite_drj                                 FALSE
_query_rewrite_maxdisjunct                         257
_query_rewrite_vop_cleanup                         TRUE
_mmv_query_rewrite_enabled                         TRUE
_bt_mmv_query_rewrite_enabled                      TRUE
_add_stale_mv_to_dependency_list                   TRUE
_max_rwgs_groupings                                8192
_full_pwise_join_enabled                           TRUE
_partial_pwise_join_enabled                        TRUE
_slave_mapping_enabled                             TRUE
_slave_mapping_group_size                          0
_local_communication_costing_enabled               TRUE
_local_communication_ratio                         50
_parallelism_cost_fudge_factor                     350
_left_nested_loops_random                          TRUE
_improved_row_length_enabled                       TRUE
_px_object_sampling                                200
_px_object_sampling_multiplier                     2
_index_join_enabled                                TRUE
_use_nosegment_indexes                             FALSE
_enable_type_dep_selectivity                       TRUE
_improved_outerjoin_card                           TRUE
_optimizer_adjust_for_nulls                        TRUE
_optimizer_degree                                  0
_use_column_stats_for_function                     TRUE
_subquery_pruning_cost_factor                      20
_subquery_pruning_reduction                        50
_subquery_pruning_enabled                          TRUE
_subquery_pruning_mv_enabled                       FALSE
_parallel_txn_global                               FALSE
_or_expand_nvl_predicate                           TRUE
_like_with_bind_as_equality                        FALSE
_table_scan_cost_plus_one                          TRUE
_cost_equality_semi_join                           TRUE
_default_non_equality_sel_check                    TRUE
_new_initial_join_orders                           TRUE
_oneside_colstat_for_equijoins                     TRUE
_column_tracking_level                             21
_optim_peek_user_binds                             TRUE
_mv_refresh_selections                             TRUE
_cursor_plan_enabled                               TRUE
_minimal_stats_aggregation                         TRUE
_mv_complete_refresh_conventional                  FALSE
_mv_refresh_eut                                    TRUE
_mav_refresh_consistent_read                       TRUE
_mav_refresh_opt                                   0
_mav_refresh_unionall_tables                       3
_mv_refresh_delta_fraction                         10
_mv_expression_extend_size                         4096
_force_temptables_for_gsets                        FALSE
__pga_aggregate_target                             197132288
_pga_max_size                                      209715200
_smm_auto_min_io_size                              56
_smm_auto_max_io_size                              248
_smm_auto_cost_enabled                             TRUE
_smm_control                                       0
_smm_trace                                         0
_smm_min_size                                      189
_smm_max_size_static                               37888
_smm_px_max_size_static                            94720
_smm_max_size                                      37888
_smm_px_max_size                                   94720
_smm_retain_size                                   0
_smm_bound                                         0
_smm_advice_log_size                               0
_smm_advice_enabled                                TRUE
_gs_anti_semi_join_allowed                         TRUE
_mv_refresh_use_stats                              FALSE
_mv_refresh_use_no_merge                           TRUE
_mv_refresh_use_hash_sj                            TRUE
_no_stale_joinback_rewrite                         FALSE
_mv_refresh_no_idx_rebuild                         FALSE
_mv_deferred_no_log_age_val                        TRUE
_mv_add_log_placeholder                            TRUE
_optim_new_default_join_sel                        TRUE
_optimizer_dyn_smp_blks                            32
_pre_rewrite_push_pred                             TRUE
_optimizer_new_join_card_computation               TRUE
_mav_refresh_double_count_prevented                FALSE
_pct_refresh_double_count_prevented                TRUE
_mv_refresh_new_setup_disabled                     FALSE
_load_without_compile                              none
_precompute_gid_values                             TRUE
_union_rewrite_for_gs                              YES_GSET_MVS
_nested_mv_fast_oncommit_enabled                   TRUE
_generalized_pruning_enabled                       TRUE
_rowsource_execution_statistics                    FALSE
_rowsource_profiling_statistics                    TRUE
_rowsource_statistics_sampfreq                     128
_bitmap_or_improvement_enabled                     TRUE
_intrapart_pdml_enabled                            TRUE
_force_tmp_segment_loads                           FALSE
_force_slave_mapping_intra_part_loads              FALSE
_intrapart_pdml_randomlocal_enabled                TRUE
_optim_adjust_for_part_skews                       TRUE
_optimizer_compute_index_stats                     TRUE
_optimizer_autostats_job                           TRUE
_optimizer_push_pred_cost_based                    TRUE
_optimizer_extend_jppd_view_types                  TRUE
_optimizer_filter_pred_pullup                      TRUE
_optimizer_connect_by_cost_based                   TRUE
_optimizer_connect_by_combine_sw                   TRUE
_optimizer_connect_by_elim_dups                    TRUE
_connect_by_use_union_all                          TRUE
_force_datefold_trunc                              FALSE
_array_update_vector_read_enabled                  FALSE
_two_pass_reverse_polish_enabled                   TRUE
_expand_aggregates                                 TRUE
_dump_common_subexpressions                        FALSE
_spr_use_AW_AS                                     TRUE
_dump_connect_by_loop_data                         FALSE
_column_compression_factor                         0
_dml_monitoring_enabled                            TRUE
_fic_algorithm_set                                 automatic
_fic_area_size                                     131072
_fic_min_bmsize                                    1024
_dtree_area_size                                   131072
_dtree_pruning_enabled                             TRUE
_dtree_binning_enabled                             TRUE
_dtree_max_surrogates                              1
_right_outer_hash_enable                           TRUE
_optimizer_mjc_enabled                             TRUE
_optimizer_sortmerge_join_enabled                  TRUE
_cursor_bind_capture_area_size                     400
_flush_plan_in_awr_sql                             0
_cursor_bind_capture_interval                      900
_dump_cursor_heap_sizes                            FALSE
_projection_pushdown                               TRUE
_projection_pushdown_debug                         0
_px_compilation_debug                              0
_px_compilation_trace                              0
_trace_virtual_columns                             FALSE
_replace_virtual_columns                           TRUE
_virtual_column_overload_allowed                   TRUE
_kdt_buffering                                     TRUE
_disable_parallel_conventional_load                FALSE
_ltc_trace                                         0
_print_refresh_schedule                            false
_optimizer_undo_changes                            FALSE
_optimizer_percent_parallel                        101
_optimizer_search_limit                            5
_mv_refresh_costing                                rule
_mv_refresh_ana                                    0
_cache_stats_monitor                               FALSE
_sta_control                                       0
_enable_automatic_sqltune                          TRUE
_sql_analyze_enable_auto_txn                       FALSE
_ds_enable_auto_txn                                FALSE
_sql_analyze_parse_model                           2
_ds_parse_model                                    2
_kes_parse_model                                   2
_ds_iocount_iosize                                 6553664
_ds_xt_split_count                                 1
_ds_progressive_no_matches_min_sample_size         50
_ds_progressive_initial_samples                    2
_ds_sampling_method                                PROGRESSIVE
_ds_enable_view_sampling                           TRUE
_mv_refsched_timeincr                              300000
_spr_push_pred_refspr                              TRUE
_optimizer_block_size                              8192
_spr_max_rules                                     10000
_idxrb_rowincr                                     100000000
_hj_bit_filter_threshold                           50
_optimizer_save_stats                              TRUE
_optimizer_cost_filter_pred                        FALSE
_optimizer_correct_sq_selectivity                  TRUE
_allow_commutativity                               TRUE
_mv_refresh_rebuild_percentage                     10
_qa_control                                        0
_qa_lrg_type                                       0
_mv_refresh_force_parallel_query                   0
_enable_fast_ref_after_mv_tbs                      FALSE
_optim_dict_stats_at_db_cr_upg                     TRUE
_utlmmig_table_stats_gathering                     TRUE
_remove_exf_component                              TRUE
_optimizer_dim_subq_join_sel                       TRUE
_optimizer_disable_strans_sanity_checks            0
_allow_level_without_connect_by                    FALSE
_optimizer_random_plan                             0
_optimizer_ceil_cost                               TRUE
_delay_index_maintain                              TRUE
_query_rewrite_setopgrw_enable                     TRUE
_cursor_plan_hash_version                          1
_disable_function_based_index                      FALSE
_optimizer_invalidation_period                     18000
_px_net_msg_cost                                   10000
_px_loc_msg_cost                                   1000
_smm_freeable_retain                               5120
_cursor_plan_unparse_enabled                       TRUE
_kill_java_threads_on_eoc                          FALSE
_optimizer_join_order_control                      3
_px_nss_planb                                      TRUE
_bloom_filter_enabled                              TRUE
_bloom_filter_debug                                0
_bloom_filter_size                                 0
_bloom_predicate_enabled                           TRUE
_bloom_predicate_offload                           TRUE
_bloom_folding_enabled                             TRUE
_bloom_folding_density                             16
_bloom_folding_min                                 0
_bloom_pushing_max                                 512
_bloom_max_size                                    262144
_bloom_pushing_total_max                           262144
_bloom_minmax_enabled                              TRUE
_bloom_rm_filter                                   FALSE
_bloom_sm_enabled                                  TRUE
_bloom_serial_filter                               ON
_enable_refresh_schedule                           TRUE
_optimizer_cartesian_enabled                       TRUE
_optimizer_starplan_enabled                        TRUE
_optimizer_join_elimination_enabled                TRUE
_gby_hash_aggregation_enabled                      TRUE
_extended_pruning_enabled                          TRUE
_globalindex_pnum_filter_enabled                   TRUE
_sql_model_unfold_forloops                         RUN_TIME
_enable_dml_lock_escalation                        TRUE
_plsql_cache_enable                                TRUE
_disable_fast_aggregation                          FALSE
_disable_adaptive_shrunk_aggregation               FALSE
_plsql_minimum_cache_hit_percent                   20
_plan_outline_data                                 TRUE
_outline_bitmap_tree                               TRUE
_drop_table_optimization_enabled                   TRUE
_drop_table_granule                                256
_xpl_trace                                         0
_dm_max_shared_pool_pct                            1
_optimizer_multiple_cenv
_kql_subheap_trace                                 0
_optimizer_cost_hjsmj_multimatch                   TRUE
_optimizer_transitivity_retain                     TRUE
_px_pwg_enabled                                    TRUE
_sql_hash_debug                                    0
_mv_rolling_inv                                    FALSE
_smm_isort_cap                                     102400
_optimizer_cbqt_no_size_restriction                TRUE
_windowfunc_optimization_settings                  0
_truncate_optimization_enabled                     TRUE
_optimizer_enhanced_filter_push                    TRUE
_compilation_call_heap_extent_size                 16384
_xpl_peeked_binds_log_size                         8192
_rowsrc_trace_level                                0
_optimizer_rownum_pred_based_fkr                   TRUE
_optimizer_better_inlist_costing                   ALL
_optimizer_self_induced_cache_cost                 FALSE
_optimizer_min_cache_blocks                        10
_enable_exchange_validation_using_check            TRUE
_optimizer_or_expansion                            DEPTH
_optimizer_outer_to_anti_enabled                   TRUE
_optimizer_order_by_elimination_enabled            TRUE
_optimizer_star_tran_in_with_clause                TRUE
_optimizer_sortmerge_join_inequality               TRUE
_selfjoin_mv_duplicates                            TRUE
_dimension_skip_null                               TRUE
_force_rewrite_enable                              FALSE
_optimizer_complex_pred_selectivity                TRUE
_build_deferred_mv_skipping_mvlog_update           TRUE
_bloom_pruning_enabled                             TRUE
_query_mmvrewrite_maxpreds                         10
_query_mmvrewrite_maxintervals                     5
_query_mmvrewrite_maxinlists                       5
_query_mmvrewrite_maxdmaps                         10
_query_mmvrewrite_maxcmaps                         20
_query_mmvrewrite_maxregperm                       512
_row_shipping_threshold                            100
_row_shipping_explain                              FALSE
_query_mmvrewrite_maxqryinlistvals                 500
_first_k_rows_dynamic_proration                    TRUE
_optimizer_aw_stats_enabled                        TRUE
_enable_row_shipping                               TRUE
_optimizer_skip_scan_guess                         FALSE
_optimizer_distinct_elimination                    TRUE
_add_col_optim_enabled                             TRUE
_optimizer_multi_level_push_pred                   TRUE
_enable_ddl_wait_lock                              TRUE
_optimizer_group_by_placement                      TRUE
_optimizer_distinct_placement                      TRUE
_optimizer_coalesce_subqueries                     TRUE
_optimizer_enable_density_improvements             TRUE
_optimizer_rownum_bind_default                     10
_enable_query_rewrite_on_remote_objs               TRUE
_enable_scn_wait_interface                         TRUE
_optimizer_improve_selectivity                     TRUE
_optimizer_aw_join_push_enabled                    TRUE
_cvw_enable_weak_checking                          TRUE
_enable_online_index_without_s_locking             TRUE
_dbop_enabled                                      1
_sqlmon_threshold                                  5
_sqlmon_max_plan                                   40
_sqlmon_max_planlines                              300
_sqlmon_recycle_time                               5
_sqlmon_binds_xml_format                           default
_optimizer_native_full_outer_join                  FORCE
_optimizer_ansi_join_lateral_enhance               TRUE
_optimizer_multi_table_outerjoin                   TRUE
_optimizer_null_accepting_semijoin                 TRUE
_optimizer_ansi_rearchitecture                     TRUE
_optimizer_aggr_groupby_elim                       TRUE
_optimizer_enable_extended_stats                   TRUE
_direct_path_insert_features                       0
_optimizer_free_transformation_heap                TRUE
_pivot_implementation_method                       CHOOSE
_optimizer_use_subheap                             TRUE
_optimizer_or_expansion_subheap                    TRUE
_optimizer_star_trans_min_cost                     0
_optimizer_star_trans_min_ratio                    0
_with_subquery                                     OPTIMIZER
_optimizer_reuse_cost_annotations                  TRUE
_optimizer_interleave_jppd                         TRUE
_optimizer_fkr_index_cost_bias                     10
_px_dump_12805_source                              TRUE
_parallel_time_unit                                10
_parallel_scalability                              50
_parallel_syspls_obey_force                        TRUE
_optimizer_nested_rollup_for_gset                  100
_plan_verify_improvement_margin                    150
_statistics_based_srf_enabled                      TRUE
_selectivity_for_srf_enabled                       FALSE
_optimizer_extended_stats_usage_control            192
_sql_plan_management_control                       0
_optimizer_fast_pred_transitivity                  TRUE
_optimizer_fast_access_pred_analysis               TRUE
_optimizer_multiple_cenv_report                    result
_optimizer_multiple_cenv_stmt                      query
_parallel_cluster_cache_pct                        80
_px_granule_batch_size                             26
_optimizer_instance_count                          0
_block_sample_readahead_prob_threshold             10
_mv_generalized_oj_refresh_opt                     TRUE
_parallel_optimization_phase_for_local             FALSE
_enable_schema_synonyms                            FALSE
_enable_rename_user                                FALSE
_mv_refresh_enhanced_dml_detection                 ON_RC
_mv_refresh_pkfk_relationship_opt                  TRUE
_mv_refresh_pkfk_data_units_opt                    auto
_optimizer_unnest_disjunctive_subq                 TRUE
_optimizer_unnest_corr_set_subq                    TRUE
_optimizer_distinct_agg_transform                  TRUE
_aggregation_optimization_settings                 0
_arch_compress_checksums                           FALSE
_optimizer_eliminate_filtering_join                TRUE
_create_table_in_any_cluster                       FALSE
_dsc_feature_level                                 0
_parallel_statement_queuing                        FALSE
_parallel_conservative_queuing                     FALSE
_px_load_factor                                    300
_dml_frequency_tracking                            FALSE
_dml_frequency_tracking_advance                    TRUE
_dml_frequency_tracking_slots                      4
_dml_frequency_tracking_slot_time                  15
_px_gim_factor                                     100
_add_trim_for_nlssort                              TRUE
_optimizer_connect_by_cb_whr_only                  FALSE
_optimizer_join_factorization                      TRUE
_optimizer_force_CBQT
_optimizer_use_cbqt_star_transformation            TRUE
_rep_base_path
_optimizer_table_expansion                         TRUE
_and_pruning_enabled                               TRUE
_dml_batch_error_limit                             0
_noseg_for_unusable_index_enabled                  TRUE
_slave_mapping_skew_ratio                          2
_px_index_sampling_objsize                         TRUE
_sql_compatibility                                 0
_optimizer_use_feedback                            TRUE
_optimizer_gather_feedback                         TRUE
_optimizer_feedback_control
_optimizer_try_st_before_jppd                      TRUE
_parallel_blackbox_size                            16384
_parallel_blackbox_enabled                         TRUE
_parallel_load_publish_threshold                   10
_px_partition_scan_enabled                         TRUE
_px_partition_scan_threshold                       64
_parallel_heartbeat_snapshot_interval              2
_parallel_heartbeat_snapshot_max                   128
_optimizer_false_filter_pred_pullup                TRUE
_px_granule_randomize                              TRUE
_optimizer_purge_stats_iteration_row_count         10000
_mv_refresh_update_analysis                        TRUE
_mv_cleanup_orphaned_metadata                      TRUE
_mv_refresh_insert_no_append                       TRUE
_part_redef_global_index_update                    TRUE
_ptn_cache_threshold                               1
_optimizer_enable_table_lookup_by_nl               TRUE
_px_max_message_pool_pct                           40
_px_chunklist_count_ratio                          32
_px_hold_time                                      0
_optimizer_generate_transitive_pred                TRUE
_optimizer_cube_join_enabled                       TRUE
_optimizer_filter_pushdown                         TRUE
_dm_enable_legacy_dmf_output_types                 FALSE
_dm_dmf_details_compatibility                      12.1.0
_cdc_subscription_owner
_optimizer_outer_join_to_inner                     TRUE
_optimizer_hybrid_fpwj_enabled                     TRUE
_px_message_compression                            TRUE
_px_object_sampling_enabled                        TRUE
_px_concurrent                                     TRUE
_px_back_to_parallel                               OFF
_scalar_type_lob_storage_threshold                 4000
_px_replication_enabled                            TRUE
_px_round_robin_rowcnt                             1000
_zonemap_use_enabled                               TRUE
_zonemap_staleness_tracking                        1
_zonemap_control                                   0
_zonemap_usage_tracking                            TRUE
_sql_plan_directive_mgmt_control                   67
_optimizer_full_outer_join_to_outer                TRUE
_px_filter_parallelized                            TRUE
_px_filter_skew_handling                           TRUE
_px_groupby_pushdown                               FORCE
_partition_advisor_srs_active                      true
_px_parallelize_expression                         TRUE
_fast_index_maintenance                            TRUE
_optimizer_gather_stats_on_load                    TRUE
_multi_transaction_optimization_enabled            TRUE
_optimizer_batch_table_access_by_rowid             TRUE
_optimizer_ads_max_table_count                     0
_optimizer_ads_time_limit                          0
_optimizer_ads_use_result_cache                    TRUE
_optimizer_ads_result_cache_life                   3600
_optimizer_ads_use_spd_cache                       TRUE
_optimizer_ads_spd_cache_owner_limit               64
_px_wif_dfo_declumping                             CHOOSE
_px_wif_extend_distribution_keys                   TRUE
_px_wif_min_ndv_per_slave                          2
_px_join_skew_handling                             TRUE
_px_join_skew_ratio                                10
_px_join_skew_minfreq                              30
_px_join_skew_use_histogram                        TRUE
_px_join_skew_sampling_time_limit                  50
_px_join_skew_sampling_percent                     0
_px_adaptive_dist_method                           CHOOSE
_px_adaptive_dist_method_threshold                 0
_px_adaptive_dist_bypass_enabled                   TRUE
_px_adaptive_dist_bypass_optimization              1
_parallel_fault_tolerance_enabled                  FALSE
_parallel_fault_tolerance_threshold                3
_px_partial_rollup_pushdown                        ADAPTIVE
_sql_diag_repo_retain
_sql_diag_repo_origin                              all
_optimizer_dsdir_usage_control                     0
_px_cpu_autodop_enabled                            TRUE
_px_cpu_process_bandwidth                          200
_px_cpu_operator_bandwidth
_px_single_server_enabled                          TRUE
_optimizer_use_gtt_session_stats                   TRUE
_optimizer_adaptive_plan_control                   0
_optimizer_adaptive_random_seed                    0
_optimizer_strans_adaptive_pruning                 TRUE
_optimizer_proc_rate_level                         BASIC
_px_hybrid_TSM_HWMB_load                           TRUE
_optimizer_use_histograms                          TRUE
_disable_directory_link_check                      FALSE
_add_nullable_column_with_default_optim            TRUE
_emx_max_sessions                                  128
_emx_session_timeout                               3600
_emx_control                                       0
_parallel_blackbox_sga                             TRUE
_rowsets_enabled                                   TRUE
_rowsets_target_maxsize                            524288
_rowsets_max_rows                                  256
_rowsets_use_encoding                              TRUE
_rowsets_max_enc_rows                              64
_px_load_monitor_threshold                         10000
_px_monitor_load                                   FALSE
_px_numa_support_enabled                           FALSE
_px_numa_stealing_enabled                          TRUE
_px_adaptive_offload_threshold                     10
_px_adaptive_offload_percentage                    30
_modify_column_index_unusable                      FALSE
_adaptive_window_consolidator_enabled              TRUE
_sql_hvshare_threshold                             0
_px_tq_rowhvs                                      TRUE
_sql_hvshare_debug                                 1
_sql_show_expreval                                 FALSE
_parallel_ctas_enabled                             TRUE
_optimizer_performance_feedback                    OFF
_optimizer_proc_rate_source                        DEFAULT
_hashops_prefetch_size                             4
_stat_aggs_one_pass_algorithm                      FALSE
_px_onepass_slave_acquisition                      TRUE
_part_access_version_by_number                     TRUE
_online_ctas_diag                                  0
_upddel_dba_hash_mask_bits                         0
_px_pwmr_enabled                                   TRUE
_px_cdb_view_enabled                               TRUE
_optimizer_cluster_by_rowid                        TRUE
_optimizer_cluster_by_rowid_control                129
_partition_cdb_view_enabled                        TRUE
_common_data_view_enabled                          TRUE
_optimizer_cbqt_or_expansion                       ON
_pred_push_cdb_view_enabled                        TRUE
_rowsets_cdb_view_enabled                          TRUE
_distinct_agg_optimization_gsets                   CHOOSE
_array_cdb_view_enabled                            TRUE
_mv_refresh_pipe_timeout                           600
_mv_refresh_truncate_log                           FALSE
_rc_sys_obj_enabled                                TRUE
_px_scalable_invdist                               TRUE
_indexable_con_id                                  TRUE
_optimizer_reduce_groupby_key                      TRUE
_optimizer_cluster_by_rowid_batched                TRUE
_optimizer_cluster_by_rowid_batch_size             100
_object_link_fixed_enabled                         TRUE
_optimizer_synopsis_min_size                       2
_merge_monitor_threshold                           10000
_optimizer_inmemory_table_expansion                TRUE
_optimizer_inmemory_gen_pushable_preds             TRUE
_optimizer_inmemory_autodop                        TRUE
_optimizer_inmemory_access_path                    TRUE
_optimizer_inmemory_quotient                       0
_optimizer_inmemory_pruning_ratio_rows             100
_parallel_inmemory_min_time_threshold              AUTO
_parallel_inmemory_time_unit                       1
_px_external_table_default_stats                   TRUE
_optimizer_nlj_hj_adaptive_join                    TRUE
_px_autodop_pq_overhead                            TRUE
_px_overhead_init_slavealloc                       1000
_px_overhead_init_endpoints                        4
_px_overhead_exec_ctrlmesg                         4
_px_overhead_teardown                              600
_grant_read_instead_of_select                      FALSE
_optimizer_inmemory_bloom_filter                   TRUE
_optimizer_inmemory_cluster_aware_dop              TRUE
_optimizer_inmemory_minmax_pruning                 TRUE
_cdb_cross_container                               65535
_cdb_view_parallel_degree                          65535
_optimizer_hll_entry                               4096
_cross_con_row_count                               FALSE
_cdb_view_prefetch_batch_size                      200
_cdb_view_rc_shelflife                             30
_approx_cnt_distinct_gby_pushdown                  choose
_approx_cnt_distinct_optimization                  0
_px_cdb_view_join_enabled                          TRUE
_external_table_smart_scan                         HADOOP_ONLY
_optimizer_db_blocks_buffers                       0
_composite_interval_partition_creation_opt_enabled TRUE
_optimizer_bushy_join                              off
_optimizer_bushy_fact_dim_ratio                    20
_optimizer_bushy_fact_min_size                     100000
_optimizer_bushy_cost_factor                       100
_query_rewrite_use_on_query_computation            TRUE
_mv_access_compute_fresh_data                      ON
_px_min_ndv_per_slave_perc_func                    2
_px_scalable_invdist_mcol                          TRUE
_rmt_for_table_redef_enabled                       TRUE
_query_execution_time_limit                        0
_optimizer_ads_use_partial_results                 TRUE
_optimizer_eliminate_subquery                      TRUE
_sqlexec_hash_based_distagg_enabled                FALSE
_sqlexec_disable_hash_based_distagg_tiv            FALSE
_sqlexec_hash_based_distagg_ssf_enabled            TRUE
_sqlexec_distagg_optimization_settings             0
_optimizer_union_all_gsets                         TRUE
_optimizer_enhanced_join_elimination               TRUE
_optimizer_multicol_join_elimination               TRUE
_optimizer_enable_plsql_stats                      TRUE
_recursive_with_parallel                           TRUE
_recursive_with_using_temp_table                   FALSE
_recursive_with_branch_iterations                  7
_recursive_with_control                            0
_partition_read_only                               TRUE
_px_dist_agg_partial_rollup_pushdown               ADAPTIVE
_mv_refresh_shrink_log                             TRUE
_pwise_distinct_enabled                            TRUE
_approx_perc_sampling_err_rate                     2
_vector_encoding_mode                              MANUAL
_optimizer_use_table_scanrate                      HADOOP_ONLY
_optimizer_use_xt_rowid                            TRUE
_xt_sampling_scan_granules                         ON
_xt_sampling_scan_granules_min_granules            1
_xt_def_compression_ratio                          4
_xt_legacy_debug_flags                             0
_sqlexec_use_rwo_aware_expr_analysis               TRUE
_monitor_workload_interval                         24
_optimizer_band_join_aware                         TRUE
_optimizer_vector_transformation                   TRUE
_optimizer_vector_fact_dim_ratio                   10
_optimizer_vector_min_fact_rows                    10000000
_optimizer_key_vector_aggr_factor                  75
_optimizer_vector_cost_adj                         100
_optimizer_key_vector_pruning_enabled              TRUE
_optimizer_vector_base_dim_fact_factor             200
_approx_percentile_optimization                    0
_optimizer_adaptive_plans_continuous               FALSE
_optimizer_adaptive_plans_iterative                FALSE
_optimizer_generate_ptf_implied_preds              TRUE
_optimizer_inmemory_capture_stored_stats           TRUE
_optimizer_inmemory_use_stored_stats               AUTO
_shard_sql_use_chunk_ids                           0
_expression_tracking_latch_count                   16
_expression_tracking_elem_per_bucket               30
_bloom_filter_ratio                                35
_optimizer_control_shard_qry_processing            65528
_optimizer_interleave_or_expansion                 TRUE
_px_partition_load_dist_threshold                  64
_optimizer_use_feedback_for_join                   FALSE
_optimizer_ads_for_pq                              FALSE
_px_slave_parse_first_with_outline_hints           TRUE
_px_join_skewed_values_count                       0
_bloom_max_wait_time                               50
_bloom_wait_on_rac                                 FALSE
_bloom_extent_size                                 0
_read_optimized_table_lookup                       TRUE
_optimizer_key_vector_payload                      TRUE
_optimizer_vector_fact_payload_ratio               20
_bloom_pruning_setops_enabled                      TRUE
_bloom_filter_setops_enabled                       TRUE
_px_pwise_wif_enabled                              TRUE
_sqlexec_reorder_wif_enabled                       TRUE
_px_partition_skew_threshold                       80
_sqlexec_pwiseops_with_sqlfuncs_enabled            TRUE
_sqlexec_pwiseops_with_binds_enabled               TRUE
_px_shared_hash_join                               FALSE
_px_reuse_server_groups                            MULTI
_px_join_skew_null_handling                        TRUE
_px_tq_memcpy_threshold                            100
_bloom_use_shared_pool                             FALSE
_enable_parallel_dml                               FALSE
_px_nlj_bcast_rr_threshold                         10
_sqlexec_use_kgghash3                              TRUE
_optimizer_gather_stats_on_load_all                FALSE
_optimizer_gather_stats_on_load_hist               FALSE
_px_granule_alignment                              1024
_optimizer_allow_all_access_paths                  TRUE
_lob_use_locator_varying_width                     FALSE
_optimizer_answering_query_using_stats             FALSE
_dmm_ipp_cutoff                                    0
_dmm_nobin_threshold                               200
_dmm_force_treetop_merge                           0
_dmm_cnt_arr_size_threshold                        500000
_dmm_sample_lower_threshold                        10000
_dmm_sample_upper_threshold                        500000
_dmm_auto_max_features                             500
_dmm_max_memory_size                               1000000
_dmm_memory_size                                   64000000
_dmsqr_qr_chunk_rows                               10000
_dm_max_chunk_size                                 2000
_dm_inmemory_threshold                             1000000
_bigram_dependency_percentage                      5
_dmm_inc_cholesky_rows                             50000
_dmm_kmean_dense_threshold                         500
_dmm_blas_library
_dmm_ts_lapack                                     1
_dmm_reduction_rate                                4
_dmm_pga_load_threshold                            3
_alter_upgrade_signature_only                      FALSE
_spadr                                             YES
_sec_enable_test_rpcs                              FALSE
_use_zero_copy_io                                  TRUE
_tstz_localtime_bypass                             FALSE
_client_tstz_error_check                           TRUE
_drain_on_ping_database                            TRUE
_client_result_cache_bypass                        FALSE
_client_result_cache_ramthreshold
_emon_regular_ntfn_slaves                          4
_emon_outbound_connect_timeout                     7200000
_emon_send_timeout                                 7200000
_emon_max_active_connections                       256
_client_ntfn_pingtimeout                           30000
_client_ntfn_pinginterval                          75
_client_ntfn_pingretries                           6
_client_enable_auto_unregister                     FALSE
_srvntfn_q_msgcount                                50
_srvntfn_q_msgcount_inc                            100
_srvntfn_jobsubmit_interval                        3
_srvntfn_max_concurrent_jobs                       20
_srvntfn_job_deq_timeout                           60
_client_ntfn_cleanup_interval                      2400
_max_clients_per_emon                              8
_emon_pool_inc                                     4
_emon_pool_min                                     4
_emon_pool_max                                     16
_tsm_connect_string
_sscr_dir
_sscr_osdir
_tsm_disable_auto_cleanup                          1
_enable_nativenet_tcpip                            FALSE
_async_scn_sync                                    OFF
_share_drcp_proxy_sessions                         TRUE
_request_boundaries                                1
_spfoc                                             OFF
_spfdc                                             OFF
_spfei
_close_deq_by_cond_curs                            FALSE
_deq_maxwait_time                                  0
_deq_max_fetch_count                               10
_deq_execute_reset_time                            30
_aq_streaming_threshold                            10485760
_aq_dly_bkt                                        2
_aq_retry_timeouts                                 0
_aq_scrambled_deqlog                               1
_aq_dqblocks_in_cache                              0
_aq_uncached_stats                                 0
_aq_free_list_pools                                10
_aq_tm_statistics_duration                         0

P_NAME                                             P_VALUE
-------------------------------------------------- --------------------------------------------------
_orph_cln_interval                                 1200
_aq_max_scan_delay                                 1500
_aq_tm_scanlimit                                   0
_aq_tm_deqcountinterval                            0
_disable_gvaq_cache                                FALSE
_rule_max_dnfp_cnt                                 1024
_re_fast_sql_operator                              all
_re_result_cache_keysiz                            20
_re_result_cache_size                              20
_re_independent_expression_cache_size              20
_enable_iee_stats                                  TRUE
_re_num_complex_operator                           1000
_re_num_rowcache_load                              2
_prop_old_enabled                                  FALSE
_bufq_stop_flow_control                            FALSE
_capture_publisher_flow_control_threshold          0
_buffered_publisher_flow_control_threshold         0
_buffered_message_spill_age                        300
_deq_log_array_size                                10000
_deq_ht_max_elements                               100000
_deq_ht_child_latches                              8
_deq_large_txn_size                                25000
_aqsharded_cache_limit                             0
_aq_Txn_ht_sz                                      1024
_aq_shard_bitmap_child_latches                     32
_aq_shard_retry_child_latches                      32
_aq_shard_txn_child_latches                        128
_aq_shard_sub_child_latches                        512
_aq_shard_sub_child_Elem_latches                   1024
_aq_shard_child_latches                            512
_aq_shard_prty_latches                             16
_aq_init_shards                                    5
_aq_x_mode                                         1
_aq_latency_relative_threshold                     100
_aq_latency_absolute_threshold                     300
_aq_max_pdb_close_msg                              1
_aq_lb_subht_bkt_ltch                              32
_aq_lb_subht_elm_ltch                              128
_aq_lb_garbage_col_interval                        600
_aq_x_msg_size                                     32768
_aq_stop_backgrounds                               FALSE
_aq_lb_cycle                                       120
_aq_lb_stats_collect_cycle                         45
_aq_pt_processes                                   10
_aq_subshard_Size                                  20000
_aq_subshards_per_qpartition                       1
_aq_subshards_per_dqpartition                      1
_aq_lookback_size                                  60
_aq_qt_prefetch_Size                               5
_aq_dq_prefetch_Siz                                5
_aq_pt_statistics_window                           60
_aq_pt_shrink_frequency                            1450
_aq_addpt_batch_size                               1
_aq_truncpt_batch_size                             1
_aq_droppt_batch_size                              5
_shrd_que_tm_processes                             1
_shrd_que_tm_statistics_duration                   0
_aq_ipc_max_slave                                  10
_aq_opt_stat_window                                21600
_aq_opt_stop_stat                                  FALSE
_aq_opt_enabled                                    TRUE
_aq_opt_background_enabled                         TRUE
_aq_opt_fudge_factor                               500
_aq_opt_prefetch_horizon                           60
_aq_opt_min_evict_memory                           0
_aq_opt_prefetch_dop                               3
_aq_opt_preevict_dop                               3
_ctx_doc_policy_stems                              FALSE
_nonce_history_buffer_size                         0
_dg_broker_trace_level
_olapi_history_retention                           FALSE
_olapi_session_history                             300
_olapi_session_history_retention                   FALSE
_olapi_iface_object_history                        1000
_olapi_iface_object_history_retention              FALSE
_olapi_interface_operation_history                 1000
_olapi_iface_operation_history_retention           FALSE
_olapi_memory_operation_history                    1000
_olapi_memory_operation_history_retention          FALSE
_olapi_memory_operation_history_pause_at_seqno     0
_olap_continuous_trace_file                        false
_olap_table_function_statistics                    FALSE
_olap_parallel_update_threshold                    1000
_olap_parallel_update_small_threshold              1000
_olap_parallel_update_server_num                   0
_olap_aggregate_buffer_size                        1048576
_olap_aggregate_min_buffer_size                    1024
_olap_aggregate_work_per_thread                    1024
_olap_aggregate_min_thread_status                  64
_olap_aggregate_statlen_thresh                     1024
_olap_aggregate_worklist_max                       5000
_olap_aggregate_max_thread_tuples                  5000
_olap_aggregate_function_cache_enabled             TRUE
_olap_aggregate_multipath_hier                     FALSE
_olap_aggregate_flags                              0
_olap_allocate_errorlog_header                     Dim      Source   Basis
                                                   %-8d %-8s %-8b Description
                                                   -------- -------- -------- -----------

_olap_allocate_errorlog_format                     %8p %8y %8z %e (%n)
_olap_dbgoutfile_echo_to_eventlog                  FALSE
_olap_eif_export_lob_size                          2147483647
_olap_sort_buffer_size                             262144
_olap_sort_buffer_pct                              10
_olap_sesscache_enabled                            TRUE
_olap_object_hash_class                            3
_olap_dimension_corehash_size                      30
_olap_dimension_corehash_pressure                  90
_olap_dimension_corehash_large                     50000
_olap_dimension_corehash_force                     FALSE
_olap_page_pool_low                                262144
_olap_page_pool_hi                                 50
_olap_page_pool_expand_rate                        20
_olap_page_pool_shrink_rate                        50
_olap_page_pool_hit_target                         100
_olap_page_pool_pressure                           90
_olap_statbool_threshold                           8100
_olap_statbool_corebits                            20000000
_olap_lmgen_dim_size                               100
_olap_lmgen_meas_size                              1000
_olap_wrap_errors                                  FALSE
_olap_analyze_max                                  10000
_olap_adv_comp_stats_max_rows                      100000
_olap_adv_comp_stats_cc_precomp                    20
_olap_row_load_time_precision                      DEFAULT
_olap_disable_loop_optimized                       FALSE
_xsolapi_fetch_type                                PARTIAL
_xsolapi_dimension_group_creation                  OVERFETCH
_xsolapi_sql_auto_measure_hints                    TRUE
_xsolapi_sql_auto_dimension_hints                  FALSE
_xsolapi_sql_hints
_xsolapi_sql_measure_hints
_xsolapi_sql_dimension_hints
_xsolapi_sql_top_measure_hints
_xsolapi_sql_top_dimension_hints
_xsolapi_sql_all_non_base_hints
_xsolapi_sql_all_multi_join_non_base_hints
_xsolapi_densify_cubes                             TABULAR
_xsolapi_sql_optimize                              TRUE
_xsolapi_sql_remove_columns                        TRUE
_xsolapi_sql_symmetric_predicate                   TRUE
_xsolapi_sql_use_bind_variables                    TRUE
_xsolapi_sql_prepare_stmt_cache_size               16
_xsolapi_sql_result_set_cache_size                 32
_xsolapi_sql_minus_threshold                       1000
_xsolapi_debug_output                              SUPPRESS
_xsolapi_materialize_sources                       TRUE
_xsolapi_load_at_process_start                     NEVER
_xsolapi_fix_vptrs                                 TRUE
_xsolapi_auto_materialization_type                 PRED_AND_RC
_xsolapi_auto_materialization_bound                20
_xsolapi_materialization_rowcache_min_rows_for_use 1
_xsolapi_source_trace                              FALSE
_xsolapi_dml_trace
_xsolapi_build_trace                               FALSE
_xsolapi_metadata_reader_mode                      DEFAULT
_xsolapi_odbo_mode                                 FALSE
_xsolapi_set_nls                                   TRUE
_xsolapi_stringify_order_levels                    FALSE
_xsolapi_suppression_chunk_size                    4000
_xsolapi_suppression_aw_mask_threshold             1000
_xsolapi_share_executors                           TRUE
_xsolapi_hierarchy_value_type                      unique
_xsolapi_use_models                                TRUE
_xsolapi_use_olap_dml                              TRUE
_xsolapi_use_olap_dml_for_rank                     TRUE
_xsolapi_remove_columns_for_materialization        TRUE
_xsolapi_precompute_subquery                       TRUE
_xsolapi_optimize_suppression                      TRUE
_xsolapi_generate_with_clause                      FALSE
_xsolapi_sql_enable_aw_join                        TRUE
_xsolapi_sql_enable_aw_qdr_merge                   TRUE
_xsolapi_opt_aw_position                           TRUE
_xsolapi_support_mtm                               FALSE
_asm_runtime_capability_volume_support             FALSE
_asm_disable_multiple_instance_check               FALSE
_asm_disable_amdu_dump                             FALSE
_asmsid                                            asm
_asm_global_dump_level                             267
_remote_asm
_ios_root_directory                                IOS
_apx_root_directory                                APX
_asm_node_site_guid
_asm_reloc_cic                                     FALSE
_asm_oda_type
_asm_allow_system_alias_rename                     FALSE
_asm_instlock_quota                                0
_asm_relocation_trace                              FALSE
_asm_disk_repair_time                              14400
_asm_disable_profilediscovery                      FALSE
_asm_relocation_ignore_hard_failure                0
_asm_max_parallelios                               256
_asm_auto_online_interval
_asm_imbalance_tolerance                           3
_asm_shadow_cycle                                  3
_asm_primary_load_cycles                           TRUE
_asm_primary_load                                  1
_asm_secondary_load_cycles                         FALSE
_asm_secondary_load                                10000
_kffmop_chunks                                     42
_kffmap_hash_size                                  1024
_kffmop_hash_size                                  2048
_kffmlk_hash_size                                  512
_asm_diskgroups2
_asm_diskgroups3
_asm_diskgroups4
_asm_zero_power_limit
_disable_rebalance_space_check                     FALSE
_asm_log_scale_rebalance                           FALSE
_asm_sync_rebalance                                FALSE
_diag_arb_before_kill                              FALSE
_asm_ausize                                        1048576
_asm_blksize                                       4096
_asm_acd_chunks                                    1
_asm_partner_target_disk_part                      8
_asm_partner_target_fg_rel                         4
_asm_automatic_rezone                              TRUE
_asm_rebalance_plan_size                           120
_asm_rebalance_space_errors                        4
_asm_relocation_scheme                             alloc_p2 alloc_s3 reb_p2 reb_s1 bal_p2 bal_s3 prep
                                                   _p2 prep_s3

_asm_disable_dangerous_failgroup_checking          FALSE
_asm_rebalance_estimates_process                   TRUE
_asm_disable_failgroup_size_checking               FALSE
_asm_disable_failgroup_count_checking              FALSE
_asm_allow_foreign_siteguids
_asm_allow_dgname_special_chars                    FALSE
_asm_libraries                                     ufs
_asm_maxio                                         1048576
_asm_allow_only_raw_disks                          TRUE
_asm_disable_vtoc_check                            FALSE
_asm_fob_tac_frequency                             9
_asm_emulate_nfs_disk                              FALSE
_asm_allow_lvm_resilvering                         TRUE
_asm_lsod_bucket_size                              67
_asm_iostat_latch_count                            31
_asm_diskerr_traces                                2
_asm_procs_trace_diskerr                           5
_asm_trace_limit_timeout                           30000
_asm_fd_cln_on_fg                                  TRUE
_asm_fd_cln_idle_sess_twait                        60000000
_asm_skip_dbfile_ios                               FALSE
_asm_offload_all                                   FALSE
_asm_read_cancel                                   AUTO
_asm_read_cancel_back_out                          5000
_asm_write_cancel                                  AUTO
_asm_cancel_delta                                  75000
_asm_cancel_alert_time                             600
_asm_enable_kfks                                   FALSE
_kfm_disable_set_fence                             FALSE
_asm_disable_smr_creation                          FALSE
_afd_disable_fence                                 FALSE
_asm_network_timeout                               1
_asm_wait_time                                     18
_asm_skip_diskval_check                            FALSE
_asm_skip_resize_check                             FALSE
_asm_skip_rename_check                             FALSE
_asm_direct_con_expire_time                        120
_asm_check_for_misbehaving_cf_clients              FALSE
_asm_diag_dead_clients                             FALSE
_asm_disable_ufg_dump                              FALSE
_asm_reserve_slaves                                TRUE
_asm_kill_unresponsive_clients                     TRUE
_asm_disable_async_msgs                            FALSE
_asm_remote_client_timeout                         300
_asm_allow_unsafe_reconnect                        TRUE
_asm_disable_ufgmemberkill                         FALSE
_asm_disable_proact_client_cleanup                 FALSE
_asm_nodekill_escalate_time                        180
_asm_healthcheck_timeout                           180
_asm_cclient_cleanup_timeout                       300
_asm_allow_older_clients                           FALSE
_asm_enable_multiple_asmb                          FALSE
_asm_disable_vxn_map_messages                      FALSE
_asm_asmb_rcvto                                    10
_asm_asmb_max_wait_timeout                         6
_asm_tcp_user_timeout                              1
_asm_resyncCkpt                                    1024
_asm_relocation_async_lock_count                   128
_asm_stripewidth                                   8
_asm_stripesize                                    131072
_disable_fastopen                                  FALSE
_asm_random_zone                                   FALSE
_asm_serialize_volume_rebalance                    FALSE
_asm_force_quiesce                                 FALSE
_asm_dba_threshold                                 0
_asm_dba_batch                                     500000
_asm_force_paritycheck_rebalance                   FALSE
_asm_dba_spcchk_thld                               100000
_asm_usd_batch                                     64
_asm_fail_random_rx                                FALSE
_relocation_commit_batch_size                      8
_asm_max_redo_buffer_size                          2097152
_asm_max_cod_strides                               10
_asm_max_aux_cods                                  5
_asm_evenread                                      2
_asm_evenread_alpha                                0
_asm_evenread_alpha2                               0
_asm_evenread_faststart                            0
_asm_noevenread_diskgroups
_asm_disable_request_tracer                        TRUE
_asm_force_parity_extent_check                     FALSE
_asm_enable_kfios                                  FALSE
_asm_networks
_asm_access_assume_local                           FALSE
_asm_iosconnect_timeout                            0
_asm_ios_network_domains                           0
_asm_ios_network_processes                         0
_asm_netp_factor                                   0
_asm_idn_processes                                 0
_asm_iowp_max_async                                0
_asm_max_clients                                   1000
_asm_netp_iosize                                   0
_asm_dbmsdg_nohdrchk                               FALSE
_asm_root_directory                                ASM
_asm_pstonpartners                                 TRUE
_asm_allowdegeneratemounts                         FALSE
_asm_hbeatiowait                                   120
_asm_hbeatwaitquantum                              2
_asm_repairquantum                                 60
_asm_emulmax                                       10000
_asm_emultimeout                                   0
_asm_kfdpevent                                     0
_asm_storagemaysplit                               FALSE
_asm_avoid_pst_scans                               TRUE
_disable_storage_type                              FALSE
_asm_min_compatibility                             11.2.0.2
_asm_compatibility                                 11.2.0.2
_rdbms_compatibility                               10.1
_kfi_version_patchmap
_kfi_software_patchmap
_asm_disable_patch_compat                          FALSE
_asm_proxy_startwait                               60
_asm_allow_dangerous_unprotected_volumes           FALSE
_asm_proxy_online_restart                          0
_allow_cell_smart_scan_attr                        TRUE
_asm_admin_with_sysdba                             FALSE
_asm_allow_appliance_dropdisk_noforce              FALSE
_disable_appliance_check                           FALSE
_disable_appliance_partnering                      FALSE
_asm_appliance_config_file
_asm_appliance_ignore_oak                          FALSE
_dirty_appliance_mode                              FALSE
_asm_appliance_slot_from_path                      FALSE
_asm_appliance_disable_fg_check                    FALSE
_asm_write_badfdata_in_contentcheck                TRUE
_asm_scrub_disable_cod                             FALSE
_asm_scrub_strict                                  FALSE
_asm_scrub_async                                   TRUE
_asm_scrub_limit                                   AUTO
_asm_scrub_unmatched_dba                           1024
_asm_enable_parity_scrub                           FALSE
_asm_async_scrub_reap_wait                         100000
_asm_enable_batch_scrub                            FALSE
_memory_max_tgt_inc_cnt                            0
_asm_enable_xrov                                   FALSE
_asm_xrov_single                                   FALSE
_asm_xrov_rsnmod                                   1
_asm_xrov_nvios                                    8
_asm_xrov_nstats                                   0
_skip_acfs_checks                                  FALSE
_asm_force_vam                                     FALSE
_usd_recent_read                                   TRUE
_usd_preload_blks                                  4
_asm_dependency_under_cfenqueue                    TRUE
_asm_max_connected_clients                         3
_asm_enable_parityfile_creation                    FALSE
_asm_enable_parity_datafile_creation               FALSE
_enable_multiple_fgprepares                        FALSE
_enable_single_dgprepare                           FALSE
_asm_allow_prepare_split                           TRUE
_enable_ios_spm                                    FALSE
_alert_expiration                                  604800
_alert_message_cleanup                             1
_alert_message_purge                               1
_alert_post_background                             1
_swrf_test_action                                  0
_sysaux_test_param                                 1
_swrf_mmon_flush                                   TRUE
_remote_awr_enabled                                FALSE
_awr_corrupt_mode                                  FALSE
_awr_restrict_mode                                 FALSE
_swrf_mmon_metrics                                 TRUE
_swrf_metric_frequent_mode                         FALSE
_awr_flush_threshold_metrics                       TRUE
_awr_flush_workload_metrics                        FALSE
_awr_disabled_flush_tables
_awr_disabled_purge_tables
_awr_snapshot_level                                BESTFIT
_swrf_on_disk_enabled                              TRUE
_awr_pdb_registration_enabled                      FALSE
_swrf_mmon_dbfus                                   TRUE
_awr_mmon_cpuusage                                 TRUE
_swrf_test_dbfus                                   FALSE
_adr_migrate_runonce                               TRUE
_awr_sql_child_limit                               200
_awr_enable_pdb_snapshots                          TRUE
_awr_incremental_flush_enabled                     TRUE
_awr_mmon_deep_purge_interval                      7
_awr_mmon_deep_purge_extent                        7
_awr_mmon_deep_purge_numrows                       5000
_awr_mmon_deep_purge_all_expired                   FALSE
_awr_cdbperf_threshold                             21
_awr_partition_interval                            0
_incremental_purge_size                            200
_awr_metrics_use_mmnl                              FALSE
_sqltune_category_parsed                           DEFAULT
_ash_sampling_interval                             1000
_ash_size                                          1048618
_ash_enable                                        TRUE
_ash_disk_write_enable                             TRUE
_ash_disk_filter_ratio                             10
_ash_eflush_trigger                                66
_ash_sample_all                                    FALSE
_ash_dummy_test_param                              0
_ash_min_mmnl_dump                                 90
_ash_compression_enable                            TRUE
_ash_progressive_flush_interval                    300
_kebm_nstrikes                                     3
_kebm_suspension_time                              104400
_kebm_sanity_check_enabled                         FALSE
_timemodel_collection                              TRUE
_disable_metrics_group                             0
_kewm_simulate_oer4031                             0
_enable_metrics_allpdb                             TRUE
_enable_metrics_pdb                                FALSE
_validate_metric_groups                            FALSE
_track_metrics_memory                              TRUE
_kewm_trace_sga                                    FALSE
_threshold_alerts_enable                           1
_enable_default_undo_threshold                     TRUE
_enable_default_temp_threshold                     TRUE
_addm_auto_enable                                  TRUE
_addm_version_check                                TRUE
_addm_skiprules
_automatic_maintenance_test                        0
_autotask_min_window                               15
_autotask_max_window                               480
_enable_automatic_maintenance                      1
_autotask_test_name                                N/A
_autotask_test_action                              0
_bsln_adaptive_thresholds_enabled                  TRUE
_wcr_control                                       0
_dbreplay_feature_control
_capture_buffer_size                               65536
_wcr_seq_cache_size                                65535
_wcr_grv_cache_size                                65535
_wcr_test_action                                   0
_max_queued_report_requests                        300
_report_capture_cycle_time                         60
_report_capture_dbtime_percent_cutoff              50
_report_capture_timeband_length                    1
_report_capture_recharge_window                    10
_max_report_flushes_percycle                       5
_report_request_ageout_minutes                     60
_kecap_cache_size                                  10240
_rtaddm_trigger_enabled                            TRUE
_rtaddm_trigger_args
_mwin_schedule                                     TRUE
_umf_remote_target_dblink
_umf_remote_enabled                                FALSE
_dbmsumf$$nn
_dbmsumf$$p
_dbmsumf$$1x
_dbmsumf$$2x
_dbmsumf$$3x
_umf_test_action                                   0
_iut_enable                                        TRUE
_iut_max_entries                                   30000
_iut_stat_collection_type                          SAMPLED
_sqlset_hash_max_size                              100000
_component_timing
_diag_verbose_error_on_init                        0
_diag_hm_rc_enabled                                TRUE
_diag_hm_tc_enabled                                FALSE
_diag_adr_enabled                                  TRUE
_diag_adr_auto_purge                               TRUE
_diag_backward_compat                              TRUE
_diag_adr_test_param                               0
_diag_adr_trace_dest                               /oracle/diag/rdbms/ora18c/ora18c/trace
_diag_pdb_purge_threshold                          97
_diag_pdb_purge_target                             90
_diag_pdb_control                                  0
_dra_enable_offline_dictionary                     FALSE
_dra_bmr_number_threshold                          1000
_dra_bmr_percent_threshold                         10
_diag_conf_cap_enabled                             TRUE
_diag_patch_cap_enabled                            TRUE
_log_segment_dump_parameter                        TRUE
_log_segment_dump_patch                            TRUE
_diag_adl_dyn_alloc                                TRUE
_diag_cdb_logging                                  short
_diag_alert_root_annotate                          FALSE
_dde_flood_control_init                            TRUE
_diag_dde_fc_enabled                               TRUE
_diag_dde_fc_implicit_time                         0
_diag_dde_fc_macro_time                            0
_diag_cc_enabled                                   TRUE
_diag_dde_inc_proc_delay                           1
_diag_dde_async_msgs                               50
_diag_dde_async_msg_capacity                       1024
_diag_dde_async_slaves                             5
_diag_dde_async_mode                               1
_diag_dde_async_age_limit                          300
_diag_dde_async_process_rate                       5
_diag_dde_async_runtime_limit                      900
_diag_dde_async_cputime_limit                      300
_diag_dde_enabled                                  TRUE
_trace_files_public                                FALSE
_max_incident_file_size
_uts_trace_segment_size                            0
_uts_trace_segments                                5
_uts_first_segment_size                            0
_uts_first_segment_retain                          TRUE
_diag_uts_control                                  0
_uts_trace_disk_threshold                          0
_diag_test_seg_reinc_mode                          FALSE
_uts_trace_files_nopurge                           FALSE
_uts_trace_buffer_size                             0
_uts_inc_inmem_trace                               0
_uts_enable_alltrc_stats                           FALSE
_trace_pool_size
_evt_system_event_propagation                      TRUE
_diag_enable_startup_events                        FALSE
_auto_manage_exadata_disks                         TRUE
_auto_manage_ioctl_bufsz                           8192
_auto_manage_num_tries                             2
_auto_manage_enable_offline_check                  TRUE
_auto_manage_max_online_tries                      3
_auto_manage_online_tries_expire_time              86400
_auto_manage_num_pipe_msgs                         1000
_auto_manage_infreq_tout                           0
_kxdbio_ctx_init_count                             32
_kxdbio_offena_timeout                             7200000
_kxdbio_disable_offload_opcode                     0
_kxdbio_enable_ds_opcode                           0
_enable_offloaded_writes                           FALSE
_block_level_offload_high_lat_thresh               40000
_kxdbio_hca_loadavg_thresh                         74
_kxdbio_ut_ctl                                     0
_cell_offload_backup_compression                   TRUE
_enable_pluggable_database                         FALSE
_oracle_script                                     FALSE
_pdb_first_script                                  FALSE
_discard_cmn_ddl_in_pdb_err                        FALSE
_pluggable_database_debug                          0
_pdb_failure_testing                               0
_cdb_compatible                                    TRUE
_deferred_seg_in_seed                              TRUE
_common_user_prefix                                C##
_relocate_pdb                                      FALSE
_set_container_service                             DEFAULT
_multiple_char_set_cdb                             TRUE
_cdb_spfile_inherit                                FALSE
_enable_pdb_close_abort                            TRUE
_enable_pdb_close_noarchivelog                     FALSE
_pdb_cluster_database                              TRUE
_enable_pdb_isolation                              FALSE
_auto_dismount_on_pdb_close                        FALSE
_pdb_mask_cdb_info                                 FALSE
_split_file_copy                                   TRUE
_test_offload_pdb_sga_init                         FALSE
_save_afns_on_subset_pdb_creation                  FALSE
_multiple_name_convert_patterns                    FALSE
_pdb_hash_table_size                               255
_pdb_hash_table_latches                            16
_pdb_lrg_auto_undots_create                        FALSE
_pdb_auto_undots_create_off                        FALSE
_proxy_connect_after_set_container                 TRUE
_pdb_seed_mcsc                                     FALSE
_pdb_strict_plugin_compat                          FALSE
_enable_pdb_process_limit                          TRUE
_pdb_inherit_cfd                                   FALSE
_auto_start_pdb_services                           FALSE
_seed_root_undo_ratio                              30
_cdb_disable_pdb_limit                             FALSE
_slow_kill_on_pdb_close_immediate                  FALSE
_restrict_local_user_dml                           TRUE
_pdb_max_diag_size                                 0
_pdb_max_audit_size                                0
_reuse_dropped_pdbid_time                          180
_pdb_auto_save_state                               FALSE
_next_pdbid                                        3
_cloud_service_type
_pdb_ignore_table_clauses                          TRUE
_no_catalog
_non_app_ignore_errors                             TRUE
_app_ignore_errors                                 TRUE
_app_default_containers                            FALSE
_exclude_seed_cdb_view                             TRUE
_disable_cdb_view_rc_invalidation                  FALSE
_object_linked_remote                              FALSE
_cdb_rac_affinity                                  TRUE
_cdb_view_recursive_px_enabled                     TRUE
_cdb_view_no_skip_migrate                          FALSE
_cdb_special_old_xplan                             TRUE
_disable_con_recurse_queuing                       TRUE
_disable_dblink_optim                              TRUE
_cdb_view_no_skip_restricted                       FALSE
_intra_cdb_dblink                                  FALSE
_partition_by_con_name                             FALSE
_error_row_predicate_evaluation                    AUTO
_cross_con_collection                              FALSE
_enable_view_pdb                                   TRUE
_enable_guid_endpoint_service                      TRUE
_enable_proxy_adg_redirect                         FALSE
_federation_max_root_clones                        5
_application_purge_enabled                         TRUE
_federation_script                                 FALSE
_application_script                                FALSE
_enable_partial_sync                               FALSE
_enable_drop_clone                                 FALSE
_disable_destructive_patch_operation               TRUE
_upgrade_optim                                     TRUE
_upgrade_capture_noops                             TRUE
_app_replay_silent_errors                          FALSE
_capture_pgadep                                    0
_sync_app_pdb_first_open                           FALSE
_disable_ptl_replay                                FALSE
_enable_module_match                               TRUE
_lrgdbcz6_ignore_restrictions                      FALSE
_enable_system_app                                 2
_modify_other_app_object                           FALSE
_enable_cdb_upgrade_capture                        FALSE
_skip_app_object_check                             FALSE
_session_aware_replay                              TRUE
_strict_dml_data_link                              FALSE
_skip_app_unconverted_check                        FALSE
_enable_auto_upgrade                               FALSE
_app_container_debug                               0
_enable_pmo_outside_begin_end                      TRUE
_record_module_name                                TRUE
_no_snapshot_root_clone                            FALSE
_root_clone_state_from_root                        TRUE
_apppdb_multi_slave_sync                           TRUE
_pdb_ldp_cascade                                   0
_pdb_lockdown_bypass_sr                            FALSE
_enable_containers_subquery                        TRUE
_force_containers_subquery                         FALSE
_cdb_fleet_sync_timeout                            10
_cdb_port                                          0
_pdb_isolation_class                               NONE
_con_map_sql_enforcement                           TRUE
_uniq_cons_sql_enforcement                         TRUE
_ref_cons_sql_enforcement                          TRUE
_xt_http_wscl                                      FALSE
_gsm
_gsm_region_list
_cloud_name
_dbpool_name
_region_name
_db_num_gsm                                        0
_gsm_config_vers                                   0
_gsm_max_instances_per_db                          8
_gsm_max_num_regions                               10
_gsm_drv_interval                                  30
_gsm_srlat_thresh                                  20
_gsm_cpu_thresh                                    75
_gsm_thresh_zone                                   10
_gsm_thresh_respct                                 50
_gsm_svcrgnmax                                     100
_gds_chunk_num                                     0
_gds_max_chunk_num                                 0
_gds_allow_nullkey                                 0
_gds_lddlid                                        0
_shardgroup_name
_gwm_db_unique_name
_gds_shardgroup_id                                 0
_shardspace_name
_shd_reptype                                       0
_gws_deployed                                      0
_gws_sharding_method                               0
_gws_cache_version                                 0
_gwm_database_flags
_user_defined_sharding                             0
_dupt_noupdate                                     0
_shd_atomic_move                                   0
_gwm_spare1
_gwm_spare2                                        0
_gwm_spare3                                        0
_rq_shm_max_size                                   1024
_hcs_perf                                          FALSE
_hcs_all_distinct                                  FALSE
_hcs_no_exists_distinct                            FALSE
_hcs_no_filter_hierarchies                         FALSE
_hcs_no_hier_join_map_nonnull                      FALSE
_hcs_no_level_ord                                  FALSE
_hcs_no_level_prune                                FALSE
_hcs_no_level_prune_gby                            FALSE
_hcs_no_level_prune_hier_join                      FALSE
_hcs_no_level_prune_hier_qry                       FALSE
_hcs_no_level_prune_hierarchize                    FALSE
_hcs_no_level_prune_in_qry                         FALSE
_hcs_no_level_prune_mbr_lookup                     FALSE
_hcs_no_level_prune_vis_lvs                        FALSE
_hcs_logging                                       SUMMARY
_hcs_no_materialize                                FALSE
_hcs_no_opt_estimate                               FALSE
_hcs_parallel_hint                                 FALSE
_hcs_no_result_cache_hint                          FALSE
_hcs_no_smooth_descendants                         FALSE
_hcs_no_smooth_drill_all                           FALSE
_hcs_no_smooth_hierarchize                         FALSE
_hcs_no_smooth_remove_all                          FALSE
_hcs_no_sql_pred_push                              FALSE
_hcs_no_vis_totals                                 FALSE
_hcs_ignore_unsupported_calcs                      FALSE
_hcs_query_hint
_hcs_no_bd_agg_opt                                 FALSE
_hcs_no_sp_jback_opt                               FALSE
_hcs_no_av_jback_opt                               FALSE
_hcs_no_all_prune                                  FALSE
_hcs_avcache_dynamic_enabled                       FALSE
_hcs_no_hord_in_oby_prune                          FALSE
_hcs_no_jback_opt_for_hord_in_oby                  FALSE
_hcs_no_mv_rewrite_check                           FALSE
_hcs_no_cell_qry_tmpls                             FALSE
_hcs_no_opt_cell_qry                               FALSE
_hcs_no_cell_qry_meas_prune_opt                    FALSE
_hcs_no_cell_qry_lvl_prune_opt                     FALSE
_hcs_no_cell_qry_no_calc_nav_opt                   FALSE
_hcs_no_cell_qry_mv_cache_opt                      FALSE
_hcs_ignore_latest_compat_check                    FALSE
_hcs_no_col_prune_optz                             FALSE
_hcs_no_rm_unused_withs_optz                       FALSE
_hcs_no_rm_like_withs_optz                         FALSE
_hcs_use_dynamic_all_cache                         FALSE
_hcs_parse_dynamic_all_cache                       FALSE
_hcs_no_mdx_cache_hint                             FALSE
_hcs_mdx_cache_name_no_sid                         FALSE
_hcs_mdx_cache_name_col                            FALSE
_hcs_use_multi_parent_gen                          FALSE
_hcs_mdx_sleep_after_pin                           FALSE
_hcs_stats_max_card                                2000000
_hcs_no_cell_qry_no_out_data_opt                   FALSE
_hcs_no_inline_tmpl_opt                            FALSE
_hcs_no_rmv_unref_top_opt                          FALSE
_hcs_no_cell_qry_atr_prune_opt                     FALSE
_hcs_no_calc_dtm_to_out_opt                        FALSE
_hcs_no_fltr_hier_star_opt                         FALSE
_hcs_no_fltr_fact_opt                              FALSE
_hcs_gen_aggr_opt_estimate                         FALSE
_hcs_no_tgt_depths_opt                             FALSE
_hcs_in_mem_cdt_hint                               FALSE
_hcs_expose_with_expr                              FALSE
_hcs_opt_av_pred_push                              TRUE
_reg_cache_status                                  FALSE

4723 rows selected.

朋友一套数据库版本10.2.0.5 on HPUX 11， 源库是noarchivelog read-only database open mode时备份，在目标库（其它主机）还原时遇到 ora-1861, 错误很简单就是遇到了不标准的时间日期字符串，如2月30日，4月31日等。

RMAN> run
{
allocate channel t1 type 'SBT_TAPE'
parms 'ENV=(NSR_SERVER=crmbeif,NSR_DATA_VOLUME_POOL=yjboss,NSR_CLIENT=yjbossdb)';
...
restore database;
release channel t1;
release channel t2;
release channel t3;

}

ORA-01861: literal does not match format string

首先定位递归SQL, 使用了系统级errorstack trace
启用

SQL> alter system set events='1861 trace name errorstack level 10';

— 运行上面的rman脚本，直至报出ora-1861错误

关闭

SQL> alter system set events='1861 trace name errorstack OFF';

查看trace file

*** ACTION NAME:(0000001 FINISHED70) 2018-08-03 17:26:31.746
*** MODULE NAME:(rman@rzcrmdb1 (TNS V1-V3)) 2018-08-03 17:26:31.746
*** SERVICE NAME:() 2018-08-03 17:26:31.746
*** SESSION ID:(10981.9) 2018-08-03 17:26:31.746
*** 2018-08-03 17:26:31.746
ksedmp: internal or fatal error
ORA-01861: literal does not match format string
Current SQL statement for this session:
select fhscn ,to_date(fhtim,'MM/DD/RR HH24:MI:SS','NLS_CALENDAR=Gregorian') ,fhcrs ,fhrls 
,to_date(fhrlc,'MM/DD/RR HH24:MI:SS','NLS_CALENDAR=Gregorian') ,fhafs ,fhrfs ,fhrft ,hxerr 
,fhfsz ,fhsta into :b1,:b2:b3,:b4,:b5,:b6:b7,:b8:b9,:b10:b11,:b12:b13,:b14,:b15,:b16 
 from x$kcvfhall where hxfil=:b17
----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
ksedst()+64          call     ksedst1()            000000000 ? 000000001 ?
ksedmp()+2176        call     ksedst()             000000000 ?
                                                   C000000000000D20 ?
                                                   4000000004043C80 ?
                                                   000000000 ? 000000000 ?
                                                   000000000 ?
$cold_ksddoa()+1424  call     ksedmp()             00000000A ?
                                                   9FFFFFFFFFFF3720 ?
                                                   60000000000BA270 ?
                                                   9FFFFFFFFFFF3CF0 ?
                                                   C000000000000F24 ?
                                                   4000000003609750 ?
ksdpcg()+576         call     $cold_ksddoa()       9FFFFFFFFFFF3D28 ?
                                                   9FFFFFFFBF3A6730 ?
                                                   9FFFFFFFFFFF3D00 ?
                                                   60000000000BA270 ?
ksdpec()+336         call     ksdpcg()             9FFFFFFFBF3B0160 ?
                                                   9FFFFFFFBF3B0170 ?
                                                   000000000 ?
ksfpec()+320         call     ksdpec()             9FFFFFFFFFFF48C0 ?
kgesev()+320         call     ksfpec()             C000000000000491 ?
                                                   000000745 ?
                                                   40000000030E4BB0 ?
                                                   000000000 ? 000000000 ?
                                                   000000000 ? 000000000 ?
                                                   000000000 ?
ksesec0()+160        call     kgesev()             60000000000318D0 ?
                                                   6000000000379320 ?
                                                   000000745 ?
                                                   6000000000032D00 ?
                                                   6000000000032CF0 ?
dteerr()+128         call     ksesec0()            000000745 ? 000000000 ?
                                                   000000000 ?
ldxerr()+144         call     dteerr()             9FFFFFFFFFFF4930 ?
                                                   000000745 ?
$cold_ldxstd()+64    call     ldxerr()             9FFFFFFFFFFF4930 ?
                                                   000000745 ?
                                                   60000000000BA270 ?
                                                   C000000000000593 ?
                                                   4000000002038020 ?
                                                   9FFFFFFFFFFF4940 ?
                                                   9FFFFFFFFFFF4A18 ?
                                                   000000000 ?
evadsi()+1152        call     $cold_ldxstd()       9FFFFFFFFFFF4930 ?
                                                   9FFFFFFFBF303390 ?
                                                   9FFFFFFFBF30282C ?
                                                   9FFFFFFFBF362AA0 ?
                                                   000000013 ?
                                                   C0000002CDDBC6A8 ?
evaopn2()+7104       call     evadsi()             C0000002CDDBCD88 ?
                                                   60000000000BA270 ?
                                                   C0000000000011AB ?
                                                   4000000002F5EAB0 ?
                                                   00002C263 ? 000000008 ?
                                                   9FFFFFFFBF302AB8 ?
                                                   C0000002CDDBC6A8 ?
evaopn2()+9472       call     evaopn2()            9FFFFFFFBEBA0088 ?
                                                   9FFFFFFFFFFF4C80 ?
                                                   60000000000BA270 ?
                                                   9FFFFFFFFFFF5240 ?
                                                   C0000000000011AB ?
                                                   4000000002F5F3F0 ?
                                                   00002C2A1 ? 000000000 ?
opifcr()+1456        call     evaopn2()            9FFFFFFFBE912CF0 ?
                                                   9FFFFFFFFFFF5270 ?
                                                   60000000000BA270 ?
                                                   9FFFFFFFFFFF5830 ?
                                                   C0000000000011AB ?
                                                   4000000002F38E90 ?
                                                   00002C261 ? 000000000 ?
qerfxFetch()+1328    call     opifcr()             9FFFFFFFFFFF6490 ?
                                                   000000002 ?
                                                   60000000000BA270 ?
                                                   C000000000001736 ?
                                                   4000000002F78D80 ?
                                                   000028427 ?
                                                   9FFFFFFFBEBA0334 ?
                                                   000000000 ?
opifch2()+6176       call     qerfxFetch()         C0000002CDDBD520 ?
                                                   4000000001CBE7D0 ?
                                                   9FFFFFFFFFFF6490 ?
                                                   000000002 ?
                                                   60000000000BA270 ?
                                                   C0000000000021CB ?
                                                   4000000002F59740 ?
                                                   000028223 ?
opiefn0()+672        call     opifch2()            9FFFFFFFFFFF7450 ?
                                                   400000000316BE00 ?
                                                   00002021F ?
                                                   9FFFFFFFFFFF6300 ?
                                                   60000000000BA270 ?
                                                   C000000000000F26 ?
                                                   6000000000031A50 ?
                                                   040202D89 ?
opiefn()+304         call     opiefn0()            C00000000000038F ?
                                                   9FFFFFFFBF3B1C4A ?
                                                   9FFFFFFFFFFF74A8 ?
                                                   000000000 ? 000000000 ?
                                                   000000000 ? 000000000 ?
                                                   000000000 ?
opiodr()+2144        call     opiefn()             00000004E ? 000000004 ?
                                                   9FFFFFFFFFFF9C30 ?
                                                   C0000000000018B7 ?
                                                   4000000002F441C0 ?
                                                   9FFFFFFFFFFF7558 ?
ttcpip()+1680        call     opiodr()             00000004E ? 000000004 ?
                                                   4000000001C03410 ?
                                                   0000046C0 ?
                                                   9FFFFFFFFFFF7560 ?
opitsk()+2368        call     ttcpip()             600000000003DF40 ?
                                                   000000001 ?
                                                   9FFFFFFFFFFF9C30 ?
                                                   000000000 ?
                                                   9FFFFFFFFFFF9DA0 ?
                                                   9FFFFFFFFFFF9B94 ?
                                                   4000000001CEBE50 ?
                                                   000000000 ?
opiino()+1664        call     opitsk()             000000000 ? 000000000 ?
                                                   60000000000BA270 ?
                                                   4000000002944200 ?
                                                   000028089 ?
                                                   4000000001C03428 ?
opiodr()+2144        call     opiino()             00000003C ?
                                                   9FFFFFFFFFFFC5F0 ?
                                                   9FFFFFFFFFFFED90 ?
                                                   9FFFFFFFFFFFBAB0 ?
                                                   60000000000BA270 ?
                                                   C0000000000018B7 ?
opidrv()+1248        call     opiodr()             00000003C ? 000000004 ?
                                                   4000000001C03140 ?
                                                   0000046C0 ?
                                                   9FFFFFFFFFFFC600 ?
                                                   60000000000BA270 ?
sou2o()+240          call     opidrv()             00000003C ?
                                                   60000000000C6CA8 ?
                                                   9FFFFFFFFFFFED90 ?
opimai_real()+496    call     sou2o()              9FFFFFFFFFFFEDB0 ?
                                                   00000003C ? 000000004 ?
                                                   9FFFFFFFFFFFED90 ?
main()+240           call     opimai_real()        000000000 ?
                                                   9FFFFFFFFFFFEDE0 ?
main_opd_entry()+80  call     main()               000000002 ?
                                                   9FFFFFFFFFFFF290 ?
                                                   60000000000BA270 ?
                                                   C000000000000004 ?
******************** Session Cursor Dump **********************

============
Plan Table
============
-----------------------------------------------------+-----------------------------------+
| Id  | Operation                | Name              | Rows  | Bytes | Cost  | Time      |
-----------------------------------------------------+-----------------------------------+
| 0   | SELECT STATEMENT         |                   |       |       |       |           |
| 1   |  FIXED TABLE FIXED INDEX | X$KCVFHALL (ind:1)|       |       |       |           |
-----------------------------------------------------+-----------------------------------+
 
Content of other_xml column
===========================
  db_version     : 10.2.0.5
  parse_schema   : SYS
  plan_hash      : 1567485612
  Outline Data:
  /*+
    BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('10.2.0.5')
      OPT_PARAM('_optim_peek_user_binds' 'false')
      OPT_PARAM('optimizer_dynamic_sampling' 1)
      OPT_PARAM('_gby_hash_aggregation_enabled' 'false')
      RBO_OUTLINE
      OUTLINE_LEAF(@"SEL$1")
      FULL(@"SEL$1" "X$KCVFHALL"@"SEL$1")
    END_OUTLINE_DATA
  */


----------------------------------------
Cursor#11(9fffffffbf3b1c48) state=FETCH curiob=9fffffffbeba0088
 curflg=66 fl2=0 par=0000000000000000 ses=c0000002f3d3c320
 sqltxt(c0000002fde2f128)=
select fhscn ,to_date(fhtim,'MM/DD/RR HH24:MI:SS','NLS_CALENDAR=Gregorian') ,fhcrs ,
fhrls ,to_date(fhrlc,'MM/DD/RR HH24:MI:SS','NLS_CALENDAR=Gregorian') ,fhafs ,fhrfs ,
fhrft ,hxerr ,fhfsz ,fhsta into :b1,:b2:b3,:b4,:b5,:b6:b7,:b8:b9,:b10:b11,:b12:b13,:b14,:b15,:b16  
from x$kcvfhall where hxfil=:b17
  hash=e780e1d2a3462ef7d179ac375eb3ad69
  parent=c0000002cef3fb90 maxchild=01 plk=c0000002cfa94c38 ppn=n
cursor instantiation=9fffffffbeba0088 used=1533288391
 child#0(c0000002fde2eef8) pcs=c0000002cef3f160
  clk=c0000002cfa94cf0 ci=c0000002cef3ee88 pn=c0000002fdbe0920 ctx=c0000002cddbed70
 kgsccflg=0 llk[9fffffffbeba0090,9fffffffbeba0090] idx=0
 xscflg=c0110c76 fl2=5440001 fl3=42222008 fl4=100
 Bind bytecodes
  Opcode = 2   Bind Twotask Scalar Sql In (may be out) Copy 
  oacdef = c0000002cef3f0b0   Offsi = 48, Offsi = 0
kkscoacd
 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000000 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=9fffffffbf10d5a8  bln=22  avl=02  flg=05
  value=47
 Frames pfr 9fffffffbf11fef8 siz=3344 efr 9fffffffbf11ff60 siz=3320

Note:
确认是在查询x$kcvfhall 时触发，SQL文本内有2个TO_DATE转换时间的字段,所以问题就出在fhtim或fhrlc. 如果找到那条记录就可以看到那个错误值 。

KCVFHALL ==》[K]enel [C]ache Reco[V]ery Component [F]ile [H]eaders [ALL]

关于这个X$KCVFHALL这是个数据文件头的视图，  没有找到太多信息，应该是recovery 时在获取数据文件头部的信息，如scn, 可以使用下面的脚本确认问题记录

set serveroutput on 

Declare 
  LIT_MISMATCH EXCEPTION;
  PRAGMA EXCEPTION_INIT(LIT_MISMATCH, -1861);
  CURSOR BP IS select hxfil,FHFNO,HXFNM, fhtim ,  fhrlc  , FHTNM  from x$kcvfhall;
  RECID NUMBER;
  D1 DATE;
  D2 DATE;
BEGIN
   <>
   FOR BPREC IN BP LOOP
     BEGIN
       select to_date(fhtim,'MM/DD/RR HH24:MI:SS') , to_date(fhrlc,'MM/DD/RR HH24:MI:SS') INTO D1,D2  from x$kcvfhall where hxfil=BPREC.hxfil;
      EXCEPTION
       WHEN LIT_MISMATCH THEN
       DBMS_OUTPUT.PUT_LINE('hxfil = ' || BPREC.hxfil || ' Error');
       DBMS_OUTPUT.PUT_LINE('HXFNM = ' || BPREC.HXFNM);
     END;
   END LOOP BP_REC;
END;
/

Note:

hxfil 是datafile_id, xhfnam是datafile name, 同样也可以去v$datafile里验证， 如果数据文件头上有错误的检查点时间也就意味着数据文件头上有错误数据对于这个平台上的checkpoint，首先要确认数据的错误程度，可以dump 数据文件头或dd.

Headers dump all data files: file_hdrs event
level 1 – represents the control file dump all the data file header.
level 2 – dump all data files represent a common header -generic header.
level 3 – level2 and contains header information in the datafile
level 10 – represent a complete file header dump all data files.

这里使用dump 数据文件头的命令：

SQL> alter session set events 
   'immediate trace name FILE_HDRS level 10';

要确认在源库和目标库之间没有经过其它平台的数据转换， 确认是否源库也存在这些错误记录，如果源库没有，目标库各别存在，可以尝试重新备份部分文件。
如果源库也存在问题，建议收集所有的参数文件和alert中的报错给ORACLE SR确认。
如果源库没有问题，目标库使用的RAW裸设置，建议用0x00清空所有ORACLE数据文件的前64K,如下：

$ dd if=/dev/zero of=/dev/....filename...   bs=8k count=8 conv=notrunc

建议对于利旧或者其它新加入数据库的设备，使用前用zero(0x00)去清空一下数据文件头，总是一个比较好的方法！

With regular logical  IOs the buffer contents are not read while holding the CBC(Cache Buffer Chian) latch:

1.  1#  Take CBC latch into shared mode
2.  2# Walk the buffer hash chain until you find the relevant buffer header
3.  3# Upgrade the CBC latch to Exclusive mode
4.  4# Pin the buffer header
5.  5 # Release the CBC latch
6.  6 # Now access the buffer data( call transaction,  data layer etc)   — if someone else wants to pin the buffer now , they’d wait for buffer busy waits
7.  7# Take the CBC latch again(in shared mode)
8.  8$ Unpin the buffer header
9.  10# Release the CBC latch

Sometimes “short” logical IOs can skip a few steps

with “short” LIOs like unique index lookup LIO(etc) Oracle can avoid the buffer pinning codepath:

1.  1# Take CBC altch in shared mode
2.  2# Walk the buffer hash chain until you find the relevant buffer header — This show up as consistent reads – examination  conter in v$sesstat
3.  3# Now access the buffer data
4.  4# Release the CBC latch

from TanelPoder’s