从Oracle 19.10 及更高版本中提供的sequence新增强功能“Sequence dynamic cache resizing ”， 在Oracle中如果应用中频繁的调用sequence的nextval值是可能存在性能问题，如enq:SQ 或row cache lock,  之前在一个客户的业务系统有一个非常糟糕的设计全库使用一个sequence, 所以及时cache值调到了2万依旧在高并发调用时sequence性能问题还是非常明显。

默认sequence cache通常为20， 我还是建议创建时配置cache 200起步。

该功能默认启用，不需要 DBA 或最终用户进行额外设置。此功能根据sequence的消耗率动态调整sequence cache的大小。因此，sequence cache大小可以远远超过配置的 DDL 缓存大小。这意味着cache可以随时间自动增长和缩小，具体取决于使用情况，同时永远不会低于 DDL 指定的缓存大小。

动态缓存大小调整可以显着提高使用序列的快速插入工作负载的性能。这是通过减少补充cache缓存所需的磁盘访问次数来实现的，这在 Oracle RAC 环境中尤其重要。如果在 RAC 设置中跨节点使用序列，这可能会导致跨实例看到的sequence存在较大差距。

禁用方法

_dynamic_sequence_cache = FALSE;

相关参数（虽然文档中是19.10引入该物性，但是19.3就已经引入了该参数）

SQL> @pd dynamic_sequ
Show all parameters and session values from x$ksppi/x$ksppcv...

       NUM N_HEX NAME                                                     VALUE                          DESCRIPTION
---------- ----- -------------------------------------------------------- ------------------------------ ---------------------------------------------------------------------------------------------------
      2882   B42 _dynamic_sequence_cache                                  TRUE                           Enable/disable sequence dynamic cache sizing
      2883   B43 _dynamic_sequence_cache_scale                            10                             Max scale factor for dynamic sequence cache size
      2884   B44 _dynamic_sequence_cache_max                              1000000                        Sequence dynamic cache maximum size

在Oracle Online文档的monitor database 章节中有描述Oracle 数据库可以自动跟踪某些SCHEMA的对象（例如表和物化视图）的活动和使用情况。

对象活动跟踪系统 (OATS) 跟踪与数据库对象相关的各种活动。可以在数据库级别和可插入数据库 (PDB) 级别执行跟踪。跟踪的活动包括对表的 DML 操作、表和分区扫描、分区维护操作 (PMOP)、物化视图重写和刷新以及索引等辅助结构的使用。统计信息用于支持automated database 功能，例如自动物化视图、Access Advisors。使用 DBMS_ACTIVITY 包中的过程和函数来控制 OATS 捕获的信息。

要启用对象活动跟踪系统：
将 STATISTICS_LEVEL 初始化参数设置为 TYPICAL 或 ALL。

可以在以下数据字典视图中查看 OATS 跟踪的统计信息：

SQL> @d DBA_ACTIVITY
Show data dictionary views and x$ tables matching the expression "DBA_ACTIVITY"...

TABLE_NAME                     COMMENTS
------------------------------ --------------------------------------------------------------------------------
DBA_ACTIVITY_CONFIG            Object Activity Tracking Configuation
DBA_ACTIVITY_MVIEW             Object Activity Tracking for Materialized Views
DBA_ACTIVITY_SNAPSHOT_META     Object Activity Tracking Metadata for Snapshot Intervals
DBA_ACTIVITY_TABLE             Object Activity Tracking for Tables

SQL> @st DBA_ACTIVITY_CONFIG

CON_DBNAME                     PARAMETER_NAME                 PARAMETER_ LAST_UPDATED_TIME                        LAST_UPDAT
------------------------------ ------------------------------ ---------- ---------------------------------------- ----------
CDB$ROOT                       ACTIVITY_INTERVAL_MINUTES      15         27-JUL-21 07.16.07.000 PM                SYS
CDB$ROOT                       ACTIVITY_RETENTION_DAYS        400        27-JUL-21 07.16.07.000 PM                SYS
CDB$ROOT                       ACTIVITY_SPACE_PERCENT         5          27-JUL-21 07.16.07.000 PM                SYS
CDB$ROOT                       ACTIVITY_SYS_TABLES            NO         27-JUL-21 07.16.07.000 PM                SYS

SQL> desc DBMS_ACTIVITY
PROCEDURE CONFIGURE
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 PARAMETER_NAME                 VARCHAR2                IN
 PARAMETER_VALUE                VARCHAR2                IN
 CON_DBNAME                     VARCHAR2                IN     DEFAULT
FUNCTION CREATE_SNAPSHOT RETURNS NUMBER
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 ALL_INSTANCES                  BOOLEAN                 IN     DEFAULT
 CON_DBNAME                     VARCHAR2                IN     DEFAULT
FUNCTION DELETE_SNAPSHOTS RETURNS BOOLEAN
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 BEFORE_SNAP_ID                 NUMBER                  IN
 CON_DBNAME                     VARCHAR2                IN     DEFAULT
FUNCTION DELETE_SNAPSHOTS RETURNS BOOLEAN
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 BEFORE_TIME                    TIMESTAMP               IN
 CON_DBNAME                     VARCHAR2                IN     DEFAULT

The database objects whose activities are tracked include tables and materialized views. Different types of activities include row insert, delete and update, table scan, load and truncate, partition maintenance operations (create, drop, move, split, merge, and exchange), materialized view rewrite and refresh.

跟踪类型
精确跟踪：以 100% 的准确度记录活动。在多处理器环境中需要某种形式的原子更新或某种锁存来获得精确的跟踪。 OATS 通常避免精确跟踪，除非客户需要那种精确度。
近似跟踪：几乎在所有情况下都会考虑一项活动。近似跟踪的一个例子是活动计数器的脏更新。不执行锁定或闩锁，并且在更改计数器时不使用原子操作。但是，脏更新会导致更新丢失，从而导致计数不足。
Probabilistic Tracking：一个活动以一定的概率被采样，当它成为样本的一部分时就被考虑在内。概率跟踪产生的活动数据不太准确，但会阻止频繁的跟踪操作
发生跟踪：记录某个时间间隔内发生的某些活动。具体而言，事件跟踪提供的信息表明特定对象活动在给定时间间隔内至少发生了一次，而无需说明它发生了多少次。
事务和回滚的计数器效果：除了精确跟踪计数器外，其他类型的计数器都不是事务性的或受回滚影响的。这意味着如果交易失败，近似计数器可能仍会记录来自该交易的任何操作。

在oracle database 21c版本开始materialized views可以自动的创建和维护，用于自冶数据库。 物化视图提供了显着提高查询性能的潜力，但需要大量的努力和技能来确定要使用的物化视图。结合了工作负载监控来确定需要哪些物化视图。基于决定，物化视图和物化视图日志被自动创建和维护，无需任何手动交互。对于复杂查询SQL的查询块(QB)，MVIEW查询重写技术可以优化此类查询，基于扩展覆盖子表达式算法(ECSE),权衡生成等价、超/子/交/合集的预查询结果。支持分区和非分区基表。支持增量物化视图刷新。Automatic Materialized Views使用对象活动跟踪系统 (OATS) 提供的工作负载信息作为自动决策过程的一部分。

使用DBMS_AUTO_MV 包管理，相关视图有DBA_ACTIVITY_* DBA_AUTO_MV_*

SQL> desc DBMS_AUTO_MV
PROCEDURE CONFIGURE
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 PARAMETER_NAME                 VARCHAR2                IN
 PARAMETER_VALUE                VARCHAR2                IN
 ALLOW                          BOOLEAN                 IN     DEFAULT
PROCEDURE CONFIGURE
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 PARAMETER_NAME                 VARCHAR2                IN
 PARAMETER_VALUE                NUMBER                  IN
 ALLOW                          BOOLEAN                 IN     DEFAULT
PROCEDURE DROP_AUTO_MVS
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 OWNER                          VARCHAR2                IN
 MV_NAME                        VARCHAR2                IN
 ALLOW_RECREATE                 BOOLEAN                 IN     DEFAULT
FUNCTION RECOMMEND RETURNS VARCHAR2
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 STS_OWNER                      VARCHAR2                IN     DEFAULT
 STS_NAME                       VARCHAR2                IN     DEFAULT
 WORKLOAD_START_TIME            TIMESTAMP               IN     DEFAULT
 WORKLOAD_END_TIME              TIMESTAMP               IN     DEFAULT
 AUTOMV_MODE                    VARCHAR2                IN     DEFAULT
PROCEDURE REFRESH
FUNCTION REPORT_ACTIVITY RETURNS CLOB
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 ACTIVITY_START                 TIMESTAMP WITH TIME ZONE IN     DEFAULT
 ACTIVITY_END                   TIMESTAMP WITH TIME ZONE IN     DEFAULT
 TYPE                           VARCHAR2                IN     DEFAULT
 SECTION                        VARCHAR2                IN     DEFAULT
 LEVEL                          VARCHAR2                IN     DEFAULT
FUNCTION REPORT_LAST_ACTIVITY RETURNS CLOB
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 TYPE                           VARCHAR2                IN     DEFAULT
 SECTION                        VARCHAR2                IN     DEFAULT
 LEVEL                          VARCHAR2                IN     DEFAULT


SQL> @d DBA_AUTO_MV
Show data dictionary views and x$ tables matching the expression "DBA_AUTO_MV"...

TABLE_NAME                               COMMENTS
---------------------------------------- --------------------------------------------------------------------------------
DBA_AUTO_MV_ANALYSIS_ACTIONS
DBA_AUTO_MV_ANALYSIS_EXECUTIONS
DBA_AUTO_MV_ANALYSIS_RECOMMENDATIONS
DBA_AUTO_MV_ANALYSIS_REPORT
DBA_AUTO_MV_ANALYSIS_TASK
DBA_AUTO_MV_CONFIG                       set of AUTO MV configuration parameters
DBA_AUTO_MV_MAINT_REPORT
DBA_AUTO_MV_REFRESH_HISTORY
DBA_AUTO_MV_VERIFICATION_REPORT
DBA_AUTO_MV_VERIFICATION_STATUS

10 rows selected.


TABLE_NAME                     COMMENTS
------------------------------ --------------------------------------------------------------------------------
DBA_ACTIVITY_CONFIG            Object Activity Tracking Configuation
DBA_ACTIVITY_MVIEW             Object Activity Tracking for Materialized Views
DBA_ACTIVITY_SNAPSHOT_META     Object Activity Tracking Metadata for Snapshot Intervals
DBA_ACTIVITY_TABLE             Object Activity Tracking for Tables

DBMS_AUTO_MV.RECOMMEND 生成并报告SYS_AUTO_STS过去 24 小时使用的建议。请注意，默认行为是REPORT_ONLY。

自动生成一组正确的物化视图是具有挑战性的任务，这是自动驾驶的一个非常理想的功能数据库。 物化视图的选择必须基于成本，可在实际数据库环境中验证。已全面实施并将部署在云上的 Oracle 自治数据库。另一方面，自动生成物化视图，本身就提出了各种挑战我们可以考虑生成每个语法相关的物化视图，用于基于所有可能的表子集的工作负载查询工作负载查询，但它会爆炸搜索空间，即使一些基于启发式的修剪；任意表子集可以在物化视图定义中引入笛卡尔积。在另一个极端，我们可以为每个查询生成，当在语法上可能，一个候选物化视图与查询的文本完全匹配；这通常会违反存储限制并导致物化视图刷新一个棘手的任务。理想化的目标是生成一个少量的物化视图，大小合理，包含连接和分组的大量预计算，并且可以重写大量当前和未来的工作量查询。这对系统提出了相互矛盾的要求。一种包含大量预计算的物化视图更多有利于它重写的查询，但它通常会重写更少的查询。此外，一个物化视图重写了许多查询往往有大量的分组列，而很少或没有选择谓词，这往往会增加物化根据它包含的行数查看大小。它提出了一种新技术，称为扩展覆盖子表达式算法(ECSE) ，用于自动化物化视图的生成。 ECSE 算法旨在相互冲突的需求之间达成妥协在两个极端之间找到平衡。自动化物化视图项目是更广泛的努力称为 Oracle 自治数据库。 其他该项目的组成部分包括任务管理、 基于机器学习的物化视图的自动刷新等。

选择：

验证：

在最后一步，我们使用名为 SQL Performance Analyzer 的 Oracle 工具验证推荐的物化视图的性能，该工具接受 SQL 工作负载并允许我们使用各种性能指标来衡量推荐的物化视图对工作负载查询执行的影响。对于验证阶段，优化器重写模块以基于成本的方式决定哪些推荐的物化视图最有利于每个工作负载查询的重写。

刷新：

自动 MV 刷新由后台作业执行，该作业每 15 分钟定期执行一次，持续时间为一小时，具有预定义的资源限制。
Auto-MV 维护使用 Oracle 数据挖掘包中提供的基于神经网络的机器学习算法 [21]。 目标是安排所有陈旧自动 MV 的刷新，以便最大化未来查询重写的次数。 每 24 小时，我们构建一个新的神经网络模型来预测未来的 DML 操作和未来的自动 MV 使用。 用于构建神经网络模型的数据——插入、删除和更新的行数，以及自动 MV 重写的数量——来自 OATS。 建立模型后，将使用五重交叉验证技术 [7] 对其进行验证，该技术将来自 OATS 的数据分成五个相等的块。 为了保证模型的准确性，神经网络算法运行了 5 次，每次使用不同的块作为测试集，其余 4 块作为训练集。

references Automated Generation of Materialized Views in Oracle. PVLDB

Oracle ANYDATA数据类型几年前在做expdp迁移时接触过，类型XML,LOB导出速度是慢之非常， 今天又一个客户在expdp时一个interval partition表时，部分分区导出报错ORA-21700: object does not exist or is marked for delete错误，依赖的对象不存在，查看表定义存在anydata列，也就是可能anydata这种任何数据类型都可以存储(但还是建议用lob类型代替)，但是存储的自定义类型又不存在了，这种情况很危险，下面演示一下这种情况。

CREATE OR REPLACE TYPE anbob.t_stu AS OBJECT (
  stu_num VARCHAR2(10),
  stu_name VARCHAR2(10)
);
/

create table anbob.test_anydata(id int,msg anydata);

INSERT INTO anbob.test_anydata (id, msg) VALUES (1, SYS.ANYDATA.convertVarchar2('This is varchar2'));
INSERT INTO anbob.test_anydata (id, msg) VALUES (2, SYS.ANYDATA.convertNumber(999));
INSERT INTO anbob.test_anydata (id, msg) VALUES (3, SYS.ANYDATA.convertNumber(1));
INSERT INTO anbob.test_anydata (id, msg) VALUES (4, SYS.ANYDATA.convertDate(sysdate));

COMMIT;


DECLARE
  l_obj     anbob.t_stu := anbob.t_stu('1','anbob');
  l_anydata SYS.ANYDATA;
BEGIN
  -- Convert Object to ANYDATA and back.
  l_anydata := SYS.ANYDATA.convertObject(l_obj);

  INSERT INTO anbob.test_anydata (id, msg) VALUES (5, l_anydata);

  IF l_anydata.getObject(l_obj) = DBMS_TYPES.SUCCESS
  THEN
    DBMS_OUTPUT.put_line('T_MY_TYPE : ' || l_obj.stu_num || ' : ' || l_obj.stu_name);
  END IF;
END;
 15  /
T_MY_TYPE : 1 : anbob

commit;

SQL> select * from anbob.test_anydata;

        ID MSG()
---------- ------------------------------
         1 ANYDATA()
         2 ANYDATA()
         3 ANYDATA()
         4 ANYDATA()
         5 ANYDATA()

SQL> @dep  anbob test_anydata % %

OWNER            DEPENDENT_NAME                 DEPENDENT_TY REF_OWNER        REF_NAME                       REF_TYPE     DEP_
---------------- ------------------------------ ------------ ---------------- ------------------------------ ------------ ----
ANBOB            TEST_ANYDATA                   TABLE        SYS              STANDARD                       PACKAGE      HARD
ANBOB            TEST_ANYDATA                   TABLE        SYS              ANYDATA                        TYPE         HARD

SQL>  select   SYS.ANYDATA.getTypeName(msg) type_name,count(*) from anbob.test_anydata group by SYS.ANYDATA.getTypeName(msg)

TYPE_NAME                        COUNT(*)
------------------------------ ----------
SYS.NUMBER                              2
ANBOB.T_STU                             1
SYS.VARCHAR2                            1
SYS.DATE                                1

TIP：
ANYDATA.getTypeName 可以取到ANYDATA数据类型的实际datatype, 但是在查看依赖对象是并不会提示表依赖我们开始创建的自定义对象， 这种情况如果删除了自定义对象就糟糕了,下面继续。

SQL> drop type anbob.t_stu ;

Type dropped.

SQL> select   SYS.ANYDATA.getTypeName(msg) type_name,count(*) from anbob.test_anydata group by SYS.ANYDATA.getTypeName(msg);
select   SYS.ANYDATA.getTypeName(msg) type_name,count(*) from anbob.test_anydata group by SYS.ANYDATA.getTypeName(msg)
                                                                                          *
ERROR at line 1:
ORA-21700: object does not exist or is marked for delete
ORA-06512: at "SYS.ANYDATA", line 174


SQL> select * from anbob.test_anydata;
ERROR:
ORA-21700: object does not exist or is marked for delete



no rows selected

[oracle@oel7db1 ~]$ expdp system/oracle@cdb1pdb1 directory=DATAPUMP dumpfile=anydata.dump tables=anbob.test_anydata

Export: Release 19.0.0.0.0 - Production on Sat Nov 6 19:58:07 2021
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Starting "SYSTEM"."SYS_EXPORT_TABLE_01":  system/********@cdb1pdb1 directory=DATAPUMP dumpfile=anydata.dump tables=anbob.test_anydata
Processing object type TABLE_EXPORT/TABLE/TABLE_DATA
Processing object type TABLE_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS
Processing object type TABLE_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS
Processing object type TABLE_EXPORT/TABLE/STATISTICS/MARKER
Processing object type TABLE_EXPORT/TABLE/TABLE
ORA-31693: Table data object "ANBOB"."TEST_ANYDATA" failed to load/unload and is being skipped due to error:
ORA-29913: error in executing ODCIEXTTABLEPOPULATE callout
ORA-21700: object does not exist or is marked for delete

TIP：
读取到表里anydata包含已删除的自定义对象的行时，anydata无法转换才有了这个错误，同时影响EXPDP.

SQL>  DECLARE
  2    l_anydata SYS.ANYDATA;
  3    my_code varchar2(100);
  4    l_typename varchar2(100);
  5  BEGIN
  6     for i in (select * from anbob.test_anydata) loop
  7      begin
  8   l_typename:=SYS.ANYDATA.getTypeName(i.msg);
  9   EXCEPTION
 10             WHEN OTHERS THEN
 11         begin
 12           my_code := SQLCODE;
 13              --DBMS_OUTPUT.put_line(my_code);
 14       if my_code in ('-21700')  then
 15        DBMS_OUTPUT.put_line('miss datatype id:'||i.id );
 16               END IF;
 17        end;
 18  end;
 19     end loop;
 20* END;
miss datatype id:5

PL/SQL procedure successfully completed.

SQL> CREATE OR REPLACE TYPE anbob.t_stu AS OBJECT (
  stu_num VARCHAR2(10),
  stu_name VARCHAR2(10)
);
 /

Type created.

SQL> select * from anbob.test_anydata;
ERROR:
ORA-21700: object does not exist or is marked for delete

note：重建对象后，依旧无法读取

解决方法，暂时发现是delete或update对象的列， 再手动补，也就是数据丢失了，如果有更好的方法，请联系我。

SQL> delete anbob.test_anydata where id=5;

1 row deleted.

SQL> commit;
Commit complete.

SQL> col msg for a30
SQL> select * from anbob.test_anydata;

        ID MSG()
---------- ------------------------------
         1 ANYDATA()
         2 ANYDATA()
         3 ANYDATA()
         4 ANYDATA()

SQL> DECLARE
  l_obj     anbob.t_stu := anbob.t_stu('1','anbob');
  l_anydata SYS.ANYDATA;
BEGIN
  -- Convert Object to ANYDATA and back.
  l_anydata := SYS.ANYDATA.convertObject(l_obj);

  INSERT INTO anbob.test_anydata (id, msg) VALUES (5, l_anydata);

  IF l_anydata.getObject(l_obj) = DBMS_TYPES.SUCCESS
  THEN
    DBMS_OUTPUT.put_line('T_MY_TYPE : ' || l_obj.stu_num || ' : ' || l_obj.stu_name);
  END IF;
END;
 15  /

PL/SQL procedure successfully completed.

SQL> commit;

Commit complete.

SQL> select * from anbob.test_anydata;

        ID MSG()
---------- ------------------------------
         1 ANYDATA()
         2 ANYDATA()
         3 ANYDATA()
         4 ANYDATA()
         5 ANYDATA()

— enjob —

Oracle 11g 提供了一种特性SCAN LISTENER, 简化客户配置应用连接串，客户请求发给scan listener后实际还是会重定向给VIP listener与客户端建立联系， 标准配置是建议把scan name的域名配置到DNS中，如果不使用DNS只能识别一个scan ip, 配置在/etc/hosts中。 如果使用DNS服务器，对DNS的响应有较高要求，同时也增加了一个故障点，对于优化DNS 延迟问题MOS中Reducing Client Connection Delays When DNS Is Unresponsive (Doc ID 1449843.1) 给出了建议， 最近一个客户的DNS突然故障影响了部分使用SCAN 特性应用连接， 这里简单的记录.

服务端

服务端的crs agent在检查监听资源时会解析联系不上DNS，而超时，oraagent_oracle.log日志如下

22292: 2021-11-03 10:02:10.337: [   AGENT][1543]{1:45270:26878} {1:45270:26878} Created alert : (:CRSAGF00113:) :  Aborting the command: check for resource: ora.LISTENER_SCAN1.lsnr 1 1
	行 22310: 2021-11-03 10:02:14.330: [    AGFW][1543]{1:45270:26878} Command: check for resource: ora.LISTENER_SCAN1.lsnr 1 1 completed with status: TIMEDOUT
	行 22349: 2021-11-03 10:05:09.543: [   AGENT][6455]{1:45270:26878} {1:45270:26878} Created alert : (:CRSAGF00113:) :  Aborting the command: check for resource: ora.LISTENER_SCAN1.lsnr 1 1
	行 22367: 2021-11-03 10:05:13.528: [    AGFW][6455]{1:45270:26878} Command: check for resource: ora.LISTENER_SCAN1.lsnr 1 1 completed with status: TIMEDOUT
	行 22484: 2021-11-03 10:13:10.364: [   AGENT][1543]{1:45270:26878} {1:45270:26878} Created alert : (:CRSAGF00113:) :  Aborting the command: check for resource: ora.LISTENER_SCAN1.lsnr 1 1
	行 22502: 2021-11-03 10:13:14.349: [    AGFW][1543]{1:45270:26878} Command: check for resource: ora.LISTENER_SCAN1.lsnr 1 1 completed with status: TIMEDOUT

crsd.log 日志

2021-11-03 10:02:14.334: [    AGFW][9768]{0:7:5} Verifying msg rid = ora.LISTENER_SCAN1.lsnr 1 1
2021-11-03 10:02:14.335: [    AGFW][9768]{0:7:5} Received state change for ora.LISTENER_SCAN1.lsnr 1 1 [old state = ONLINE, new state = UNKNOWN]
2021-11-03 10:02:14.335: [    AGFW][9768]{0:7:5} Received state LABEL change for ora.LISTENER_SCAN1.lsnr 1 1 [old label  = , new label = CHECK TIMED OUT]
2021-11-03 10:02:14.344: [   CRSPE][11053]{0:7:5} State change received from hbjcdb02 for ora.LISTENER_SCAN1.lsnr 1 1
2021-11-03 10:02:14.454: [   CRSPE][11053]{0:7:5} Processing PE command id=96064. Description: [Resource State Change (ora.LISTENER_SCAN1.lsnr 1 1) : 11ada1550]
2021-11-03 10:02:14.461: [   CRSPE][11053]{0:7:5} RI [ora.LISTENER_SCAN1.lsnr 1 1] new external state [INTERMEDIATE] old value: [ONLINE] on hbjcdb02 label = [CHECK TIMED OUT]
2021-11-03 10:02:14.461: [   CRSPE][11053]{0:7:5} Set State Details to [CHECK TIMED OUT] from [ ] for [ora.LISTENER_SCAN1.lsnr 1 1]
2021-11-03 10:02:14.462: [   CRSPE][11053]{0:7:5} Processing unplanned state change for [ora.LISTENER_SCAN1.lsnr 1 1]
2021-11-03 10:02:14.463: [  CRSRPT][11310]{0:7:5} Published to EVM CRS_RESOURCE_STATE_CHANGE for ora.LISTENER_SCAN1.lsnr
2021-11-03 10:02:14.471: [   CRSPE][11053]{0:7:5} PE Command [ Resource State Change (ora.LISTENER_SCAN1.lsnr 1 1) : 11ada1550 ] has completed

lsnrctl status可能会hang, Scan监听日志中出现:

 grep -B 1 TNS listener_scan1.log
03-NOV-2021 10:04:16 *  * (ADDRESS=(PROTOCOL=tcp)(HOST=::1)(PORT=9996)) * status *  * 12525
TNS-12525: TNS:listener has not received client's request in time allowed
 TNS-12535: TNS:operation timed out
  TNS-12606: TNS: Application timeout occurred

客户端

在DNS正常运行时间有问题或 DNS 延迟可能造成问题的情况下，可以将客户端连接描述符配置为使用 IP 地址对与SCAN Listener的连接进行负载平衡, 例如

RAC =
   (DESCRIPTION=
     (ADDRESS_LIST=
       (LOAD_BALANCE=on)(FAILOVER=ON)
       (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
       (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
       (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
     )
     (CONNECT_DATA=(SERVICE_NAME= rac.example.com))
   )

但是注意SCAN LISTENER还是把VIP name重定向返回给客户端，需要客户端联系DNS再次解析，如果files 不是在dns 之前的话, 而如果是使用JDBC thin 驱动则必须要过DNS检查，及时是files在前。Mos中有如下描述:

There is an additional DNS query performed after the client is redirected by the scan listener to the vip/database listener. The scan listener supplies the client with the vip name which the client must again resolve through DNS. Having each of the node vip names mapped to IPs in the client /etc/hosts file will suppress the DNS query provided nsswitch.conf is configured to use files before dns.

hosts: files dns

This workaround is suitable perhaps for a single or small group of systems (e.g., application servers) and will eliminate a bit more of any DNS related connection delays.

JDBC thin driver architecture differs from OCI in that the Listener must perform a DNS query with each incoming (JDBC thin) client connection. In that situation and if DNS response is slow all incoming client connections through the listener may be affected or delayed including other coinciding OCI connections. A recent performance enhancement to the JDBC thin client modifies the connection properties to lessen this DNS dependency at the listener. This improved JDBC connection behavior takes effect starting with Oracle 12.1.0.2.0 but the same enhancement can also be backported to earlier versions using bug 18369949.

如果正在使用oracle 19c rac 安装了19.10,19.11 RU后，关注一下使用了global temporary table(GTT)的运行时间，根据MOS Global temporary table (GTT) usage increased elapsed time after Jan RU 2021 (Doc ID 2798017.1) 记录响应时间会比之前慢9-10倍以上，属于 BUG 33127032，等待事件主要是“end: KI – contention”一个跨实例调用的序列化操作并不常见，通常P2值为59表示全局对象失效。

诊断方法通常是根据WAIT EVENT去MOS里search一下，不过有时并不是总有答案，就像MOS上面提到的ID 只有简单的描述，没有任何关键字。 这时可以dump wait 的call stack, 然后根据call stack 去猜测调用。

alter system set events 'wait_event["enq: KI - contention"] trace("anbob_debug %s\n", shortstack())'; 
-- run sql
alter system set events 'wait_event["enq: KI - contention"] off';

-- cd trace path  then grep "anbob_debug" *.trc

猜测与GTT有关， 对于“ON COMMIT DELETE ROWS” GTT的清理和持久表一样需要跨实例锁定，并刷新脏块， Jonathan Lewis 解释是说ORACLE 的开发人员使用GTT复用了持久表的一些代码原因。后来有很快有人表示也遇到过相同案例，安装了patch后消失。

这是一个国外savvinov描述的问题，感叹oracle提供了独特诊断功能和OWI能力，及广大爱好者多年知识的积累与分享，还有活跃的twitter技术圈社区讨论，可以在24小时内MOS上没有太多关连时把问题解决， 这是国内目前数据库和生态所不具备的，国产路任重而道远。

最近同事遇到一个案例，Oracle数据库频繁重启，数据库日志也是各种后台进程IO hang, 环境VMWARE，RHEL6, multipath, 华为存储。iostat显示问题时间段共享存储设备io 为0，%util 100%. 有几个相似案例，从操作系统WATCHDOG的日志中的call stack trace给出几点建议方向，这里简单的记录。

操作系统日志显示

Nov 11 20:46:24 anbob1 kernel: oracle D 0000000000000001 0 5265 1 0x00000084
Nov 11 20:46:24 anbob1 kernel: ffff882ed3b27b18 0000000000000086 0000000000000000 ffff8817a8a56800
Nov 11 20:46:24 anbob1 kernel: ffffea009f764e38 000000007b53e2c5 0000000000000000 ffff881291cac080
Nov 11 20:46:24 anbob1 kernel: ffff882f17ef85f8 ffff882ed3b27fd8 000000000000fb88 ffff882f17ef85f8
Nov 11 20:46:24 anbob1 kernel: Call Trace:
Nov 11 20:46:24 anbob1 kernel: [<ffffffff814fdfc3>] io_schedule+0x73/0xc0
Nov 11 20:46:24 anbob1 kernel: [<ffffffff811b663e>] __blockdev_direct_IO_newtrunc+0x6fe/0xb90
Nov 11 20:46:24 anbob1 kernel: [<ffffffff8118fec0>] ? pollwake+0x0/0x60
Nov 11 20:46:24 anbob1 kernel: [<ffffffff811b6b2e>] __blockdev_direct_IO+0x5e/0xd0
Nov 11 20:46:24 anbob1 kernel: [<ffffffff811b33e0>] ? blkdev_get_blocks+0x0/0xc0
Nov 11 20:46:24 anbob1 kernel: [<ffffffff811b4247>] blkdev_direct_IO+0x57/0x60
Nov 11 20:46:24 anbob1 kernel: [<ffffffff811b33e0>] ? blkdev_get_blocks+0x0/0xc0
Nov 11 20:46:24 anbob1 kernel: [<ffffffff81115fcb>] generic_file_aio_read+0x6bb/0x700
Nov 11 20:46:24 anbob1 kernel: [<ffffffff814278b3>] ? move_addr_to_user+0x93/0xb0
Nov 11 20:46:24 anbob1 kernel: [<ffffffff8117aeaa>] do_sync_read+0xfa/0x140
Nov 11 20:46:24 anbob1 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40
Nov 11 20:46:24 anbob1 kernel: [<ffffffff8121fd8b>] ? selinux_file_permission+0xfb/0x150
Nov 11 20:46:24 anbob1 kernel: [<ffffffff81213136>] ? security_file_permission+0x16/0x20
Nov 11 20:46:24 anbob1 kernel: [<ffffffff8117b8b5>] vfs_read+0xb5/0x1a0
Nov 11 20:46:24 anbob1 kernel: [<ffffffff810d69e2>] ? audit_syscall_entry+0x272/0x2a0
Nov 11 20:46:24 anbob1 kernel: [<ffffffff8117bbe2>] sys_pread64+0x82/0xa0
Nov 11 20:46:24 anbob1 kernel: [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b

建议：
请与 VMware 支持联系以验证是否存在虚拟机管理程序级别的任何问题，导致 VMware 虚拟磁盘上的 IO 请求无法完成？管理程序日志中是否有关于虚拟磁盘的任何错误消息？

— 原文 www.anbob.com 转载注明出处—

而如果日志是下面的情况，可能是多路径原因

ul 9 16:56:42,node1,[daemon.notice],multipathd:, 36006016061602d008296e08eb281e211: remaining active paths: 0
[...]
Jul 9 17:00:12,node1,[kern.err],kernel:,INFO: task qdiskd:24750 blocked for more than 120 seconds.
Jul 9 17:00:12,node1,[kern.err],kernel:,"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 9 17:00:12,node1,[kern.info],kernel:,qdiskd D 0000000000000004 0 24750 1 0x00000000
Jul 9 17:00:12,node1,[kern.warning],kernel:, ffff8804706dfb18 0000000000000082 0000000000000000 ffffffffa00041fc
Jul 9 17:00:12,node1,[kern.warning],kernel:, ffff8804706dfae8 00000000e4bd82fc ffff8804706dfb08 ffff880875e5ad80
Jul 9 17:00:12,node1,[kern.warning],kernel:, ffff8804726ad0f8 ffff8804706dffd8 000000000000f4e8 ffff8804726ad0f8
Jul 9 17:00:12,node1,[kern.warning],kernel:,Call Trace:
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] ? dm_table_unplug_all+0x5c/0x100 [dm_mod]
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] io_schedule+0x73/0xc0
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] __blockdev_direct_IO_newtrunc+0x6fe/0xb90
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] __blockdev_direct_IO+0x5e/0xd0
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] ? blkdev_get_blocks+0x0/0xc0
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] blkdev_direct_IO+0x57/0x60
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] ? blkdev_get_blocks+0x0/0xc0
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] generic_file_aio_read+0x6bb/0x700
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] ? futex_requeue+0x310/0x890
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] do_sync_read+0xfa/0x140
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] ? autoremove_wake_function+0x0/0x40
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] ? unmap_region+0x110/0x130
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] ? security_file_permission+0x16/0x20
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] vfs_read+0xb5/0x1a0
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] sys_read+0x51/0x90
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] system_call_fastpath+0x16/0x1b

建议：
似乎设备以某种方式被完全删除，从而导致 scsi_dh_alua 的 SCSI_DH_NOSYS 错误以及卡在 dm_table_unplug_all() 中的 qdisk 问题。主要假设是删除设备实际上是问题所在，一个可能的原因可能是在 RHEL 6 中，当路径丢失时，设备被删除。将 dev_loss_tmo 更改为“infinity”并赋予 fast_io_fail_tmo 值“30”似乎可以解决问题。

而如果是下面的

Apr 26 02:25:01 kernel: INFO: task lvcreate:15067 blocked for more than 120 seconds.
Apr 26 02:25:01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 26 02:25:01 kernel: lvcreate      D 0000000000000000     0 15067  15066 0x00000080
Apr 26 02:25:01 kernel: ffff8801b385bac8 0000000000000082 ffff8801b385ba88 ffffffffa00043fc
Apr 26 02:25:01 kernel: ffff8801b385ba98 00000000628b5d4f ffff8801b385bab8 ffff880194e7d180
Apr 26 02:25:01 kernel: ffff88061fd51ab8 ffff8801b385bfd8 000000000000fb88 ffff88061fd51ab8
Apr 26 02:25:01 kernel: Call Trace:
Apr 26 02:25:01 kernel: [<ffffffffa00043fc>] ? dm_table_unplug_all+0x5c/0x100 [dm_mod]
Apr 26 02:25:01 kernel: [<ffffffff810a2431>] ? ktime_get_ts+0xb1/0xf0
Apr 26 02:25:01 kernel: [<ffffffff8150e8c3>] io_schedule+0x73/0xc0
Apr 26 02:25:01 kernel: [<ffffffff811bed0e>] __blockdev_direct_IO_newtrunc+0x6de/0xb30
Apr 26 02:25:01 kernel: [<ffffffff811bf1be>] __blockdev_direct_IO+0x5e/0xd0
Apr 26 02:25:01 kernel: [<ffffffff811bb590>] ? blkdev_get_blocks+0x0/0xc0
Apr 26 02:25:01 kernel: [<ffffffff811bc657>] blkdev_direct_IO+0x57/0x60
Apr 26 02:25:01 kernel: [<ffffffff811bb590>] ? blkdev_get_blocks+0x0/0xc0
Apr 26 02:25:01 kernel: [<ffffffff8111bcab>] generic_file_aio_read+0x6bb/0x700
Apr 26 02:25:01 kernel: [<ffffffff812235d1>] ? avc_has_perm+0x71/0x90
Apr 26 02:25:01 kernel: [<ffffffff811bd029>] ? __blkdev_get+0x1a9/0x3b0
Apr 26 02:25:01 kernel: [<ffffffff811bbba3>] blkdev_aio_read+0x53/0xc0
Apr 26 02:25:01 kernel: [<ffffffff811811aa>] do_sync_read+0xfa/0x140
Apr 26 02:25:01 kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Apr 26 02:25:01 kernel: [<ffffffff810d9e7d>] ? audit_filter_rules+0x2d/0xdd0
Apr 26 02:25:01 kernel: [<ffffffff81228ffb>] ? selinux_file_permission+0xfb/0x150
Apr 26 02:25:01 kernel: [<ffffffff8121bed6>] ? security_file_permission+0x16/0x20
Apr 26 02:25:01 kernel: [<ffffffff81181a95>] vfs_read+0xb5/0x1a0
Apr 26 02:25:01 kernel: [<ffffffff81181bd1>] sys_read+0x51/0x90
Apr 26 02:25:01 kernel: [<ffffffff810dc685>] ? __audit_syscall_exit+0x265/0x290
Apr 26 02:25:01 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b

建议：
恢复挂起的设备。
# dmsetup resume <设备>
建议禁用 vmtoolsd 服务。

每个事务都有一个唯一的 ID，称为 XID，基于Undo segment number 、Undo segment slot number and Undo segment sequence number (wrap#)，为每个事务在第一个 DML语句期间分配一个事务 ID（XID）, 例如：0002.028.000004DA. 要查明transaction是否真的被提交，需要查看transaction table的描述符。transaction table在每个undo 段头可以转储如alter system dump undo header ‘xxx’, 每次transaction table的slot重用时, slot wrap(sequence number) number就会增加.当slot重用(或wrap#)达到maximum value 4,294,967,295 (2^32 – 1)(0xffffffff)次数时，当再次增加 wrap number 1或多值时，通常会提示ORA-1558 或ORA-600[4187]出现，由于撤销段分配的 AUM 算法，这也应该是非常罕见的。但是执行量非常大事务的库，在undo段非常少时还是有发生的概率，最近这个客户就遇到了（ORACLE RAC 11.2.0.4）。

现象

Fri Nov 19 15:10:12 2021
Thread 1 advanced to log sequence 54671 (LGWR switch)
  Current log# 5 seq# 54671 mem# 0: +DATA/anbob/onlinelog/group_5.287.904064243
Fri Nov 19 15:10:16 2021
Archived Log entry 52856 added for thread 1 sequence 54670 ID 0xfe89eeec dest 1:
Fri Nov 19 15:12:09 2021
Errors in file /u01/oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_23451.trc  (incident=189693):
ORA-00600: internal error code, arguments: [4187], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_189693/anbob1_ora_23451_i189693.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

Alert中出现的ora-600 [4187]，部分事务无法继续，随后系统会生成较大的trace日志，有可能会导致ORACLE_HOME磁盘空间耗尽， 通常这种情况重启实例也无法彻底解决，客户在反复重启后，可能会出现下面其它的ORA-600 错误。

	行 5958: ORA-00600: internal error code, arguments: [4187], [], [], [], [], [], [], [], [], [], [], []
	行 5963: ORA-00600: internal error code, arguments: [kfncSlaveMsgFree1], [0], [0], [0], [], [], [], [], [], [], [], []
	行 5965: ORA-00600: internal error code, arguments: [4187], [], [], [], [], [], [], [], [], [], [], []
	行 5972: ORA-00600: internal error code, arguments: [kfgRecoverCreation_0], [], [], [], [], [], [], [], [], [], [], []
	行 5973: ORA-00600: internal error code, arguments: [kfncSlaveMsgFree1], [0], [0], [0], [], [], [], [], [], [], [], []
	行 5975: ORA-00600: internal error code, arguments: [4187], [], [], [], [], [], [], [], [], [], [], []
	行 5984: ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
	行 5985: ORA-00600: internal error code, arguments: [kfgRecoverCreation_0], [], [], [], [], [], [], [], [], [], [], []
	行 5986: ORA-00600: internal error code, arguments: [kfncSlaveMsgFree1], [0], [0], [0], [], [], [], [], [], [], [], []

因为事务无法继续，可能在数据库层分析性能SQL就容易被误解，毕竟事务挂起无法继续。事务中断回滚清理、UNDO header Block recovery等，甚至可能存在SMON实例回滚时的内部错误，如果smon遇到100次内部错误则会重启实例（参数_smon_internal_errlimit控制）。在实例重启后出现ORA-15221 raised in ASM I/O path等ora-600 KF 相关的ASM事务报错，PMON会终止实例。

PMON TRACE

*** 2021-11-19 16:27:44.257
*** SESSION ID:(567.1) 2021-11-19 16:27:44.257
*** CLIENT ID:() 2021-11-19 16:27:44.257
*** SERVICE NAME:(SYS$BACKGROUND) 2021-11-19 16:27:44.257
*** MODULE NAME:() 2021-11-19 16:27:44.257
*** ACTION NAME:() 2021-11-19 16:27:44.257
 
Doing block recovery for file 3 block 272
Block header before block recovery:
buffer tsn: 2 rdba: 0x00c00110 (3/272)
scn: 0x0f9b.ed0bc9e7 seq: 0x01 flg: 0x04 tail: 0xc9e72601
frmt: 0x02 chkval: 0x5ec8 type: 0x26=KTU SMU HEADER BLOCK
Resuming block recovery (PMON) for file 3 block 272
Block recovery from logseq 54683, block 996 to scn 17162371681671

*** 2021-11-19 16:27:44.270
Recovery of Online Redo Log: Thread 1 Group 5 Seq 54683 Reading mem 0
Block recovery completed at rba 54683.998.16, scn 3995.3977334373
==== Redo read statistics for thread 1 ====
Total physical reads (from disk and memory): 1541Kb
-- Redo read_disk statistics --
Read rate (ASYNC): 0Kb in 0.01s => 0.00 Mb/sec
-- Redo read_memory statistics --
Read disk 0Kb and read memory 1541Kb, hit-ratio=1.00
Precise schema: 1541Kb valid, 0Kb invalid, valid-ratio 1.00
Longest record: 0Kb, moves: 0/4 (0%)
Longest LWN: 1Kb, moves: 0/3 (0%), moved: 0Mb
Last redo scn: 0x0f9b.ed114d64 (17162371681636)
----------------------------------------------
Block image after block recovery:
buffer tsn: 2 rdba: 0x00c00110 (3/272)
scn: 0x0f9b.ed0bc9e7 seq: 0x01 flg: 0x04 tail: 0xc9e72601
frmt: 0x02 chkval: 0x5ec8 type: 0x26=KTU SMU HEADER BLOCK

 Retention Table 
  -----------------------------------------------------------
 Extent Number:0  Commit Time: 1637305903
 Extent Number:1  Commit Time: 1637305905
 Extent Number:2  Commit Time: 1637305905
 Extent Number:3  Commit Time: 1637303989
 Extent Number:4  Commit Time: 1637304038
...
...
 Extent Number:28  Commit Time: 1637305842
 Extent Number:29  Commit Time: 1637305901
 Extent Number:30  Commit Time: 1637305901
  
  TRN CTL:: seq: 0xd14f chd: 0x0009 ctl: 0x0004 inc: 0x00000000 nfb: 0x0000
            mgc: 0xb000 xts: 0x0068 flg: 0x0001 opt: 2147483646 (0x7ffffffe)
            uba: 0x00c06434.d14f.2c scn: 0x0f9b.ed0bc826
Version: 0x01
  FREE BLOCK POOL::
    uba: 0x00000000.d14f.2b ext: 0x2  spc: 0x1dc   
    uba: 0x00000000.d14f.2a ext: 0x2  spc: 0x70e   
    uba: 0x00000000.d14b.02 ext: 0x1e spc: 0x1f02  
    uba: 0x00000000.ce3f.02 ext: 0x12 spc: 0x14da  
    uba: 0x00000000.3226.02 ext: 0x32 spc: 0x14ae  
  TRN TBL::
 
  index  state cflags  wrap#    uel         scn            dba            parent-xid    nub     stmt_num    cmt
  ------------------------------------------------------------------------------------------------
   0x00    9    0x00  0xfffffa0c  0x000c  0x0f9b.ed0bc8f1  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305928
   0x01    9    0x00  0xfffff4ab  0x0016  0x0f9b.ed0bc84e  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305927
   0x02    9    0x00  0xfffff3aa  0x0008  0x0f9b.ed0bc934  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305928
   0x03    9    0x00  0xfffff8d9  0x000e  0x0f9b.ed0bc985  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305928
   0x04    9    0x00  0xfffffce8  0xffff  0x0f9b.ed0bc9e7  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305928
   0x05    9    0x00  0xfffff627  0x001a  0x0f9b.ed0bc833  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305927
   0x06    9    0x00  0xfffff4e6  0x0004  0x0f9b.ed0bc9cb  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305928
   0x07    9    0x00  0xffffece5  0x000b  0x0f9b.ed0bc85c  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305927
   0x08    9    0x00  0xfffff724  0x0021  0x0f9b.ed0bc93a  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305928
   0x09    9    0x00  0xfffffff3  0x0015  0x0f9b.ed0bc828  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305927
   0x0a    9    0x00  0xfffffaf2  0x0018  0x0f9b.ed0bc90c  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305928
   0x0b    9    0x00  0xfffff671  0x0010  0x0f9b.ed0bc867  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305927
   0x0c    9    0x00  0xfffffec0  0x001e  0x0f9b.ed0bc900  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305928
   0x0d    9    0x00  0xfffff8bf  0x0020  0x0f9b.ed0bc889  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305928
   0x0e    9    0x00  0xfffff4ce  0x0013  0x0f9b.ed0bc9ab  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305928
   0x0f    9    0x00  0xfffff64d  0x000d  0x0f9b.ed0bc875  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305927
   0x10    9    0x00  0xfffff5ec  0x000f  0x0f9b.ed0bc86b  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305927
...
   0x1d    9    0x00  0xfffff23f  0x0019  0x0f9b.ed0bc91e  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305928
   0x1e    9    0x00  0xfffff67e  0x000a  0x0f9b.ed0bc908  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305928
   0x1f    9    0x00  0xfffff1ad  0x0003  0x0f9b.ed0bc982  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305928
   0x20    9    0x00  0xfffffb0c  0x001b  0x0f9b.ed0bc8ba  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305928
   0x21    9    0x00  0xfffff1eb  0x0011  0x0f9b.ed0bc943  0x00c06434  0x0000.000.00000000  0x00000001   0x00000000  1637305928
  EXT TRN CTL::

Incident 328031 created, dump file: /u01/oracle/app/oracle/diag/rdbms/lcfa/LCFA1/incident/incdir_328031/LCFA1_pmon_11307_i328031.trc
ORA-00600: internal error code, arguments: [kfgRecoverCreation_0], [], [], [], [], [], [], [], [], [], [], []

PMON: fatal error while deleting s.o. 0x6ae9d8a18 in this tree:
----------------------------------------
SO: 0x6d91e2ea0, type: 2, owner: (nil), flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0x6d91e2ea0, name=process, file=ksu.h LINE:12721, pg=0
(process) Oracle pid:50, ser:3, calls cur/top: 0x65863abc0/0x65863abc0
          flags : (0x1) DEAD
          flags2: (0x8000),  flags3: (0x10) 
          intr error: 0, call error: 0, sess error: 0, txn error 0
          intr queue: empty
  ksudlp FALSE at location: 0
  Cleanup details:
    Found dead = 1 sec ago
    Total Cleanup attempts = 1, Cleanup time = 0 sec, Cleanup timer = 0.000 sec
    Last attempt (full) started 1 sec ago, Length = in progress
  (post info) last post received: 0 0 51
              last post received-location: ksv2.h LINE:1679 ID:ksvpst: workdone
              last process to post me: 0x6d51f8310 1 2
              last post sent: 0 0 26
              last post sent-location: ksa2.h LINE:285 ID:ksasnd
              last process posted by me: 0x6ed26d1a0 1 6
  (latch info) wait_event=0 bits=0x0
  Process Group: DEFAULT, pseudo proc: 0x6ed5819c8
  O/S info: user: grid, term: UNKNOWN, ospid: 12107 (DEAD)
  OSD pid info: Unix process pid: 12107, image: oracle@
...

Kernel Transaction Undo tranXaction Control (KTUXC): Maintains control information to rollback changes to the transaction table itself.

从上面transaction table header(ktuxc)显示chd 9# slot， 而slot 9的wrap#为 0xfffffff3已经非常接近上限0xffffffff，并且其它slot同样也非常高。
wrap#增长当前采用的是在执行ktubnd函数为事务绑定undo段时，会调用kjqghd去计算出一个重用slot递增值delta，这个delta限制必须小于16(由KTU_MAX_KSQN_DELTA定义)，所以就可能会出现 0xfffffff3 +delta的值超过KSQNMAXVAL，显示为负值。

从MOS中可以确认存在该问题Doc ID 19700135.8

ORA-600 [4187] can occur for undo segments where wrap# is close to the max value of 0xffffffff (KSQNMAXVAL).
This normally affects databases with high transaction rate that have existed for a relatively long time.

To identify undo segments causing the above error and others that may potentially cause it
in the future, run the next query:

select b.segment_name, b.tablespace_name
,a.ktuxeusn “Undo Segment Number”
,a.ktuxeslt “Slot”
,a.ktuxesqn “Wrap#”
from x$ktuxe a, dba_rollback_segs b
where a.ktuxesqn > -429496730 and a.ktuxesqn < 0
and a.ktuxeusn = b.segment_id;

Then drop the undo segments or the undo tablespace from the output above.

数据库中查询以上SQL

SQL> set lines 999
SQL> select b.segment_name, b.tablespace_name 
  2           ,a.ktuxeusn "Undo Segment Number"
  3           ,a.ktuxeslt "Slot"
  4           ,a.ktuxesqn "Wrap#"
  5     from  x$ktuxe a, dba_rollback_segs b
  6     where a.ktuxesqn > -429496730 and a.ktuxesqn < 0
  7         and a.ktuxeusn = b.segment_id;

SEGMENT_NAME                     TABLESPACE_NAME          Undo Segment Number       Slot      Wrap#
-------------------------------- ------------------------ ------------------- ---------- ----------
_SYSSMU10_562384330$             UNDOTBS1                                  10          0      -1524
_SYSSMU10_562384330$             UNDOTBS1                                  10          1      -2901
_SYSSMU10_562384330$             UNDOTBS1                                  10          2      -3158
_SYSSMU10_562384330$             UNDOTBS1                                  10          3      -1831
_SYSSMU10_562384330$             UNDOTBS1                                  10          4       -792
_SYSSMU10_562384330$             UNDOTBS1                                  10          5      -2521
_SYSSMU10_562384330$             UNDOTBS1                                  10          6      -2842
_SYSSMU10_562384330$             UNDOTBS1                                  10          7      -4891
_SYSSMU10_562384330$             UNDOTBS1                                  10          8      -2268
_SYSSMU10_562384330$             UNDOTBS1                                  10          9        -13
_SYSSMU10_562384330$             UNDOTBS1                                  10         10      -1294
...
...
_SYSSMU10_562384330$             UNDOTBS1                                  10         22       -666
_SYSSMU10_562384330$             UNDOTBS1                                  10         23       -555
_SYSSMU10_562384330$             UNDOTBS1                                  10         24      -3596
_SYSSMU10_562384330$             UNDOTBS1                                  10         25      -3325
_SYSSMU10_562384330$             UNDOTBS1                                  10         26      -2670
_SYSSMU10_562384330$             UNDOTBS1                                  10         27      -1551
_SYSSMU10_562384330$             UNDOTBS1                                  10         28      -4384
_SYSSMU10_562384330$             UNDOTBS1                                  10         29      -3521
_SYSSMU10_562384330$             UNDOTBS1                                  10         30      -2434
_SYSSMU10_562384330$             UNDOTBS1                                  10         31      -3667
_SYSSMU10_562384330$             UNDOTBS1                                  10         32      -1268
'''

Note:
大量slot的wrap# 已经出现的溢出为负值。

transaction table的说明

To find out whether the transaction has really been committed, it is necessary to look at its descriptor in the transaction table.
The transaction ID (or XID) enables you to find the transaction descriptor:
• usn: Number of the undo segment which was assigned to the transaction
• slot: Number of the entry in the transaction table (contained in the header of the undo segment) corresponding to the transaction descriptor. 
This is the index column in the transaction table.
• seq: Sequence number indicating reuse of the slot. This number is incremented as each slot is reused. It corresponds to the wrap# column in the transaction table.
All this information is collected in a memory area called the cleanout information area. This information is then used to actually apply the changes.

The transaction table holds a pointer to the head and tail of the committed transaction table slots. Each inactive entry in the transaction table points to the next committed (inactive) slot. 
They are ordered by commit SCN, starting at the oldest transaction to have committed. If the committed list is empty after several attempts at traversing it, the transaction table remains full 
and an error is reported. This indicates that there are not enough undo segments for the amount of concurrent transactions in the database. However, since AUM tries to allocate each concurrent transaction to their 
own undo segment, this should be extremely rare to see. If this error does occur, the solution is to increase the size of the undo tablespace to allow more undo segments to be created.

At instance startup each undo tablespace will contain 10 undo segments. The number of additional segments brought online during instance startup is based on the SESSIONS parameter.
 Oracle allocates approximately one undo segment for every five sessions. These are sized according to the auto-allocate algorithm for locally managed tablespaces. The basic algorithm 
is that the first 16 extents are 64 KB in size. During subsequent allocation the next 63 extents are 1 MB, the next 120 extents are 8 MB and all additional extents are 64 MB.

事务绑定UNDO段的原则是尽可能的将活动事务平均的各个undo段上，具体算法为：
1，在当前undo tablespace中的online undo segment中寻找事务表中没有活动事务的undo segment
2，如果没有找到则尝试在当前undo tablespace online 那些处于offline状态的undo segment
3，如果没有找到则尝试在当前undo tablespace创建undo segment并online
4，如果无法创建则会寻找最近最少使用的undo segment

到这一步既然确认了是业务特性高频事务导致undo 段的slot wrap#耗尽，检查一下UNDO段个数，发现当前的undo 表空间为大小1.5G （数据文件自动扩展），判断应该是大量的小事务，如loop 中insert value ommit等， 对于这么高频率的事务的数据库节点,同时因为UNDO段个数少，即使是undo段使用平衡,也会存在耗尽的风险， 如果个别segment的slot wrap#高，可以drop undo segment，但是如果普遍undo段wrap#使用都高，可以先创建新的UNDO表空间，修改数据库默认表空间解决，后期删聊原UNDO表空间。

预防
1， 初始化更多的UNDO段，使用_rollback_segment_count参数

2， 增加巡检项

select b.segment_name, b.tablespace_name 
         ,a.ktuxeusn "Undo Segment Number"
         ,a.ktuxeslt "Slot"
         ,a.ktuxesqn "Wrap#"
   from  x$ktuxe a, dba_rollback_segs b
   where  a.ktuxeusn = b.segment_id 
   and ktuxesqn>power(2,20); -- change

最近一起Oracle数据库自动crash, 再次打开是提示需要做datafile recover, 分析alert日志提示ORA-63999 ORA-01122 ORA-01110 ORA-01208， 问题时logfileswitch并不频繁，当时也没有做duplicat 操作(bug 13498382)，ora-1208是因为数据文件头上的ckpt比控制文件上的旧，reover后随后数据文件头被更新，因此可以正常访问而不会发生损坏,判断是IO写丢失导致. 仅记录

2021-11-22 22:25:10.543000 +08:00
Archived Log entry 3751 added for thread 1 sequence 3767 ID 0x83ca5242 dest 1:
ALTER SYSTEM ARCHIVE LOG
Thread 1 advanced to log sequence 3769 (LGWR switch)
  Current log# 1 seq# 3769 mem# 0: /oradata/ANBOB/onlinelog/o1_mf_1_gv0my4b8_.log
Archived Log entry 3752 added for thread 1 sequence 3768 ID 0x83ca5242 dest 1:
2021-11-22 22:26:15.984000 +08:00
ALTER SYSTEM ARCHIVE LOG
Thread 1 advanced to log sequence 3770 (LGWR switch)
  Current log# 2 seq# 3770 mem# 0: /oradata/ANBOB/onlinelog/o1_mf_2_gv0mygsx_.log
Archived Log entry 3753 added for thread 1 sequence 3769 ID 0x83ca5242 dest 1:
2021-11-23 02:00:00.004000 +08:00
Closing scheduler window
Closing Resource Manager plan via scheduler window
Clearing Resource Manager plan via parameter
2021-11-23 02:00:59.344000 +08:00
Thread 1 advanced to log sequence 3771 (LGWR switch)
  Current log# 3 seq# 3771 mem# 0: /oradata/ANBOB/onlinelog/o1_mf_3_gv0mys8m_.log
2021-11-23 02:01:06.292000 +08:00
Archived Log entry 3754 added for thread 1 sequence 3770 ID 0x83ca5242 dest 1:
2021-11-23 03:11:52.148000 +08:00
datafile header validation failure for file /dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf
2021-11-23 03:14:00.276000 +08:00
Read of datafile '/dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf' (fno 22) header failed with ORA-01208
Rereading datafile 22 header failed with ORA-01208
Checker run found 1 new persistent data failures
2021-11-23 03:47:05.194000 +08:00
Read of datafile '/dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf' (fno 22) header failed with ORA-01208
Rereading datafile 22 header failed with ORA-01208
2021-11-23 04:21:58.388000 +08:00
Read of datafile '/dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf' (fno 22) header failed with ORA-01208
Rereading datafile 22 header failed with ORA-01208
2021-11-23 19:30:41.151000 +08:00
Thread 1 advanced to log sequence 3772 (LGWR switch)
  Current log# 4 seq# 3772 mem# 0: /oradata/ANBOB/onlinelog/o1_mf_4_gv0n7srg_.log
2021-11-23 19:30:47.133000 +08:00
Archived Log entry 3755 added for thread 1 sequence 3771 ID 0x83ca5242 dest 1:
2021-11-23 19:35:42.016000 +08:00
Read of datafile '/dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf' (fno 22) header failed with ORA-01208
Rereading datafile 22 header failed with ORA-01208
Errors in file /u01/oracle/diag/rdbms/anbob/anbob/trace/anbob_ckpt_10232.trc:
ORA-63999: data file suffered media failure
ORA-01122: database file 22 failed verification check
ORA-01110: data file 22: '/dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf'
ORA-01208: data file is an old version - not accessing current version
Errors in file /u01/oracle/diag/rdbms/anbob/anbob/trace/anbob_ckpt_10232.trc:
ORA-63999: data file suffered media failure
ORA-01122: database file 22 failed verification check
ORA-01110: data file 22: '/dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf'
ORA-01208: data file is an old version - not accessing current version
CKPT (ospid: 10232): terminating the instance due to error 63999        《《《《《《《《《《《《
System state dump requested by (instance=1, osid=10232 (CKPT)), summary=[abnormal instance termination].
System State dumped to trace file /u01/oracle/diag/rdbms/anbob/anbob/trace/anbob_diag_10216_20211123193542.trc
2021-11-23 19:35:47.118000 +08:00
Instance terminated by CKPT, pid = 10232

提示：
从数据库告警日志同时间可以看到从3:14开始在读取数据文件#22号’/dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf’ (fno 22)时，做数据库一致性校验时失败，触发ORA-01208错误，读到了一个老版本的数据. 一直持续到2021-11-23 19:35 数据库的核心后台进程CKPT做检查点时因为相同的问题直接终止了数据库实例。 此日10：04数据库打开，所以提有启动时提示要做recover，前滚数据文件一致性后才能OPEN。

ckpt trace file

*** 2021-11-23 19:35:42.028
Rereading datafile 22 header failed with ORA-01208
 V10 STYLE FILE HEADER:
        Compatibility Vsn = 186647552=0xb200400
        Db ID=2211061570=0x83ca1f42, Db Name='ANBOB'
        Activation ID=0=0x0
        Control Seq=810175=0xc5cbf, File size=3932160=0x3c0000
        File Number=22, Blksiz=8192, File Type=3 DATA
Tablespace #6 - HFMIS_ZHFW  rel_fn:22
Creation   at   scn: 0x0000.e0672b45 08/21/2021 14:35:35
Backup taken at scn: 0x0000.00000000 01/01/1988 00:00:00 thread:0
 reset logs count:0x3cf14dc4 scn: 0x0000.000e2006
 prev reset logs count:0x3121c97a scn: 0x0000.00000001
 recovered at 10/08/2021 09:10:15
 status:0x4 root dba:0x00000000 chkpt cnt: 757 ctl cnt:756
begin-hot-backup file size: 0
Checkpointed at scn:  0x0000.ed6df7ab 11/23/2021 02:00:59
 thread:1 rba:(0xebb.2.10)
 enabled  threads:  01000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
。。。
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000
Backup Checkpointed at scn:  0x0000.00000000
 thread:0 rba:(0x0.0.0)
 enabled  threads:  00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
...
  00000000 00000000 00000000 00000000 00000000 00000000
External cache id: 0x0 0x0 0x0 0x0
Absolute fuzzy scn: 0x0000.00000000
Recovery fuzzy scn: 0x0000.00000000 08/21/2021 14:35:35
Terminal Recovery Stamp  01/01/1988 00:00:00
Platform Information:    Creation Platform ID: 13
Current Platform ID: 13 Last Platform ID: 13
DATA FILE #22:
  name #30: /dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf
creation size=3932160 block size=8192 status=0xe head=30 tail=30 dup=1
 tablespace 6, index=6 krfil=22 prev_file=21
 unrecoverable scn: 0x0000.ed6a50d0 11/22/2021 22:37:42
 Checkpoint cnt:758 scn: 0x0000.ed6f887f 11/23/2021 03:11:45
 Stop scn: 0xffff.ffffffff 10/08/2021 02:05:50
 Creation Checkpointed at scn:  0x0000.e0672b45 08/21/2021 14:35:35
 thread:1 rba:(0xc4d.74214.10)
 enabled  threads:  01000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
...
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000
 Offline scn: 0x0000.00000000 prev_range: 0
 Online Checkpointed at scn:  0x0000.00000000
 thread:0 rba:(0x0.0.0)
 enabled  threads:  00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
...
  00000000 00000000 00000000 00000000 00000000 00000000
 Hot Backup end marker scn: 0x0000.00000000
 aux_file is NOT DEFINED
 Plugged readony: NO
 Plugin scnscn: 0x0000.00000000
 Plugin resetlogs scn/timescn: 0x0000.00000000 01/01/1988 00:00:00
 Foreign creation scn/timescn: 0x0000.00000000 01/01/1988 00:00:00
 Foreign checkpoint scn/timescn: 0x0000.00000000 01/01/1988 00:00:00
 Online move state: 0
DDE rules only execution for: ORA 1110

ORA-63999: data file suffered media failure
ORA-01122: database file 22 failed verification check
ORA-01110: data file 22: '/dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf'
ORA-01208: data file is an old version - not accessing current version
error 63999 detected in background process
ORA-63999: data file suffered media failure
ORA-01122: database file 22 failed verification check
ORA-01110: data file 22: '/dbdata/ANBOB/datafile/o1_mf_hfmis_zh_jl17mn6d_.dbf'
ORA-01208: data file is an old version - not accessing current version
kjzduptcctx: Notifying DIAG for crash event
----- Abridged Call Stack Trace -----
ksedsts()+465<-kjzdssdmp()+267<-kjzduptcctx()+232<-kjzdicrshnfy()+63<-ksuitm()+5594<-ksbrdp()+3507<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+250<-ssthrdmain()+265<-main()+201<-__libc_start_main()+253
----- End of Abridged Call Stack Trace -----

*** 2021-11-23 19:35:42.048
CKPT (ospid: 10232): terminating the instance due to error 63999
ksuitm: waiting up to [5] seconds before killing DIAG(10216)

提示：
数据文件头记录的Checkpointed at scn: 0x0000.ed6df7ab 11/23/2021 02:00:59，而控制文件记录的是Checkpoint cnt:758 scn: 0x0000.ed6f887f 11/23/2021 03:11:45， 两者不一致，数据文件头丢失了1次ckpt信息，通常发生在磁盘的写丢失或人为的把历史数据覆盖现有数据，但这么近的时间可能性不大。

一套oracle single instance on AIX环境，数据库未启动，客户硬件只是扩容了内存OS重启后，sqlplus启库无法登录，sqlplus / as sysdba命令挂起。

1, rename  sqlnet.ora

2, 检查了/etc/hosts

3, 尝试了listener可以启动

4，relink sqlplus后依旧hang， sqlplus /nolog可以

5, 配置监听静态，sqlplus使用密码登录依旧hang

/* Linux */
strace -o sqlplus_hang.log -ttt sqlplus / as sysdba

/* AIX */
truss -aefo sqlplus_hang.log sqlplus / as sysdba

# grep open  sqlplus_hang.log

	行 5: 32373 16:42:18.426092 [00007f3ff87df7a7] open("/etc/ld.so.preload", O_RDONLY|O_CLOEXEC) = 3 <0.000018>
	行 9: 32373 16:42:18.426252 [00007f3ff87df7a7] open("/home/dbsectde/libdbsr.so", O_RDONLY|O_CLOEXEC) = 3 <0.000017>
	行 18: 32373 16:42:18.426572 [00007f3ff87df7a7] open("/u01/app/oracle/product/11.2.0/db_1/lib/tls/x86_64/libsqlplus.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000015>
	行 20: 32373 16:42:18.426647 [00007f3ff87df7a7] open("/u01/app/oracle/product/11.2.0/db_1/lib/tls/libsqlplus.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000014>
	行 22: 32373 16:42:18.426707 [00007f3ff87df7a7] open("/u01/app/oracle/product/11.2.0/db_1/lib/x86_64/libsqlplus.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000013>
	行 24: 32373 16:42:18.426764 [00007f3ff87df7a7] open("/u01/app/oracle/product/11.2.0/db_1/lib/libsqlplus.so", O_RDONLY|O_CLOEXEC) = 3 <0.000018>
	行 33: 32373 16:42:18.427049 [00007f3ff87df7a7] open("/u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1", O_RDONLY|O_CLOEXEC) = 3 <0.000015>
	行 41: 32373 16:42:18.427303 [00007f3ff87df7a7] open("/u01/app/oracle/product/11.2.0/db_1/lib/libnnz11.so", O_RDONLY|O_CLOEXEC) = 3 <0.000016>
	行 49: 32373 16:42:18.427567 [00007f3ff87df7a7] open("/u01/app/oracle/product/11.2.0/db_1/lib/libdl.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000013>
	行 50: 32373 16:42:18.427599 [00007f3ff87df7a7] open("/u01/app/oracle/product/11.2.0/db_1/lib/libdl.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000012>
	行 955: 32374 16:42:18.467506 [00007f5f6f830a30] open("/proc/self/cmdline", O_RDONLY) = 3 <0.000015>
	行 963: 32374 16:42:18.467736 [00007f5f6f830a30] open("/home/dbsectde/Oracle_address", O_RDONLY) = 3 <0.000014>
	行 1013: 32374 16:42:18.469280 [00007f5f6f830a30] open("/tmp/tdehook/cdump/20211214164218-32374.core", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3 <0.000089>
	行 1033: 32374 16:42:19.470180 [00007f5f6f830a30] open("/tmp/tdehook/cdump/20211214164219-32374.core", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3 <0.000078>
	行 1053: 32374 16:42:20.471090 [00007f5f6f830a30] open("/tmp/tdehook/cdump/20211214164220-32374.core", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3 <0.000113>
	行 1073: 32374 16:42:21.472079 [00007f5f6f830a30] open("/tmp/tdehook/cdump/20211214164221-32374.core", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3 <0.000103>
	行 1093: 32374 16:42:22.473046 [00007f5f6f830a30] open("/tmp/tdehook/cdump/20211214164222-32374.core", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3 <0.000105>
	行 1113: 32374 16:42:23.474061 [00007f5f6f830a30] open("/tmp/tdehook/cdump/20211214164223-32374.core", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3 <0.000145>
	行 1133: 32374 16:42:24.475002 [00007f5f6f830a30] open("/tmp/tdehook/cdump/20211214164224-32374.core", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3 <0.000106>
	行 1153: 32374 16:42:25.475925 [00007f5f6f830a30] open("/tmp/tdehook/cdump/20211214164225-32374.core", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3 <0.000170>
	行 1173: 32374 16:42:26.476960 [00007f5f6f830a30] open("/tmp/tdehook/cdump/20211214164226-32374.core", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3 <0.000101>
...
循环中

# 查看open /emp/tdehook前部分内容

32374 16:42:18.467911 [00007f5f6f83a0f7] munmap(0x7f5f722cc000, 4096) = 0 <0.000009>
32374 16:42:18.467961 [00007f5f6f83a127] mprotect(0xcb23000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC) = 0 <0.000010>
32374 16:42:18.467996 [00007f5f6f83a127] mprotect(0x2060000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC) = 0 <0.000017>
32374 16:42:18.468044 [00007f5f6f83a127] mprotect(0x2060000, 8192, PROT_READ|PROT_EXEC) = 0 <0.000009>
32374 16:42:18.468074 [00007f5f6f83a127] mprotect(0xf74000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC) = 0 <0.000014>
32374 16:42:18.468110 [00007f5f6f83a127] mprotect(0xf74000, 8192, PROT_READ|PROT_EXEC) = 0 <0.000008>
32374 16:42:18.468135 [00007f5f6f83a127] mprotect(0x5805000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC) = 0 <0.000013>
32374 16:42:18.468170 [00007f5f6f83a127] mprotect(0x5805000, 8192, PROT_READ|PROT_EXEC) = 0 <0.000009>
32374 16:42:18.468196 [00007f5f6f83a127] mprotect(0x5805000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC) = 0 <0.000008>
32374 16:42:18.468222 [00007f5f6f83a127] mprotect(0x5805000, 8192, PROT_READ|PROT_EXEC) = 0 <0.000009>
32374 16:42:18.468247 [00007f5f6f83a127] mprotect(0x5805000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC) = 0 <0.000008>
32374 16:42:18.468272 [00007f5f6f83a127] mprotect(0x5805000, 8192, PROT_READ|PROT_EXEC) = 0 <0.000008>
32374 16:42:18.468298 [00007f5f6f83a127] mprotect(0x8545000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC) = 0 <0.000013>
32374 16:42:18.468331 [00007f5f6f83a127] mprotect(0x8545000, 8192, PROT_READ|PROT_EXEC) = 0 <0.000009>
32374 16:42:18.468357 [00007f5f6f83a127] mprotect(0x8544000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC) = 0 <0.000011>
32374 16:42:18.468389 [00007f5f6f83a127] mprotect(0x8544000, 8192, PROT_READ|PROT_EXEC) = 0 <0.000011>
32374 16:42:18.468433 [00007f5f6f83a127] mprotect(0x8545000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC) = 0 <0.000010>
32374 16:42:18.468460 [00007f5f6f83a127] mprotect(0x8545000, 8192, PROT_READ|PROT_EXEC) = 0 <0.000009>
32374 16:42:18.468487 [00007f5f6f83a127] mprotect(0x8544000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC) = 0 <0.000009>
32374 16:42:18.468513 [00007f5f6f83a127] mprotect(0x8544000, 8192, PROT_READ|PROT_EXEC) = 0 <0.000008>
32374 16:42:18.468539 [00007f5f6f83a127] mprotect(0x93e3000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC) = 0 <0.000014>
32374 16:42:18.468574 [00007f5f6f83a127] mprotect(0x93e3000, 8192, PROT_READ|PROT_EXEC) = 0 <0.000008>
32374 16:42:18.468600 [00007f5f6f83a127] mprotect(0x1ec9000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC) = 0 <0.000013>
32374 16:42:18.468634 [00007f5f6f83a127] mprotect(0x1ec9000, 8192, PROT_READ|PROT_EXEC) = 0 <0.000009>
32374 16:42:18.468659 [00007f5f6f83a127] mprotect(0x961b000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC) = 0 <0.000012>
32374 16:42:18.468691 [00007f5f6f83a127] mprotect(0x961b000, 8192, PROT_READ|PROT_EXEC) = 0 <0.000008>
32374 16:42:18.468715 [00007f5f6f83a127] mprotect(0x961b000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC) = 0 <0.000008>
32374 16:42:18.468741 [00007f5f6f83a127] mprotect(0x961b000, 8192, PROT_READ|PROT_EXEC) = 0 <0.000008>
32374 16:42:18.468766 [00007f5f6f83a127] mprotect(0x1bd9000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC) = 0 <0.000013>
32374 16:42:18.468799 [00007f5f6f83a127] mprotect(0x1bd9000, 8192, PROT_READ|PROT_EXEC) = 0 <0.000009>
32374 16:42:18.468824 [00007f5f6f83a127] mprotect(0xa54a000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC) = 0 <0.000012>
32374 16:42:18.468858 [00007f5f6f83a127] mprotect(0xa54a000, 8192, PROT_READ|PROT_EXEC) = 0 <0.000008>
32374 16:42:18.468885 [00007f5f6f83a127] mprotect(0x6478000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC) = 0 <0.000012>
32374 16:42:18.468919 [00007f5f6f83a127] mprotect(0x6478000, 8192, PROT_READ|PROT_EXEC) = 0 <0.000008>
32374 16:42:18.468944 [00007f5f6f83a127] mprotect(0x1860000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC) = 0 <0.000012>
32374 16:42:18.468981 [00007f5f6f83a127] mprotect(0x1860000, 8192, PROT_READ|PROT_EXEC) = 0 <0.000008>
32374 16:42:18.469017 [00007f5f6f8414f7] shmctl(0, SHM_INFO, 0x7ffc719f9120) = 0 <0.000009>
32374 16:42:18.469045 [00007f5f6f8414f7] shmctl(0, SHM_STAT, 0x7ffc719f90b0) = 65536 <0.000009>
32374 16:42:18.469076 [00007f5f6f841467] shmat(65536, 0, 0) = 0x7f5f6ef41000 <0.000013>
32374 16:42:18.469116 [00007f5f6f841497] shmdt(0x7f5f6ef41000) = 0 <0.000012>
32374 16:42:18.469145 [00007f5f6f841467] shmat(65536, 0, SHM_RDONLY) = 0x7f5f6ef41000 <0.000010>
32374 16:42:18.469176 [00007f5f6f841497] shmdt(0x7f5f6ef41000) = 0 <0.000010>
32374 16:42:18.469218 [00007f5f6f830d27] access("/tmp/tdehook", F_OK) = 0 <0.000011>
32374 16:42:18.469247 [00007f5f6f830d27] access("/tmp/tdehook/cdump", F_OK) = 0 <0.000009>
32374 16:42:18.469280 [00007f5f6f830a30] open("/tmp/tdehook/cdump/20211214164218-32374.core", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3 <0.000089>
32374 16:42:18.469389 [00007f5f6f83fbe0] lseek(3, 0, SEEK_END) = 0 <0.000018>
32374 16:42:18.469436 [00007f5f6f830907] fchmod(3, 0666) = 0 <0.000012>

# grep access sqlplus_hang.log

 行 4: 32373 16:42:18.426057 [00007f3ff87df807] access("/etc/ld.so.preload", R_OK) = 0 <0.000010>
 行 226: 32373 16:42:18.439559 [00007f3ff4b00d27] access("/u01/app/oracle/product/11.2.0/db_1/network/admin/sqlnet.ora", F_OK) = 0 <0.000012>
 行 235: 32373 16:42:18.439865 [00007f3ff4b00d27] access("/home/oracle/.sqlnet.ora", F_OK) = -1 ENOENT (No such file or directory) <0.000012>
 行 315: 32373 16:42:18.443221 [00007f3ff4b00d27] access("/u01/app/oracle/product/11.2.0/db_1/network/admin/sqlnet.ora", F_OK) = 0 <0.000012>
 行 324: 32373 16:42:18.443543 [00007f3ff4b00d27] access("/etc/intchg.ora", F_OK) = -1 ENOENT (No such file or directory) <0.000009>
 行 325: 32373 16:42:18.443574 [00007f3ff4b00d27] access("/u01/app/oracle/product/11.2.0/db_1/network/admin/intchg.ora", F_OK) = -1 ENOENT (No such file or directory) <0.000010>
 行 326: 32373 16:42:18.443609 [00007f3ff4b00d27] access("/etc/tnsnav.ora", F_OK) = -1 ENOENT (No such file or directory) <0.000009>
 行 327: 32373 16:42:18.443638 [00007f3ff4b00d27] access("/u01/app/oracle/product/11.2.0/db_1/network/admin/tnsnav.ora", F_OK) = -1 ENOENT (No such file or directory) <0.000010>
 行 725: 32374 16:42:18.457644 [00007f5f721c9807] access("/etc/ld.so.preload", R_OK) = 0 <0.000010>
 行 1011: 32374 16:42:18.469218 [00007f5f6f830d27] access("/tmp/tdehook", F_OK) = 0 <0.000011>
 行 1012: 32374 16:42:18.469247 [00007f5f6f830d27] access("/tmp/tdehook/cdump", F_OK) = 0 <0.000009>
 行 1031: 32374 16:42:19.470119 [00007f5f6f830d27] access("/tmp/tdehook", F_OK) = 0 <0.000010>
 行 1032: 32374 16:42:19.470150 [00007f5f6f830d27] access("/tmp/tdehook/cdump", F_OK) = 0 <0.000009>
 行 1051: 32374 16:42:20.471027 [00007f5f6f830d27] access("/tmp/tdehook", F_OK) = 0 <0.000011>
 行 1052: 32374 16:42:20.471060 [00007f5f6f830d27] access("/tmp/tdehook/cdump", F_OK) = 0 <0.000010>
 行 1071: 32374 16:42:21.472015 [00007f5f6f830d27] access("/tmp/tdehook", F_OK) = 0 <0.000011>
-- 循环

Pstack 进程

#0  0x00007fd1aec4b7e0 in __read_nocancel () from /lib64/libpthread.so.0
#1  0x00007fd1afe4d517 in sntpcall_stg2 () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#2  0x00007fd1afe4948b in sntpcall () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#3  0x00007fd1afe45a27 in ntpcon () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#4  0x00007fd1afd7ba15 in ntconn () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#5  0x00007fd1afd337d2 in nsopen () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#6  0x00007fd1afcf71de in nscall1 () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#7  0x00007fd1afcf48c4 in nscall () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#8  0x00007fd1afd91e40 in niotns () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#9  0x00007fd1afe236d1 in nigcall () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#10 0x00007fd1afd9cf58 in osncon () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#11 0x00007fd1afc2c2ab in kpuadef () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#12 0x00007fd1afc07df7 in upiini () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#13 0x00007fd1afbffb82 in upiah0 () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#14 0x00007fd1afc25775 in kpuatch () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#15 0x00007fd1afc0bfbb in OCIServerAttach () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#16 0x00007fd1b24c66a0 in afioci () from /u01/app/oracle/product/11.2.0/db_1/lib/libsqlplus.so
#17 0x00007fd1b24960d2 in aficntatt () from /u01/app/oracle/product/11.2.0/db_1/lib/libsqlplus.so
#18 0x00007fd1b2495901 in aficntcon () from /u01/app/oracle/product/11.2.0/db_1/lib/libsqlplus.so
#19 0x00007fd1b249bb19 in aficoncon () from /u01/app/oracle/product/11.2.0/db_1/lib/libsqlplus.so
#20 0x00007fd1b24bea55 in afilon () from /u01/app/oracle/product/11.2.0/db_1/lib/libsqlplus.so
#21 0x00007fd1b24b795c in afidrv () from /u01/app/oracle/product/11.2.0/db_1/lib/libsqlplus.so
#22 0x0000000000400753 in main ()
#0  0x00007fd1aec4b7e0 in __read_nocancel () from /lib64/libpthread.so.0
#1  0x00007fd1afe4d517 in sntpcall_stg2 () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#2  0x00007fd1afe4948b in sntpcall () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#3  0x00007fd1afe45a27 in ntpcon () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#4  0x00007fd1afd7ba15 in ntconn () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#5  0x00007fd1afd337d2 in nsopen () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#6  0x00007fd1afcf71de in nscall1 () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#7  0x00007fd1afcf48c4 in nscall () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#8  0x00007fd1afd91e40 in niotns () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#9  0x00007fd1afe236d1 in nigcall () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#10 0x00007fd1afd9cf58 in osncon () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#11 0x00007fd1afc2c2ab in kpuadef () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#12 0x00007fd1afc07df7 in upiini () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#13 0x00007fd1afbffb82 in upiah0 () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#14 0x00007fd1afc25775 in kpuatch () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#15 0x00007fd1afc0bfbb in OCIServerAttach () from /u01/app/oracle/product/11.2.0/db_1/lib/libclntsh.so.11.1
#16 0x00007fd1b24c66a0 in afioci () from /u01/app/oracle/product/11.2.0/db_1/lib/libsqlplus.so
#17 0x00007fd1b24960d2 in aficntatt () from /u01/app/oracle/product/11.2.0/db_1/lib/libsqlplus.so
#18 0x00007fd1b2495901 in aficntcon () from /u01/app/oracle/product/11.2.0/db_1/lib/libsqlplus.so
#19 0x00007fd1b249bb19 in aficoncon () from /u01/app/oracle/product/11.2.0/db_1/lib/libsqlplus.so
#20 0x00007fd1b24bea55 in afilon () from /u01/app/oracle/product/11.2.0/db_1/lib/libsqlplus.so
#21 0x00007fd1b24b795c in afidrv () from /u01/app/oracle/product/11.2.0/db_1/lib/libsqlplus.so
#22 0x0000000000400753 in main ()
循环中

note:
the __read_nocancel function comes from the libpthread operating system library , which is a place a server process is expected to wait and is actually in an oracle wait event (SQL*Net message from client).

更换一个同平台同版本的登录正常的测试库，跟踪sqlplus / as sysdba

[oracle@oel7db1 ~]$ grep access sqlplus_hang.log
5629 1639624344.255472 access(“/etc/ld.so.preload”, R_OK) = -1 ENOENT (No such file or directory)

….

发现/etc/ld.so.preload 在测试环境是不存在的， ld.so.preload用于预加载一些类库，通常sudo使用，甚至有时会有安全风险，尝试去掉该文件。

$ move /etc/ld.so.preload  /etc/ld.so.preload_bak

sqlplus / as sysdba恢复正常。

在做 db benchmarks 时，qps、tps 是衡量数据库性能的关键指标,TPS : Transactions Per Second 是每秒事务数，即数据库服务器在单位时间内处理的事务数。 横向对比计划几类数据库计算tps的方法。

Oracle database

定期采样

select sysdate, name, value from v$sysstat where name in ( 'user commits','user rollbacks');

您可以看到发生了多少次提交和划分——但这绝对不是任何有意义的“TPS”。用户提交将不会反映应用层的“事务”。

oracle自己的metriy视图

select sum(decode(metric_name,'User Transaction Per Sec',round(value,2),0)) as UTPS 
FROM v$sysmetric 
where group_id   = 2 AND metric_name IN ( 'User Transaction Per Sec') 
group by begin_time,end_time ;

也可以参考DBA_TAB_MODIFICATIONS的记录数。

WITH hist_snaps
AS (SELECT instance_number,
snap_id,
round(begin_interval_time,'MI') datetime,
(  begin_interval_time + 0 - LAG (begin_interval_time + 0)
OVER (PARTITION BY dbid, instance_number ORDER BY snap_id)) * 86400 diff_time
FROM dba_hist_snapshot), hist_stats
AS (SELECT dbid,
instance_number,
snap_id,
stat_name,
VALUE - LAG (VALUE) OVER (PARTITION BY dbid,instance_number,stat_name ORDER BY snap_id)
delta_value
FROM dba_hist_sysstat
WHERE stat_name IN ('user commits', 'user rollbacks'))
SELECT datetime,
ROUND (SUM (delta_value) / 3600, 2) "Transactions/s"
FROM hist_snaps sn, hist_stats st
WHERE     st.instance_number = sn.instance_number
AND st.snap_id = sn.snap_id
AND diff_time IS NOT NULL
GROUP BY datetime
ORDER BY 1 desc;

现成的工具可以使用AWR or STATSPACK，Number of Transactions = (DeltaCommits+DeltaRollbacks)/Time

MySQL

Com_commit和Com_rollback在information_schema.global_status. 您需要定期查询它们并计算读数之间的差异才能获得quantity per duration.

SELECT  SUM(IF(variable_name IN ('Com_commit', 'Com_rollback'),
                   variable_value, 0)) /
        SUM(IF(variable_name = 'Uptime',
                   variable_value, 0))
    FROM  information_schema.GLOBAL_STATUS

根据question和com_*的状态变量计算tps和qps

com_commit = show global status like 'com_commit';
com_rollback = show global status like 'com_rollback';
uptime = show global status like 'uptime';
tps=(com_commit + com_rollback)/uptime

show global status where variable_name in('com_select','com_insert','com_delete','com_update');
Get the value of com_* at an interval of 1s, and do the difference calculation
del_diff = (int(mystat2['com_delete'])   - int(mystat1['com_delete']) ) / diff
ins_diff = (int(mystat2['com_insert'])    - int(mystat1['com_insert']) ) / diff
sel_diff = (int(mystat2['com_select'])    - int(mystat1['com_select']) ) / diff
upd_diff = (int(mystat2['com_update'])   - int(mystat1['com_update']) ) / diff

tps= Com_insert/s + Com_update/s + Com_delete/s
qps=Com_select/s + Com_insert/s + Com_update/s + Com_delete/s

如果数据库中有更多的myisam表，问题更适合计算。
如果数据库中有更多的innodb表，com_*数据源更适合计算。

postgresql

使用此查询读取在所有数据库中执行的事务总数：

SELECT sum(xact_commit+xact_rollback) FROM pg_stat_database;

如果您只想为一个数据库使用相同的计数器，请使用：

SELECT xact_commit+xact_rollback FROM pg_stat_database WHERE datname = 'mydb';

要计算 TPS（每秒事务数），请多次运行查询并计算时间间隔内的差异。

有现成的工具，其中之一是http://bucardo.org/wiki/Check_postgres

SQL SERVER

如果您的服务器上有多个实例，您可以运行以下脚本来了解所有实例在过去 10 秒内发生了多少事务。

-- First PASS
DECLARE @First INT
DECLARE @Second INT
SELECT @First = cntr_value
FROM sys.dm_os_performance_counters
WHERE counter_name = 'Transactions/sec'
-- Following is the delay
WAITFOR DELAY '00:00:10'
-- Second PASS
SELECT @Second = cntr_value
FROM sys.dm_os_performance_counters
WHERE counter_name = 'Transactions/sec';
SELECT (@Second - @First) 'TotalTransactions'
GO

实例特定事务
如果您的服务器上有多个实例，并且想要获取有关任何特定实例的详细信息，则可以运行以下脚本以了解该特定实例在过去 10 秒内发生了多少事务。

 -- First PASS
DECLARE @First INT
DECLARE @Second INT
SELECT @First = cntr_value
FROM sys.dm_os_performance_counters
WHERE
OBJECT_NAME = 'MSSQL$SQL19:Databases' AND -- Change name of your server ' AND -- Change name of your server
counter_name = 'Transactions/sec' AND
instance_name = '_Total';
-- Following is the delay
WAITFOR DELAY '00:00:10'
-- Second PASS
SELECT @Second = cntr_value
FROM sys.dm_os_performance_counters
WHERE
OBJECT_NAME = 'MSSQL$SQL19:Databases' AND -- Change name of your server
counter_name = 'Transactions/sec' AND
instance_name = '_Total';
SELECT (@Second - @First) 'TotalTransactions'
GO

数据库特定事务
如果您的服务器上有多个实例，并且想要获取有关任何特定实例和特定数据库的详细信息，则可以运行以下脚本以了解过去 10 秒内针对该特定实例发生了多少事务数据库。

-- First PASS
DECLARE @First INT
DECLARE @Second INT
SELECT @First = cntr_value
FROM sys.dm_os_performance_counters
WHERE
OBJECT_NAME = 'MSSQL$SQL19:Databases' AND -- Change name of your server ' AND -- Change name of your server
counter_name = 'Transactions/sec' AND
instance_name = 'tempdb'; -- Change name of your database
-- Following is the delay
WAITFOR DELAY '00:00:10'
-- Second PASS
SELECT @Second = cntr_value
FROM sys.dm_os_performance_counters
WHERE
OBJECT_NAME = 'MSSQL$SQL19:Databases' AND -- Change name of your server
counter_name = 'Transactions/sec' AND
instance_name = 'tempdb'; -- Change name of your database
SELECT (@Second - @First) 'TotalTransactions'
GO

Oracle 数据库中的在线索引、表、数据文件维护和操作

Oracle Groundbreakers APAC Virtual Tour 2021

演讲主题： Online index、table、datafile maintenance operations in Oracle database
DATE : 12/05/2021

墨天轮
https://www.modb.pro/doc/51651

https://www.modb.pro/event/416

Youtube
https://youtu.be/mVf7DMg6jao

APACOUC

https://apacgbt.mykommu.com/
https://apacgbt.mykommu.com/all_speaker

2021数据技术嘉年华

演讲主题： Oracle数据库中的一些Automatic特性
DATE : 12/24/2021

墨天轮

https://www.modb.pro/dtc2021

链接：https://pan.baidu.com/s/19kkdp-0K19cyU7adCdhTSw
提取码：gwa7

PPT: 2021DTC张维照

环境oracle 19c(19.7) on linux, 数据库做了failover后在open resetlogs数据库时，等待了很久的时间，数据库里查询wait event是’row cache lock’ & ‘cursor: pin S wait on X’， DB alert log显示下面的信息:
Note:
可以看到不断的在输出TTnn Sleep 10 seconds and then try to clear SRLs in N time(s)， TTnn进程为异步模式下的REDO传输进程， 在清理standby redo log时hang, 还出现了row cache enqueue生成了SSD（system state dump）trace, 下面分析trace内容。

分析SSD trace是个烧脑的过程，可以先看汇总信息再看明细，首先是ass 脚本

oracle@oel7db1 SSD]$ awk -f ass1046.awk anbob_p001_47286.trc
awk: ass1046.awk:463: warning: escape sequence `\+' treated as plain `+'

Starting Systemstate 1
..............................................................................
...............................................................................
........................................................
Ass.Awk Version 1.0.46
~~~~~~~~~~~~~~~~~~~~~~
Source file : anbob_p001_47286.trc

System State 1  (2021-12-24 16:29:12.304)
~~~~~~~~~~~~~~   ~~~~~~~~~~~~~~~~~~~~~~~
1:
2:
3: waiting for 'pmon timer'
4: waiting for 'rdbms ipc message'
5: waiting for 'VKTM Logical Idle Wait'
6: waiting for 'rdbms ipc message'
7: waiting for 'rdbms ipc message'
8: waiting for 'SQL*Net message from client'

...
47: waiting for 'Data Guard: Gap Manager'
48: waiting for 'Data Guard: Timer'
49: waiting for 'PX Deq: Txn Recovery Reply'
    Cmd: Unknown(35)
50: waiting for 'heartbeat redo informer'
51: waiting for 'Data Guard: rtrt work'
52: waiting for 'cursor: pin S wait on X'[Mutex 4d078f90]
    Final Blocker: inst: 1, sid: 5036, ser: 11929
53: waiting for 'row cache lock'       [Rcache object=0xcb7e8b50,]
    Cmd: Update
54: waiting for 'cursor: pin S wait on X'[Mutex 4d078f90]
    Final Blocker: inst: 1, sid: 5036, ser: 11929
55: waiting for 'cursor: pin S wait on X'[Mutex 4d078f90]
    Final Blocker: inst: 1, sid: 5036, ser: 11929
56: waiting for 'cursor: pin S wait on X'[Mutex 4d078f90]
    Final Blocker: inst: 1, sid: 5036, ser: 11929
57: waiting for 'cursor: pin S wait on X'[Mutex 4d078f90]
    Final Blocker: inst: 1, sid: 5036, ser: 11929
58: waiting for 'cursor: pin S wait on X'[Mutex 4d078f90]
    Final Blocker: inst: 1, sid: 5036, ser: 11929
59: waiting for 'cursor: pin S wait on X'[Mutex 4d078f90]
    Final Blocker: inst: 1, sid: 5036, ser: 11929
60: waiting for 'cursor: pin S wait on X'[Mutex 4d078f90]
    Final Blocker: inst: 1, sid: 5036, ser: 11929

...
78: waiting for 'cursor: pin S wait on X'[Mutex 4d078f90]
    Final Blocker: inst: 1, sid: 287, ser: 3759
79: waiting for 'cursor: pin S wait on X'[Mutex 4d078f90]
    Final Blocker: inst: 1, sid: 287, ser: 3759
80: waiting for 'cursor: pin S wait on X'[Mutex 4d078f90]
    Final Blocker: inst: 1, sid: 287, ser: 3759
81: waiting for 'cursor: pin S wait on X'[Mutex 4d078f90]
    Final Blocker: inst: 1, sid: 287, ser: 3759
82: waiting for 'cursor: pin S wait on X'[Mutex 4d078f90]
    Final Blocker: inst: 1, sid: 287, ser: 3759
83: waiting for 'cursor: pin S wait on X'[Mutex 4d078f90]
    Final Blocker: inst: 1, sid: 5036, ser: 11929
84: waiting for 'cursor: pin S wait on X'[Mutex 4d078f90]
    Final Blocker: inst: 1, sid: 287, ser: 3759
85: waiting for 'cursor: pin S wait on X'[Mutex 4d078f90]
    Final Blocker: inst: 1, sid: 287, ser: 3759
86: waiting for 'cursor: pin S wait on X'[Mutex 4d078f90]
    Final Blocker: inst: 1, sid: 287, ser: 3759
87: waiting for 'cursor: pin S wait on X'[Mutex 4d078f90]
...


                    Resource Holder State
              Mutex 4d078f90    53: 53: is waiting for 83:
   Rcache object=0xcb7e8b50,    83: 83: is waiting for 53:

PID to SID Mapping
~~~~~~~~~~~~~~~~~~
Pid 53 maps to Sid(s): 5036 5038
Pid 83 maps to Sid(s): 287 290

Object Names
~~~~~~~~~~~~
Mutex 4d078f90
Rcache object=0xcb7e8b50,       cid=3(dc_rollback_segments)

Summary of Wait Events Seen (count>10)
~~~~~~~~~~~~~~~~~~~~~~~~~~~
  No wait events seen more than 10 times

Note:
大量的cursor: pin S wait on X等待，SQL并发解析等待，大量会话在等SID 5036和287,  再看resource holder是产生了deadlock,  PID 53等83 mutex, 83等53 row cache obj,cache_id=3  dc_rollback_segments.

除了ass脚本，Oracle还有一个工具STRA， 格式也不错。

1.0 WAIT EVENTS: Summary of all current wait events for all sessions found in the System State Dump.

HOWTO: Look for wait events related to your issue OR any other that can block or slown down the database.

=================================================================================================================================================

 Occurence  Wait Event                      
 ---------  ----------                      
 159        cursor: pin S wait on X         
 26         rdbms ipc message               
 5          Space Manager: slave idle wait  
 2          watchdog main loop              
 1          wait for unread message on broadcast channel
 1          VKTM Logical Idle Wait          
 1          VKRM Idle                       
 1          SQL*Net message from client     
 1          smon timer                      
 1          row cache lock                  
 1          PX Deq: Txn Recovery Reply      
 1          pmon timer                      
 1          pman timer                      
 1          OFS idle                        
 1          lreg timer                      
 1          heartbeat redo informer         
 1          DIAG idle wait                  
 1          Data Guard: Timer               
 1          Data Guard: rtrt work           
 1          Data Guard: Gap Manager  

2.1 FINAL BLOCKERS: Final Blockers and Occurances.

HOWTO: Look for the blockers that have more occurrencies which means that those sessions are holding more waiters in the database, but keep in mind 
that they are not ROOT blockers or FINAL blockers. Some of them might have or not another blocker which would be also listed here. 

==================================================================================
Occurrence      Blocker
----------      --------- 
     87         inst: 1, sid: 287, ser: 3759
     72         inst: 1, sid: 5036, ser: 11929

2.3 FINAL BLOCKERS: Most likely final blockers found

HOWTO: All blockers that blocking at least one session that is not being blocked or do not have Dumping final blocker in the SSD. In case of Mutex issue the waiters might not appear in this section since the Dumping Final Blocker might not be in the SSD trace for those waiters.
go after MUTEXES section in this report. Start analyzing from the blockers that are holding more waiters per 2.2 FINAL BLOCKERS and 2.1 FINAL BLOCKERS sections.
Consider the time that the blocker is blocking the waiter in the section 3.0 BLOCKED SESSIONS. Keep in mind that deadlocks are not being showed here. 

==================================================================================
Most likely a final blocker is 5036,11929

PROCESS 53: P001
  SO: 0xafef0be8, type: process (2), map: 0xaf58e6b0
  (process) Oracle pid:53, ser:4, calls cur/top: 0xb2341840/0xcb3bc308
            flags : (0x0) -  icon_uid:0 logon_pdbid=0
            flags2: (0x1000),  flags3: (0x10) 
            call error: 0, sess error: 0, txn error 0
            intr queue: empty
    (latch info) hold_bits=0x0 ud_influx=0x329
    (osp latch info) hold_bits=0x0 ud_influx=0x0
    Process Group: DEFAULT, pseudo proc: 0xad70acd0
    O/S info: user: oracle, term: UNKNOWN, ospid: 47286 
    OSD pid info: 
    Short stack dump: 
ksedsts()+426<-ksdxfstk()+58<-ksdxdocmdmultex()+6119<-ksdxdocmdmult()+55<-ksudmp_proc()+1594<-ksudss_main()+1976<-ksudss_opt()+1544<-kqrget()+3744<-kqrLockPo()+246<-kqrpre2()+1043<-kqrpre1()+40<-ktuGetRowCache1()+82<-ktuGetUsegDba()+598<-ktuGetCommitTimesMain_Int()+604<-ktuGetCommitTimesMain()+542<-ktuGetCommitTimes()+149<-ktbgcl1()+2482<-ktrgcm()+9998<-ktrget2()+971<-kdsgrp()+511<-qetlbr()+287<-qertbFetchByRowID()+1176<-opifch2()+3242<-opiall0()+1560<-opikpr()+595<-opiodr()+1202<-rpidrus()+198<-skgmstack()+65<-rpidru()+132<-rpiswu2()+543<-kprball()+1046<-kqdhsr()+1640<-kqrReadFromDB()+338<-kqrpre2()+1931<-kqrpre()+35<-kkestRCSBase()+458<-kkercs()+211<-kkehrd_prealo()+1059<-kkeaprWithFro()+894<-kkepslCompCtl()+907<-kkepsl()+35<-kkecdn()+1291<-kkotap()+16226<-kkoiqb()+9323<-kkooqb()+579<-kkoqbc()+2722<-apakkoqb()+159<-apaqbdDescendents()+488<-apaqbd()+140<-apadrv()+1236<-opitca()+2207<-kksFullTypeCheck()+81<-rpiswu2()+2004<-kksLoadChild()+7818<-kxsGetRuntimeLock()+1983<-kksfbc()+18320<-opiexe()+2834<-opiodr()+1202<-rpidrus()+198<-skgmstack()+65<-rpidru()+132<-rpiswu2()+543<-rpidrv()+1266<-rpiexe()+68<-ktsucr()+1565<-kqrCreateUsingSecondaryKey()+451<-kqrpre2()+3750<-kqrpre1()+40<-kklmsle()+140<-kkmupd()+1090<-opiprsdml()+783<-opiSem()+7418<-opiprs()+321<-kksParseChildCursor()+527<-rpiswu2()+2004<-kksLoadChild()+5283<-kxsGetRuntimeLock()+1983<-kksfbc()+18320<-kkspsc0()+1566<-kksParseCursor()+114<-opiosq0()+2330<-opiodr()+1202<-rpidrus()+198<-skgmstack()+65<-rpidru()+132<-rpiswu2()+543<-rpidrv()+1266<-rpisplu_internal()+474<-ktuscu()+294<-kqrcmt()+978<-ktcCommitTxn_new()+5261<-ktcCommitTxn()+94<-kturfptrSlaveWork()+869<-kturfptrSlaveMain()+762<-kxfprdp_int()+1915<-ksbdispatch()+365<-opirip()+522<-opidrv()+581<-sou2o()+165<-opimai_real()+173<-ssthrdmain()+417<-main()+256<-__libc_start_main()+245
    (session) sid: 5036 ser: 11929 trans: 0x789674d0, creator: 0xaf58e6b0
              flags: (0x41) USR/- flags2: (0x9) -/-/INC
              flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/-
              DID: 0001-0035-000000070000-0000-00000000, short-term DID: 
              txn branch: (nil)
              edition#: 0              user#/name: 0/SYS
              oct: 6, prv: 0, sql: 0xc4f3b268, psql: 0xc3e35de0
              stats: 0xcb452d48, PX stats: 0x12a12c04
    service name: SYS$USERS
    client details:
      O/S info: user: oracle, term: pts/0, ospid: 47286
      machine: zdbm01 program: oracle@zdbm01 (P001)
    Current Wait Stack:
     0: waiting for 'row cache lock'
        cache id=0x3, mode=0x0, request=0x3
        wait_id=63 seq_num=165 snap_id=51
        wait times: snap=0.052141 sec, exc=5 min 4 sec, total=5 min 7 sec
        wait times: max=infinite, heur=5 min 7 sec
        wait counts: calls=99 os=99
        in_wait=1 iflags=0x15a2
    There are 159 sessions blocked by this session.
    Dumping one waiter:
      inst: 1, sid: 5987, ser: 57799
      wait event: 'cursor: pin S wait on X'
        p1: 'idn'=0x4d078f90
        p2: 'value'=0x13ac00000000
      min_blocked_time: 2 secs, waiter_cache_ver: 97
    Wait State:
      fixed_waits=0 flags=0x23 boundary=(nil)/-1
==================================================================================

2.4 FINAL BLOCKERS: Getting the summarized information from blockers found as final blockers.

HOWTO: This section provides information about ALL blockers, not only the ROOT or FINAL ones. The most important thing is to verify if you have any BACKGROUND process 
blocking others sessions. If you want to verify only the ROOT blockers of FINAL blockers to go 2.3 FINAL BLOCKERS section. This section also actually creates the blockers file. 
Be aware that in same cases the blocker is not found in the SSD, so not details will be shown. Make sure if you are analyzing a RAC environment that you are following the correctly the sh stra.sh intructions.
==================================================================================
Occurrence      Blocker
----------      --------- 
      87         inst: 1, sid: 287, ser: 3759


========= More details about the blocker =========

PROCESS 83: P00V
  SO: 0xafef24f8, type: process (2), map: 0xaf5a2840
  (process) Oracle pid:83, ser:1, calls cur/top: 0xb2344ef0/0xcabec0c0
            flags : (0x0) -  icon_uid:0 logon_pdbid=0
            flags2: (0x0),  flags3: (0x10) 
            call error: 0, sess error: 0, txn error 0
            intr queue: empty
    (latch info) hold_bits=0x0 ud_influx=0x5
    Process Group: DEFAULT, pseudo proc: 0xad70acd0
    O/S info: user: oracle, term: UNKNOWN, ospid: 47347 
    OSD pid info: 
    Short stack dump: 
ksedsts()+426<-ksdxfstk()+58<-ksdxcb()+872<-sspuser()+200<-__sighandler()<-__select()+19<-skgpwwait()+420<-kgxWait()+836<-kgxSharedExamine()+785<-kxsGetRuntimeLock()+246<-kkscsCheckCursor()+550<-kkscsSearchChildList()+1324<-kksfbc()+15645<-kkspsc0()+1566<-kksParseCursor()+114<-opiosq0()+2330<-opiodr()+1202<-rpidrus()+198<-skgmstack()+65<-rpidru()+132<-rpiswu2()+543<-rpidrv()+1266<-rpisplu_internal()+474<-ktuscu()+294<-kqrcmt()+978<-ktcCommitTxn_new()+5261<-ktcCommitTxn()+94<-kturfptrSlaveWork()+869<-kturfptrSlaveMain()+762<-kxfprdp_int()+1915<-ksbdispatch()+365<-opirip()+522<-opidrv()+581<-sou2o()+165<-opimai_real()+173<-ssthrdmain()+417<-main()+256<-__libc_start_main()+245
    (session) sid: 287 ser: 3759 trans: 0x76e67cb0, creator: 0xaf5a2840
              flags: (0x41) USR/- flags2: (0x40009) -/-/INC
              flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/-
              DID: 0001-0053-000000020000-0000-00000000, short-term DID: 
              txn branch: (nil)
              edition#: 0              user#/name: 0/SYS
              oct: 0, prv: 0, sql: 0xc4f3b268, psql: 0xc3e35de0
              stats: 0xae2d1810, PX stats: 0x12a12c04
    service name: SYS$USERS
    client details:
      O/S info: user: oracle, term: pts/0, ospid: 47347
      machine: zdbm01 program: oracle@zdbm01 (P00V)
    Current Wait Stack:
     0: waiting for 'cursor: pin S wait on X'
        idn=0x4d078f90, value=0x13ac00000000, where=0x500000000
        wait_id=22 seq_num=24 snap_id=1
        wait times: snap=13.060294 sec, exc=13.060294 sec, total=13.060294 sec
        wait times: max=infinite, heur=5 min 11 sec
        wait counts: calls=0 os=0
        in_wait=1 iflags=0x5a2
    There is at least one session blocking this session.
      Dumping 1 direct blocker(s):
        inst: 1, sid: 5036, ser: 11929
      Dumping final blocker:
        inst: 1, sid: 5036, ser: 11929
    There are 159 sessions blocked by this session.
    Dumping one waiter:
      inst: 1, sid: 5036, ser: 11929
      wait event: 'row cache lock'
        p1: 'cache id'=0x3
        p2: 'mode'=0x0
      min_blocked_time: 0 secs, waiter_cache_ver: 98
    Wait State:
      fixed_waits=0 flags=0x22 boundary=(nil)/-1

If more than one PROCESS is found this is because you have more than 1 SSD in the same file.

Final Blockers - Current/Previous SQLs
(session) sid: 287 ser: 3759

  Current SQL:    c4f3b268
        ObjectName:  Name=update /*+ rule */ undo$ set name=:2,file#=:3,block#=:4,status$=:5,user#=:6,undosqn=:7,xactsqn=:8,scnbas=:9,scnwrp=:10,inst#=:11,ts#=:12,spare1=:13 where us#=:1 
          FullHashValue=8f7b9f1e491021278dfa3acf4d078f90 Namespace=SQL AREA(00) Type=CURSOR(00) ContainerId=0 ContainerUid=0 Identifier=1292341136 OwnerIdn=0 
        Statistics:  InvalidationCount=0 ExecutionCount=0 LoadCount=2 ActiveLocks=160 TotalLockCount=324 TotalPinCount=37 

    Previous SQL:    c3e35de0
        ObjectName:  Name=select /*+ rule */ name,file#,block#,status$,user#,undosqn,xactsqn,scnbas,scnwrp,DECODE(inst#,0,NULL,inst#),ts#,spare1 from undo$ where us#=:1 
          FullHashValue=f9484ec6c26299ad75b4690c15920bf2 Namespace=SQL AREA(00) Type=CURSOR(00) ContainerId=0 ContainerUid=0 Identifier=361892850 OwnerIdn=0 
        Statistics:  InvalidationCount=0 ExecutionCount=176 LoadCount=4 ActiveLocks=160 TotalLockCount=165 TotalPinCount=1 



(session) sid: 5036 ser: 11929

  Current SQL:    c4f3b268
        ObjectName:  Name=update /*+ rule */ undo$ set name=:2,file#=:3,block#=:4,status$=:5,user#=:6,undosqn=:7,xactsqn=:8,scnbas=:9,scnwrp=:10,inst#=:11,ts#=:12,spare1=:13 where us#=:1 
          FullHashValue=8f7b9f1e491021278dfa3acf4d078f90 Namespace=SQL AREA(00) Type=CURSOR(00) ContainerId=0 ContainerUid=0 Identifier=1292341136 OwnerIdn=0 
        Statistics:  InvalidationCount=0 ExecutionCount=0 LoadCount=2 ActiveLocks=160 TotalLockCount=324 TotalPinCount=37 

    Previous SQL:    c3e35de0
        ObjectName:  Name=select /*+ rule */ name,file#,block#,status$,user#,undosqn,xactsqn,scnbas,scnwrp,DECODE(inst#,0,NULL,inst#),ts#,spare1 from undo$ where us#=:1 
          FullHashValue=f9484ec6c26299ad75b4690c15920bf2 Namespace=SQL AREA(00) Type=CURSOR(00) ContainerId=0 ContainerUid=0 Identifier=361892850 OwnerIdn=0 
        Statistics:  InvalidationCount=0 ExecutionCount=176 LoadCount=4 ActiveLocks=160 TotalLockCount=165 TotalPinCount=1 



Mutexes Holders - Current/Previous SQLs


(session) sid: 5036 ser: 11929

  Current SQL:    c4f3b268

        ObjectName:  Name=update /*+ rule */ undo$ set name=:2,file#=:3,block#=:4,status$=:5,user#=:6,undosqn=:7,xactsqn=:8,scnbas=:9,scnwrp=:10,inst#=:11,ts#=:12,spare1=:13 where us#=:1 
          FullHashValue=8f7b9f1e491021278dfa3acf4d078f90 Namespace=SQL AREA(00) Type=CURSOR(00) ContainerId=0 ContainerUid=0 Identifier=1292341136 OwnerIdn=0 
        Statistics:  InvalidationCount=0 ExecutionCount=0 LoadCount=2 ActiveLocks=160 TotalLockCount=324 TotalPinCount=37 

    Previous SQL:    c3e35de0

        ObjectName:  Name=select /*+ rule */ name,file#,block#,status$,user#,undosqn,xactsqn,scnbas,scnwrp,DECODE(inst#,0,NULL,inst#),ts#,spare1 from undo$ where us#=:1 
          FullHashValue=f9484ec6c26299ad75b4690c15920bf2 Namespace=SQL AREA(00) Type=CURSOR(00) ContainerId=0 ContainerUid=0 Identifier=361892850 OwnerIdn=0 
        Statistics:  InvalidationCount=0 ExecutionCount=176 LoadCount=4 ActiveLocks=160 TotalLockCount=165 TotalPinCount=1 

信息比较多，如果只有ass，自己也可以手动分析，如果大量session在cursor pin等sid 287 ，287等5036, 5036等row cache lock ，

    ----------------------------------------
    SO: 0xabff1228, type: session (4), map: 0xa9420f58
        state: LIVE (0x4532), flags: 0x1
        owner: 0xafef0be8, proc: 0xafef0be8
        link: 0xabff1248[0xc7298d68, 0xafef0c58]
        child list count: 26, link: 0xabff1298[0xc7297d68, 0x76fd3020]
        pg: 0
    SOC: 0xa9420f58, type: session (4), map: 0xabff1228
         state: LIVE (0x99fc), flags: INIT (0x1)
    (session) sid: 5036 ser: 11929 trans: 0x789674d0, creator: 0xaf58e6b0
              flags: (0x41) USR/- flags2: (0x9) -/-/INC
              flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/-
              DID: 0001-0035-000000070000-0000-00000000, short-term DID: 
              txn branch: (nil)
              edition#: 0              user#/name: 0/SYS
              oct: 6, prv: 0, sql: 0xc4f3b268, psql: 0xc3e35de0
              stats: 0xcb452d48, PX stats: 0x12a12c04
    service name: SYS$USERS
    client details:
      O/S info: user: oracle, term: pts/0, ospid: 47286
      machine: zdbm01 program: oracle@zdbm01 (P001)
    Current Wait Stack:
     0: waiting for 'row cache lock'
        cache id=0x3, mode=0x0, request=0x3
        wait_id=63 seq_num=165 snap_id=51
        wait times: snap=0.052141 sec, exc=5 min 4 sec, total=5 min 7 sec
        wait times: max=infinite, heur=5 min 7 sec
        wait counts: calls=99 os=99
        in_wait=1 iflags=0x15a2
    There are 159 sessions blocked by this session.


 LibraryHandle:  Address=0xc4f3b268 Hash=4d078f90 LockMode=N PinMode=0 LoadLockMode=0 Status=VALD 
        ObjectName:  Name=update /*+ rule */ undo$ set name=:2,file#=:3,block#=:4,status$=:5,user#=:6,undosqn=:7,xactsqn=:8,scnbas=:9,scnwrp=:10,inst#=:11,ts#=:12,spare1=:13 where us#=:1 

根据proc cur调用查call
SOC: 0xb2341840, type: call (3), map: 0xc72981c8
                   state: LIVE (0x99fc), flags: INIT (0x1)
              (call) sess: cur a9425f28, rec a9425f28, usr a9420f58; flg:0 fl2:1; depth:5
              svpt(xcb:0x789674d0 sptn:0x2d5 uba: 0x00000000.0000.00 uba: 0x00000000.0000.00)
                ----------------------------------------
                SO: 0xc72a2ec8, type: row cache enqueues (111), map: 0xb3d3b008
                    state: LIVE (0x4532), flags: 0x0
                    owner: 0xc72981c8, proc: 0xafef0be8
                    link: 0xc72a2ee8[0x91ce36c0, 0xc7298238]
                    pg: 0
                SOC: 0xb3d3b008, type: row cache enqueues (111), map: 0xc72a2ec8
                     state: LIVE (0x99fc), flags: INIT (0x1)
                row cache enqueue: count=1 session=0xa9420f58 object=0xcb7e8b50, request=S
                flag=03 WAI/TRC/-/-/-/-/-/- savepoint=0x2d7
                row cache parent object: addr=0xcb7e8b50 cid=3(dc_rollback_segments) conid=0 conuid=0
                hash=fc81f131 typ=21 transaction=0x76e67cb0 flags=0000012a inc=1, pdbinc=1
                version=2 mtx version=1152
                own=0xcb7e8c20[0xcb77f970,0xcb77f970] wat=0xcb7e8c30[0xb3d3b088,0xb3d3b088] mode=X req=N
                status=VALID/UPDATE/-/-/IO/-/-/-/-  KGH pinned

--查找0xcb7e8b50
     ----------------------------------------
        SO: 0xc72bb098, type: row cache enqueues (111), map: 0xcb77f8f0
            state: LIVE (0x4532), flags: 0x0
            owner: 0x789cd770, proc: 0xafef24f8
            link: 0xc72bb0b8[0x789cd7e0, 0x789cd7e0]
            pg: 0
        SOC: 0xcb77f8f0, type: row cache enqueues (111), map: 0xc72bb098
             state: LIVE (0x99fc), flags: INIT (0x1)
        row cache enqueue: count=1 session=0xad9ea458 object=0xcb7e8b50, mode=X
        flag=00 -/-/-/-/-/-/-/- savepoint=0x3
        row cache parent object: addr=0xcb7e8b50 cid=3(dc_rollback_segments) conid=0 conuid=0
        hash=fc81f131 typ=21 transaction=0x76e67cb0 flags=0000012a inc=1, pdbinc=1
        version=2 mtx version=1156
        own=0xcb7e8c20[0xcb77f970,0xcb77f970] wat=0xcb7e8c30[0xb3d3b088,0xb3d3b088] mode=X req=N
        status=VALID/UPDATE/-/-/IO/-/-/-/-  KGH pinned

-- 查找0xad9ea458
    ----------------------------------------
    SO: 0xaff95298, type: session (4), map: 0xad9ea458
        state: LIVE (0x4532), flags: 0x1
        owner: 0xafef24f8, proc: 0xafef24f8
        link: 0xaff952b8[0xc72bb1b8, 0xafef2568]
        child list count: 6, link: 0xaff95308[0xc72bae38, 0x76fd4430]
        pg: 0
    SOC: 0xad9ea458, type: session (4), map: 0xaff95298
         state: LIVE (0x99fc), flags: INIT (0x1)
    (session) sid: 287 ser: 3759 trans: 0x76e67cb0, creator: 0xaf5a2840
              flags: (0x41) USR/- flags2: (0x40009) -/-/INC
              flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/-
              DID: 0001-0053-000000020000-0000-00000000, short-term DID: 
              txn branch: (nil)
              edition#: 0              user#/name: 0/SYS
              oct: 0, prv: 0, sql: 0xc4f3b268, psql: 0xc3e35de0
              stats: 0xae2d1810, PX stats: 0x12a12c04
    service name: SYS$USERS
    client details:
      O/S info: user: oracle, term: pts/0, ospid: 47347
      machine: zdbm01 program: oracle@zdbm01 (P00V)
    Current Wait Stack:
     0: waiting for 'cursor: pin S wait on X'
        idn=0x4d078f90, value=0x13ac00000000, where=0x500000000
        wait_id=22 seq_num=24 snap_id=1
        wait times: snap=13.060294 sec, exc=13.060294 sec, total=13.060294 sec
        wait times: max=infinite, heur=5 min 11 sec
        wait counts: calls=0 os=0
        in_wait=1 iflags=0x5a2
    There is at least one session blocking this session.
      Dumping 1 direct blocker(s):
        inst: 1, sid: 5036, ser: 11929
      Dumping final blocker:
        inst: 1, sid: 5036, ser: 11929
    There are 159 sessions blocked by this session.

Note:
5036 等待row cache lock, 0xcb7e8b50 请求S,  但是该资源被 287 以X 持有，它有在等待cursor pin,等5036, 构成列锁。

此问题比较像Bug 30931981 Open Reset Logs Hangs With ‘row cache lock’ and ‘cursor: pin s wait for x’ Waits，但是不是DC_USER, 包括c函数调用也有出现

ktuscu ==> kernel transaction undo segment row Cache update
kklmsle ==> kernel compile look? m? supplemental logging enabled

which is used to check see if we have supplemental logging enabled on the table. But since this lookup is for a data dictionary object, we should not be having
logging enabled.

还有一个Bug 31747989 中提到的参数 “_min_undosegs_for_parallel_fptr”=0 之前已经调整。

本次安装one-off patch后重启正常。

–enjoy —

Oracle 19c(19.9) RAC on linux, 应用执行一个update sql时报错ora-600, 错误日志如下

ORA-00600: internal error code, arguments: [kkshhcdel:wrong-bucket], [0x58445A790], [0x000000000], [0x67DFB5820], [], [], [], [], [], [], [], []

----- Current SQL Statement for this session (sql_id=1jkhptdvurr4q) -----
update XXX_LOG set sendstatus=:1 ,ERROR_INFO=:2 ,process_time=sysdate where rowid = chartorowid(:3 )
[TOC00003-END]

[TOC00004]
----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
...
dbgePostErrorKGE()+  call     dbgexProcessError()  7FFFF7F436D0 7FFFF7F488A0
1853                                               000000001 000000000
                                                   000000000 ? 000000082 ?
dbkePostKGE_kgsf()+  call     dbgePostErrorKGE()   7FFFF7F839A0 7FFFF7E0AA88
71                                                 000000258 000000000 ?
                                                   000000000 ? 000000082 ?
kgeadse()+447        call     dbkePostKGE_kgsf()   7FFFF7F839A0 7FFFF7E0AA88
                                                   000000258 000000000 ?
                                                   000000000 ? 000000082 ?
kgerinv_internal()+  call     kgeadse()            7FFFF7F839A0 ? 7FFFF7E0AA88
44                                                 000000258 ? 013D772D0
                                                   000000000 000000003
kgerinv()+40         call     kgerinv_internal()   7FFFF7F839A0 ? 7FFFF7E0AA88 ?
                                                   000000258 ? 013D772D0 ?
                                                   000000000 ? 000000003 ?
kgeasnmierr()+146    call     kgerinv()            7FFFF7F839A0 ? 7FFFF7E0AA88 ?
                                                   000000258 ? 013D772D0 ?
                                                   000000000 ? 000000003 ?
kkshhcdel()+1375     call     kgeasnmierr()        7FFFF7F839A0 ? 7FFFF7E0AA88 ?
                                                   000000258 ? 013D772D0 ?
                                                   000000002 58445A790
kksfbc()+17471       call     kkshhcdel()          7FFFF7F839A0 ? 7FFFF7E0AA88 ?
                                                   58445A7E0 ? 67DFB5800 ?
                                                   000000002 ? 58445A790 ?
opiexe()+2895        call     kksfbc()             7FFFF268C4A8 7FFFF7E0AA88 ?
                                                   58445A7E0 ? 67DFB5800 ?
                                                   000000002 ? 58445A790 ?
kpoal8()+2387        call     opiexe()             000000049 7FFFF7E0AA88 ?
                                                   7FFFFFFF7FE0 67DFB5800 ?
                                                   000000002 ? 58445A790 ?
opiodr()+1202        call     kpoal8()             00000005E 000000026
                                                   7FFFFFFFBA40 67DFB5800 ?
                                                   000000002 ? 58445A790 ?

kkshhcdel – kernel compile shared objects (cursor) cursor hash table hash chain child delete

kksfbc – kernel compile shared objects (cursor) find bound cursor

An Ora-600 [kkshindel:wrong-bucket] might have been seen if we have concurrent access to the same cursor across multiple sessions. This has
now be addressed.

应该是在SQL解析阶段，多个session一致性访问相同的sql cursor时， 获取了hash chain后，遍历hash table时，查找hash bucket时出错。还是发生在shared pool内存中， 判断问题是SQL级，影响该SQL, 建议尝试flush shared pool。

目前oracle中未发现匹配bug和one-off patch.  本案例在flush shared pool恢复正常。

BTW， 19c RU选择时，建议19.11以上，相对问题少一些。

开始了19c的躺雷模式， 再次建议选择ORACLE 19C版本时安装19.11 以上RU。 最近一客户升级了19C, 本月拆分区时遇到了ora-1 内部字典表数据唯一性冲突， 下面简单记录，报错信息如下:

ALTER TABLE ANBOB.TLOG SPLIT PARTITION PART_110_MAX AT (110, TO_DATE(' 2022-02-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS')) INTO (PARTITION PART_110_202201 ,PARTITION PART_310_MAX )
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 1
ORA-00001: unique constraint (SYS.I_INDPART_BOPART$) violated

分析递归SQL,当然做个10046 event trace 就可以，从报错的索引也能知道对象。 下面先看10046 trace

-- 先找exec error
ERROR #140737256599760:err=1 tim=3429453098383

--  根据cursor# 找binds
EXEC #140737256599760:c=184,e=184,p=0,cr=2,cu=7,mis=0,r=1,dep=1,og=4,plh=2739666332,tim=3429453097448
CLOSE #140737256599760:c=1,e=1,dep=1,type=3,tim=3429453097465
PARSE #140737256599760:c=4,e=3,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=2739666332,tim=3429453097475
BINDS #140737256599760:

 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=48 off=0
  kxsbbbfp=7ffff2f4b010  bln=22  avl=03  flg=05
  value=220
 Bind#1
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=0 off=24
  kxsbbbfp=7ffff2f4b028  bln=22  avl=05  flg=01
  value=5832439

-- 根据 currsor# 找sql parse
PARSING IN CURSOR #140737256599760 len=46 dep=1 uid=0 oct=6 lid=0 tim=3429453095567 hv=3952657371 ad='559374eb0' sqlid='gk5aj2zptjhyv'
update indpart$ set part# = :1 where obj# = :2
END OF STMT

Note:
可见是在update indpart$表，把part#更新为220， obj#能确认业务索引对象， 违反了indpart$的唯一约束。

SQL> @ind SYS.I_INDPART_BOPART$
Display indexes where table or index name matches %SYS.I_INDPART_BOPART$%...

TABLE_OWNER          TABLE_NAME                     INDEX_NAME                     POS# COLUMN_NAME                    DSC
-------------------- ------------------------------ ------------------------------ ---- ------------------------------ ----
SYS                  INDPART$                       I_INDPART_BOPART$                 1 BO#
                                                                                      2 PART#


INDEX_OWNER          TABLE_NAME                     INDEX_NAME                     IDXTYPE    UNIQ STATUS   PART TEMP  H     LFBLKS           NDK   NUM_ROWS       CLUF LAST_ANAL DEGREE VISIBILIT
-------------------- ------------------------------ ------------------------------ ---------- ---- -------- ---- ---- -- ---------- ------------- ---------- ---------- --------- ------ ---------
SYS                  INDPART$                       I_INDPART_BOPART$              NORMAL     YES  VALID    NO   N     1          1           332        332        170 28-DEC-21 1      VISIBLE

SQL> @desc indpart$
           Name                            Null?    Type
           ------------------------------- -------- ----------------------------
    1      OBJ#                            NOT NULL NUMBER
    2      DATAOBJ#                                 NUMBER
    3      BO#                             NOT NULL NUMBER
    4      PART#                           NOT NULL NUMBER
    5      HIBOUNDLEN                      NOT NULL NUMBER
    6      HIBOUNDVAL                               LONG
    7      FLAGS                           NOT NULL NUMBER
    8      TS#                             NOT NULL NUMBER
    9      FILE#                           NOT NULL NUMBER
   10      BLOCK#                          NOT NULL NUMBER
   11      PCTFREE$                        NOT NULL NUMBER
   12      PCTTHRES$                                NUMBER
   13      INITRANS                        NOT NULL NUMBER
   14      MAXTRANS                        NOT NULL NUMBER
   15      ANALYZETIME                              DATE
   16      SAMPLESIZE                               NUMBER
   17      ROWCNT                                   NUMBER
   18      BLEVEL                                   NUMBER
   19      LEAFCNT                                  NUMBER
   20      DISTKEY                                  NUMBER
   21      LBLKKEY                                  NUMBER
   22      DBLKKEY                                  NUMBER
   23      CLUFAC                                   NUMBER
   24      SPARE1                                   NUMBER
   25      SPARE2                                   NUMBER
   26      SPARE3                                   NUMBER
   27      INCLCOL                                  NUMBER
   28      BHIBOUNDVAL                              BLOB


-- cdpart.bsq

create table indpart$ (
  obj#        number not null,                 /* object number of partition */
  /* DO NOT CREATE INDEX ON DATAOBJ#  AS IT WILL BE UPDATED IN A SPACE
   * TRANSACTION DURING TRUNCATE */
  dataobj#             number,                   /* data layer object number */
  bo#         number not null,                /* object number of base index */
  part#       number not null,
                         /* partition number (see discussion under TABPART$) */
  hiboundlen  number not null,      /* length of high bound value expression */
  hiboundval  long ,                  /* text of high bound value expression */
...

NOTE:
内部在更新 indpart$时，因为违反了bo#, part#唯一性冲突。索引基于的对象bo#是不变的，拆分区增加的就是part#. 查询indpart$表，确实记录已存在。 正常情况下基于某个段分区local索引和段分区顺序相同的part#也相同，如表分区part# 对应的分区索引part#是相等的。 这里是因为local索引更新出现了不一致。

下面手动创建个表并还原一下问题。

  CREATE TABLE "ANBOB"."T"
   (    "ID" NUMBER(*,0),
        "CTIME" DATE,
        "NAME" VARCHAR2(100)
   ) PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
  STORAGE(
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS"
  PARTITION BY RANGE ("CTIME")
 (PARTITION "P1"  VALUES LESS THAN (TO_DATE(' 2015-01-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION DEFERRED
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
  ...
;

  CREATE INDEX "ANBOB"."T_I1" ON "ANBOB"."T" ("CTIME", "NAME") LOCAL;

SQL> select  obj#,part# ,subobject_name from indpart$ a,dba_objects b where a.obj#=b.object_id and b.object_name='T_I1' and part#>150 order by 2;

      OBJ#      PART# SUBOBJECT_NAME
---------- ---------- ------------------------------
     78804        160 P16
     78805        170 P17
     78806        180 P18
     78807        190 P19
     79152        199 P19_1
     79156        200 P19_2
     79160        201 P19_3
     79164        202 P19_4
     79168        203 P19_5
     79172        204 P19_6
     79176        205 P19_7
     79180        206 P19_8
     79184        207 P19_9
     79188        208 P19_10
     79192        209 P19_11
     79196        219 P19_12
     79200        228 P19_13
     79199        230 P20
     79112        239 P21
     79116        249 P22
     79120        259 P23
     79124        269 P24
     79128        279 P25
     79132        289 P26
     79136        300 P27
     79140        311 P28
     79144        321 P29
     79148        331 P29_1
     79147        341 P30

29 rows selected.

SQL> update indpart$ set part#=280 where obj#=79132;
1 row updated.

SQL> commit;
Commit complete.

SQL> alter system flush shared_pool;
System altered.

SQL> alter table anbob.t split partition p25 into(partition p24_1 values less than (to_date('20190401 01','yyyymmdd hh24')),partition p25);

Table altered.

SQL> select  obj#,part# ,subobject_name from indpart$ a,dba_objects b where a.obj#=b.object_id and b.object_name='T_I1' and part#>150 order by 2;

      OBJ#      PART# SUBOBJECT_NAME
---------- ---------- ------------------------------
     78804        160 P16
     78805        170 P17
     78806        180 P18
     78807        190 P19
     79152        199 P19_1
     79156        200 P19_2
     79160        201 P19_3
     79164        202 P19_4
     79168        203 P19_5
     79172        204 P19_6
     79176        205 P19_7
     79180        206 P19_8
     79184        207 P19_9
     79188        208 P19_10
     79192        209 P19_11
     79196        219 P19_12
     79200        228 P19_13
     79199        230 P20
     79112        239 P21
     79116        249 P22
     79120        259 P23
     79124        269 P24
     79206        278 P24_1
     79205        280 P25
     79132        289 P26
     79136        300 P27
     79140        311 P28
     79144        321 P29
     79148        331 P29_1
     79147        341 P30

SQL> alter table anbob.t split partition p25 into(partition p24_2 values less than (to_date('20190401 02','yyyymmdd hh24')),partition p25);
Table altered.

SQL> select  obj#,part# ,subobject_name from indpart$ a,dba_objects b where a.obj#=b.object_id and b.object_name='T_I1' and part#>150 order by 2;
      OBJ#      PART# SUBOBJECT_NAME
---------- ---------- ------------------------------
     78804        160 P16
     78805        170 P17
     78806        180 P18
     78807        190 P19
     79152        199 P19_1
     79156        200 P19_2
     79160        201 P19_3
     79164        202 P19_4
     79168        203 P19_5
     79172        204 P19_6
     79176        205 P19_7
     79180        206 P19_8
     79184        207 P19_9
     79188        208 P19_10
     79192        209 P19_11
     79196        219 P19_12
     79200        228 P19_13
     79199        230 P20
     79112        239 P21
     79116        249 P22
     79120        259 P23
     79124        269 P24
     79206        278 P24_1
     79210        279 P24_2
     79209        281 P25
     79132        289 P26
     79136        300 P27
     79140        311 P28
     79144        321 P29
     79148        331 P29_1
     79147        341 P30

31 rows selected.

SQL> update indpart$ set part#=280 where obj#=79148;

1 row updated.

SQL> commit;

Commit complete.

SQL> select  obj#,part# ,subobject_name from indpart$ a,dba_objects b where a.obj#=b.object_id and b.object_name='T_I1' and part#>150 order by 2;

      OBJ#      PART# SUBOBJECT_NAME
---------- ---------- ------------------------------
     78804        160 P16
     78805        170 P17
     78806        180 P18
     78807        190 P19
     79152        199 P19_1
     79156        200 P19_2
     79160        201 P19_3
     79164        202 P19_4
     79168        203 P19_5
     79172        204 P19_6
     79176        205 P19_7
     79180        206 P19_8
     79184        207 P19_9
     79188        208 P19_10
     79192        209 P19_11
     79196        219 P19_12
     79200        228 P19_13
     79199        230 P20
     79112        239 P21
     79116        249 P22
     79120        259 P23
     79124        269 P24
     79206        278 P24_1
     79210        279 P24_2
     79148        280 P29_1
     79209        281 P25
     79132        289 P26
     79136        300 P27
     79140        311 P28
     79144        321 P29
     79147        341 P30

31 rows selected.

SQL> alter table anbob.t split partition p25 into(partition p24_3 values less than (to_date('20190401 03','yyyymmdd hh24')),partition p25);
alter table anbob.t split partition p25 into(partition p24_3 values less than (to_date('20190401 03','yyyymmdd hh24')),partition p25)
                  *
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 1
ORA-00001: unique constraint (SYS.I_INDPART_BOPART$) violated


SQL> select  obj#,part# ,subobject_name from indpart$ a,dba_objects b where a.obj#=b.object_id and b.object_name='T_I1' and part#>150 order by 2;

      OBJ#      PART# SUBOBJECT_NAME
---------- ---------- ------------------------------
     78804        160 P16
     78805        170 P17
     78806        180 P18
     78807        190 P19
     79152        199 P19_1
     79156        200 P19_2
     79160        201 P19_3
     79164        202 P19_4
     79168        203 P19_5
     79172        204 P19_6
     79176        205 P19_7
     79180        206 P19_8
     79184        207 P19_9
     79188        208 P19_10
     79192        209 P19_11
     79196        219 P19_12
     79200        228 P19_13
     79199        230 P20
     79112        239 P21
     79116        249 P22
     79120        259 P23
     79124        269 P24
     79206        278 P24_1
     79210        279 P24_2
     79148        280 P29_1
     79209        281 P25
     79132        289 P26
     79136        300 P27
     79140        311 P28
     79144        321 P29
     79147        341 P30

31 rows selected.

note:
因为oracle part#增长的控制分几种情况， 前2次没有触发,下一篇再分享part#的生成方法， 第3次模拟出了问题。

下面分析问题

select i.obj#, ip.obj#, tp.phypart#, ip.phypart#
from tabpartv$ tp, indpartv$ ip, ind$ i
where ip.bo# = i.obj#
and tp.bo# = i.bo#
and tp.part# = ip.part#
  6  and tp.phypart# != ip.phypart#;

      OBJ#       OBJ#   PHYPART#   PHYPART#
---------- ---------- ---------- ----------
     78788      79148        281        280
     78788      79209        289        281
     78788      79132        300        289
     78788      79136        311        300
     78788      79140        321        311
     78788      79144        331        321

6 rows selected.

create or replace view tabpartv$
  (obj#, dataobj#, bo#, part#, hiboundlen, hiboundval, ts#, file#, block#,
   pctfree$, pctused$, initrans, maxtrans, flags, analyzetime, samplesize,
   rowcnt, blkcnt, empcnt, avgspc, chncnt, avgrln, phypart#)
as select obj#, dataobj#, bo#,
          row_number() over (partition by bo# order by part#),
          hiboundlen, hiboundval, ts#, file#, block#, pctfree$, pctused$,
          initrans, maxtrans, flags, analyzetime, samplesize, rowcnt, blkcnt,
          empcnt, avgspc, chncnt, avgrln, part#
from tabpart$
where bitand(flags, 8388608) = 0           /* filter out hidden partitions */
/

create or replace view indpartv$
  (obj#, dataobj#, bo#, part#, hiboundlen, hiboundval, flags, ts#, file#,
   block#, pctfree$, pctthres$, initrans, maxtrans, analyzetime, samplesize,
   rowcnt, blevel, leafcnt, distkey, lblkkey, dblkkey, clufac, spare1,
   spare2, spare3, inclcol, phypart#)
as select obj#, dataobj#, bo#,
          row_number() over (partition by bo# order by part#),
          hiboundlen, hiboundval, flags, ts#, file#, block#,
          pctfree$, pctthres$, initrans, maxtrans, analyzetime, samplesize,
          rowcnt, blevel, leafcnt, distkey, lblkkey, dblkkey, clufac, spare1,
          spare2, spare3, inclcol, part#
from indpart$
where bitand(flags, 8388608) = 0           /* filter out hidden partitions */
/

note：
这里用到了2个内部视图tabpartv$ 和indpartv$，也是基于tabpart$和indpart$， 列出了问题part#是从拆的分区开始到倒数第二个分区(最大分区除外).

这里能想到的有3种解决方法：
1， 把索引删除，重新创建(not rebuild)
但是有特殊情况，对于本次出问题的是个lob index（SYS_ILxxxxx）, 不可以删除index。并且move lob级联rebuild lob index也无法解决。
2， 安装one-off patch
Bug 32259535 ORA-1/ORA-00001: unique constraint (sys.i_indpart_bopart$) during ALTER TABLE SPLIT PARTITION, 影响范围RU<=19.10.
3, 更新字典基表

下面用使用最快的第3种方法

-- s1
SQL> lock table tabpart$ in exclusive mode;
Table(s) Locked.

SQL> lock table partobj$ in exclusive mode;
Table(s) Locked.

-- s2
create table bakpart_byanbob(ind_obj#, ip_obj#, tbl_phypart#, idx_phypart#) as
select i.obj#, ip.obj#, tp.phypart#, ip.phypart#
from tabpartv$ tp, indpartv$ ip, ind$ i
where ip.bo# = i.obj#
and tp.bo# = i.bo#
and tp.part# = ip.part#
and tp.phypart# != ip.phypart#;
 
update indpart$ ip
set part# = (select tbl_phypart# from bakpart_byanbob
where ip.obj# = ip_obj#)
   where ip.obj# in (select ip_obj# from bakpart_byanbob);

SQL> commit;
Commit complete.

SQL> alter system flush shared_pool;
System altered.

-- s1
SQL> commit;
Commit complete.

SQL> alter table anbob.t split partition p25 into(partition p24_3 values less than (to_date('20190401 03','yyyymmdd hh24')),partition p25);
Table altered.

注意：

问题解决。

以上操作危险，未在专业人员指导下，请勿操作。

Troubleshooting 19c ORA-1: unique constraint (sys.i_indpart_bopart$) during ALTER TABLE SPLIT PARTITION

上一篇blog中提到LOCAL 分区，分区索引part#和分区表part#是相等的，上一篇在还原那个问题时，让part#占用时发现还不是那么简单，如果用10046跟split partition,时而insert,时而存在update, 其实oracle在这方面也是做了优化， 在split分区位置和个数也有性能上的细微的差别。

SQL> @ddl anbob.t

PL/SQL procedure successfully completed.


DBMS_METADATA.GET_DDL(OBJECT_TYPE,OBJECT_NAME,OWNER)
------------------------------------------------------------------------------------------------------------- 

  CREATE TABLE "ANBOB"."T"
   (    "ID" NUMBER(*,0),
        "CTIME" DATE,
        "NAME" VARCHAR2(100)
   ) PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
  STORAGE(
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS"
  PARTITION BY RANGE ("CTIME")
 (PARTITION "P1"  VALUES LESS THAN (TO_DATE(' 2015-01-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION DEFERRED
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ,
 PARTITION "P2"  VALUES LESS THAN (TO_DATE(' 2015-02-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION IMMEDIATE
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(INITIAL 8388608 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ,
 PARTITION "P3"  VALUES LESS THAN (TO_DATE(' 2015-03-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION IMMEDIATE
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(INITIAL 8388608 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ,
 PARTITION "P4"  VALUES LESS THAN (TO_DATE(' 2015-04-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION IMMEDIATE
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(INITIAL 8388608 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ,
 PARTITION "P5"  VALUES LESS THAN (TO_DATE(' 2015-05-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION IMMEDIATE
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(INITIAL 8388608 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ,
 PARTITION "P6"  VALUES LESS THAN (TO_DATE(' 2015-06-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION DEFERRED
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ,
 PARTITION "P7"  VALUES LESS THAN (TO_DATE(' 2015-07-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION DEFERRED
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ,
 PARTITION "P8"  VALUES LESS THAN (TO_DATE(' 2015-08-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION DEFERRED
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ,
 PARTITION "P9"  VALUES LESS THAN (TO_DATE(' 2015-09-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION DEFERRED
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ,
 PARTITION "P10"  VALUES LESS THAN (TO_DATE(' 2015-10-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION DEFERRED
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ,
 PARTITION "P11"  VALUES LESS THAN (TO_DATE(' 2015-11-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION DEFERRED
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ,
 PARTITION "P12"  VALUES LESS THAN (TO_DATE(' 2015-12-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION DEFERRED
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ,
 PARTITION "P13"  VALUES LESS THAN (TO_DATE(' 2016-01-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION DEFERRED
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ,
 PARTITION "P14"  VALUES LESS THAN (TO_DATE(' 2016-02-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION DEFERRED
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ,
 PARTITION "P15"  VALUES LESS THAN (TO_DATE(' 2016-03-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION DEFERRED
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ,
 PARTITION "P16"  VALUES LESS THAN (TO_DATE(' 2016-04-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION DEFERRED
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ,
 PARTITION "P17"  VALUES LESS THAN (TO_DATE(' 2016-05-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION DEFERRED
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ,
 PARTITION "P18"  VALUES LESS THAN (TO_DATE(' 2016-06-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION DEFERRED
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ,
 PARTITION "P19"  VALUES LESS THAN (TO_DATE(' 2016-07-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION DEFERRED
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ,
 PARTITION "P20"  VALUES LESS THAN (TO_DATE(' 2016-08-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN')) SEGMENT CREATION IMMEDIATE
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 ROW STORE COMPRESS ADVANCED LOGGING
  STORAGE(INITIAL 8388608 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "USERS" ) ;


1 row selected.

SQL> @ind anbob.t_i1
Display indexes where table or index name matches %anbob.t_i1%...

TABLE_OWNER          TABLE_NAME                     INDEX_NAME                     POS# COLUMN_NAME                    DSC
-------------------- ------------------------------ ------------------------------ ---- ------------------------------ ----
ANBOB                T                              T_I1                              1 CTIME
                                                                                      2 NAME


INDEX_OWNER          TABLE_NAME                     INDEX_NAME                     IDXTYPE    UNIQ STATUS   PART TEMP  H     LFBLKS           NDK   NUM_ROWS       CLUF LAST_ANALYZED       DEGREE VISIBILIT
-------------------- ------------------------------ ------------------------------ ---------- ---- -------- ---- ---- -- ---------- ------------- ---------- ---------- ------------------- ------ ---------
ANBOB                T                              T_I1                           NORMAL     NO   N/A      YES  N     3       4314       1000000    1000000     195853 2021-12-18 22:00:50 1      VISIBLE

SQL> @o anbob.t_i1

owner                     object_name                    subname                        object_type          status           OID      D_OID CREATED             LAST_DDL_TIME
------------------------- ------------------------------ ------------------------------ -------------------- --------- ---------- ---------- ------------------- -------------------
ANBOB                     T_I1                                                          INDEX                VALID          78788            2021-11-19 15:14:56 2021-11-19 15:14:56
ANBOB                     T_I1                           P10                            INDEX PARTITION      VALID          78798      78798 2021-11-19 15:14:56 2021-11-19 15:14:56
ANBOB                     T_I1                           P11                            INDEX PARTITION      VALID          78799      78799 2021-11-19 15:14:56 2021-11-19 15:14:56
ANBOB                     T_I1                           P12                            INDEX PARTITION      VALID          78800      78800 2021-11-19 15:14:56 2021-11-19 15:14:56
ANBOB                     T_I1                           P13                            INDEX PARTITION      VALID          78801      78801 2021-11-19 15:14:56 2021-11-19 15:14:56
ANBOB                     T_I1                           P14                            INDEX PARTITION      VALID          78802      78802 2021-11-19 15:14:56 2021-11-19 15:14:56
ANBOB                     T_I1                           P15                            INDEX PARTITION      VALID          78803      78803 2021-11-19 15:14:56 2021-11-19 15:14:56
ANBOB                     T_I1                           P16                            INDEX PARTITION      VALID          78804      78804 2021-11-19 15:14:56 2021-11-19 15:14:56
ANBOB                     T_I1                           P17                            INDEX PARTITION      VALID          78805      78805 2021-11-19 15:14:56 2021-11-19 15:14:56
ANBOB                     T_I1                           P18                            INDEX PARTITION      VALID          78806      78806 2021-11-19 15:14:56 2021-11-19 15:14:56
ANBOB                     T_I1                           P19                            INDEX PARTITION      VALID          78807      78807 2021-11-19 15:14:56 2021-11-19 15:14:56
ANBOB                     T_I1                           P2                             INDEX PARTITION      VALID          78790      78790 2021-11-19 15:14:56 2021-11-19 15:14:56
ANBOB                     T_I1                           P20                            INDEX PARTITION      VALID          78947      78950 2021-11-19 15:14:56 2021-11-19 15:14:56
ANBOB                     T_I1                           P3                             INDEX PARTITION      VALID          78791      78791 2021-11-19 15:14:56 2021-11-19 15:14:56
ANBOB                     T_I1                           P4                             INDEX PARTITION      VALID          78792      78792 2021-11-19 15:14:56 2021-11-19 15:14:56
ANBOB                     T_I1                           P5                             INDEX PARTITION      VALID          78793      78793 2021-11-19 15:14:56 2021-11-19 15:14:56
ANBOB                     T_I1                           P6                             INDEX PARTITION      VALID          78794      78794 2021-11-19 15:14:56 2021-11-19 15:14:56
ANBOB                     T_I1                           P7                             INDEX PARTITION      VALID          78795      78795 2021-11-19 15:14:56 2021-11-19 15:14:56
ANBOB                     T_I1                           P8                             INDEX PARTITION      VALID          78796      78796 2021-11-19 15:14:56 2021-11-19 15:14:56
ANBOB                     T_I1                           P9                             INDEX PARTITION      VALID          78797      78797 2021-11-19 15:14:56 2021-11-19 15:14:56
ANBOB                     T_I1                           P1                             INDEX PARTITION      VALID          78789      78789 2021-11-19 15:14:56 2021-11-19 15:14:56

21 rows selected.


SQL> r
  1*  select obj#,part# ,SUBOBJECT_NAME from indpart$ a,dba_objects b where a.obj#=b.object_id and b.object_name='T_I1'

      OBJ#      PART# subname
---------- ---------- ------------------------------
     78790         20 P2
     78791         30 P3
     78792         40 P4
     78793         50 P5
     78794         60 P6
     78795         70 P7
     78796         80 P8
     78797         90 P9
     78789         10 P1
     78798        100 P10
     78799        110 P11
     78800        120 P12
     78801        130 P13
     78802        140 P14
     78803        150 P15
     78804        160 P16
     78805        170 P17
     78806        180 P18
     78807        190 P19
     78947        200 P20

20 rows selected.

Note:
能看到创建后默认part# 间隔为10.

1, 增加分区

SQL> alter table anbob.t add partition p30 values less than (to_date('20200101','yyyymmdd'));

Table altered.

SQL> select  obj#,part# ,subobject_name from indpart$ a,dba_objects b where a.obj#=b.object_id and b.object_name='T_I1';

      OBJ#      PART# subname
---------- ---------- ------------------------------
...
     78802        140 P14
     78803        150 P15
     78804        160 P16
     78805        170 P17
     78806        180 P18
     78807        190 P19
     78947        200 P20
     79108        210 P30

21 rows selected.

Note：
增加分区还是part#增加10

2， 从最大分区拆

SQL> alter table anbob.t split partition p30 into(partition p21 values less than (to_date('20190101','yyyymmdd')),partition p30);

Table altered.

SQL> select  obj#,part# ,subobject_name from indpart$ a,dba_objects b where a.obj#=b.object_id and b.object_name='T_I1' order by 2;

      OBJ#      PART# subname
---------- ---------- ------------------------------
...
...
     78805        170 P17
     78806        180 P18
     78807        190 P19
     78947        200 P20
     79112        211 P21
     79111        221 P30

22 rows selected.

SQL> alter table anbob.t split partition p30 into(partition p22 values less than (to_date('20190201','yyyymmdd')),partition p30);
Table altered.

SQL> select  obj#,part# ,subobject_name from indpart$ a,dba_objects b where a.obj#=b.object_id and b.object_name='T_I1' order by 2;

      OBJ#      PART# subname
---------- ---------- ------------------------------
...
     78803        150 P15
     78804        160 P16
     78805        170 P17
     78806        180 P18
     78807        190 P19
     78947        200 P20
     79112        211 P21
     79116        222 P22
     79115        232 P30
23 rows selected.

SQL> alter table anbob.t split partition p30 into(partition p23 values less than (to_date('20190301','yyyymmdd')),partition p30);

Table altered.

SQL> select  obj#,part# ,subobject_name from indpart$ a,dba_objects b where a.obj#=b.object_id and b.object_name='T_I1' order by 2;

      OBJ#      PART# subname
---------- ---------- ------------------------------
     78789         10 P1
     78790         20 P2
     78791         30 P3
     78792         40 P4
     78793         50 P5
     78794         60 P6
     78795         70 P7
     78796         80 P8
     78797         90 P9
     78798        100 P10
     78799        110 P11
     78800        120 P12
     78801        130 P13
     78802        140 P14
     78803        150 P15
     78804        160 P16
     78805        170 P17
     78806        180 P18
     78807        190 P19
     78947        200 P20
     79112        211 P21
     79116        222 P22
     79120        233 P23
     79119        243 P30
24 rows selected.

SQL> alter table anbob.t split partition p30 into(partition p24 values less than (to_date('20190401','yyyymmdd')),partition p30);
Table altered.

      OBJ#      PART# subname
---------- ---------- ------------------------------
...
     78804        160 P16
     78805        170 P17
     78806        180 P18
     78807        190 P19
     78947        200 P20
     79112        211 P21
     79116        222 P22
     79120        233 P23
     79124        244 P24
     79123        254 P30
25 rows selected.

... 就以上规律继续拆

SQL> alter table anbob.t split partition p30 into(partition p28 values less than (to_date('20190801','yyyymmdd')),partition p30);
Table altered.

      OBJ#      PART# subname
---------- ---------- ------------------------------
...
     78803        150 P15
     78804        160 P16
     78805        170 P17
     78806        180 P18
     78807        190 P19
     78947        200 P20
     79112        211 P21
     79116        222 P22
     79120        233 P23
     79124        244 P24
     79128        255 P25
     79132        266 P26
     79136        277 P27
     79140        288 P28
     79139        298 P30
29 rows selected.

SQL> alter table anbob.t split partition p30 into(partition p29 values less than (to_date('20190901','yyyymmdd')),partition p30);
Table altered.

SQL> select  obj#,part# ,subobject_name from indpart$ a,dba_objects b where a.obj#=b.object_id and b.object_name='T_I1' order by 1;

      OBJ#      PART# subname
---------- ---------- ------------------------------
...
     78804        160 P16
     78805        170 P17
     78806        180 P18
     78807        190 P19
     78947        200 P20
     79112        211 P21
     79116        222 P22
     79120        233 P23
     79124        244 P24
     79128        255 P25
     79132        266 P26
     79136        277 P27
     79140        288 P28
     79144        299 P29
     79143        309 P30
30 rows selected.

SQL> alter table anbob.t split partition p30 into(partition p29_1 values less than (to_date('20191001','yyyymmdd')),partition p30);
Table altered.

      OBJ#      PART# subname
---------- ---------- ------------------------------
...
     78805        170 P17
     78806        180 P18
     78807        190 P19
     78947        200 P20
     79112        211 P21
     79116        222 P22
     79120        233 P23
     79124        244 P24
     79128        255 P25
     79132        266 P26
     79136        277 P27
     79140        288 P28
     79144        299 P29
     79148        310 P29_1
     79147        320 P30
31 rows selected.

note:
如果10046 跟的话是delete 1条，insert 2条， 看到part# 规律是 split X into (PARTITION Y , PARTITION X), X的part#是原part#+10+1, Y的part#是新x的part# -10.

3, 从中间分区拆

SQL> alter table anbob.t split partition p20 into(partition p19_1 values less than (to_date('20160701 01','yyyymmdd hh24')),partition p20);
Table altered.

SQL> select  obj#,part# ,subobject_name from indpart$ a,dba_objects b where a.obj#=b.object_id and b.object_name='T_I1' order by 2;

      OBJ#      PART# subname
---------- ---------- ------------------------------
...
     78804        160 P16
     78805        170 P17
     78806        180 P18
     78807        190 P19
     79152        199 P19_1
     79151        201 P20
     79112        211 P21
     79116        222 P22
     79120        233 P23
     79124        244 P24
     79128        255 P25
     79132        266 P26
     79136        277 P27
     79140        288 P28
     79144        299 P29
     79147        320 P30
     79148        310 P29_1
32 rows selected.

SQL> alter table anbob.t split partition p20 into(partition p19_2 values less than (to_date('20160701 02','yyyymmdd hh24')),partition p20);
Table altered.

SQL> select  obj#,part# ,subobject_name from indpart$ a,dba_objects b where a.obj#=b.object_id and b.object_name='T_I1' order by 2;

      OBJ#      PART# subname
---------- ---------- ------------------------------
...
     78805        170 P17
     78806        180 P18
     78807        190 P19
     79152        199 P19_1
     79156        200 P19_2
     79155        202 P20
     79112        211 P21
     79116        222 P22
     79120        233 P23
     79124        244 P24
     79128        255 P25
     79132        266 P26
     79136        277 P27
     79140        288 P28
     79144        299 P29
     79148        310 P29_1
     79147        320 P30

33 rows selected.

SQL> alter table anbob.t split partition p20 into(partition p19_3 values less than (to_date('20160701 03','yyyymmdd hh24')),partition p20);

Table altered.

SQL> select  obj#,part# ,subobject_name from indpart$ a,dba_objects b where a.obj#=b.object_id and b.object_name='T_I1' and part#>150 order by 2;

      OBJ#      PART# subname
---------- ---------- ------------------------------
     78804        160 P16
     78805        170 P17
     78806        180 P18
     78807        190 P19
     79152        199 P19_1
     79156        200 P19_2
     79160        201 P19_3
     79159        203 P20
     79112        211 P21
     79116        222 P22
     79120        233 P23
     79124        244 P24
     79128        255 P25
     79132        266 P26
     79136        277 P27
     79140        288 P28
     79144        299 P29
     79148        310 P29_1
     79147        320 P30

19 rows selected.

SQL> alter table anbob.t split partition p20 into(partition p19_4 values less than (to_date('20160701 04','yyyymmdd hh24')),partition p20);
Table altered.

SQL> select  obj#,part# ,subobject_name from indpart$ a,dba_objects b where a.obj#=b.object_id and b.object_name='T_I1' and part#>150 order by 2;

      OBJ#      PART# subname
---------- ---------- ------------------------------
     78804        160 P16
     78805        170 P17
     78806        180 P18
     78807        190 P19
     79152        199 P19_1
     79156        200 P19_2
     79160        201 P19_3
     79164        202 P19_4
     79163        204 P20
     79112        211 P21
     79116        222 P22
     79120        233 P23
     79124        244 P24
     79128        255 P25
     79132        266 P26
     79136        277 P27
     79140        288 P28
     79144        299 P29
     79148        310 P29_1
     79147        320 P30

20 rows selected.

SQL> alter table anbob.t split partition p20 into(partition p19_5 values less than (to_date('20160701 05','yyyymmdd hh24')),partition p20);
Table altered.

SQL> alter table anbob.t split partition p20 into(partition p19_6 values less than (to_date('20160701 06','yyyymmdd hh24')),partition p20);
Table altered.

SQL> alter table anbob.t split partition p20 into(partition p19_7 values less than (to_date('20160701 07','yyyymmdd hh24')),partition p20);
Table altered.

SQL> alter table anbob.t split partition p20 into(partition p19_8 values less than (to_date('20160701 08','yyyymmdd hh24')),partition p20);
Table altered.

SQL> alter table anbob.t split partition p20 into(partition p19_9 values less than (to_date('20160701 09','yyyymmdd hh24')),partition p20);
Table altered.

SQL> select  obj#,part# ,subobject_name from indpart$ a,dba_objects b where a.obj#=b.object_id and b.object_name='T_I1' and part#>150 order by 2;

      OBJ#      PART# subname
---------- ---------- ------------------------------
     78804        160 P16
     78805        170 P17
     78806        180 P18
     78807        190 P19
     79152        199 P19_1
     79156        200 P19_2
     79160        201 P19_3
     79164        202 P19_4
     79168        203 P19_5
     79172        204 P19_6
     79176        205 P19_7
     79180        206 P19_8
     79184        207 P19_9
     79183        209 P20
     79112        211 P21
     79116        222 P22
     79120        233 P23
     79124        244 P24
     79128        255 P25
     79132        266 P26
     79136        277 P27
     79140        288 P28
     79144        299 P29
     79148        310 P29_1
     79147        320 P30

25 rows selected.

SQL> alter table anbob.t split partition p20 into(partition p19_10 values less than (to_date('20160701 10','yyyymmdd hh24')),partition p20);
Table altered.

SQL> select  obj#,part# ,subobject_name from indpart$ a,dba_objects b where a.obj#=b.object_id and b.object_name='T_I1' and part#>150 order by 2;

      OBJ#      PART# subname
---------- ---------- ------------------------------
     78804        160 P16
     78805        170 P17
     78806        180 P18
     78807        190 P19
     79152        199 P19_1
     79156        200 P19_2
     79160        201 P19_3
     79164        202 P19_4
     79168        203 P19_5
     79172        204 P19_6
     79176        205 P19_7
     79180        206 P19_8
     79184        207 P19_9
     79188        208 P19_10
     79187        210 P20
     79112        211 P21
     79116        222 P22
     79120        233 P23
     79124        244 P24
     79128        255 P25
     79132        266 P26
     79136        277 P27
     79140        288 P28
     79144        299 P29
     79148        310 P29_1
     79147        320 P30

26 rows selected.

Note:
注意到拆中间的分区时，看到part# 规律是 split X into (PARTITION Y , PARTITION X),  X的part# 是原X的part#+1, Y的part#是原X的part#-1. 也就是相差2， 但这有一个问题，就是如果X的值，每次增加1，如果把开始预留的10个part#耗尽了呢？

4， 中间拆分区， X的next  part# 已使用时

SQL> alter table anbob.t split partition p20 into(partition p19_11 values less than (to_date('20160701 11','yyyymmdd hh24')),partition p20);
Table altered.

SQL> select  obj#,part# ,subobject_name from indpart$ a,dba_objects b where a.obj#=b.object_id and b.object_name='T_I1' and part#>150 order by 2;

      OBJ#      PART# subname
---------- ---------- ------------------------------
     78804        160 P16
     78805        170 P17
     78806        180 P18
     78807        190 P19
     79152        199 P19_1
     79156        200 P19_2
     79160        201 P19_3
     79164        202 P19_4
     79168        203 P19_5
     79172        204 P19_6
     79176        205 P19_7
     79180        206 P19_8
     79184        207 P19_9
     79188        208 P19_10
     79192        209 P19_11
     79191        210 P20
     79112        211 P21
     79116        222 P22
     79120        233 P23
     79124        244 P24
     79128        255 P25
     79132        266 P26
     79136        277 P27
     79140        288 P28
     79144        299 P29
     79148        310 P29_1
     79147        320 P30

27 rows selected.

Note:
X如果part# next+1的方式在下一个号被使用时， 看到part# 规律是 split X into (PARTITION Y , PARTITION X), X的part#未改变，Y的part#是X的PART#-1， 这样x与y的part#就相差1， 如果再拆就面临X 如果不增加，就把Y next part#占用的情况，oracle会如何做呢？

5, 中间拆分区， Y的next part#补占用时

SQL> alter table anbob.t split partition p20 into(partition p19_12 values less than (to_date('20160701 12','yyyymmdd hh24')),partition p20);
Table altered.

SQL> select  obj#,part# ,subobject_name from indpart$ a,dba_objects b where a.obj#=b.object_id and b.object_name='T_I1' and part#>150 order by 2;

      OBJ#      PART# subname
---------- ---------- ------------------------------
     78804        160 P16
     78805        170 P17
     78806        180 P18
     78807        190 P19
     79152        199 P19_1
     79156        200 P19_2
     79160        201 P19_3
     79164        202 P19_4
     79168        203 P19_5
     79172        204 P19_6
     79176        205 P19_7
     79180        206 P19_8
     79184        207 P19_9
     79188        208 P19_10
     79192        209 P19_11
     79196        219 P19_12
     79195        229 P20
     79112        239 P21
     79116        249 P22
     79120        259 P23
     79124        269 P24
     79128        279 P25
     79132        289 P26
     79136        300 P27
     79140        311 P28
     79144        321 P29
     79148        331 P29_1
     79147        341 P30

28 rows selected.

Note:
Y next+1, X在第10次未+1,y next part#被使用时， 看到part# 规律是 split X into (PARTITION Y , PARTITION X), Y的part# 是原来X的part#+10-1,而后面所有的分区part#基本都增加了10, 但是， 但是p27这个300 打破了规律，有点蒙。

10046中显示也是直接 update 成300,无征兆。

就到这里， 2021年最后一篇。

— enjoy —

OSW工具不用多说oracle数据库环境建议采集OS系统数据的脚本集， 采集的数据可以拿到其它机器，如WINDOWS上分析输出图形， 在ORACLE 12c后 AHF自动健康框架中已自带， 同时还有oracheck等，当troubleshooting时使用tfactl 可以一并收集相对全面的日志数据，目录比较深,如收集node: anbob2节点:

…\collection_Tue_Jan_11_17_34_19_CST_2022_node_anbob2\anbob2.tfa_Tue_Jan_11_17_34_19_CST_2022.zip\anbob2\oracle\app\grid\oracle.ahf\data\repository\suptools\anbob2\oswbb\grid\archive

几年前记录过2篇OSW，不做过多介绍，这里简单记录一下在Windows上使用oswbba.jar分析时的一些小问题。

OS Watcher (OSW)系统性能监控软件

Fixed OSWatcher v7.3.3 some problems on HPUX ia 11-31 platform

1, archive 路径不能含空格

C:\Users\zhang>java -jar “E:\google drive\Oracle\Oracle tool\oswbb840\oswbb\oswbba.jar” -i “C:\Users\zhang\Desktop\0111 orcl crash\oswbb\grid\archive”

ERROR. oswbba can not find any vmstat files in you archive location.
Either you are pointing to an invalid directory location or your archive does not contain vmstat files.
Check that C:\Users\zhang\Desktop\0111 orcl crash\oswbb\grid\archive is a valid archive directory and there are vmstat files under oswvmstat.
This tool requires vmstat data in order to run.
Program aborting…

2， 环境变量日期格式不对

C:\Users\zhang>java -jar “E:\google drive\Oracle\Oracle tool\oswbb840\oswbb\oswbba.jar” -i D:\archive

Starting OSW Analyzer V8.4.0
OSWatcher Analyzer Written by Oracle Center of Expertise
Copyright (c) 2019 by Oracle Corporation

Parsing Data. Please Wait…

Scanning file headers for version and platform info…

Parsing file ldzwd2_prvtnet_22.01.11.0200.dat …
Parsing file ldzwd2_prvtnet_22.01.11.0200.dat …
An exception occured in insertVmstatLists on line 0
An exception occured in insertVmstatLists on line 5
An exception occured in insertVmstatLists on line 10
An exception occured in insertVmstatLists on line 15
An exception occured in insertVmstatLists on line 20

3， 内存溢出

C:\Users\zhang>java -Duser.language=en -Duser.country=us -jar “E:\google drive\Oracle\Oracle tool\oswbb840\oswbb\oswbba.jar” -i D:\archive

Starting OSW Analyzer V8.4.0
OSWatcher Analyzer Written by Oracle Center of Expertise
Copyright (c) 2019 by Oracle Corporation

Parsing Data. Please Wait…

Scanning file headers for version and platform info…

Parsing file ldzwd2_prvtnet_22.01.11.0200.dat …
Parsing file ldzwd2_prvtnet_22.01.11.0200.dat …
Parsing file ldzwd2_prvtnet_22.01.11.0300.dat …
Parsing file ldzwd2_prvtnet_22.01.11.0400.dat …
Parsing file ldzwd2_prvtnet_22.01.11.0500.dat …
Parsing file ldzwd2_prvtnet_22.01.11.0600.dat …

Exception in thread “main” java.lang.OutOfMemoryError: GC overhead limit exceeded
at u.c(Unknown Source)
at u.a(Unknown Source)
at OSWGraph.OSWGraph.main(Unknown Source)

— 有时可以清理掉一些不需要的文件 如ps
C:\Users\zhang>java -Duser.language=en -Duser.country=us -Xmx1024M -jar “E:\google drive\Oracle\Oracle tool\oswbb840\oswbb\oswbba.jar” -i D:\archive

Parsing Completed.

Enter 1 to Display CPU Process Queue Graphs
Enter 2 to Display CPU Utilization Graphs
Enter 3 to Display CPU Other Graphs
Enter 4 to Display Memory Graphs
Enter 5 to Display Disk IO Graphs
Enter 61 to Display Individual OS Process I/O RPS Graphs
Enter 62 to Display Individual OS Process I/O WPS Graphs
Enter 63 to Display Individual OS Process Percent User CPU Graphs
Enter 64 to Display Individual OS Process Percent System CPU Graphs
Enter 65 to Display Individual OS Process Percent Total CPU (User + System) Graphs
Enter 66 to Display Individual OS Process Percent Memory Graphs

Enter GP to Generate Individual Process Profile
Enter GC to Generate All CPU Gif Files
Enter GM to Generate All Memory Gif Files
Enter GD to Generate All Disk Gif Files
Enter GN to Generate All Network Gif Files

Enter L to Specify Alternate Location of Gif Directory
Enter Z to Zoom Graph Time Scale (Does not change analysis dataset)
Enter B to Returns to Baseline Graph Time Scale (Does not change analysis dataset)
Enter R to Remove Currently Displayed Graphs

Enter X to Export Parsed Data to Flat File
Enter S to Analyze Subset of Data(Changes analysis dataset including graph time scale)
Enter A to Analyze Data
Enter D to Generate DashBoard

Enter Q to Quit Program

某年某月某一天，有个客户的oracle归档日志空间突然满了，实例挂起，客户的告警系统一如既往的保持沉默， 我们自己部署的“保镖”脚本被友军打上了#号，友军的“打手”脚本说sys用户密码已过期，  wait!wait! are you kidding me?  SYS user 密码过期？  逻辑是这样的这套脚本是在DATAGUARD的standby 使用sys@tns方式远程查询已applyed 日志删除，在调用日志里提示“ORA-28002: the password will expire within 7 days“。

ORA-28002&ORA-28001

这种错误并不陌生， 就是db user因为配置了user profile中的PASSWORD_LIFE_TIME和PASSWORD_GRACE_TIME，以用户登录时对比上次更改密码时间与当前时间判断，是否需要修改密码的策略。 应用程序用户通常创建新user profile，对于懒的定期改的就把Password life time搞成unlimted 永不过期， 数据库安装部署里需要考虑这问题，免的半年后应用用户突然因为密码过期原因影响了全业务。 那sys对于PASSWORD_LIFE_TIME是否生效呢？以前不会但现在会。

User SYS Does Not Get ORA-28002 Nor ORA-28001 Even When PASSWORD_LIFE_TIME or PASSWORD_GRACE_TIME are Set (Doc ID 289898.1)
中也有指出”The PASSWORD_LIFE_TIME and PASSWORD_GRACE_TIME parameters of a resource profile seem to be working for all the users but not for SYS“
也就是说user profile中配置的这两个参数并不对sys启作用， 这是oracle的预期行为，这也是问了几个做了十几、二十年的ORACLE DBA从来没听说过sys还密码过期的原因，但是，从Oracle 12c开始安全增强，改变了这个行为， SYS也开始受profile 中PASSWORD_LIFE_TIME and PASSWORD_GRACE_TIME影响。如果没有使用SYS远程登录的习惯可能也不太容易发现，在12c 安全增强中还有另一个关于用户状态的profile limit INACTIVE_ACCOUNT_TIME顾名思义，就是用户多长时间不用可以自动lock， 但该参数默认没有配置天数限制。

v$pwfile_users

我们知道sys（sysdba）和其它用户不一样，属于内部的管理用户，可以启动或关闭数据库实例，如果密码和用户状态只存在数据库内的数据字典，那在数据库open前就无法校验，所以它的密码是存在数据库外的密码文件，实例启动时同步到数据库字典， 查看密码文件中用户有个视图v$pwfile_users， 那这个视图是否可以看到用户状态呢？

11g

12C（12.2）

Note:
其实从12c开始该view增加了很多字段，其中就有用户状态和过期时间，所以理论上就可以在数据库打开前从密码文件就可以验证用户是否过期。

实践验证理论

Note: test in Oracle 19.3 CDB on linux

[oracle@oel7db1 ~]$ cd $ORACLE_HOME
[oracle@oel7db1 db_1]$ cd dbs
l[oracle@oel7db1 dbs]$ ls
arch  c-3414393273-20201126-00  hc_anbob19c.dat  initanbob19c.ora  init.ora  lkANBOB19C  orapwanbob19c  snapcf_anbob19c.f  spfileanbob19c.ora
[oracle@oel7db1 dbs]$ ll orapwanbob19c
-rw-r----- 1 oracle oinstall 6144 Dec 28 12:49 orapwanbob19c
[oracle@oel7db1 dbs]$ md5sum orapwanbob19c
0b3ec322d07035edbabc8afe8d062594  orapwanbob19c

SQL> @cc pdb1
ALTER SESSION SET container = pdb1;

Session altered.


USERNAME             INST_NAME            HOST_NAME                  I# SID   SERIAL#  VERSION    STARTED  SPID       OPID  CPID            SADDR            PADDR
-------------------- -------------------- ------------------------- --- ----- -------- ---------- -------- ---------- ----- --------------- ---------------- ----------------
SYS                  PDB1-anbob19c        oel7db1                     1 1     14       19.0.0.0.0 20220107 2186       33    2093            0000000078481028 00000000790F7F48


SQL> @us sys
Show database usernames from dba_users matching %sys%

USERNAME                  DEFAULT_TABLESPACE        TEMPORARY_TABLESPACE              USER_ID CREATED   ACCOUNT_STATUS                   PROFILE
------------------------- ------------------------- ------------------------------ ---------- --------- -------------------------------- --------------------------------------------------------------------------------------------------------------------------------
SYS                       SYSTEM                    TEMP                                    0 17-APR-19 OPEN                             DEFAULT
SYSTEM                    SYSTEM                    TEMP                                    9 17-APR-19 OPEN                             DEFAULT
LBACSYS                   SYSTEM                    TEMP                                  100 17-APR-19 LOCKED                           DEFAULT
APPQOSSYS                 SYSAUX                    TEMP                                   58 17-APR-19 LOCKED                           DEFAULT


SQL> alter profile default limit PASSWORD_LIFE_TIME 5/24/60;
Profile altered.

SQL> select * from dba_profiles where profile='DEFAULT' and RESOURCE_NAME like '%PASSWORD%';

PROFILE         RESOURCE_NAME                    RESOURCE LIMIT                                                        COM INH IMP
--------------- -------------------------------- -------- ------------------------------------------------------------ --- --- ---
DEFAULT         PASSWORD_LIFE_TIME               PASSWORD .0034                                                        NO  NO  NO
DEFAULT         PASSWORD_REUSE_TIME              PASSWORD UNLIMITED                                                    NO  NO  NO
DEFAULT         PASSWORD_REUSE_MAX               PASSWORD UNLIMITED                                                    NO  NO  NO
DEFAULT         PASSWORD_VERIFY_FUNCTION         PASSWORD NULL                                                         NO  NO  NO
DEFAULT         PASSWORD_LOCK_TIME               PASSWORD 1                                                            NO  NO  NO
DEFAULT         PASSWORD_GRACE_TIME              PASSWORD .0001                                                        NO  NO  NO

select 'DBA_USERS', USERNAME, PASSWORD_VERSIONS,PROFILE, ACCOUNT_STATUS, LOCK_DATE,EXPIRY_DATE, PASSWORD_CHANGE_DATE PTIME  from dba_users where username='SYS'
union
select 'PWFILE', USERNAME, NULL, PASSWORD_PROFILE, ACCOUNT_STATUS,LOCK_DATE,EXPIRY_DATE, null PTIME   from v$pwfile_users where username='SYS'
union
  select 'USER$', NAME, NULL, NULL, to_char(ASTATUS), LTIME, EXPTIME, PTIME from user$ where name='SYS'
 
select 'DBA_USERS', USERNAME, PASSWORD_VERSIONS,PROFILE, ACCOUNT_STATUS, LOCK_DATE,EXPIRY_DATE, PASSWORD_CHANGE_DATE PTIME from dba_users where username='SYS'
union
select 'PWFILE', USERNAME, NULL, PASSWORD_PROFILE, ACCOUNT_STATUS,LOCK_DATE,EXPIRY_DATE, null PTIME from v$pwfile_users where username='SYS'
union
  5    select 'USER$', NAME, NULL, NULL, to_char(ASTATUS), LTIME, EXPTIME, PTIME from user$ where name='SYS';

'DBA_USER USERNAME                  PASSWORD_VERSIONS PROFILE         ACCOUNT_ST LOCK_DATE         EXPIRY_DATE       PTIME
--------- ------------------------- ----------------- --------------- ---------- ----------------- ----------------- -----------------
DBA_USERS SYS                       11G 12C           DEFAULT         OPEN                         20210624 17:12:42 20210624 17:07:42
PWFILE    SYS                                         DEFAULT         OPEN                         20220626 12:49:10
USER$     SYS                                                         0                                              20210624 17:07:42


SQL> @printtab "select * from v$pwfile_users";
USERNAME                      : SYS
SYSDBA                        : TRUE
SYSOPER                       : TRUE
SYSASM                        : FALSE
SYSBACKUP                     : FALSE
SYSDG                         : FALSE
SYSKM                         : FALSE
ACCOUNT_STATUS                : OPEN
PASSWORD_PROFILE              : DEFAULT
LAST_LOGIN                    : 07-JAN-22 04.45.16.000000000 PM +08:00
LOCK_DATE                     :
EXPIRY_DATE                   : 26-jun-2022 12:49:10
EXTERNAL_NAME                 :
AUTHENTICATION_TYPE           : PASSWORD
COMMON                        : YES
CON_ID                        : 0

SQL> select to_date('20220626','yyyymmdd')-180 from dual;

TO_DATE('20220626
-----------------
20211228 00:00:00

Note:
这里是切到了PDB中，修改了PDB的user profile. 可以v$pwfile的时间是密码文件的最后一个Modify time+180天，而dba_users(user$)计算的才是我们刚修改的5分钟,但是一个过去的时间，也未提示用户过期

通过alter user改密码

SQL> alter user sys identified by oracle container=all;
alter user sys identified by oracle container=all
*
ERROR at line 1:
ORA-65040: operation not allowed from within a pluggable database

SQL> conn / as sysdba
Connected.

USERNAME             INST_NAME            HOST_NAME                  I# SID   SERIAL#  VERSION    STARTED  SPID       OPID  CPID            SADDR            PADDR
-------------------- -------------------- ------------------------- --- ----- -------- ---------- -------- ---------- ----- --------------- ---------------- ----------------
SYS                  CDB$ROOT-anbob19c    oel7db1                     1 1     18919    19.0.0.0.0 20220107 4372       33    3461            0000000078481028 00000000790F7F48


SQL> alter user sys identified by oracle container=all;
User altered.

shudown immediate
startup

SQL>
SQL> @cc pdb1
ALTER SESSION SET container = pdb1;

Session altered.

USERNAME             INST_NAME            HOST_NAME                  I# SID   SERIAL#  VERSION    STARTED  SPID       OPID  CPID            SADDR            PADDR
-------------------- -------------------- ------------------------- --- ----- -------- ---------- -------- ---------- ----- --------------- ---------------- ----------------
SYS                  PDB1-anbob19c        oel7db1                     1 1     18919    19.0.0.0.0 20220107 4372       33    3461            0000000078481028 00000000790F7F48


select 'DBA_USERS', USERNAME, PASSWORD_VERSIONS,PROFILE, ACCOUNT_STATUS, LOCK_DATE,EXPIRY_DATE, PASSWORD_CHANGE_DATE PTIME,LAST_LOGIN LLST from dba_users where username='SYS'
union
select 'PWFILE', USERNAME, NULL, PASSWORD_PROFILE, ACCOUNT_STATUS,LOCK_DATE,EXPIRY_DATE, null PTIME ,NULL from v$pwfile_users where username='SYS'
union
  5    select 'USER$', NAME, NULL, NULL, to_char(ASTATUS), LTIME, EXPTIME, PTIME,NULL from user$ where name='SYS';

'DBA_USER USERNAME                  PASSWORD_VERSIONS PROFILE              ACCOUNT_STATUS                 LOCK_DATE         EXPIRY_DATE       PTIME             LLST
--------- ------------------------- ----------------- -------------------- ------------------------------ ----------------- ----------------- ----------------- --------------------
DBA_USERS SYS                       11G 12C           DEFAULT              OPEN                                             20210624 17:12:42 20210624 17:07:42
PWFILE    SYS                                         DEFAULT              OPEN                                             20220706 17:02:17
USER$     SYS                                                              0                                                                  20210624 17:07:42

SQL> host
[oracle@oel7db1 dbs]$ ls -l orapwanbob19c
-rw-r----- 1 oracle oinstall 6144 Jan  7 17:04 orapwanbob19c

Note:
使用alter user会同步修改密码文件，PDB中的 user$的过期时间并不会更新， v$pwfile_users的过期时间继续推后180天。可能会想是不是oracle bug, PDB都改成5分钟了，这里还是按180天算，不是的，下面再说原因。 那我们先改操作系统时间到未来那个时间，看是不是真的会过期。

# 改OS 时间

[root@oel7db1 ~]# date -s '20220807'
Sun Aug  7 00:00:00 CST 2022
[root@oel7db1 ~]# date
Sun Aug  7 00:00:07 CST 2022
[root@oel7db1 ~]# su - oracle
Last login: Fri Jan  7 16:24:22 CST 2022 on pts/0
[oracle@oel7db1 ~]$ ora

SQL*Plus: Release 19.0.0.0.0 - Production on Sun Aug 7 00:00:13 2022
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area  700445040 bytes
Fixed Size                  9138544 bytes
Variable Size             637534208 bytes
Database Buffers           50331648 bytes
Redo Buffers                3440640 bytes
Database mounted.
Database opened.
SQL> select sysdate from dual;

SYSDATE
---------
07-AUG-22

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
[oracle@oel7db1 ~]$ sqlplus sys/oracle@cdb1pdb1 as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Sun Aug 7 00:00:45 2022
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

ERROR:
ORA-28002: the password will expire within 7 days


SQL> r
  1  select 'DBA_USERS', USERNAME, PASSWORD_VERSIONS,PROFILE, ACCOUNT_STATUS, LOCK_DATE,EXPIRY_DATE, PASSWORD_CHANGE_DATE PTIME  from dba_users where username='SYS'
  2  union
  3  select 'PWFILE', USERNAME, NULL, PASSWORD_PROFILE, ACCOUNT_STATUS,LOCK_DATE,EXPIRY_DATE, null PTIME   from v$pwfile_users where username='SYS'
  4  union
  5*   select 'USER$', NAME, NULL, NULL, to_char(ASTATUS), LTIME, EXPTIME, PTIME from user$ where name='SYS'

'DBA_USER USERNAME                  PASSWORD_VERSIONS PROFILE    ACCOUNT_STATUS                           LOCK_DATE           EXPIRY_DATE         PTIME
--------- ------------------------- ----------------- ---------- ---------------------------------------- ------------------- ------------------- -------------------
DBA_USERS SYS                       11G 12C           DEFAULT    OPEN                                                         2021-06-24 17:12:42 2021-06-24 17:07:42
PWFILE    SYS                                         DEFAULT    EXPIRED(GRACE)                                               2022-08-14 00:00:45
USER$     SYS                                                    0                                                                                2021-06-24 17:07:42

# 日期继续改

SQL> shu immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
[oracle@oel7db1 ~]$ exit
logout
[root@oel7db1 ~]# date -s '20220815'
Mon Aug 15 00:00:00 CST 2022
[root@oel7db1 ~]# su - oracle
Last login: Sun Aug  7 00:00:09 CST 2022 on pts/0


[oracle@oel7db1 ~]$ sqlplus sys/oracle@cdb1pdb1 as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Aug 15 00:00:54 2022
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

ERROR:
ORA-28001: the password has expired


Changing password for sys
New password:


SQL> r
  1  select 'DBA_USERS', USERNAME, PASSWORD_VERSIONS,PROFILE, ACCOUNT_STATUS, LOCK_DATE,EXPIRY_DATE, PASSWORD_CHANGE_DATE PTIME  from dba_users where username='SYS'
  2  union
  3  select 'PWFILE', USERNAME, NULL, PASSWORD_PROFILE, ACCOUNT_STATUS,LOCK_DATE,EXPIRY_DATE, null PTIME   from v$pwfile_users where username='SYS'
  4  union
  5*   select 'USER$', NAME, NULL, NULL, to_char(ASTATUS), LTIME, EXPTIME, PTIME from user$ where name='SYS'

'DBA_USER USERNAME                  PASSWORD_VERSIONS PROFILE              ACCOUNT_STATUS                           LOCK_DATE           EXPIRY_DATE         PTIME
--------- ------------------------- ----------------- -------------------- ---------------------------------------- ------------------- ------------------- -------------------
DBA_USERS SYS                       11G 12C           DEFAULT              OPEN                                                         2021-06-24 17:12:42 2021-06-24 17:07:42
PWFILE    SYS                                         DEFAULT              EXPIRED                                                      2022-08-14 00:00:45
USER$     SYS                                                              0                                                                                2021-06-24 17:07:42

NOTE:
可以看到这里仅向后修改操作系统的时间，模拟出了SYS过期，但是并不是日期大于开始的20220706就直接过期，而是在大于这个日期首次登录时会提示EXPIRED(GRACE)，同时把v$pwfile_user.expire_time过期时间有当前时间再加上grace 7天时间。如果这7点后还没改，状态变为EXPIRED已过期， 但是PDB中的dba_users还是OPEN 没有参考意义了。可以看出可以以v$pwfile_user的过期时间巡检，那再看180天根据哪来的。

刚才一直是在PDB查询，下面切到CDB.
SQL> conn / as sysdba
Connected.

USERNAME             INST_NAME            HOST_NAME                  I# SID   SERIAL#  VERSION    STARTED  SPID       OPID  CPID            SADDR            PADDR
-------------------- -------------------- ------------------------- --- ----- -------- ---------- -------- ---------- ----- --------------- ---------------- ----------------
SYS                  CDB$ROOT-anbob19c    oel7db1                     1 68    28670    19.0.0.0.0 20220815 7187       33    7059            00000000785281E0 00000000790F7F48


SQL> select * from dba_profiles where profile='DEFAULT' and RESOURCE_NAME like '%PASSWORD%';

PROFILE              RESOURCE_NAME                    RESOURCE LIMIT                          COM INH IMP
-------------------- -------------------------------- -------- ------------------------------ --- --- ---
DEFAULT              PASSWORD_LIFE_TIME               PASSWORD 180                            NO  NO  NO
DEFAULT              PASSWORD_REUSE_TIME              PASSWORD UNLIMITED                      NO  NO  NO
DEFAULT              PASSWORD_REUSE_MAX               PASSWORD UNLIMITED                      NO  NO  NO
DEFAULT              PASSWORD_VERIFY_FUNCTION         PASSWORD NULL                           NO  NO  NO
DEFAULT              PASSWORD_LOCK_TIME               PASSWORD 1                              NO  NO  NO
DEFAULT              PASSWORD_GRACE_TIME              PASSWORD 7                              NO  NO  NO

6 rows selected.


select 'DBA_USERS', USERNAME, PASSWORD_VERSIONS,PROFILE, ACCOUNT_STATUS, LOCK_DATE,EXPIRY_DATE, null PTIME from dba_users where username='SYS'
union
select 'PWFILE', USERNAME, NULL, PASSWORD_PROFILE, ACCOUNT_STATUS,LOCK_DATE,EXPIRY_DATE, null PTIME from v$pwfile_users where username='SYS'
union
 select 'USER$', NAME, NULL, NULL, to_char(ASTATUS), LTIME, EXPTIME, PTIME from user$ where name='SYS';

'DBA_USER USERNAME                  PASSWORD_VERSIONS PROFILE              ACCOUNT_STATUS                           LOCK_DATE           EXPIRY_DATE         PTIME
--------- ------------------------- ----------------- -------------------- ---------------------------------------- ------------------- ------------------- -------------------
DBA_USERS SYS                       11G 12C           DEFAULT              EXPIRED                                                      2022-08-14 00:00:45
PWFILE    SYS                                         DEFAULT              EXPIRED                                                      2022-08-14 00:00:45
USER$     SYS                                                              1                                                            2022-08-14 00:00:45 2019-04-17 00:56:32

Note:
注意 原来sys的计划user profile是根据CDB而来，修改PDB 并不会改变， 而且在CDB中dba_users已显示为expired.  另外注意user$.ptime就是意思是上次修改密码时间是2019-4-17这可能是ORACLE安装介质中镜像生成的，好多数据库都是这个日期。 接下来再改一下密码。

SQL> alter user sys identified by oracle1234 container=all;

User altered.

select 'DBA_USERS', USERNAME, PASSWORD_VERSIONS,PROFILE, ACCOUNT_STATUS, LOCK_DATE,EXPIRY_DATE, null PTIME from dba_users where username='SYS'
union
select 'PWFILE', USERNAME, NULL, PASSWORD_PROFILE, ACCOUNT_STATUS,LOCK_DATE,EXPIRY_DATE, null PTIME from v$pwfile_users where username='SYS'
union
select 'USER$', NAME, NULL, NULL, to_char(ASTATUS), LTIME, EXPTIME, PTIME from user$ where name='SYS';

'DBA_USER USERNAME                  PASSWORD_VERSIONS PROFILE              ACCOUNT_STATUS                           LOCK_DATE           EXPIRY_DATE         PTIME
--------- ------------------------- ----------------- -------------------- ---------------------------------------- ------------------- ------------------- -------------------
DBA_USERS SYS                       11G 12C           DEFAULT              OPEN                                                         2019-10-14 00:56:32
PWFILE    SYS                                         DEFAULT              OPEN                                                         2023-02-11 00:11:24
USER$     SYS                                                              0                                                            2023-02-11 00:11:24 2019-04-17 00:56:32

SQL> select to_date('20230211','yyyymmdd')-180 from dual;

TO_DATE('20230211',
-------------------
2022-08-15 00:00:00

SQL> host
[oracle@oel7db1 ~]$ cd $ORACLE_HOME
[oracle@oel7db1 db_1]$ cd dbs

-rw-r----- 1 oracle oinstall 6144 Aug 15 00:14 orapwanbob19c

Note:
再次修改密码后，SYS用户显示OPEN, 并且v$pwfile_users显示的过期时间已自动延后，而dba_users的过期时间是根据user$.ptime计算而来，没有什么参考意义了。 接下来我们再扩大user profile的密码过期天数

select 'DBA_USERS', USERNAME, PASSWORD_VERSIONS,PROFILE, ACCOUNT_STATUS, LOCK_DATE,EXPIRY_DATE, null PTIME from dba_users where username='SYS'
union
select 'PWFILE', USERNAME, NULL, PASSWORD_PROFILE, ACCOUNT_STATUS,LOCK_DATE,EXPIRY_DATE, null PTIME from v$pwfile_users where username='SYS'
union
  5  select 'USER$', NAME, NULL, NULL, to_char(ASTATUS), LTIME, EXPTIME, PTIME from user$ where name='SYS';

'DBA_USER USERNAME                  PASSWORD_VERSIONS PROFILE              ACCOUNT_STATUS                           LOCK_DATE           EXPIRY_DATE         PTIME
--------- ------------------------- ----------------- -------------------- ---------------------------------------- ------------------- ------------------- -------------------
DBA_USERS SYS                       11G 12C           DEFAULT              OPEN                                                         2019-10-14 00:56:32
PWFILE    SYS                                         DEFAULT              OPEN                                                         2023-02-11 00:11:24
USER$     SYS                                                              0                                                            2023-02-11 00:11:24 2019-04-17 00:56:32


SQL>
SQL> alter profile default limit PASSWORD_LIFE_TIME 200;

Profile altered.

select 'DBA_USERS', USERNAME, PASSWORD_VERSIONS,PROFILE, ACCOUNT_STATUS, LOCK_DATE,EXPIRY_DATE, null PTIME from dba_users where username='SYS'
union
select 'PWFILE', USERNAME, NULL, PASSWORD_PROFILE, ACCOUNT_STATUS,LOCK_DATE,EXPIRY_DATE, null PTIME from v$pwfile_users where username='SYS'
union
select 'USER$', NAME, NULL, NULL, to_char(ASTATUS), LTIME, EXPTIME, PTIME from user$ where name='SYS';

'DBA_USER USERNAME                  PASSWORD_VERSIONS PROFILE              ACCOUNT_STATUS                           LOCK_DATE           EXPIRY_DATE         PTIME
--------- ------------------------- ----------------- -------------------- ---------------------------------------- ------------------- ------------------- -------------------
DBA_USERS SYS                       11G 12C           DEFAULT              OPEN                                                         2019-11-03 00:56:32
PWFILE    SYS                                         DEFAULT              OPEN                                                         2023-02-11 00:11:24
USER$     SYS                                                              0                                                            2023-02-11 00:11:24 2019-04-17 00:56:32

NOTE:
注意，增加了profile的密码过期时间，但v$pwfile.expiry_date并没有自动延长。即使使用sys再次登录也不会更新。

SQL> alter user sys identified by oracle;

User altered.

select 'DBA_USERS', USERNAME, PASSWORD_VERSIONS,PROFILE, ACCOUNT_STATUS, LOCK_DATE,EXPIRY_DATE, null PTIME from dba_users where username='SYS'
union
select 'PWFILE', USERNAME, NULL, PASSWORD_PROFILE, ACCOUNT_STATUS,LOCK_DATE,EXPIRY_DATE, null PTIME from v$pwfile_users where username='SYS'
union
select 'USER$', NAME, NULL, NULL, to_char(ASTATUS), LTIME, EXPTIME, PTIME from user$ where name='SYS';

'DBA_USER USERNAME                  PASSWORD_VERSIONS PROFILE    ACCOUNT_STATUS                           LOCK_DATE           EXPIRY_DATE         PTIME
--------- ------------------------- ----------------- ---------- ---------------------------------------- ------------------- ------------------- -------------------
DBA_USERS SYS                       11G 12C           DEFAULT    OPEN                                                         2019-11-03 00:56:32
PWFILE    SYS                                         DEFAULT    OPEN                                                         2023-03-03 00:21:16
USER$     SYS                                                    0                                                            2023-03-03 00:21:16 2019-04-17 00:56:32

NOTE:
修改profile password life time天数后，直到再次变更密码，expiry_date才发生改变。那这里就要注意了，很多时候我们的理解如果密码周期就只增加profile的密码周期，又没有再改密码，到期还是会过期，如果我们把password_life_time改成unlimited呢？

SQL> alter profile default limit PASSWORD_LIFE_TIME unlimited;

Profile altered.

select 'DBA_USERS', USERNAME, PASSWORD_VERSIONS,PROFILE, ACCOUNT_STATUS, LOCK_DATE,EXPIRY_DATE, null PTIME from dba_users where username='SYS'
union
select 'PWFILE', USERNAME, NULL, PASSWORD_PROFILE, ACCOUNT_STATUS,LOCK_DATE,EXPIRY_DATE, null PTIME from v$pwfile_users where username='SYS'
union
  5  select 'USER$', NAME, NULL, NULL, to_char(ASTATUS), LTIME, EXPTIME, PTIME from user$ where name='SYS';

'DBA_USER USERNAME                  PASSWORD_VERSIONS PROFILE    ACCOUNT_STATUS                           LOCK_DATE           EXPIRY_DATE         PTIME
--------- ------------------------- ----------------- ---------- ---------------------------------------- ------------------- ------------------- -------------------
DBA_USERS SYS                       11G 12C           DEFAULT    OPEN
PWFILE    SYS                                         DEFAULT    OPEN                                                         2023-03-03 00:21:16
USER$     SYS                                                    0                                                            2023-03-03 00:21:16 2019-04-17 00:56:32

SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area  700445040 bytes
Fixed Size                  9138544 bytes
Variable Size             637534208 bytes
Database Buffers           50331648 bytes
Redo Buffers                3440640 bytes
Database mounted.
Database opened.
select 'DBA_USERS', USERNAME, PASSWORD_VERSIONS,PROFILE, ACCOUNT_STATUS, LOCK_DATE,EXPIRY_DATE, null PTIME from dba_users where username='SYS'
union
select 'PWFILE', USERNAME, NULL, PASSWORD_PROFILE, ACCOUNT_STATUS,LOCK_DATE,EXPIRY_DATE, null PTIME from v$pwfile_users where username='SYS'
union
 select 'USER$', NAME, NULL, NULL, to_char(ASTATUS), LTIME, EXPTIME, PTIME from user$ where name='SYS';

'DBA_USER USERNAME                  PASSWORD_VERSIONS PROFILE    ACCOUNT_STATUS                           LOCK_DATE EXPIRY_DA PTIME
--------- ------------------------- ----------------- ---------- ---------------------------------------- --------- --------- ---------
DBA_USERS SYS                       11G 12C           DEFAULT    OPEN
PWFILE    SYS                                         DEFAULT    OPEN                                               03-MAR-23
USER$     SYS                                                    0                                                  03-MAR-23 17-APR-19

NOTE:
注意改完PASSWORD_LIFE_TIME为 unlimited后，dba_user.expire_date显示为空，但是v$pwfile.expiry_date还是之前的过期日期。重启实例也并未改变。那可能就又埋了个雷，即使sys的profile当前PASSWORD_LIFE_TIME已经为无限，是不是到时还会过期呢？

再改日期
SQL> shut immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
[oracle@oel7db1 ~]$ exit
logout
[root@oel7db1 ~]# date -s '20230815'
Tue Aug 15 00:00:00 CST 2023
[root@oel7db1 ~]# su - oracle
Last login: Mon Aug 15 00:00:03 CST 2022 on pts/0
[oracle@oel7db1 ~]$ ora

SQL> startup
ORACLE instance started.

Total System Global Area  700445040 bytes
Fixed Size                  9138544 bytes
Variable Size             637534208 bytes
Database Buffers           50331648 bytes
Redo Buffers                3440640 bytes
Database mounted.
Database opened.

SQL> r
  1  select 'DBA_USERS', USERNAME, PASSWORD_VERSIONS,PROFILE, ACCOUNT_STATUS, LOCK_DATE,EXPIRY_DATE, null PTIME from dba_users where username='SYS'
  2  union
  3  select 'PWFILE', USERNAME, NULL, PASSWORD_PROFILE, ACCOUNT_STATUS,LOCK_DATE,EXPIRY_DATE, null PTIME from v$pwfile_users where username='SYS'
  4  union
  5*  select 'USER$', NAME, NULL, NULL, to_char(ASTATUS), LTIME, EXPTIME, PTIME from user$ where name='SYS'

'DBA_USER USERNAME                  PASSWORD_VERSIONS PROFILE                        ACCOUNT_STATUS                           LOCK_DATE EXPIRY_DA PTIME
--------- ------------------------- ----------------- ------------------------------ ---------------------------------------- --------- --------- ---------
DBA_USERS SYS                       11G 12C           DEFAULT                        OPEN
PWFILE    SYS                                         DEFAULT                        OPEN                                               03-MAR-23
USER$     SYS                                                                        0                                                  03-MAR-23 17-APR-19


用sys登录一下

[oracle@oel7db1 ~]$ sqlplus sys/oracle@cdb1pdb1 as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Aug 15 00:02:18 2023
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

ERROR:
ORA-28002: the password will expire within 7 days


SQL> r
  1  select 'DBA_USERS', USERNAME, PASSWORD_VERSIONS,PROFILE, ACCOUNT_STATUS, LOCK_DATE,EXPIRY_DATE, null PTIME from dba_users where username='SYS'
  2  union
  3  select 'PWFILE', USERNAME, NULL, PASSWORD_PROFILE, ACCOUNT_STATUS,LOCK_DATE,EXPIRY_DATE, null PTIME from v$pwfile_users where username='SYS'
  4  union
  5*  select 'USER$', NAME, NULL, NULL, to_char(ASTATUS), LTIME, EXPTIME, PTIME from user$ where name='SYS'

'DBA_USER USERNAME                  PASSWORD_VERSIONS PROFILE                        ACCOUNT_STATUS                           LOCK_DATE           EXPIRY_DATE         PTIME
--------- ------------------------- ----------------- ------------------------------ ---------------------------------------- ------------------- ------------------- -------------------
DBA_USERS SYS                       11G 12C           DEFAULT                        EXPIRED(GRACE)                                               2023-08-22 00:02:18
PWFILE    SYS                                         DEFAULT                        EXPIRED(GRACE)                                               2023-08-22 00:02:18
USER$     SYS                                                                        2                                                            2023-08-22 00:02:18 2019-04-17 00:56:32

Note:
SYS还是过期了。 如果再改一下密码会是什么样子?

SQL> alter user sys identified by oracle;
User altered.


select 'DBA_USERS', USERNAME, PASSWORD_VERSIONS,PROFILE, ACCOUNT_STATUS, LOCK_DATE,EXPIRY_DATE, null PTIME from dba_users where username='SYS'
union
select 'PWFILE', USERNAME, NULL, PASSWORD_PROFILE, ACCOUNT_STATUS,LOCK_DATE,EXPIRY_DATE, null PTIME from v$pwfile_users where username='SYS'
union
  5   select 'USER$', NAME, NULL, NULL, to_char(ASTATUS), LTIME, EXPTIME, PTIME from user$ where name='SYS';

'DBA_USER USERNAME                  PASSWORD_VERSIONS PROFILE                        ACCOUNT_STATUS                           LOCK_DATE           EXPIRY_DATE         PTIME
--------- ------------------------- ----------------- ------------------------------ ---------------------------------------- ------------------- ------------------- -------------------
DBA_USERS SYS                       11G 12C           DEFAULT                        OPEN
PWFILE    SYS                                         DEFAULT                        OPEN
USER$     SYS                                                                        0           

Note:
注意在当前PASSWORD_LIFE_TIME为 unlimited后,现次变更密码这里dba_users还是v$pwfile_users的EXPIRY_DATE都显示为空，才是真正的永不过期，看这里是不是需要赶紧去查一下生产呢？

SUMMARY:
1, oracle 12c的安全增强, sys用户也受到了user profile中的PASSWORD_LIFE_TIME的参数影响。
2， 12c后的密码文件中增加了用户状态和过期时间
3， 变更profile的 password life time后，没有再次修改密码，过期时间不会生效
4， 确保sys不过期是v$pwfile_users.EXPIRY_DATE 显示为null.

What is a 'core dump' ?
~~~~~~~~~~~~~~~~~~~~~~~        
        A core dump is an image copy of a processes state at the instant
        it 'aborted'. It is produced in the form of a file called 'core' 
        usually located in the current directory. 


What causes a core dump ?
~~~~~~~~~~~~~~~~~~~~~~~~~
        There are many situations which can cause a core dump to be produced, 
        but it is usually because the process has attempted to do something
        which the operating system does not like. The most common causes
        of this are:

                The program tried to access memory outside its allowed range.

                The program tried to obtain a resource which was either
                exhausted or unavailable.

                An attempt was made to execute illegal instructions.

		An attempt was made to read unaligned data

        In Unix systems the offending process is sent one of a number of
        signals which force a core dump to be produced. It is also possible
        for a user to produce a core dump by sending one of these signals
        to a process manually. 

What should I do if I get a core dump ?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        As with any problem you should first note down the FULL version
        numbers of the product, the RDBMS, PL/SQL (if used) and any 
        related products.

        You should also note the EXACT command you were running when 
        this occurred. Eg: If it was a SQL*Forms problem and you were
        using 'mrunform30', write this down.  This command will be referred
        to as 'program' below. 

        Now follow the instructions below in order:

        1) Check if you have a 'core' file, it should be in the directory where 
           the command was issued, or in CORE_DUMP_DEST/core_NNNNNN
           if it is the 'oracle' executable. "oracle" can also produce
           core files in $ORACLE_HOME/dbs/core_NNNNN or $ORACLE_HOME/dbs .


        2) Log in as ORACLE and cd to the directory containing the core file.  
           Then issue:

                         file core

           This should identify the "program" name to use in the next step, 
           e.g.: oracle


        3) Log in as ORACLE and change in to the $ORACLE_HOME/bin
           directory. Enter the command:

                         file program

           and write the result down letter for letter. If the word 'dynamic' 
           or 'dynamically linked' appears in the output of this command
           then please make a note of this as there are a few platforms on
           which Oracle does NOT support dynamic linking and this may be
           the cause of your problem.

 4) Now enter:

                        chmod +r program

           to add read permission to the program. 


        5) Log out , then log in as the user who encountered the error.
           The next step will vary slightly depending on which version of
           Unix you are using. One of the following commands should exist
           on your machine - try each in order until you find one that exists.
           In some cases you may be asked for stacks from all threads so 
           use the "thread" version of the command if it exists. 

	   An alternative to the commands below is to use the stackx.sh 
	   script from Note:362791.1 . That script will try to find a 
	   suitable debugger and extract the stack tracee for you .
           

	     Common Debuggers and commands to show a symbolic stack trace:

              Command   NB           Exit command     Stack Trace command
              -------   --           ------------     -------------------
                dbx                  quit             where
                xdb     (HPUX 10)    quit             t
                gdb     (HPUX 11)    q                bt
                dde     (HPUX 11)    q                bt
                sdb                  q                t 
                adb                  $q (or Ctrl-D)   $c
		debug	(PTX only)   quit	      stack
                gdb     (Linux)      quit             bt
                                                   or thread apply all where
                pstack  (HPUX, Linux, Solaris)

           Change to the directory where the core dump is located and enter
           the commands as in the relevant example below. If you are not
           sure which program produced the 'core' file then on some Unix 
           platforms the command 'file core' will tell you the executable
           name that the core file is from (this does not work on ALL 
           Unix platforms, see note below.)
. 
           Example commands:

                DBX:        $ script /tmp/mystack
                            $ dbx $ORACLE_HOME/bin/<program> core 
                            (dbx) where
                            ...                 << Stack should appear here
                            (dbx) quit
                            $ exit

                XDB:        $ script /tmp/mystack
                            $ xdb $ORACLE_HOME/bin/<program> core 
                            (xdb) t     
                            ...                 << Stack should appear here
                            (xdb) quit
                            $ exit

                SDB:        $ script /tmp/mystack
                            $ sdb $ORACLE_HOME/bin/<program> core 
                            (sdb) t      
                            ...                 << Stack should appear here
                            (sdb) q
                            $ exit

		(NOTE: In the 'adb' commands below literally type the $c & $q)
                ADB:        $ script /tmp/mystack
                            $ adb $ORACLE_HOME/bin/<program> core 
                             $c      << NB: adb has no prompt so just enter $c
                            ...                 
                             $q
                            $ exit

                DEBUG:      $ script /tmp/mystack
                            $ debug -c core $ORACLE_HOME/bin/<program> 
                            debug> stack
                            ...                 << Stack should appear here
                            debug> quit
                            $ exit

                GDB:        $ script /tmp/mystack
                            $ gdb $ORACLE_HOME/bin/<program> core
                            (gdb) bt
                            ...                 << Stack should appear here
                            (gdb) thread apply all where
                            ...                 << Stacks for all threads here
                            (gdb) quit
                            $ exit

                DDE:        $ script /tmp/mystack
                            $ dde -ui line core $ORACLE_HOME/bin/<program>
                            dde> bt
                            ...
                            dde> q
                            $exit 

                PSTACK:     $ script /tmp/mystack
                            $ pstack core
                            $ exit

[root@oel7db1 ~]# ./stackx gdb core.59248
######################################################################
stackx core / stack trace extraction utility
version 1.3 (c) 2005, 2006, 2007 Oracle Corp. all rights reserved.
Reporting: Mon Jan 17 16:03:37 CST 2022
Machine : oel7db1
OS Name : Linux
OS Ver : 4.14.35-1902.3.2.el7uek.x86_64
CPU Type : 64
Core file: core.59248
Core date: Jan 17 15:11
./stackx[503]: whence: /u01/app/11.2.0/grid/bin/ocssd.bin: not found
Tool(s) : /usr/bin/gdb
Program :
Core extraction section follows
######################################################################
No executables found for /u01/app/11.2.0/grid/bin/ocssd.bin in corefile core.59248

Read more Getting a Stack Trace from a CORE file on Unix (Doc ID 1812.1)