Quantcast
Channel: ANBOB
Viewing all 704 articles
Browse latest View live

Oracle 哪些进程可以KILL不会导致实例重启?

$
0
0

oracle后台进程当出现问题时,有些进程kill会导致实例立即重启,像smon, pmon,ckpt.. , 而有些进程kill并不会影响实例可用性, 甚至会立即做进程级重启从db alert log可以观测到,如mmon,rec,jnnn, pnnn等, 前两天看到Poder在其BLOG分享v$process的基表X$KSUPR中中有记录哪些是oracle的致命进程,在X$KSUPR.KSUPRFLG第3位, 下面我做个测试, kill 点X$KSUPR.KSUPRFLG第3位都不为1的进程。

— 环境 oracle 19.3 on-primise

致命进程

SELECT indx,ksuprpnm,TO_CHAR(ksuprflg,'XXXXXXXXXXXXXXXX'),KSUPROSID
FROM x$ksupr
WHERE BITAND(ksuprflg,4) = 4 ORDER BY indx
  4  /

      INDX KSUPRPNM                                         TO_CHAR(KSUPRFLG, KSUPROSID
---------- ------------------------------------------------ ----------------- ------------------------
         2 oracle@oel7db1 (PMON)                                            E 2245
         3 oracle@oel7db1 (CLMN)                                            E 2247
         4 oracle@oel7db1 (PSP0)                                            6 2249
         5 oracle@oel7db1 (VKTM)                                            6 2251
         6 oracle@oel7db1 (GEN0)                                            6 2255
         8 oracle@oel7db1 (MMAN)                                            6 2259
        13 oracle@oel7db1 (DBRM)                                            6 2270
        16 oracle@oel7db1 (PMAN)                                            6 2276
        18 oracle@oel7db1 (DBW0)                                            6 2280
        19 oracle@oel7db1 (LGWR)                                            6 2282
        20 oracle@oel7db1 (CKPT)                                            6 2284
        21 oracle@oel7db1 (SMON)                                           16 2286
        25 oracle@oel7db1 (LREG)                                            6 2294

非致命进程

SELECT indx,ksuprpnm,TO_CHAR(ksuprflg,'XXXXXXXXXXXXXXXX'),KSUPROSID
FROM x$ksupr
WHERE BITAND(ksuprflg,4) != 4 and KSUPROSID is not null ORDER BY indx
/

      INDX KSUPRPNM                                         TO_CHAR(KSUPRFLG, KSUPROSID
---------- ------------------------------------------------ ----------------- ------------------------
         0 oracle@oel7db1 (MZ00)                                            0 3949
         7 oracle@oel7db1 (DIAG)                                            2 2265
         9 oracle@oel7db1 (GEN1)                                            2 2261_2263
        10 oracle@oel7db1 (SCMN)                                            2 2261_2261
        11 oracle@oel7db1 (OFSD)                                            2 2267_2268
        12 oracle@oel7db1 (SCMN)                                            2 2267_2267
        14 oracle@oel7db1 (VKRM)                                            2 2272
        15 oracle@oel7db1 (SVCB)                                            2 2274
        17 oracle@oel7db1 (DIA0)                                            2 2278
        22 oracle@oel7db1 (SMCO)                                            2 2288
        23 oracle@oel7db1 (RECO)                                            2 2290
        24 oracle@oel7db1 (W000)                                            2 2292
        26 oracle@oel7db1 (W001)                                            2 2296
        27 oracle@oel7db1 (PXMN)                                            2 2298
        28 oracle@oel7db1 (S000)                                           40 2308
        29 oracle@oel7db1 (MMON)                                            2 2302
        30 oracle@oel7db1 (MMNL)                                            2 2304
        31 oracle@oel7db1 (D000)                                           80 2306
        32 oracle@oel7db1 (TMON)                                            2 2310
        33 oracle@oel7db1 (TNS V1-V3)                                       0 2722
        34 oracle@oel7db1 (M000)                                            2 2316
        35 oracle@oel7db1 (TT00)                                            2 2324
        36 oracle@oel7db1 (TT01)                                            2 2326
        37 oracle@oel7db1 (TT02)                                            2 2328
        38 oracle@oel7db1 (W002)                                            2 2332
        39 oracle@oel7db1 (AQPC)                                            2 2336
        40 oracle@oel7db1 (W003)                                            2 2339
        41 oracle@oel7db1 (W004)                                            2 2349
        42 oracle@oel7db1 (P000)                                            0 2344
        43 oracle@oel7db1 (P001)                                            0 2346
        44 oracle@oel7db1 (M004)                                            0 3498
        45 oracle@oel7db1 (CJQ0)                                            2 2357
        46 oracle@oel7db1 (QM02)                                            2 2435
        47 oracle@oel7db1 (Q001)                                            2 2437
        48 oracle@oel7db1 (W005)                                            2 3179
        49 oracle@oel7db1 (Q003)                                            2 2442
        50 oracle@oel7db1 (W006)                                            2 3183
        51 oracle@oel7db1 (W007)                                            2 3188
        52 oracle@oel7db1 (MZ00)                                            0 3949
        53 oracle@oel7db1 (J001)                                            0 3802
        54 oracle@oel7db1 (MZ00)                                            0 3793
        55 oracle@oel7db1 (J004)                                            0 2523
        56 oracle@oel7db1 (J005)                                            0 2525
        57 oracle@oel7db1 (J006)                                            0 2527
        58 oracle@oel7db1 (J007)                                            0 2530
        59 oracle@oel7db1 (J008)                                            0 2532
        60 oracle@oel7db1 (J009)                                            0 2534
        61 oracle@oel7db1 (J00A)                                            0 2536
        62 oracle@oel7db1 (J00B)                                            0 2538
        63 oracle@oel7db1 (J00C)                                            0 2540
        64 oracle@oel7db1 (J00D)                                            0 2543
        65 oracle@oel7db1 (J00E)                                            0 2545
        66 oracle@oel7db1 (M001)                                            2 2569
        67 oracle@oel7db1 (M002)                                            2 2571
        68 oracle@oel7db1 (M003)                                            2 2573
        69 oracle@oel7db1 (Q005)                                            0 2610
        70 oracle@oel7db1 (Q006)                                            0 2612
        71 oracle@oel7db1 (Q007)                                            0 2614
        72 oracle@oel7db1 (Q008)                                            0 2616
        73 oracle@oel7db1 (Q009)                                            0 2618
        74 oracle@oel7db1 (Q00A)                                            0 2620
        75 oracle@oel7db1 (Q00B)                                            0 2622
        76 oracle@oel7db1 (Q00C)                                            0 2624
        77 oracle@oel7db1 (Q00D)                                            0 2626
        78 oracle@oel7db1 (Q00E)                                            0 2628
        79 oracle@oel7db1 (Q00F)                                            0 2630
        80 oracle@oel7db1 (Q00G)                                            0 2633
        81 oracle@oel7db1 (Q00H)                                            0 2635
        82 oracle@oel7db1 (Q00I)                                            0 2637
        83 oracle@oel7db1 (Q00J)                                            0 2639
        84 oracle@oel7db1 (Q00K)                                            0 2641
        85 oracle@oel7db1 (Q00L)                                            0 2643
        86 oracle@oel7db1 (Q00M)                                            0 2645
       299 oracle@oel7db1 (TNS V1-V3)                                       0 2233

KILL ALL 非致命进程

SELECT 'host kill -9 '||KSUPROSID
FROM x$ksupr
WHERE BITAND(ksuprflg,4) != 4 and KSUPROSID is not null ORDER BY indx
/

— 果然实例没有重启

DB ALERT LOG

Restarting dead background process DIAG
Starting background process DIAG
DIAG started with pid=17, OS id=4520
Restarting dead background process OFSD
Starting background process OFSD
OFSD started with pid=22, OS id=4522_4524
Restarting dead background process VKRM
Starting background process VKRM
Oracle running with ofslib:'Oracle File Server Library' version=2
VKRM started with pid=24, OS id=4526
Restarting dead background process SVCB
Starting background process SVCB
SVCB started with pid=26, OS id=4528
Restarting dead background process DIA0
Starting background process DIA0
DIA0 started with pid=27, OS id=4530
Restarting dead background process TMON
Starting background process TMON
TMON started with pid=28, OS id=4532
Restarting dead background process SMCO
Starting background process SMCO
SMCO started with pid=29, OS id=4534
Restarting dead background process RECO
Starting background process RECO
RECO started with pid=30, OS id=4536
Restarting dead background process CJQ0
Starting background process CJQ0
CJQ0 started with pid=32, OS id=4540
Restarting dead background process PXMN
Starting background process PXMN
PXMN started with pid=35, OS id=4544
Restarting dead background process AQPC
Starting background process AQPC
AQPC started with pid=36, OS id=4546
Restarting dead background process MMON
Starting background process MMON
MMON started with pid=37, OS id=4548
Restarting dead background process MMNL
Starting background process MMNL
MMNL started with pid=38, OS id=4550
2020-05-12 11:25:58.025000 -04:00
TT00 (PID:4560): Gap Manager starting

进程自动重启, cool!


Troubleshooting ORA-01578 ORA-01110 ORA-26040 NOLOGGING corrupted block

$
0
0

The following errors have recently occurred in our database( Oracle 19c RAC):

ORA-01578: ORACLE data block corrupted (file # xxx, block # yyyyy)
ORA-01110: data file xxx: 'xxxxxxxxxxxxxxxxx.dbf'
ORA-26040: Data block was loaded using the NOLOGGING option

There are many possible causes of a block corruption including:
1. Bad IO hardware / firmware
2. OS problems
3. Oracle Bugs
4. Recovering through “UNRECOVERABLE” or “NOLOGGING” database actions

When a segment is defined with the NOLOGGING attribute and if a NOLOGGING/UNRECOVERABLE operation updates the segment or if datapump import parameter disable_archive_logging:y is used, the online redo log file is updated with minimal information to invalidate the affected blocks when a RECOVERY is later performed.

If a NOLOGGING (or UNRECOVERABLE) operation is performed on an object and the datafile containing that object is subsequently recovered then the data blocks affected by the NOLOGGING operation are marked as corrupt and will signal an ORA-1578 error when accessed.
In Oracle8i and greater an ORA-26040 is also signalled (“ORA-26040: Data block was loaded using the NOLOGGING option” ) which makes the cause fairly obvious, but earlier releases have no additional error message.

“NOLOGGING ” It almost equals to “No Recover” in terms of data protection, even though your database is in ARCHIVELOG mode. So don’t expect the corrupted data is salvageable.Since there’s no way to recover the corrupted data file, so we need to treat the object differently. First, we have to know what type and object was affected.

SQL> select * from dba_extents where file_id = xxx and yyyyy  between block_id and block_id + blocks - 1;

if RMAN validate was run:

alter session set nls_date_format = 'DD-MON-YY HH24:MI:SS';

select file#, block#, first_time, next_time
from   v$archived_log, v$database_block_corruption
where  CORRUPTION_CHANGE# between first_change# and next_change#
  and CORRUPTION_TYPE='NOLOGGING';

#In 12c:
select file#, block#, first_time, next_time
from v$nonlogged_block, v$archived_log
where NONLOGGED_START_CHANGE# between first_change# and next_change#;

The “VALIDATE” RMAN command is used to identify NOLOGGING blocks and populates the view v$database_block_corruption (versions lower than 12c) and v$nonlogged_block (12c and greater).

In versions lower than 10.2.0.5 and 11.1.0.7, RMAN validate reports it with a generic message like:
10.2.0.4 and lower, 11.1.0.6, 11.1.0.7:

The VALIDATE RMAN command reports the NOLOGGING blocks v$database_block_corruption with CORRUPTION_TYPE=LOGICAL

In version 10.2.0.5 or in 11.2.0.1 and forward, RMAN has been enhanced to report it with CORRUPTION_TYPE=NOLOGGING. Reference Doc ID 7396077.8 :
10.2.0.5 and 11.2.0.1+:

The VALIDATE RMAN command reports the NOLOGGING blocks in v$database_block_corruption with CORRUPTION_TYPE=NOLOGGING

In version 12c and forward RMAN validate no longer populates view v$database_block_corruption; instead the new view v$nonlogged_block is updated:

RMAN backups do not fail due to NOLOGGING corrupt blocks. In general RMAN does not fails with soft corrupt blocks; the MAXCORRUPT clause is not necessary in such cases. The backup will contain the soft corrupt block and a restore will leave the corruption as when the backup was made.

How to fixed?

FREE Block
until the block is reused which will automatically re-format the block or force re-formatting the block using Doc ID 336133.1

INDEX Block
drop and create the index

TABLE Block
DBMS_REPAIR.SKIP_CORRUPT_BLOCKS can be used to skip, move it to other tablespace. then to modify the table back to NOSKIP_FLAG , insert data manual if possible.

At that time if the corruption is still reported in v$database_block_corruption or v$nonlogged_block (12c+), run a rman validate to clear that view. RMAN commands are introduced:

RMAN> validate [database / datafile] nonlogged block;
RMAN> recover [database / datafile] nonlogged block;  -> for Standby Databases

in my case ,the Nologging Corrupted Blocks are AWR objects stored in the SYSAUX tablespace. it’s WRH$_ACTIVE_SESSION_HISTORY, the v$active_session_history in ASH buffer (Circular Buffer )- 1M to 128M (~2% of SGA) Flushed every snapshot interval to disk or when buffer 2/3 full (it protects itself so you can relax) Avg row around 150bytes 3600 secs in an hour ~ ½ Meg per Active Session per hour That’s generally over an hour of ASH.

We can manually clean up this object as written in my previous article ORA-1688: unable to extend table SYS.WRH$_ACTIVE_SESSION_HISTOR.

 

How to check the time it takes AWR to refresh each table?

SQL> select table_name_kewrtb name, end_time-begin_time time 
2 from wrm$_snapshot_details, x$kewrtb 
3 where snap_id = :snap_id 
4 and dbid = :dbid 
5 and table_id = table_id_kewrtb 
6 order by table_id; 

References Mos Doc ID 794505.1

Oracle 12c Alert log show ” ADVISORY: Please collect redo for investigation of ORA-8103″ frequently

$
0
0

叕一个oracle 12c频繁生成日志文件的问题,最近一套12.2的RAC突然又文件系统告警,生成了大量的trace文件,db alert log也在不停的显示如下内容。该表分区较多,前一晚有做该表move操作。

# alert log

ADVISORY: Please collect redo for investigation of ORA-8103. Use command:
   ALTER SYSTEM DUMP REDO scn min 1 scn max 16716635042430 dba min 32 2094058 dba max 32 2094058;
2020-05-20T13:36:19.202533+08:00
ADVISORY: Please collect redo for investigation of ORA-8103. Use command:
   ALTER SYSTEM DUMP REDO scn min 1 scn max 16716635042540 dba min 32 2093731 dba max 32 2093731;
2020-05-20T13:36:19.254255+08:00
ADVISORY: Please collect redo for investigation of ORA-8103. Use command:
   ALTER SYSTEM DUMP REDO scn min 1 scn max 16716635042621 dba min 32 2093862 dba max 32 2093862;
2020-05-20T13:36:19.368628+08:00
ADVISORY: Please collect redo for investigation of ORA-8103. Use command:
   ALTER SYSTEM DUMP REDO scn min 1 scn max 16716635042798 dba min 32 2093822 dba max 32 2093822;
2020-05-20T13:36:19.478326+08:00
ADVISORY: Please collect redo for investigation of ORA-8103. Use command:
   ALTER SYSTEM DUMP REDO scn min 1 scn max 16716635042961 dba min 32 2093976 dba max 32 2093976;
2020-05-20T13:36:19.534631+08:00
ADVISORY: Please collect redo for investigation of ORA-8103. Use command:
   ALTER SYSTEM DUMP REDO scn min 1 scn max 16716635043070 dba min 32 2093944 dba max 32 2093944;
2020-05-20T13:36:19.584696+08:00

# trace file contents

*** SESSION ID:(1994.64263) 2020-05-20T11:47:30.893213+08:00
OBJD MISMATCH typ=6, seg.obj=286445, diskobj=286437, dsflg=100000, dsobj=286445, tid=286445, cls=1
kcbz_exec_ckf: check function 0x10b053f0 returned error=8103:1
Suspect buffer header:
BH (0xb9e386738) file#: 29 rdba: 0x07600b64 (29/2100068) class: 1 ba: 0xb93f1a000
set: 161 pool: 3 bsz: 8192 bsi: 0 sflg: 0 pwc: 0,25
dbwrid: 0 obj: 286445 objn: 286445 tsn: [0/18] afn: 29 hint: f
hash: [0x1ede6dc30,0x1ede6dc30] lru: [0xbfe2bf630,0x69e3717d0]
ckptq: [NULL] fileq: [NULL]
objq: [0xafe4d5b38,0xc24f76b0] objaq: [0xafe4d5b48,0xc24f76a0]
use: [0x20cb44d20,0x20cb44d20] wait: [NULL]
st: READING md: EXCL tch: 0 le: 0x5bef2df68
flags: only_sequential_access
Printing buffer operation history (latest change first):
cnt: 4
01. sid:10 L122:zgb:set:st 02. sid:10 L830:olq1:clr:WRT+CKT
03. sid:10 L951:zgb:lnk:objq 04. sid:10 L372:zgb:set:MEXCL
05. sid:10 L123:zgb:no:FEN 06. sid:10 L896:z_mkfr:ulnk:objq
07. sid:10 L083:zgb:ent:fn 08. sid:10 L836:z_age_hot:mv:cld
09. sid:01 L192:kcbbic2:bic:FBD 10. sid:01 L191:kcbbic2:bic:FBW
11. sid:01 L822:bic1_int:ent:rtn 12. sid:01 L832:oswmqbg1:clr:WRT
13. sid:01 L930:kubc:sw:mq 14. sid:01 L913:bxsv:sw:objq
15. sid:01 L608:bxsv:bis:FBW 16. sid:01 L607:bxsv:bis:FFW
buffer tsn: 18 rdba: 0x07600b64 (29/2100068)

Block header dump: 0x07600b64
Object id on Block? Y
seg/obj: 0x45ee5 csc: 0x00000ef425897404 itc: 40 flg: E typ: 1 - DATA
brn: 0 bdba: 0x7600a83 ver: 0x01 opc: 0
inc: 0 exflg: 0

Itl Xid Uba Flag Lck Scn/Fsc
0x01 0x0230.006.00002453 0x00c34b45.0c15.20 C--- 0 scn 0x00000ef425895b78
0x02 0x01fd.002.00003d1e 0x2301879f.1322.16 C--- 0 scn 0x00000ef425896acf
0x03 0x00a2.006.000055fe 0x2301855a.30b6.37 --U- 5 fsc 0x0000.2589744a
0x04 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x05 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x06 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x07 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x08 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x09 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x0a 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x0b 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x0c 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x0d 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x0e 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x0f 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x10 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x11 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x12 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x13 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x14 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x15 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x16 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x17 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x18 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x19 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x1a 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x1b 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x1c 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x1d 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x1e 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x1f 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x20 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x21 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x22 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x23 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x24 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x25 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x26 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x27 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
0x28 0x0000.000.00000000 0x00000000.0000.00 ---- 0 fsc 0x0000.00000000
bdba: 0x07600b64
data_block_dump,data header at 0xb93f1a3f4
===============
tsiz: 0x1c08
hsiz: 0x26
pbl: 0xb93f1a3f4
76543210
flag=--------
ntab=1
nrow=10
frre=-1
fsbo=0x26
fseo=0x71e
avsp=0x6f8
tosp=0x6f8
0xe:pti[0] nrow=10 offs=0
0x12:pri[0] offs=0x14a8
0x14:pri[1] offs=0x19e0
0x16:pri[2] offs=0x1a32
0x18:pri[3] offs=0x1b17
0x1a:pri[4] offs=0x10a1
0x1c:pri[5] offs=0x71e
0x1e:pri[6] offs=0xbb9
0x20:pri[7] offs=0xe1a
0x22:pri[8] offs=0xe80
0x24:pri[9] offs=0xf8d
block_row_dump:

Dumping Short Stack
ksedsts()+346<-kcbz_exec_ckf_debug()+1422<-kcbz_check_objd_typ()+1756<-kcbzib()+3884<-kcbgtcr()+11492<-ktrget2()+1056<-kdsgrp()+527<-qetlbr()+835<-qertbFetchByRowID()+1216<-qergiFetch()+567<-qersoProcessULS()+300<-qersoFetchSimple()+1433<-qersoFetch()+210<-opifch2()+3267
<-kpoal8()+3490<-opiodr()+1229<-ttcpip()+1257<-opitsk()+1940<-opiino()+941<-opiodr()+1229<-opidrv()+1021<-sou2o()+145<-opimai_real()+455<-ssthrdmain()+417<-main()+262<-__libc_start_main()+245Cursor valid: 1
Dumping kcb descriptor and buffer header(s):
kcbds 0x7fc8aa04cec0: pdb 0, tsn 18, rdba 0x081fe6ac, afn 32, objd 286445, cls 1, tidflg 0xc8 0x80 0x0
dsflg 0x100000, dsflg2 0x4000, lobid 0x0:0, cnt 0, addr 0x682a68014, exf 0x10b02510, dx 0x0, ctx 0
whr: 'kdswh05: kdsgrp'
env [0x7fc8aa04e95c]: (scn: 0x00000f3421175bae xid: 0x0000.000.00000000 uba: 0x00000000.0000.00 statement num=0 parent xid: 0x0000.000.00000000 st-scn: 0x0000000000000000 hi-scn: 0x0000000000000000 ma-scn: 0x00000f342117338c flg: 0x00000660)
Dump of buffer cache at level 8 for pdb=0 tsn=18 rdba=136308396

查看对象

SQL> @oid 286445,286437
owner                     object_name                    object_type        SUBOBJECT_NAME       CREATED           LAST_DDL_TIME     status    DATA_OBJECT_ID
------------------------- ------------------------------ ------------------ -------------------- ----------------- ----------------- --------- --------------
ANBOB                      CS_REC_TABXXX                 TABLE PARTITION    PART_336_20190602    20190523 23:42:03 20190523 23:42:03 VALID             286445

SQL> @doid 286445,286437
owner                     object_name                    O_PARTITION                                                                                                                      object_type        CREATED           LAST_DDL_TIME     status    DATA_OBJECT_ID
------------------------- ------------------------------ -------------------------------------------------------------------------------------------------------------------------------- ------------------ ----------------- ----------------- --------- --------------
ANBOB                      CS_REC_TABXXX                 PART_336_20190602                                                                                                                TABLE PARTITION    20190523 23:42:03 20190523 23:42:03 VALID             286445

SQL> select /*+full(t)*/count(*) from ANBOB.CS_REC_TABXXX PARTITION();
--无报错

SQL> @hex 45ee5
                               DEC                  HEX
----------------------------------- --------------------
                      286437.000000                45EE5

刷新内存,无法解决

ALTER SYSTEM FLUSH BUFFER_CACHE;
ALTER SYSTEM FLUSH Shared_pool;

验证数据文件

SQL> select block_size,name,ts# from v$datafile where file#=29

BLOCK_SIZE NAME                                                                                    TS#
---------- -------------------------------------------------------------------------------- ----------
      8192 +DATADG/ORDER/DATAFILE/datacommon.304.992108133                                          18

oracle@ANBOB.COM:/home/oracle> dbv userid=system/xxxxxxxx blocksize=8192 file='+DATADG/ORDER/DATAFILE/datacommon.304.992108133'
DBVERIFY: Release 12.2.0.1.0 - Production on Wed May 20 14:01:17 2020
Copyright (c) 1982, 2017, Oracle and/or its affiliates.  All rights reserved.
DBVERIFY - Verification starting : FILE = +DATADG/ORDER/DATAFILE/datacommon.304.992108133
DBVERIFY - Verification complete
Total Pages Examined         : 3932032
Total Pages Processed (Data) : 3516720
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 193799
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 28892
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 192621
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 622502163 (0.622502163)

重启实例应该也可以解决,但是不允许。MOS 中与Bug 28844866 较匹配,该BUG在12c,18c, 19c中都存在。

尝试使用12c online move table partition解决。

SQL> alter table ANBOB.CS_REC_TABXXX move partition PART_336_20190602 online  tablespace xxx;
Table altered.

注意如果做以上操作在高并发业务时还是会出现短时的library cache lock.

Enable DDL logging in Oracle databaase (安全审计)

$
0
0

Oracle database use DDL statements to define structures such as tables to store data and functions to store code. Monitoring, auditing and logging changes to DDL are key components of a database security program. By default Oracle database does not log any DDL operations performed by any user.When someone do some destructive DDL in DB, we often need the DDL log.

The options for DDL auditing include:
• Standard auditing
• System Triggers
• Oracle Database Lifecycle Management Pack
• Third party tools
• logminer

In this article I will  only record the method of using Oracle Database Lifecycle Management Pack( Enable_ddl_logging),The init.ora parameter ENABLE_DDL_LOGGING is licensed as part of the Database Lifecycle Management Pack when set to TRUE.

ENABLE_DDL_LOGGING

In Oracle 11G oracle has introduced new parameter ENABLE_DDL_LOGGING. This parameter enables or disables the writing of a subset of data definition language (DDL) statements into a log.

Oracle 11g
DDL statements are written to the alert log in: $ADR_BASE/diag/rdbms/${DBNAME}/${ORACLE_SID}/trace/alert_${ORACLE_SID}.log

Oracle 12c R1
12.1 DDLs were written into alter.log  and  written to dedicated log stored in new directory directory.There are two DDL logs that contain the same information. One is an XML file, and the other is a text file. The DDL log is stored in the log/ddl subdirectory of the ADR home.

Oracle 12c R2 and newer
12.2 DDL statement not written alert log, written to dedicated log only. Now if you look in the following text file:
$ADR_BASE/diag/rdbms/${DBNAME}/${ORACLE_SID}/log/ddl_${ORACLE_SID}.log

There is also a XML version:
$ADR_BASE/diag/rdbms/${DBNAME}/${ORACLE_SID}/log/ddl/log.xml

Enable DDL logging

sing the enabling a DDL logging feature built into the database. By default it is turned off and you can turn it on by setting the value of ENABLE_DDL_LOGGING initialization parameter to true.

SQL> show parameter DDL_logging;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
ENABLE_DDL_LOGGING boolean FALSE

We can turn it on using the following command. The parameter is dynamic and you can turn it on/off on the go.

SQL> alter system set ENABLE_DDL_LOGGING=true;
System altered.

Once it is turned on, every DDL command will be logged in the alert log file and also the log.xml file. The information in the alert log will be very concise.

Demo

version oracle 12.2 on OEL6
JAVA CODE

[oracle@anbob ~]$ more DDL.java
import java.sql.*;
public class DDL{

public static Connection getConnection() throws Exception {
    String driver = "oracle.jdbc.driver.OracleDriver";
    String url = "jdbc:oracle:thin:@anbob.com:1521/pdbanbob.com";
    String username = "anbob";
    String password = "anbob";

    Class.forName(driver);
    Connection conn = DriverManager.getConnection(url, username, password);
    return conn;
  }

  public static void  droptable(Connection conn, String tableName) throws SQLException {
    // select the number of rows in the table
    Statement stmt = null;
    ResultSet rs = null;
    try {
      stmt = conn.createStatement();
      String sql="drop table "+ tableName;
      System.out.println("SQL Text: "+sql);
      stmt.executeUpdate(sql);
    } finally {
      stmt.close();
    }
  }

 public static void main(String[] args) {
    Connection conn = null;
    try {
      conn = getConnection();
      String tableName = "test100";
      System.out.println("tableName=" + tableName);
      System.out.println("conn=" + conn);
          // to do
      droptable(conn, tableName);

    } catch (Exception e) {
      e.printStackTrace();
      System.exit(1);
    } finally {
      // release database resources
      try {
        conn.close();
      } catch (SQLException e) {
        e.printStackTrace();
      }
    }
  }
}

create table with sqlplus and drop table with java jdbc.

DDL LOG

[oracle@anbob log]$ pwd
/u02/app/oracle/diag/rdbms/anbob/anbob/log

[oracle@anbob log]$ ls
ddl ddl_anbob.log debug debug.log hcs imdb test
[oracle@anbob log]$ more ddl_anbob.log
2020-05-22T21:51:55.794795+08:00
diag_adl:create table test1(id int)
2020-05-22T21:52:02.180012+08:00
diag_adl:drop table test1
2020-05-22T21:56:33.734658+08:00
diag_adl:create table anbob.t100(id int)
2020-05-22T21:59:16.404601+08:00
diag_adl:create table test100(id int)
2020-05-22T21:59:55.817096+08:00
diag_adl:drop table test100
[oracle@anbob log]$ ls ddl
log.xml
[oracle@anbob log]$ more ddl/log.xml
<msg time='2020-05-22T21:51:55.793+08:00' org_id='oracle' comp_id='rdbms'
msg_id='kpdbLogDDL:21798:2946163730' type='UNKNOWN' group='diag_adl'
level='16' host_id='anbob' host_addr='192.168.56.101'
pid='4590' version='1' con_uid='3629755513'
con_id='3' con_name='PDBANBOB'>
<txt>create table test1(id int)
</txt>
</msg>
<msg time='2020-05-22T21:52:02.179+08:00' org_id='oracle' comp_id='rdbms'
msg_id='kpdbLogDDL:21798:2946163730' type='UNKNOWN' group='diag_adl'
level='16' host_id='anbob' host_addr='192.168.56.101'
pid='4590' con_uid='3629755513' con_id='3'
con_name='PDBANBOB'>
<txt>drop table test1
</txt>
</msg>
...

Or using ADRCI

adrci> show log -l ddl

ADR Home = /u02/app/oracle/diag/rdbms/anbob/anbob:
*************************************************************************
Output the results to file: /tmp/utsout_3020_140224_1.ado

2020-05-22 21:51:55.793000 +08:00
create table test1(id int)
2020-05-22 21:52:02.179000 +08:00
drop table test1
2020-05-22 21:56:33.734000 +08:00
create table anbob.t100(id int)
2020-05-22 21:59:16.404000 +08:00
create table test100(id int)
2020-05-22 21:59:55.815000 +08:00
drop table test100

— OVER–

Troubleshooting ORA-00600 [ORA_NPI_ERROR] ORA-00600: internal error code [kffilCreate01]

$
0
0

Oracle 11.2.0.3 2nodes RAC on AIX, node1  DB instance crash and db alert log show “ORA-00600 [ORA_NPI_ERROR] ORA-00600: internal error code [kffilCreate01]” and “ORA-15064: communication failure with ASM instance” then Instance terminated by ASMB.

# db alert log

Fri May 22 04:16:23 2020
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
NOTE: ASMB terminating
Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_asmb_40373526.trc:
ORA-15064: communication failure with ASM instance
ORA-00600: internal error code, arguments: [ORA_NPI_ERROR], [600], [ORA-00600: internal error code, arguments: [kffilCreate01], [anbob1:anbob], [], [], [], [], [], [], [], [], [], []
], [], [], [], [], [], [], [], [], []
ASMB (ospid: 40373526): terminating the instance due to error 15064
Fri May 22 04:16:33 2020
Instance terminated by ASMB, pid = 40373526 

# asm instance alert

2020-05-22 04:16:18.672000 +08:00
Errors in file /oracle/app/11.2.0.3/grid/log/diag/asm/+asm/+ASM1/trace/+ASM1_ora_27264968.trc  (incident=312217):
ORA-00600: internal error code, arguments: [kffilCreate01], [anbob1:anbob], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/11.2.0.3/grid/log/diag/asm/+asm/+ASM1/incident/incdir_312217/+ASM1_ora_27264968_i312217.trc

# tracefile

dump file: /oracle/app/11.2.0.3/grid/log/diag/asm/+asm/+ASM1/incident/incdir_312217/+ASM1_ora_27264968_i312217.trc
ORA-00600: internal error code, arguments: [kffilCreate01], [anbob1:anbob], [], [], [], [], [], [], [], [], [], []


*** 2020-05-22 04:16:33.782
-------------------------------------------------------------------------------
Trace Bucket Dump Begin: default bucket for process 27 (osid: 27264968)
TIME(*=approx):SEQ:COMPONENT:FILE@LINE:FUNCTION:SECT/DUMP: [EVENT#:PID:SID] DATA
-------------------------------------------------------------------------------
2020-05-22 04:16:18.656615 :C88A0662:KFNS:kfn.c@741:kfnDispatch(): completed KFNOP=5 callcnt=-1077812999
2020-05-22 04:16:18.656714 :C88A0667:KFNS:kfn.c@710:kfnDispatch(): calling server stub for KFNOP=5
2020-05-22 04:16:18.656715 :C88A0668:KFNU:kfns.c@1727:kfnsBackground(): kfnsBackground consuming in-progress message typ=14 unread=1 status=0xfffffff1
2020-05-22 04:16:18.656716 :C88A0669:KFNU:kfns.c@1965:kfnsConsume(): kfnsConsume message 0x70000013d66d2f8 status=0xfffffff1 flags=0x1
2020-05-22 04:16:18.656722 :C88A066A:KFNU:kfn.c@5315:kfnmsg_Dump(): kfnsBg (kfnmsg) opcode=15 (KFNMS_MAPLOAD) fd 0x7000001177b7e08 mid 4294967291, gn [2.1165624170], fn [1147.963253597] exts 4480, sta
rt 0, cnt 60
2020-05-22 04:16:18.656723 :C88A066B:KFNU:kfns.c@1786:kfnsBackground(): kfnsBackground got targeted message
2020-05-22 04:16:18.656726 :C88A066C:db_trace:kfns.c@5385:kfnsFillMapMsg(): [10491:27:838] MAP_LOAD: gn=2 fn=1147, mapid=-5 start=0 cnt=60
2020-05-22 04:16:18.656727 :C88A066D:KFNU:kfns.c@1942:kfnsBackground(): kfnsBackground completed in 0 seconds (KFNPM=2)
2020-05-22 04:16:18.656728 :C88A066E:KFNS:kfn.c@741:kfnDispatch(): completed KFNOP=5 callcnt=-1077812998
2020-05-22 04:16:18.656813 :C88A0673:KFNS:kfn.c@710:kfnDispatch(): calling server stub for KFNOP=5
2020-05-22 04:16:18.656814 :C88A0674:KFNU:kfns.c@1727:kfnsBackground(): kfnsBackground consuming in-progress message typ=15 unread=1 status=0xfffffff1
2020-05-22 04:16:18.656814 :C88A0675:KFNU:kfns.c@1965:kfnsConsume(): kfnsConsume message 0x70000011769cfa8 status=0xfffffff1 flags=0x1
2020-05-22 04:16:18.657300 :C88A06AA:KFNU:kfn.c@5366:kfnmsg_Dump(): kfnsBg (kfnmsg) op=25 (KFNMS_FILEUSE) (filuse) gn 2 un 0 mid 4294967292 cod 65535

----- Call Stack Trace -----
calling              call     entry               
location             type     point               
-------------------- -------- --------------------
dbkePostKGE_kgsf()+  call     dbgePostErrorKGE()  
68                                                
kgeadse()+380        call     dbkePostKGE_kgsf()  
kgerinv_internal()+  call     kgeadse()           
48                                                
kgerinv()+48         call     kgerinv_internal()  
kgeasnmierr()+72     call     kgerinv()           
kffilCreate()+540    call     kgeasnmierr()       
kfnsUFG()+4036       call     kffilCreate()       
kfnsBackground()+39  call     kfnsUFG()           
88                                                
kfnDispatch()+1128   call     kfnsBackground()    
opiodr()+720         call     kfnDispatch()       
ttcpip()+1028        call     opiodr()            
opitsk()+1508        call     ttcpip()            
opiino()+940         call     opitsk()            
opiodr()+720         call     opiino()            
opidrv()+1132        call     opiodr()            
sou2o()+136          call     opidrv()            
opimai_real()+608    call     sou2o()             
ssthrdmain()+268     call     opimai_real()       
main()+204           call     ssthrdmain()        
__start()+112        call     main()              

kffilCreate()+540 kernel automatic storage management file Create
kfnsUFG()+4036 kernel automatic storage management networking subsystem server networking [partial hit for: kfns ]

# grep -i mapid /oracle/app/11.2.0.3/grid/log/diag/asm/+asm/+ASM1/incident/incdir_312217/+ASM1_ora_27264968_i312217.trc
            (kffil) netnm: , mapid: 
            (kffil) netnm: anbob1:anbob, mapid: 4294967295
            (kffil) netnm: anbob1:anbob, mapid: 4294967294
            (kffil) netnm: anbob1:anbob, mapid: 4294946774
            (kffil) netnm: anbob1:anbob, mapid: 3849552408
            (kffil) netnm: anbob1:anbob, mapid: 3562194217
            (kffil) netnm: anbob1:anbob, mapid: 3487906158
            (kffil) netnm: anbob1:anbob, mapid: 3230128050
            (kffil) netnm: anbob1:anbob, mapid: 3185598356
            (kffil) netnm: anbob1:anbob, mapid: 3128598163
            (kffil) netnm: anbob1:anbob, mapid: 3080044389
            (kffil) netnm: anbob1:anbob, mapid: 3078043616
            (kffil) netnm: anbob1:anbob, mapid: 3044289890
            (kffil) netnm: anbob1:anbob, mapid: 2866698183
            (kffil) netnm: anbob1:anbob, mapid: 2764660939
            (kffil) netnm: anbob1:anbob, mapid: 2603804616
            (kffil) netnm: anbob1:anbob, mapid: 2412295670
            (kffil) netnm: anbob1:anbob, mapid: 2367686953
            (kffil) netnm: anbob1:anbob, mapid: 2358398223
            (kffil) netnm: anbob1:anbob, mapid: 2358377032
            (kffil) netnm: anbob1:anbob, mapid: 1469824483
            (kffil) netnm: anbob1:anbob, mapid: 1029941947
            (kffil) netnm: anbob1:anbob, mapid: 501493211
            (kffil) netnm: anbob1:anbob, mapid: 501288479
            (kffil) netnm: anbob1:anbob, mapid: 104836
            (kffil) netnm: anbob1:anbob, mapid: 89301
            (kffil) netnm: anbob1:anbob, mapid: 49316
            (kffil) netnm: anbob1:anbob, mapid: 47922
            (kffil) netnm: anbob1:anbob, mapid: 1067
            (kffil) netnm: anbob1:anbob, mapid: 1066
            (kffil) netnm: anbob1:anbob, mapid: 1065
            (kffil) netnm: anbob1:anbob, mapid: 1064
            (kffil) netnm: anbob1:anbob, mapid: 1062
            (kffil) netnm: anbob1:anbob, mapid: 1061
			...
			...
            (kffil) netnm: anbob1:anbob, mapid: 8
            (kffil) netnm: anbob1:anbob, mapid: 7
            (kffil) netnm: anbob1:anbob, mapid: 6
            (kffil) netnm: anbob1:anbob, mapid: 5
            (kffil) netnm: anbob1:anbob, mapid: 4
            (kffil) netnm: anbob1:anbob, mapid: 3
            (kffil) netnm: anbob1:anbob, mapid: 2
2020-05-22 04:16:18.656726 :C88A066C:db_trace:kfns.c@5385:kfnsFillMapMsg(): [10491:27:838] MAP_LOAD: gn=2 fn=1147, mapid=-5 start=0 cnt=60
2020-05-22 04:16:18.657439 :C88A06CD:db_trace:kfns.c@5310:kfnsFillMapMsg(): [10491:27:838] MAP_ALLOCATE: gn=2 fn=1135, mapid=-4 size=4480
2020-05-22 04:16:18.658157 :C88A079C:db_trace:kfns.c@5385:kfnsFillMapMsg(): [10491:27:838] MAP_LOAD: gn=2 fn=1135, mapid=-4 start=0 cnt=60
2020-05-22 04:16:18.658236 :C88A07BF:db_trace:kfns.c@5385:kfnsFillMapMsg(): [10491:27:838] MAP_LOAD: gn=2 fn=1147, mapid=-5 start=60 cnt=4420

[oracle@Xanbob1:/home/oracle> 
    ----------------------------------------
    SO: 0x7000001417a46a0, type: 4, owner: 0x7000001421bb628, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
     proc=0x7000001421bb628, name=session, file=ksu.h LINE:12624 ID:, pg=0
    (session) sid: 838 ser: 19 trans: 0x0, creator: 0x7000001421bb628
              flags: (0x41) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
              flags2: (0x9) -/-/INC
              DID: , short-term DID:
              txn branch: 0x0
              oct: 0, prv: 0, sql: 0x0, psql: 0x0, user: 0/SYS
    ksuxds FALSE at location: 0
    service name: SYS$USERS
    client details:
      O/S info: user: oracle, term: , ospid: 40373526
      machine: Xanbob1 program: oracle@Xanbob1 (TNS V1-V3)
      application name: oracle@Xanbob1 (TNS V1-V3), hash value=2834275364
    Current Wait Stack:
     1: waiting for 'ASM file metadata operation'
        msgop=0x19, locn=0x2, =0x0
        wait_id=3031433093 seq_num=59721 snap_id=1
        wait times: snap=3.413248 sec, exc=3.413248 sec, total=3.413248 sec
        wait times: max=infinite, heur=3.413248 sec
        wait counts: calls=0 os=0
        in_wait=1 iflags=0x520
		
 ----------------------------------------
      SO: 0x70000012919e930, type: 130, owner: 0x7000001417a46a0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
       proc=0x7000001421bb628, name=ASM db client, file=kfn2.h LINE:1353 ID:, pg=0
      (kfncl) netnm: anbob1:anbob, proc: 0x7000001421bb628, serial: 10
              state: 0x1 type: 257 compat: 11020003 version: 11020003 hbeat: 1590092178 (3) asmb:
        ----------------------------------------
        SO: 0x70000012a41f510, type: 125, owner: 0x70000012919e930, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
         proc=0x7000001421bb628, name=KFG state obj, file=kfg2.h LINE:1178 ID:, pg=0
        (kfgso) flags: 00000000 clt: 4 err: 0 hint: 29193b70
        (kfgpn) rpi: 23 itrn:0 gst:0 usrp:0
        busy: 0 rep: 0 grp: 0 check: 0/0 glink: 70000012a41f5d0 70000012a41f5d0
 ----------------------------------------
          SO: 0x700000129193b70, type: 126, owner: 0x70000012a41f510, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
           proc=0x7000001421bb628, name=KFG SO child, file=kfg2.h LINE:1181 ID:, pg=0
          (kfgsc) pwr: 0 flag: 0 op: 0 recur: 0
          (kfgpn) rpi: 3891 itrn:0 gst:0 usrp:0
 
           ----------------------------------------
            SO: 0x7000001177b8c30, type: 132, owner: 0x700000129193b70, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
             proc=0x7000001421bb628, name=ASM file, file=kff2.h LINE:855 ID:, pg=0
            (kffil) netnm: anbob1:anbob, mapid: 4294967295
                    gnum: 2.1165624170, fnum: 1135.963253681, flgs: 0x182
                    family: 0.0, parent: 0.0
                    opennm: +DATADG/rzwb_redo12
                    openflags: 0x80000000
                    fullnm: +datadg/rzwb_redo12
                    redun: 0x11, fdflg: 0x0, blksiz: 512, fsiz: 9172993
                    ftype: 3, extsz: [4294967295,0,0]
                    extents: 4480, start: 60, count: 4420, xp: 0x0
                    base: 2754, lxcnt: 1
                    enq: 0x70000013b03deb0, cod: 0x0
                    unlock[0].au: 0, unlock[0].disk: 0, unlock[0].flags 0x0

SQL> @dec 4294967295
                                DEC                  HEX
----------------------------------- --------------------
                  4294967295.000000             FFFFFFFF

Note:
We can find the “MAPID” has reached the maximum value.

The mapid I think is a sequence number mapped to ASM FILE id in each process, and the same as the file handle number called in the program.Due to a bug, a larger span of the serial number jumps. Search in MOS, hit Bug 16357438.

 

 

Wait Event: Wait for Table Lock

$
0
0

Wait for Table Lock

Meaning:
Wait for exclusive TM enqueue in materialized view operations.

Optimization steps:
In the SAP environment, materialized views (Note 741478) are only implicitly used as standard when you carry out an online reorganization. Therefore, the “Wait for Table Lock” wait event generally only occurs in connection with an online reorganization. If you reorganize a table online and execute commands such as DBMS_REDEFINITION.START_REDEF_TABLE or DBMS_REDEFINITION.FINISH_REDEF_TABLE, and if another session simultaneously holds an exclusive TM enqueue (Note 745639) on the same table, the reorg session waits for “Wait for Table Lock”.

Therefore, this wait event only affects the runtime of reorganizations, and does not affect the production operation. You can reduce the number of times “Wait for Table Lock” occurs by not carrying out any other activities that set an exclusive TM enqueue at the same time as the online reorganization (also see Note 745639).

To check table indexes or anther data Dict was wrong.

Know issue
Bug 20111004 “COMMENT ON COLUMN” statement waits 1 second on “Wait for Table Lock”

Alert: Remember to specify the table name alias for column of the subquery (不存在列名)

$
0
0

前几日同事那遇到的灾难性的SQL, 一个简单的子查询,不知是一开始就写错了子查询中的列名,还是子查询的表后期有改列名,导致主查询表所有数据被更改,在未指定列所属表名时,列的查询顺序先检查子查询再检查主查询,这是oracle的预期行为,并不会报列不存在,只是在后期书写时子查询记的增加列上的所属别名。

-DEMO

SQL> @st anbob.tp
        ID NAME
---------- ----------
         1 a
         2 a
         3 a
         4 a
         5 a

SQL> @st anbob.tc
       CID        PID NAME
---------- ---------- ----------
         1          5 a

SQL> @desc anbob.tc
           Name                            Null?    Type
           ------------------------------- -------- ----------------------------
    1      CID                                      NUMBER(38)
    2      PID                                      NUMBER(38)
    3      NAME                                     VARCHAR2(10)

SQL> select * from anbob.tp where id in (select id from anbob.tc);

        ID NAME
---------- ----------
         1 a
         2 a
         3 a
         4 a
         5 a

Note:
TC表中并不存在ID列,但是也未报错,而是直接返回了主查询的所有数据。

# enable opertimizer query transformation trace

SQL> oradebug setmypid
Statement processed.
SQL> @53on
alter session set events '10053 trace name context forever, level 1';
alter session set "_optimizer_trace"=all;

SQL> select * from anbob.tp where id in (select id from anbob.tc);
        ID NAME
---------- ----------
         1 a
         2 a
         3 a
         4 a
         5 a

SQL> @53off
alter session set events '10053 trace name context off';

# trace file

=====================================
SPD: BEGIN context at statement level
=====================================
Stmt: ******* UNPARSED QUERY IS *******
SELECT "TP"."ID" "ID","TP"."NAME" "NAME" 
   FROM "ANBOB"."TP" "TP" 
   WHERE "TP"."ID"=ANY (SELECT "TP"."ID" "ID" FROM "ANBOB"."TC" "TC")


*************************
Common Subexpression elimination (CSE)
*************************
CSE:     CSE not performed on query block SEL$2 (#2).
CSE:     CSE not performed on query block SEL$1 (#1).
Final query after transformations:******* UNPARSED QUERY IS *******
SELECT "TP"."ID" "ID","TP"."NAME" "NAME" 
FROM "ANBOB"."TP" "TP" 
WHERE  EXISTS (SELECT 0 FROM "ANBOB"."TC" "TC" WHERE "TP"."ID"="TP"."ID")
*************************

Note
在CBO查询转换后因为子查询不存在ID列, 优化认为是主查询的ID列,而条件成了”TP”.”ID”=”TP”.”ID”。 Always Be Ture!!!

SQL> delete anbob.tp where id in (select id from anbob.tc);

5 rows deleted.

SQL> @st anbob.tp
no rows selected

--全部被删除

SQL> rollback;

SQL> alter table anbob.tc rename column cid to id;
Table altered.

SQL> select * from anbob.tp where id in (select id from anbob.tc);

        ID NAME
---------- ----------
         1 a

在MOS Doc ID 124014.1 记录这是一种预期行为,不是oracle的BUG, 测试环境19c.

 

Oracle 20c新特性: dbms_xplan.display_awr 增加了谓词信息

$
0
0

执行计划中的谓词信息非常的重要,有助于我们判断是否进行了隐式转换,为什么没有使用索引等, 使用dbms_xplan.display_cursor可以从shared_pool中取到sq cursor的谓词,但是在20c之前dbms_xplan.display_AWR 在之前的版本中并不能, 但是在Oracle社区的投票和诸多人的推动下,终于在oracle 20c所谓词信息也在dbms_xplan.display_AWR中显示。

— demo

[oracle@oel7db1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 20.0.0.0.0 - Production on Sun May 24 01:46:12 2020
Version 20.2.0.0.0
Copyright (c) 1982, 2020, Oracle.  All rights reserved.
Connected to:
Oracle Database 20c Enterprise Edition Release 20.0.0.0.0 - Production
Version 20.2.0.0.0


USERNAME             INST_NAME            HOST_NAME                  I# SID   SERIAL#  VERSION    STARTED  SPID       OPID  CPID            SADDR            PADDR
-------------------- -------------------- ------------------------- --- ----- -------- ---------- -------- ---------- ----- --------------- ---------------- ----------------
SYS                  CDB$ROOT-anbob20c    oel7db1                     1 78    32736    20.0.0.0.0 20200524 10981      49    10980           00000000707F5EF0 000000007058B6B0

SQL> create table tobj as select * from dba_objects where rownum<=1000; Table created. SQL> set feedback on sql_id
SQL> select owner,count(*) from tobj where owner='SYS' group by owner;

OWNER                            COUNT(*)
------------------------------ ----------
SYS                                   985

SQL_ID: gsmdpptgv34w5
SQL> set feedback off
SQL> exec  dbms_workload_repository.add_colored_sql('gsmdpptgv34w5');
PL/SQL procedure successfully completed.

SQL> exec dbms_workload_repository.create_snapshot;

SQL> select owner,count(*) from tobj where owner='SYS' group by owner;

OWNER                            COUNT(*)
------------------------------ ----------
SYS                                   985

SQL> exec dbms_workload_repository.create_snapshot;
SQL> @st DBA_HIST_COLORED_SQL;

      DBID SQL_ID        CREATE_TIME             CON_ID
---------- ------------- ------------------- ----------
4232312917 gsmdpptgv34w5 2020-05-24 02:06:22          0

SQL> select * from dbms_xplan.display_awr(sql_id=>'gsmdpptgv34w5');

PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------
SQL_ID gsmdpptgv34w5
--------------------
select owner,count(*) from tobj where owner='SYS' group by owner

Plan hash value: 1334607550

-----------------------------------------------------------------------------
| Id  | Operation            | Name | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------
|   0 | SELECT STATEMENT     |      |       |       |     6 (100)|          |
|   1 |  SORT GROUP BY NOSORT|      |   985 | 65010 |     6   (0)| 00:00:01 |
|*  2 |   TABLE ACCESS FULL  | TOBJ |   985 | 65010 |     6   (0)| 00:00:01 |
-----------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - filter("OWNER"='SYS')

Note
-----
   - dynamic statistics used: dynamic sampling (level=2)

当然,这也可以从?/rdbms/admin/awrsqrpt.sql报告中看到谓词信息。

Note:Information will be captured in each snapshot for sqls marked using DBMS_WORKLOAD_REPOSITORY.ADD_COLORED_SQL. All the information related to the specific sql (sql plan, executions statistics, etc) will be stored in the AWR tables. However the SQLs will not neccesarily appear in the AWR Report sections related to SQLS unless they are indeed amongst the top SQLs for that sections as determined by the setting for DBMS_WORKLOAD_REPOSITORY.AWR_SET_REPORT_THRESHOLDS


DB Time 去哪了? Oracle 12C AWR 增加了on cpu runqueue

$
0
0

是否遇到过在分析AWR报告时,明明AAS很高,但从Top 10 Foreground Events by Total Wait Time 上看top event使用的百分比加起来离100%很远? 那DB TIME去哪了?下面我附一个11G(11.2.0.4 RAC on AIX)  AWR 案例,这个问题在12c的AWR中提供了TOP EVENT。

Host Name Platform CPUs Cores Sockets Memory (GB)
ANBOB.COM AIX-Based Systems (64-bit) 64 16 241.00
Snap Id Snap Time Sessions Cursors/Session Instances
Begin Snap: 30330 05-Mar-19 16:00:50 1422 3.0 2
End Snap: 30331 05-Mar-19 17:00:06 904 4.1 2
Elapsed: 59.28 (mins)
DB Time: 24,019.53 (mins)

Top 10 Foreground Events by Total Wait Time

Event Waits Total Wait Time (sec) Wait Avg(ms) % DB time Wait Class
DB CPU 51.4K 3.6
direct path read 64,724 3903.5 60 .3 User I/O
db file sequential read 219,201 1459.1 7 .1 User I/O
gc buffer busy acquire 273,842 1062.4 4 .1 Cluster
log file sync 36,584 882.1 24 .1 Commit
reliable message 102,567 854.4 8 .1 Other
gc current block 2-way 188,043 738.8 4 .1 Cluster
gc cr block 2-way 65,141 625.5 10 .0 Cluster
gc cr grant 2-way 103,189 241 2 .0 Cluster
gc current block busy 5,347 114.1 21 .0 Cluster
Statistic Name Time (s) % of DB Time
sql execute elapsed time 1,437,961.72 99.78
DB CPU 51,438.20 3.57
parse time elapsed 389.48 0.03
connection management call elapsed time 270.42 0.02
hard parse elapsed time 244.00 0.02
PL/SQL execution elapsed time 161.11 0.01
hard parse (sharing criteria) elapsed time 46.25 0.00
sequence load elapsed time 11.70 0.00
hard parse (bind mismatch) elapsed time 6.41 0.00
PL/SQL compilation elapsed time 1.16 0.00
repeated bind elapsed time 0.13 0.00
failed parse elapsed time 0.03 0.00
DB time 1,441,171.91
background elapsed time 2,121.28
background cpu time 84.06

从上面的数据看24,019/59.28 ≈407    AAS  >64 Cpus *3, 说明数据库有严重的性能问题。但是从event的百分比看加起来都不足5%. 那db Time去哪了?

DB Time
• Total time in database calls by foreground sessions
• Includes CPU time, IO time and non-idle wait time +Wait on CPU queue
• Total DB time = sum of DB time for all active sessions

过高的应用高并发或CPU耗尽,大量的前台进程就会等待在CPU queue队列,在cpu分片上频发switch , interrupt.  而且db cpu 不包含on queue, 下面看12c 的AWR, 增加了top event,  “CPU+ WAIT ON CPU”。 这样就可以就可以计算”on cpu runqueue”的百分比.

Oracle数据库许可(License)

$
0
0

Oracle产品可从Oracle网站免费下载。但是,在使用它之前,您需要同意Oracle Technology Network(OTN)开发人员许可条款。最近几年虚拟化、云化环境的普及,Oracle的license许可有些疑惑,在这里特意整理一下。Oracle文档中关于此主题的信息确实很少,稍不注意可能会带了百万级的费用消耗, 前提是你有严格对待ORACLE的软件许可,当然还是以Oracle合同中指定条款为主。

根据Oracle的定义,许可证是“在约定的条款和条件下使用Oracle软件的非排他性和有限权利”。Oracle主协议(OMA)中描述了总体许可权,订购文档中描述了有关特定产品和服务的权利。

对于开发和测试环境,通常可以听到也是需要够买License的,当然如果确认没有在Oracle的定义内,但是oracle定义确实比较严格明细,可以不购买。下面是oracle开发、测试、生产环境的定义:

Development Environment: Customers may use Oracle Full Use licenses in a development environment. Customers also may download Oracle technology products from the Oracle Technology Network (OTN) at http://otn.oracle.com/software/. In order to download an Oracle product from OTN, customers must signify their agreement to the terms of the OTN Development License. This limited license gives the user the right to develop, but not to deploy, applications using the licensed products. It also limits the use of the downloaded product to one person, and limits installation of the product to one server. Customers may not use products licensed under the OTN Development License in connection with any classroom activity, internal data processing operations, or any other commercial or production use purposes.

Test Environment: All programs used in a test environment must be licensed under an OLSA or other appropriate Oracle (or Oracle authorized reseller) license agreement.

Production Environment: The environment used by end users for business or other operations is called a production environment. All programs used in the production environment must be licensed under an OLSA or other appropriate Oracle (or Oracle authorized reseller) license agreement

Licensing Oracle Software in the Cloud Computing Environment  online

主要列出了– Amazon Elastic Compute Cloud (EC2), Amazon Relational Database Service (RDS) and  Microsoft Azure Platform (collectively, the ‘Authorized Cloud Environments’).

  1.  Amazon EC2 and RDS – count two vCPUs as equivalent to one Oracle Processor license if hyper-threading is enabled, and one vCPU as equivalent to one Oracle Processor license if hyper-threading is not enabled.
  2.   Microsoft Azure – count two vCPUs as equivalent to one Oracle Processor license if hyperthreading is enabled, and one vCPU as equivalent to one Oracle Processor license if hyperthreading is not enabled.

Licensing Oracle Software VMware

在VMware ESX基础结构上安装的Oracle数据库,是比常见的应用场景, 如果在您在VMware上运行Oracle,Oracle on VMware:受支持,但未经认证。Oracle不保证这些产品在VMware体系结构上正常工作,但是在发生与VMware基础架构无关的事件时将为您提供支持。

Oracle仅对已知在本机OS上发生的问题或由于不是在VMware上运行而导致的问题提供支持。如果问题是已知的Oracle问题,则Oracle支持人员将在本机OS上建议适当的解决方案。如果该解决方案在VMware虚拟环境中不起作用,则将向客户寻求VMware支持。

License 量化

为了量化和衡量客户使用Oracle软件的方式,Oracle引入了许可度量的概念。Oracle使用两个指标:包括cpu个数和用户数量。

对于企业版,客户必须计算核心数乘以核心因子,才能确定Oracle处理器核心数。核心因素决定了所使用的系数,具体取决于处理器类型。Oracle在以下地址提供的核心因子表中定义了该核心因子:http://www.oracle.com/us/corporate/contracts/processor-core-factor-table-070634.pdf

 

供应商和处理器

核心处理器许可因子

Sun和Fujitsu UltraSPARC T1处理器(1.0或1.2 GHz)仅命名服务器包括:

Sun Fire T1000服务器,SPARC Enterprise T1000服务器*,具有6或8核1.0 GHz UltraSPARC T1处理器

Sun Fire T2000服务器,SPARC Enterprise T2000服务器*,具有4、6或8核1.0 GHz或8核1.2 GHz UltraSPARC T1处理器

0.25

Sun Netra T2000、1.0或1.2 GHz UltraSPARC T1处理器

0.25

S PARC T3处理器

0.25

Sun和富士通UltraSPARC T1 1.4 GHz

仅命名服务器包括:

具有8核的Sun Fire T2000服务器和SPARC Enterprise T2000服务器*,

1.4 GHz UltraSPARC T1处理器

0.5

一个MD皓龙模型13XX,23XX,24XX,32XX,41XX,42XX,43XX,

61XX,62XX,63XX,83XX,84XX或更早的多核芯片

0.5

Intel Xeon系列56XX,65XX系列,75XX系列,E7-28XX系列,E7-28XX v2,E7-48XX系列,E7-48XX v2,E7-88XX系列,E7-88XX v2,E5-24XX系列,E5-26XX系列,E5-26XX v2,E5-46XX系列,E5-46XX v2,E5-16XX系列,E3-12XX系列或更早的多核芯片

0.5

Intel Itanium系列93XX或更早版本的多核芯片(适用于2010年12月1日之前购买的服务器)

0.5

英特尔或AMD台式机,笔记本电脑/笔记本电脑或上网本多核芯片

0.5

Sun UltraSPARC T2 +

0.5

SPARC64 VII +

0.5

SPARC64 X,SPARC64 X +

0.5

SPARC T4处理器

0.5

SPARC T5

0.5

SPARC M5

0.5

SPARC M6

0.5

Sun和Fujitsu SPARC64 VI,VII

0.75

Sun UltraSPARC IV,IV +或更早的多核芯片

0.75

Sun UltraSPARC T2

0.75

惠普PA-RISC

0.75

IBM POWER5 +或更早版本的多核芯片

0.75

所有单核芯片

1.0

我NTEL安腾系列93XX(有关或之后2010年12月1日购买的服务器)

英特尔安腾系列95XX

1.0

IBM POWER6

1.0

IBM POWER7,IBM POWER7 +

1.0

IBM POWER8

1.0

IBM System z(z10和更早版本)

1.0

所有其他多核芯片

1.0

购买oracle时的用户数怎么算?Oracle数据库按照用户数授权,是指最终端的连接到Oracle数据库的用户数。按照用户数来买的时候只能用于一个系统,不允许在多台机器上安装。每一个访问Oracle数据库的用户,无论是自然人还是设备,都算作一个用户 (Named User)。如果是B/S架构,那么是指连接到中间件上的用户数。

 

Oracle 官方要求的各类版本的最低用户数为:

个人版本:1user*PC数

标准版1=服务器的CPU数*5

标准版=服务器的CPU数*10user

企业版=服务器的CPU数*25user;

不同Oracle数据库版本之间的差异

  • Oracle数据库企业版没有许可证限制。这是唯一允许许可数据库选项(例如分区,OLAP,数据挖掘,空间,企业管理包等)的版本。此外,Oracle数据库企业版还包含其他版本中未提供的其他功能集(例如,Data Guard,可移动表空间,物化视图查询重写,跨平台备份,闪回表,数据库,事务查询等)。此外,该版本是Grid Control或Cloud Control允许监视的唯一版本。
  • Oracle数据库标准版只能在最大容量为4个套接字的服务器上获得许可。该版本包含Real Application Cluster,但是,在群集节点中增加套接字的数量不能超过4。它不是每个节点的最大值。
  • Oracle Database Standard Edition One只能在最大容量为2个套接字的服务器上获得许可。与标准版不同,标准版One不包含Real Application Cluster。此外,与标准版和企业版不同,标准版一不包括自动工作负载管理。
  • 可以将Oracle Database Express Edition(XE)安装在具有任意数量CPU的任何大小的主机上(每台计算机一个数据库),但是XE将最多存储11GB的用户数据,最多使用1GB的内存和一个CPU。主机。Express Edition中没有其他版本中集成的许多功能。更多可以看我之前的blog《浅谈ORACLE免费数据库Oracle Database XE版》

 

硬服务器和软服务器分区

“将服务器上的CPU分为多个独立分区时,其中每个部分都充当一个单独的系统。

有几种可用的硬件和软件虚拟化技术可提供分区功能,并具有不同程度的资源分配灵活性。”
客户越来越频繁地使用分区策略。基础技术和产品旨在实现以下目标:

  • 通过为特定系统分配或多或少的CPU能力来实现工作负载平衡
  • 通过在同一台物理服务器上运行多个不同的操作系统或同一操作系统的多个版本的架构合并,以优化资源使用
  • 部署“按需付费”和“按需容量”等经济模型

Oracle区分两种主要的分区类型:

1.软分区

Oracle将软分区定义为使用OS资源管理器对操作系统进行分段的一种灵活方式,因为分配给运行Oracle数据库的操作系统的CPU容量可以很容易地更改。如AIX工作负载管理器,Oracle VM和VMware。使用这种软分区技术的客户必须许可物理系统的全部核心或插槽。

在VMware 5.0之前的版本中,运行装有Oracle的虚拟机的客户必须完全许可组成VMware群集的物理主机。从VMware 5.1开始,如果只有一个虚拟机运行在vCenter Server实例中管理的Oracle,则必须完全许可由该vCenter Server实例管理的所有物理主机,而不仅仅是群集实例。

2.硬分区

硬分区被Oracle定义为“一种通过物理方式分割服务器的方法,即将一台大型服务器分离成不同的较小的系统。每个分离的系统都充当物理上独立的独立服务器,通常具有自己的CPU,操作系统,单独的引导区,内存,输入/输出子系统和网络资源。”ORACLE允许使用它们来限制任何给定服务器或服务器集群所需的软件许可证数量。包括:动态系统域(DSD),Oracle Linux for KVM, Oracle OVM Server, Oracle Solaris Zones, LPAR(在AIX 5.2中添加了DLPAR)Fujitsu的PPAR等。

Oracle Technology Licensing:

Oracle Engineered Systems Price List 2020

MySQL Price

the Price List and the Database Licensing Documents

Virtualized Environments, Partitioning technology

 

References:

https://www.oracle.com/a/ocom/docs/corporate/oracle-software-licensin

All you need to know about Oracle Database licensing with VMware

 

BBED simulates and fixes ORA-08102 error (Oracle 19c)

$
0
0

Sometimes due to sudden power failure and other reasons, the database data dictionary is inconsistent, such as hits ora-8102, the  indexes key and the table key value does not match, often delete the index, rebuild the index can be resolved, but if the object_id <60 bootstrap$ internal The index is damaged, and the normal situation needs to be backed up and restored, because some of the indexes in these bootstrap $ cannot be rebuilt by setting event 38003, which results in the database not being used properly, and the rebuilt index reports an error:

ORA-00701: object necessary for warmstarting database cannot be altered

Do not to test in the production environment!

env 19.3 pdb,    to  fix corrupted index I_OBJ4.

SQL> select banner from v$version;
BANNER
--------------------------------------------------------------------------------
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

SQL> show con_name
CON_NAME
------------------------------
PDB1

SQL> @desc bootstrap$ ;
           Name                            Null?    Type
           ------------------------------- -------- ----------------------------
    1      LINE#                           NOT NULL NUMBER
    2      OBJ#                            NOT NULL NUMBER
    3      SQL_TEXT                        NOT NULL VARCHAR2(4000)

SQL> select max(obj#),count(*) from bootstrap$ ;

 MAX(OBJ#)   COUNT(*)
---------- ----------
        59         60

SQL> select * from bootstrap$ where obj#=39;

     LINE#       OBJ#
---------- ----------
SQL_TEXT
--------------------------------------------------------------------------------
        39         39
CREATE INDEX I_OBJ4 ON OBJ$(DATAOBJ#,TYPE#,OWNER#) PCTFREE 10 INITRANS 2 MAXTRAN
S 255 STORAGE (  INITIAL 64K NEXT 1024K MINEXTENTS 1 MAXEXTENTS 2147483645 PCTIN
CREASE 0 OBJNO 39 EXTENTS (FILE 1 BLOCK 360))

backup system datafile

SQL> select file#,name from v$datafile;

     FILE# NAME
---------- -----------------------------------------------
         1 /u01/app/oracle/oradata/ANBOB19C/system01.dbf
         3 /u01/app/oracle/oradata/ANBOB19C/sysaux01.dbf
         4 /u01/app/oracle/oradata/ANBOB19C/undotbs01.dbf
         5 /u01/app/oracle/oradata/ANBOB19C/pdbseed/system01.dbf
         6 /u01/app/oracle/oradata/ANBOB19C/pdbseed/sysaux01.dbf
         7 /u01/app/oracle/oradata/ANBOB19C/users01.dbf
         8 /u01/app/oracle/oradata/ANBOB19C/pdbseed/undotbs01.dbf
         9 /u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf
        10 /u01/app/oracle/oradata/ANBOB19C/pdb1/sysaux01.dbf
        11 /u01/app/oracle/oradata/ANBOB19C/pdb1/undotbs01.dbf
        12 /u01/app/oracle/oradata/ANBOB19C/pdb1/users01.dbf

11 rows selected.

[oracle@oel7db1 admin]$ rman target sys/oracle@cdb1pdb1

Recovery Manager: Release 19.0.0.0.0 - Production on Thu May 28 09:59:56 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ANBOB19C:PDB1 (DBID=2777603868, not open)

RMAN> backup datafile 9;

Starting backup at 28-MAY-20
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00009 name=/u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf
channel ORA_DISK_1: starting piece 1 at 28-MAY-20
channel ORA_DISK_1: finished piece 1 at 28-MAY-20
piece handle=/u01/app/oracle/product/19.2.0/db_1/dbs/02v1antb_1_1 tag=TAG20200528T100219 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:07
Finished backup at 28-MAY-20

simulates ora-8102

SQL> @desc obj$
           Name                            Null?    Type
           ------------------------------- -------- ----------------------------
    1      OBJ#                            NOT NULL NUMBER
    2      DATAOBJ#                                 NUMBER
    3      OWNER#                          NOT NULL NUMBER
    4      NAME                            NOT NULL VARCHAR2(128)
    5      NAMESPACE                       NOT NULL NUMBER
    6      SUBNAME                                  VARCHAR2(128)
    7      TYPE#                           NOT NULL NUMBER
    8      CTIME                           NOT NULL DATE
    9      MTIME                           NOT NULL DATE
   10      STIME                           NOT NULL DATE
   11      STATUS                          NOT NULL NUMBER
   12      REMOTEOWNER                              VARCHAR2(128)
   13      LINKNAME                                 VARCHAR2(128)
   14      FLAGS                                    NUMBER
   15      OID$                                     RAW(16)
   16      SPARE1                                   NUMBER
   17      SPARE2                                   NUMBER
   18      SPARE3                                   NUMBER
   19      SPARE4                                   VARCHAR2(1000)
   20      SPARE5                                   VARCHAR2(1000)
   21      SPARE6                                   DATE
   22      SIGNATURE                                RAW(16)
   23      SPARE7                                   NUMBER
   24      SPARE8                                   NUMBER
   25      SPARE9                                   NUMBER
   26      DFLCOLLID                                NUMBER
   27      CREAPPID                                 NUMBER
   28      CREVERID                                 NUMBER
   29      CREPATCHID                               NUMBER
   30      MODAPPID                                 NUMBER
   31      MODVERID                                 NUMBER
   32      MODPATCHID                               NUMBER
   33      SPARE10                                  NUMBER
   34      SPARE11                                  NUMBER
   35      SPARE12                                  VARCHAR2(1000)
   36      SPARE13                                  VARCHAR2(1000)
   37      SPARE14                                  TIMESTAMP(6)

SQL> select file#,rfile#,name from v$datafile
     FILE#     RFILE# NAME
---------- ---------- ------------------------------------------------------------------------------------------
         9          1 /u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf
        10          4 /u01/app/oracle/oradata/ANBOB19C/pdb1/sysaux01.dbf
        11          9 /u01/app/oracle/oradata/ANBOB19C/pdb1/undotbs01.dbf
        12         12 /u01/app/oracle/oradata/ANBOB19C/pdb1/users01.dbf


SQL> alter system dump datafile 9 block 241;
System altered.

SQL> @t
TRACEFILE
-------------------------------------------------------------------
/u01/app/oracle/diag/rdbms/anbob19c/anbob19c/trace/anbob19c_ora_4735.trc

SQL> select max(dataobj#) from OBJ$;

MAX(DATAOBJ#)
-------------
        73509
 
SQL> select  dump(73509,16) from dual;
DUMP(73509,16)
----------------------
Typ=2 Len=4: c3,8,24,a

block dump trace file

tab 0, row 44, @0x3b9
tl: 87 fb: --H-FL-- lb: 0x1 cc: 26
col 0: [ 2] c1 02
col 1: [ 4] c3 08 24 0a   <<< data object id 
col 2: [ 1] 80
col 3: [12] 5f 4e 45 58 54 5f 4f 42 4a 45 43 54
col 4: [ 2] c1 02

bbed modify table block

[oracle@oel7db1 ~]$ bbed blocksize=8192 mode=edit filename='/u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf'
Password:

BBED: Release 2.0.0.0.0 - Limited Production on Sat May 30 06:08:23 2020

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

************* !!! For Oracle Internal Use only !!! ***************

BBED> show all;
        FILE#           0
        BLOCK#          1
        OFFSET          0
        DBA             0x00000000 (0 0,1)
        FILENAME        /u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf
        BIFILE          bifile.bbd
        LISTFILE
        BLOCKSIZE       8192
        MODE            Edit
        EDIT            Unrecoverable
        IBASE           Dec
        OBASE           Dec
        WIDTH           80
        COUNT           512
        LOGFILE         log.bbd
        SPOOL           No

BBED> set block 241
        BLOCK#          241

BBED> p *kdbr[44]
rowdata[202]
------------
ub1 rowdata[202]                            @1021     0x2c

BBED> dump /v offset 1021 count 16
 File: /u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf (0)
 Block: 241     Offsets: 1021 to 1036  Dba:0x00000000
-------------------------------------------------------
 2c011a02 c10204c3 08240a01 800c5f4e l ,...▒..▒.$...._N

BBED> x /rnnncncnt
rowdata[202]                                @1021
------------
flag@1021: 0x2c (KDRHFL, KDRHFF, KDRHFH)
lock@1022: 0x01
cols@1023:   26

col    0[2] @1024: 1
col    1[4] @1027: 73509
col    2[1] @1032: 0
col   3[12] @1034: _NEXT_OBJECT
col    4[2] @1047: 1
col    5[0] @1050: *NULL*
col    6[1] @1051: 0
col    7[7] @1053: 17-APR-19
col    8[7] @1061: 23-MAY-20
col    9[7] @1069: 17-APR-19
col   10[1] @1077:  0x80
col   11[0] @1079: *NULL*
col   12[0] @1080: *NULL*
col   13[1] @1081:  0x80
col   14[0] @1083: *NULL*
col   15[1] @1084:  0x80
col   16[4] @1086:  0xc3  0x07  0x38  0x24
col   17[1] @1091:  0x80
col   18[0] @1093: *NULL*
col   19[0] @1094: *NULL*
col   20[0] @1095: *NULL*
col   21[0] @1096: *NULL*
col   22[1] @1097:  0x80
col   23[1] @1099:  0x80
col   24[1] @1101:  0x80
col   25[4] @1103:  0xc3  0x02  0x40  0x53


BBED> set  block 241 offset 1028
        BLOCK#          241
        OFFSET          1028

BBED> d
 File: /u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf (0)
 Block: 241              Offsets: 1028 to 1043           Dba:0x00000000
------------------------------------------------------------------------
 c308240a 01800c5f 4e455854 5f4f424a

 <32 bytes per line>

BBED> m /x c308240e
BBED-00209: invalid number (c308240b)


BBED> set offset +3
        OFFSET          1031

BBED> d
 File: /u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf (0)
 Block: 241              Offsets: 1031 to 1046           Dba:0x00000000
------------------------------------------------------------------------
 0a01800c 5f4e4558 545f4f42 4a454354

 <32 bytes per line>

BBED> m /x e
 File: /u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf (0)
 Block: 241              Offsets: 1031 to 1046           Dba:0x00000000
------------------------------------------------------------------------
 0e01800c 5f4e4558 545f4f42 4a454354

 BBED> set  block 241 offset 1028
        BLOCK#          241
        OFFSET          1028

BBED> d
 File: /u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf (0)
 Block: 241              Offsets: 1028 to 1043           Dba:0x00000000
------------------------------------------------------------------------
 c308240e 01800c5f 4e455854 5f4f424a

BBED> sum apply
Check value for File 0, Block 241:
current = 0xc5d5, required = 0xc5d5

BBED> verify
DBVERIFY - Verification starting
FILE = /u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf
BLOCK = 241

DBVERIFY - Verification complete

Total Blocks Examined         : 1
Total Blocks Processed (Data) : 1
Total Blocks Failing   (Data) : 0
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0
Message 531 not found;  product=RDBMS; facility=BBED

Note:

 To  modify c308240a(73509)  to c308240e(73513).

restart pdb verify

SQL> alter pluggable database pdb1 close;
Pluggable database altered.

SQL> alter pluggable database pdb1 open;
Pluggable database altered.

SQL> @cc pdb1
ALTER SESSION SET container = pdb1;

Session altered.
 
SQL> create table anbob.t2 (id int);
create table anbob.t2 (id int)
*
ERROR at line 1:
ORA-08102: index key not found, obj# 39, file 9, block 25268 (2)

SQL> @ind I_OBJ4
Display indexes where table or index name matches %I_OBJ4%...

TABLE_OWNER          TABLE_NAME                     INDEX_NAME                     POS# COLUMN_NAME                    DSC
-------------------- ------------------------------ ------------------------------ ---- ------------------------------ ----
SYS                  OBJ$                           I_OBJ4                            1 DATAOBJ#
                                                                                      2 TYPE#
                                                                                      3 OWNER#
SQL> select /*+ index(t i_obj4) */ DATAOBJ#,type#,owner# from obj$  t
minus
  3  select /*+ full(t1) */ DATAOBJ#,type#,owner# from obj$  t1;

  DATAOBJ#      TYPE#     OWNER#
---------- ---------- ----------
     73509          0          0

SQL> select /*+ full(t1) */ DATAOBJ#,type#,owner# from obj$  t1
  minus
  3    select /*+ index(t i_obj4) */ DATAOBJ#,type#,owner# from obj$  t ;

  DATAOBJ#      TYPE#     OWNER#
---------- ---------- ----------
     73513          0          0

SQL> select /*+full(t)*/ rowid from  obj$ t where DATAOBJ#=73509;
no rows selected

SQL> select /*+full(t)*/ dump(rowid,16) from  obj$ t where DATAOBJ#=73513;
DUMP(ROWID,16)
---------------------------------------------------------------------------
Typ=69 Len=10: 0,0,0,12,0,40,0,f1,0,2c

1 row selected.

db alert log

Opening pdb with no Resource Manager plan active
joxcsys_required_dirobj_exists: directory object exists with required path /u01/app/oracle/product/19.2.0/db_1/javavm/admin/, pid 6791 cid 3

Pluggable database PDB1 opened read write
Completed: alter pluggable database pdb1 open
2020-05-30 06:41:31.465000 -04:00
Errors in file /u01/app/oracle/diag/rdbms/anbob19c/anbob19c/trace/anbob19c_ora_6791.trc:
2020-05-30 06:41:32.981000 -04:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************

trace file anbob19c_ora_6791

you can do 10046 event trace also.

*** 2020-05-30T06:49:11.706513-04:00 (PDB1(3))
oer 8102.2 - obj# 39, rdba: 0x004062b4(afn 9, blk# 25268)
kdk key 8102.2:
  ncol: 4, len: 16
  key: (16):  04 c3 08 24 0e 01 80 01 80 06 00 40 00 f1 00 2c    --table block key
  mask: (4096):
 a1 02 7f 00 00 d0 cd 4a b7 2f 7f 00 00 70 02 00 00 00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 20 cb 4a b7 2f
 7f 00 00 20 74 c9 12 00 00 00 00 98 79 64 6a 00 00 00 00 c8 b2 4a b7 2f 7f
 00 00 a0 59 4a b7 2f 7f 00 00 00 20 00 00 00 00 00 00 10 70 ad e3 ff 7f 00
 00 e3 0f 04 12 00 00 00 00 98 24 20 60 00 00 00 00 e8 0f 00 00 00 00 00 00
 88 31 17 60 00 00 00 00 b0 79 64 6a 00 00 00 00 0e 00 00 00 00 00 00 00 03
 00 00 00 2f 7f 00 00 ac 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 02 00
 00 00 00 00 00 00 a0 59 4a b7 2f 7f 00 00 00 00 00 00 00 00 00 00 e8 0f 00
 00 00 00 00 00 ff 2f 07 90 ff 7f 00 00 c0 63 4f b7 2f 7f 00 00 01 00 00 00
 ff 7f 00 00 00 00 00 00 00 00 00 00 00 20 00 00 ff 7f 00 00 20 74 c9 12 00
 00 00 00 88 31 17 60 00 00 00 00 28 1c 30 67 00 00 00 00 a0 59 4a b7 2f 7f
 00 00 00 20 00 00 00 00 00 00 90 71 ad e3 ff 7f 00 00 00 18 82 12 00 00 00
 00 34 1c 30 67 00 00 00 00 4a 21 82 12 00 00 00 00 b0 1e 00 60 00 00 00 00
 58 02 00 00 00 00 00 00 f0 0f 00 00 ff 7f 00 00 00 00 00 00 00 00 00 00 20
 cb 4a b7 2f 7f 00 00 c8 b2 4a b7 2f 7f 00 00 d0 a6 34 60 00 00 00 00 03 00

----- Current SQL Statement for this session (sql_id=c3utnxsnrx8tk) -----
update obj$ set obj#=:4, type#=:5,ctime=:6,mtime=:7,stime=:8,status=:9,dataobj#=:10,flags=:11,oid$=:12,spare1=:13,spare2=:14,spare3=:15,signature=:16,spare7=:17,spare8=:18,spare9=:19, dflcollid=decode(:20,0,null,:20),creappid=:21,creverid=:22, modappid=:23,modverid=:24,crepatchid=:25,modpatchid=:26 where owner#=:1 and name=:2 and namespace=:3 and remoteowner is null and linkname is null and subname is null

...

Block header dump:  0x004062b4
 Object id on Block? Y
 seg/obj: 0x27  csc:  0x000000000039bd2e  itc: 4  flg: -  typ: 2 - INDEX
     fsl: 0  fnx: 0x0 ver: 0x01

 Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   0x0008.001.000002a9  0x02402c9c.010e.01  CB--    0  scn  0x000000000022ac9c
0x02   0x0006.006.00000339  0x024005c2.00e2.39  C---    0  scn  0x000000000039bd2c
0x03   0x0003.009.00000396  0x02401c52.01c2.23  --U-    1  fsc 0x0000.0039bd42
0x04   0x0005.012.00000338  0x02400221.014f.5a  --U-    1  fsc 0x0000.0039bd37
Leaf block dump
===============
header address 2206580876=0x8385c08c
kdxcolev 0
KDXCOLEV Flags = - - -
kdxcolok 0
kdxcoopc 0x80: opcode=0: iot flags=--- is converted=Y
kdxconco 4
kdxcosdc 1
kdxconro 302
kdxcofbo 640=0x280
kdxcofeo 1844=0x734
kdxcoavs 1618
kdxlespl 0
kdxlende 0
kdxlenxt 4201842=0x401d72
kdxleprv 4219571=0x4062b3
kdxledsz 0
kdxlebksz 7984
row#0[7965] flag: -------, lock: 0, len=19

...

row#282[1844] flag: -------, lock: 3, len=21
col 0; len 4; (4):  c3 08 24 06
col 1; len 2; (2):  c1 03
col 2; len 3; (3):  c2 02 07
col 3; len 6; (6):  00 40 73 b3 00 0f
row#283[1886] flag: -------, lock: 0, len=18
col 0; len 4; (4):  c3 08 24 0a   <<< index key   diff table key
col 1; len 1; (1):  80
col 2; len 1; (1):  80
col 3; len 6; (6):  00 40 00 f1 00 2c   ---### rowid
row#284[2657] flag: -------, lock: 0, len=15
col 0; NULL
col 1; len 2; (2):  c1 02
col 2; len 1; (1):  80
col 3; len 6; (6):  00 40 11 c2 00 14
row#285[2642] flag: -------, lock: 0, len=15

bbed fixed index block

SQL> SELECT utl_raw.cast_to_number(replace('c3 08 24 0a',' ')) value FROM dual;
     VALUE
----------
     73509

1 row selected.

offset = 1886 + 44 + 24* number of itls  (in the case itl is 4)= 2026

BBED> set block 25268 offset 2026
        BLOCK#          25268
        OFFSET          2026

BBED> d count 32
 File: /u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf (0)
 Block: 25268            Offsets: 2026 to 2057           Dba:0x00000000
------------------------------------------------------------------------
 000004c3 08240a01 80018006 004000f1 002c0000 04c30823 5b02c103 03c20207

 <32 bytes per line>

BBED> set offset +6
        OFFSET          2032

BBED> d
 File: /u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf (0)
 Block: 25268            Offsets: 2032 to 2063           Dba:0x00000000
------------------------------------------------------------------------
 0a018001 80060040 00f1002c 000004c3 08235b02 c10303c2 02070600 4073b300

BBED> m /x e
 File: /u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf (0)
 Block: 25268            Offsets: 2032 to 2063           Dba:0x00000000
------------------------------------------------------------------------
 0e018001 80060040 00f1002c 000004c3 08235b02 c10303c2 02070600 4073b300

 <32 bytes per line>

BBED> d
 File: /u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf (0)
 Block: 25268            Offsets: 2032 to 2063           Dba:0x00000000
------------------------------------------------------------------------
 0e018001 80060040 00f1002c 000004c3 08235b02 c10303c2 02070600 4073b300

 <32 bytes per line>

BBED> sum apply
Check value for File 0, Block 25268:
current = 0xfb9e, required = 0xfb9e

restart pdb  to verify

SQL> alter pluggable database pdb1 close;
Pluggable database altered.

SQL> alter pluggable database pdb1 open;
Pluggable database altered.

SQL> create table anbob.t2 (id int);
Table created.

SQL> select max(data_object_id) from dba_objects;
MAX(DATA_OBJECT_ID)
-------------------
              73513

1 row selected.

— over —

如果您遇到了问题,自己解决不了,请联系 www.anbob.com 的首页上的联系方式。

BBED simulates and fixes ORA-08102 error (Oracle 19c) (二)

$
0
0

BBED simulates and fixes ORA-08102 error (Oracle 19c)

方法一使用了bbed修改 index key的方法, 因为表列上只有这一个索引,所以只改一个索引就可以。这里还使用相同的方法模拟ora-8102,使用第二种方法,删除bootstrap$中的index I_OBJ4 记录解决。

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ WRITE NO
 
SQL> @cc pdb1
ALTER SESSION SET container = pdb1;

Session altered.


USERNAME             INST_NAME            HOST_NAME                  I# SID   SERIAL#  VERSION    STARTED  SPID       OPID  CPID            SADDR            PADDR
-------------------- -------------------- ------------------------- --- ----- -------- ---------- -------- ---------- ----- --------------- ---------------- ----------------
SYS                  PDB1-anbob19c        oel7db1                     1 390   22700    19.0.0.0.0 20200531 2179       33    2068            0000000077881028 00000000785069A8


SQL> select file#,rfile#,name from v$datafile;

     FILE#     RFILE# NAME
---------- ---------- ----------------------------------------------------------------------
         9          1 /u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf
        10          4 /u01/app/oracle/oradata/ANBOB19C/pdb1/sysaux01.dbf
        11          9 /u01/app/oracle/oradata/ANBOB19C/pdb1/undotbs01.dbf
        12         12 /u01/app/oracle/oradata/ANBOB19C/pdb1/users01.dbf

SQL> select   max(DATAOBJ#)from obj$  t;
MAX(DATAOBJ#)
-------------
        73523
 
SQL> select /*+ index(t i_obj4) */ dump(rowid,16) from obj$  t where dataobj#=73523;

DUMP(ROWID,16)
----------------------------------------------------
Typ=69 Len=10: 0,0,0,12,0,40,0,f1,0,2c


SQL> select dump(73523,16) from dual;
DUMP(73523,16)
-----------------------
Typ=2 Len=4: c3,8,24,18


BBED> d
 File: /u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf (0)
 Block: 25268            Offsets: 1933 to 2444           Dba:0x00000000
------------------------------------------------------------------------
 18018001 80060040 00f1002c 000404c3 08240e02 c10303c2 02070600 4073b300
 10010304 c3082413 01800180 06004000 f1002c00 0004c308 240602c1 0303c202

BBED> m /x 1a
Warning: contents of previous BIFILE will be lost. Proceed? (Y/N) y
 File: /u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf (0)
 Block: 25268            Offsets: 1933 to 2444           Dba:0x00000000
------------------------------------------------------------------------
 1a018001 80060040 00f1002c 000404c3 08240e02 c10303c2 02070600 4073b300

BBED> sum apply
Check value for File 0, Block 25268:
current = 0x0126, required = 0x0126


SQL> alter pluggable database pdb1 close;
Pluggable database altered.

SQL> alter pluggable database pdb1 open;
Pluggable database altered.

SQL> create table anbob.test1 as select 1 id from dual;
create table anbob.test1 as select 1 id from dual
                                             *
ERROR at line 1:
ORA-08102: index key not found, obj# 39, file 9, block 25268 (2)


bbed fixed

SQL> select  distinct dbms_rowid.rowid_relative_fno(rowid) file#,dbms_rowid.rowid_block_number(rowid) block#   from bootstrap$ ;

     FILE#     BLOCK#
---------- ----------
         1        521
         1        523
         1        522


SQL> select   dbms_rowid.rowid_relative_fno(rowid) file#,dbms_rowid.rowid_block_number(rowid) block#,
     dbms_rowid.rowid_row_number(rowid) row#  from bootstrap$ where obj#=39;

     FILE#     BLOCK#       ROW#
---------- ---------- ----------
         1        523          7

BBED> set block 523
        BLOCK#          523

BBED> map
 File: /u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf (0)
 Block: 523                                   Dba:0x00000000
------------------------------------------------------------
 KTB Data Block (Table/Cluster)
 struct kcbh, 20 bytes                      @0
 struct ktbbh, 48 bytes                     @20
 struct kdbh, 14 bytes                      @68
 struct kdbt[1], 4 bytes                    @82
 sb2 kdbr[14]                               @86
 ub1 freespace[1557]                        @114
 ub1 rowdata[6517]                          @1671
 ub4 tailchk                                @8188

BBED> p * kdbr[7]
rowdata[3766]
-------------
ub1 rowdata[3766]                           @5437     0x2c

BBED> set offset 5437
        OFFSET          5437

BBED> d count 32
 File: /u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf (0)
 Block: 523              Offsets: 5437 to 5468           Dba:0x00000000
------------------------------------------------------------------------
 2c000302 c12802c1 28cd4352 45415445 20494e44 45582049 5f4f424a 34204f4e

 <32 bytes per line>

SQL> @desc  bootstrap$
           Name                            Null?    Type
           ------------------------------- -------- ----------------------------
    1      LINE#                           NOT NULL NUMBER
    2      OBJ#                            NOT NULL NUMBER
    3      SQL_TEXT                        NOT NULL VARCHAR2(4000)


BBED> x /rnnc
rowdata[3766]                               @5437
-------------
flag@5437: 0x2c (KDRHFL, KDRHFF, KDRHFH)   ## row flag
lock@5438: 0x00
cols@5439:    3 ## columns

col    0[2] @5440: 39
col    1[2] @5443: 39
col  2[205] @5446: CREATE INDEX I_OBJ4 ON OBJ$(DATAOBJ#,TYPE#,OWNER#) PCTF
REE 10 INITRANS 2 MAXTRANS 255 STORAGE (  INITIAL 64K NEXT 1024K MINEXTENTS
 1 MAXEXTENTS 2147483645 PCTINCREASE 0 OBJNO 39 EXTENTS (FILE 1 BLOCK 360))

BBED> m /x 3c
 File: /u01/app/oracle/oradata/ANBOB19C/pdb1/system01.dbf (0)
 Block: 523              Offsets: 5437 to 5468           Dba:0x00000000
------------------------------------------------------------------------
 3c000302 c12802c1 28cd4352 45415445 20494e44 45582049 5f4f424a 34204f4e

 <32 bytes per line>
 
BBED> x /rnnc
rowdata[3766]                               @5437
-------------
flag@5437: 0x3c (KDRHFL, KDRHFF, KDRHFD, KDRHFH)
lock@5438: 0x00
cols@5439:    0

BBED> sum apply
Check value for File 0, Block 523:
current = 0x7186, required = 0x7186

Note:
delete will change ‘row flag’ from 0x2c to 32+16+8+4 = 60 or 0x3c.

verify

SQL> select *  from bootstrap$ where obj#=39;
no rows selected

SQL> @ind obj$
Display indexes where table or index name matches %obj$%...

TABLE_OWNER          TABLE_NAME                     INDEX_NAME                     POS# COLUMN_NAME                    DSC
-------------------- ------------------------------ ------------------------------ ---- ------------------------------ ----
                     OBJ$                           I_OBJ1                            1 OBJ#
                                                                                      2 OWNER#
                                                                                      3 TYPE#
                                                    I_OBJ2                            1 OWNER#
                                                                                      2 NAME
                                                                                      3 NAMESPACE
                                                                                      4 REMOTEOWNER
                                                                                      5 LINKNAME
                                                                                      6 SUBNAME
                                                                                      7 TYPE#
                                                                                      8 SPARE3
                                                                                      9 OBJ#
                                                    I_OBJ3                            1 OID$
                                                    I_OBJ4                            1 DATAOBJ#
                                                                                      2 TYPE#
                                                                                      3 OWNER#
                                                    I_OBJ5                            1 SPARE3
                                                                                      2 NAME
                                                                                      3 NAMESPACE
                                                                                      4 TYPE#
                                                                                      5 OWNER#
                                                                                      6 REMOTEOWNER
                                                                                      7 LINKNAME
                                                                                      8 SUBNAME
                                                                                      9 OBJ#


SQL> create table anbob.test1 as select 1 id from dual;
Table created.

SQL> show pdbs
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         3 PDB1                           READ WRITE NO
SQL> select   max(DATAOBJ#)from obj$  t;

MAX(DATAOBJ#)
-------------
        73530

1 row selected.

— over —

Wait event: enq: HW – contention

$
0
0

这是一套11G数据库,一线DBA反应是数据库活动会话很高, 要来一份AWR,显示数据库top event中较高的是enq: HW – contention和enq: TX – row lock contention,而另一节点是有较高的direct path read.

DB Name DB Id Instance Inst num Startup Time Release RAC
ANBOB 3391174999 anbob1 1 03-May-20 18:05 11.2.0.4.0 YES
Host Name Platform CPUs Cores Sockets Memory (GB)
anbob.com1 Solaris[tm] OE (64-bit) 512 64 4 1023.00
Snap Id Snap Time Sessions Cursors/Session Instances
Begin Snap: 88631 03-Jun-20 20:00:17 549 .6 2
End Snap: 88633 03-Jun-20 21:00:36 2293 2.7 2
Elapsed: 60.30 (mins)
DB Time: 47,824.25 (mins)

Report Summary

Load Profile

Per Second Per Transaction Per Exec Per Call
DB Time(s): 793.1 11.6 0.48 0.21
DB CPU(s): 21.5 0.3 0.01 0.01
Redo size (bytes): 397,274.5 5,831.3
Logical read (blocks): 1,254,716.0 18,417.0
Block changes: 2,235.9 32.8
Physical read (blocks): 576.7 8.5
Physical write (blocks): 107.4 1.6
Read IO requests: 223.2 3.3
Write IO requests: 79.5 1.2
Read IO (MB): 4.5 0.1
Write IO (MB): 0.8 0.0
Global Cache blocks received: 290.0 4.3
Global Cache blocks served: 205.6 3.0
User calls: 3,830.6 56.2
Parses (SQL): 1,433.6 21.0
Hard parses (SQL): 4.1 0.1
SQL Work Area (MB): 11.2 0.2
Logons: 1.9 0.0
Executes (SQL): 1,651.4 24.2
Rollbacks: 22.1 0.3
Transactions: 68.1

Instance Efficiency Percentages (Target 100%)

Buffer Nowait %: 99.98 Redo NoWait %: 99.99
Buffer Hit %: 99.98 In-memory Sort %: 100.00
Library Hit %: 99.65 Soft Parse %: 99.71
Execute to Parse %: 13.19 Latch Hit %: 98.14
Parse CPU to Parse Elapsd %: 85.74 % Non-Parse CPU: 99.55

Top 10 Foreground Events by Total Wait Time

Event Waits Total Wait Time (sec) Wait Avg(ms) % DB time Wait Class
enq: HW – contention 1,409 773.6K 549069 27.0 Configuration
enq: TX – row lock contention 4,853 600.8K 123792 20.9 Application
row cache lock 183,475 207.7K 1132 7.2 Concurrency
gc buffer busy acquire 386,790 169.4K 438 5.9 Cluster
db file sequential read 737,067 161.2K 219 5.6 User I/O
gc buffer busy release 178,230 139.9K 785 4.9 Cluster
read by other session 185,264 123.8K 668 4.3 User I/O
enq: US – contention 87,502 83.3K 951 2.9 Other
DB CPU 77.7K 2.7
log file sync 181,402 73.7K 406 2.6 Commit

从以上信息得到1小时内会话数据上涨2000, 每秒dbtime 793 已高于512 CPUs说明这个系统负载已经很高,每秒逻辑读125万,DB IOPS约300多, IO吞吐量几M,transaction 和SQL 执行量也并不高, % Non-Parse CPU 达99%+,从当前实例看不是解析和事务量大导致的。

从TOP EVENT看是enq: HW – contention,enq: TX – row lock contention 占了前台db time 50%, avg ms已经高的失真, 下面看IO 类的avg 也在200ms。

Operating System Statistics – Detail

Snap Time Load %busy %user %sys %idle %iowait
03-Jun 20:00:17 13.59
03-Jun 20:30:40 6.61 7.23 6.26 0.97 92.77 0.00
03-Jun 21:00:36 13.77 2.99 2.11 0.88 97.01 0.00

Back to Wait Events Statistics
Back to Top

Foreground Wait Class

  • s – second, ms – millisecond – 1000th of a second
  • ordered by wait time desc, waits desc
  • %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
  • Captured Time accounts for 87.1% of Total DB time 2,869,455.22 (s)
  • Total FG Wait Time: 2,421,211.45 (s) DB CPU time: 77,748.59 (s)
Wait Class Waits %Time -outs Total Wait Time (s) Avg wait (ms) %DB time
Configuration 4,316 34 777,663 180181 27.10
Application 16,739 0 600,782 35891 20.94
Cluster 1,900,124 4 340,004 179 11.85
User I/O 1,148,504 0 296,017 258 10.32
Concurrency 983,527 6 245,365 249 8.55
Other 414,877 70 83,754 202 2.92
DB CPU 77,749 2.71
Commit 181,402 0 73,654 406 2.57
System I/O 32,000 0 3,941 123 0.14
Network 9,675,267 0 32 0 0.00

数据库负载AAS高,但CPU并不高,最高的是configuration CLASS也就是enq: hw.

Background Wait Events

  • ordered by wait time desc, waits desc (idle events last)
  • Only events with Total Wait Time (s) >= .001 are shown
  • %Timeouts: value of 0 indicates value was < .5%. Value of null is truly 0
Event Waits %Time -outs Total Wait Time (s) Avg wait (ms) Waits /txn % bg time
db file parallel write 223,056 0 18,135 81 0.90 39.93
gcs log flush sync 560,810 45 8,156 15 2.28 17.96
control file sequential read 15,802 0 3,755 238 0.06 8.27
log file parallel write 276,181 0 1,470 5 1.12 3.24
log file sync 488 0 1,302 2668 0.00 2.87
db file sequential read 3,224 0 1,160 360 0.01 2.55
control file parallel write 4,928 0 594 121 0.02 1.31

后台db file 和controlfile 的IO 也并不好。

Global Cache Load Profile

Per Second Per Transaction
Global Cache blocks received: 290.03 4.26
Global Cache blocks served: 205.60 3.02
GCS/GES messages received: 655.58 9.62
GCS/GES messages sent: 959.28 14.08
DBWR Fusion writes: 13.02 0.19
Estd Interconnect traffic (KB) 4,280.46

Global Cache and Enqueue Services – Workload Characteristics

Avg global enqueue get time (ms): 47.8
Avg global cache cr block receive time (ms): 79.1
Avg global cache current block receive time (ms): 18.6
Avg global cache cr block build time (ms): 0.0
Avg global cache cr block send time (ms): 0.0
Global cache log flushes for cr blocks served %: 16.1
Avg global cache cr block flush time (ms): 349.8
Avg global cache current block pin time (ms): 0.9
Avg global cache current block send time (ms): 0.0
Global cache log flushes for current blocks served %: 1.3
Avg global cache current block flush time (ms): 1,395.7

Global Cache and Enqueue Services – Messaging Statistics

Avg message sent queue time (ms): 0.1
Avg message sent queue time on ksxp (ms): 0.6
Avg message received queue time (ms): 0.0
Avg GCS message process time (ms): 0.1
Avg GES message process time (ms): 0.0
% of direct sent messages: 45.61
% of indirect sent messages: 52.52
% of flow controlled messages: 1.86

Global CURRENT Served Stats

  • Pins = CURRENT Block Pin Operations
  • Flushes = Redo Flush before CURRENT Block Served Operations
  • Writes = CURRENT Block Fusion Write Operations
Statistic Total % <1ms % <10ms % <100ms % <1s % <10s
Pins 1,912 47.59 34.21 7.79 4.55 5.86
Flushes 6,047 29.67 37.95 1.03 2.40 28.96
Writes 48,372 36.00 56.30 0.61 1.56 5.53

Note:

网络带宽使用也就4MB, 但cr/current block receive time不是理想,cr/ current block flush time 存在严重问题,Flush 指脏块被LMS进程传输出去之前,其相关的redo必须由LGWR已经flush 到磁盘上。 对于这个案例我们从SQL ordered by Elapsed Time找到了insert SQL, 该表中确实有lob字段。同时该时间段另一实例的direct path 是一个几十GB的大表在全表扫描。

What is an enqueue?

Enqueues are shared memory structures (locks) that serialize access to database resources. They can be associated with a session or transaction.

What is high water mark (HW)

The boundary between used and unused space in a segment.

enq: HW – contention

. In Oracle, the High-water Mark (HWM) of a segment is a pointer to a data block up to which free blocks are formatted and are available to insert new data. If data is inserted at a high rate, new blocks may have to be made available after a search for the freelists or L1 blocks was unable to return any space. This involves formatting the blocks, inserting them into a segment header or bitmap block and pushing up the HWM. This has to happen while holding the HWM enqueue and acquiring exclusive access to the new blocks. Normally, a batch of blocks with contiguous block addresses is  “newed” and the HW enqueue is not released until the end of the operation.

he former being a consequence of the serialization on the High-water Mark enqueue (HW), and the latter relating to the fact that current access to the new data blocks is required for the “newing” operation. In a RAC environment, the length of this space management operation is proportional to the time it takes to acquire the HW enqueue and the time it takes to acquire global locks for all the new blocks. This time is small under normal circumstances because there will never be any access conflict for the new blocks.

Waits on High Water enqueue occur when allocating space beyond the high water mark of a segment during insert operations. Also, this wait occurs when reclaiming free space in lob segments after delete or update operations.  When lob segments are deleted or updated, the space being used is not immediately freed up so consistent reads can be maintained. When more space is needed, a HW enqueue lock is acquired to reclaim the free space from the previous deletes or updates. The PCTVERSION or RETENTION settings on the log segment object control the frequency of when the space is reclaimed.

To find the root cause of the problem, determine which segment is growing fast, either by extrapolating from the rates at which data is inserted (V$SQLAREA) or by capturing the id1 and id2 of the HW enqueue waited for, where the id1 is the tablespace number in which the segment resides and id2 will be the block address of the segment header。

Parameters:

You can get the information about the event from dba_hist_active_sess_history.

SELECT event, 
 p1, 
 p2, 
 p3, 
 Count(1) 
FROM dba_hist_active_sess_history 
WHERE event_id = 1645217925  --enq hw
GROUP BY event,

tablespace#
This is the tablespace id for the segment we want an HW lock on.
You can find the tablespace name using P2 thus:

SELECT *
FROM dba_tablespaces
WHERE tablespace_name = ( select name from sys.ts$ where ts#=&P2 )
;

block (RDBA)
This parameter depends on whether the tablespace is a BIGFILE tablespace or not:
If the tablespace is BIGFILE then P3 is the block number in the file for that tablespace.

If the tablespace is NOT a BIGFILE tablespace then P2 is a relative DBA (RDBA) rather than a block#. The value encodes the relative file# and block# of the segment header that the HW lock is requested for in a 4 byte value. You can convert P3 to a relative file and block# using DBMS_UTILITY:

SELECT DBMS_UTILITY.DATA_BLOCK_ADDRESS_FILE(rdba) RELFILE#,
DBMS_UTILITY.DATA_BLOCK_ADDRESS_BLOCK(rdba) BLOCK#
FROM (select &P3 rdba from dual)
;

Now we can use this information to get which segment it belongs too.

SELECT owner,
       segment_type,
       segment_name
FROM   dba_extents
WHERE  file_id = 14
       AND 266 BETWEEN block_id AND block_id + blocks - 1;

if It’s a lob segment, We can find out which table and column by checking dba_lobs table.

SELECT owner, 
 table_name, 
 column_name, 
 chunk, 
 securefile 
FROM dba_lobs 
WHERE segment_name = 'SYS_LOB00000xxxx'

Finding Blockers:

You can find current blockers by querying GV$LOCK like this:

SELECT distinct w.HW, w.p2 OBJECT_ID, l.inst_id, l.sid, l.lmode, l.request
 FROM 
  ( SELECT p2, p3, 'HW-'||substr(p2raw,-8)||'-'||lpad(p3,8,'0') HW
      FROM v$session_wait 
     WHERE event='enq: HW - contention'
       and state='WAITING'
  ) W, 
  gv$lock L
 WHERE l.type(+)='HW'
   and l.id1(+)=w.p2
   and l.id2(+)=w.p3
 ORDER BY hw, lmode desc, request desc
There are 10 bugs listed.
NB Prob Bug Fixed Description
28067846 19.1 System state did not dump lock elements which was considered busy
III 27986817 12.1.0.2.190416, 19.1 L1 Validation Waits With High Concurrent Workload
III 20687474 12.2.0.1 Excessive “enq: HW – contention” / “latch: cache buffers chains” inserting into very large non-partitioned table
II 18221857 11.2.0.4.4.SAPEXADBBP, 11.2.0.4.BP15, 12.1.0.2, 12.2.0.1 “enq: HW – contention” on large segments with lots of extents and high insert rate
IIII 7319120 HW contention for BasicFile LOBs – special workaround
III 13357632 11.2.0.4, 12.1.0.1 High ST contention from parallel direct load & MERGE
III 9801919 11.2.0.1.BP08, 11.2.0.2.10, 11.2.0.2.BP20, 11.2.0.3, 12.1.0.1 “enq: HW – contention” against segments that add an extent frequently during high concurrency
D III 6413373 10.2.0.5, 11.1.0.7, 11.2.0.1 Lob HW enqueue contention in RAC environments with ASSM space management
D IIII 6376915 10.2.0.4, 11.1.0.7, 11.2.0.1 HW enqueue contention for ASSM LOB segments
II 4867884 10.2.0.3, 11.1.0.6 Lob HW lock contention with space reclaimation

HW enqueue根本存在目的是为了串行化对segment高水位线的移动以及回收lob segment中的空间。造成HW enqueue contention争用的根本原因多种多样,但最终HW enqueue总是只在数据段segment High Water Mark高水位线需要移动时才被持有。

常见的原因:
1, 频繁的insert
2, IO性能差
3, LOB 对象回收
4, LOB相关的BUG

常见的解决方法:

1, move 到ASSM 管理的表空间
2, 手动预分配空间allocate extent
3, 查找物理IO读较高的SQL 或 查找存储性能问题
4, 如果lob对象存在较频繁的更新,可以考虑增加LOB PCTVERSION to 20%,保留更多的空间用于lob的旧版本,考虑lob 分区或增加lob 段的chunk大小
5,   由于ASSM LOB空间分析一次只获取一个块,导致HW争用,应用 Bug 6376915实现批量回收,或通过此修复程序,ASSM LOB将基于事件44951的值获得最小数量的块(最大设置为1024)。

alter system set events '44951 trace name context forever, level 1024';

6, 迁移basicfile LOBs到Securefiles LOB对于11g or later. Securefile LOBs对于并发的性能更好

How to debug

From Oracle version 10g and above, x$ksqst is externalized as v$enqueue_statistics.

However, this statistics shows activity at instance level. While we can use this statistics to measure HWenqueue activity , we need to make sure that there is no other session acquiring HW enqueue. Event 10704 can be used to trace enqueues and every call to get an enqueue prints few lines in the trace file. SQL statement to dump this information to trace file is :

alter session set events ’10704 trace name context forever, level 15′;

Event 10704 is documented as below:

10704, 00000, "Print out information about what enqueues are being obtained"
// *Cause:  When enabled, prints out arguments to calls to ksqcmi and
//          ksqlrl and the return values.
// *Action: Level indicates details:
//   Level: 1-4: print out basic info for ksqlrl, ksqcmi
//          5-9: also print out stuff in callbacks:  ksqlac, ksqlop
//          10+: also print out time for each line

Few lines from the trace files printed below. ksq is internal Oracle module names for enqueues and ksqgtl is to get locks on a resource. From the lines below, we can see that HW enqueue is acquired in mode 6, exclusive mode. Timestamp is also printed since we enabled this event at level 15. If we need count number of HW enqueue gets, we need to count occurrences of HW- string in the trace file.

Troubleshooting ORA-27102: out of memory Linux-x86_64 Error: 12: Cannot allocate memory

$
0
0

Insufficent memory related errors (i.e: ORA-27xxx).ORA-27102 error may occur because of insufficient shared memory segments.

ORA-27102 on Startup

A] Ensure that the physical RAM is greater than the sga_max_size / sga_target
B] Ensure SHMMAX kernel setting is equal or larger than the sga max size Usually occurs with  Error: 22: Invalid argument,Shmall is the total amount of shared memory
C] Ensure Ulimit settings are unlimited for the memory related parameters
C.1] The “ulimit -l” parameter is not set
D] Ensure that the swap size Usually occurs with Error: 12: Cannot allocate memory
E] If oracle version is 32bit, the maximum SGA_TARGET / SGA_MAX_SIZE is 1.75gb
unix> file $ORACLE_HOME/bin/oracle
F] Automatic Memory Management MEMORY_TARGET and MEMORY_MAX_TARGET cannot be used when LOCK_SGA is enabled or with huge pages on Linux.

 

ORA-27102 To Connect To Database

A] Bug 13691225 – File handle leak for processes accessing /proc/stat
B] Bug 24921392 Linux:Background process fails to start under memory/swap pressure
C] BUG 24304753 – RESTORE SHOULD CHECK AVAILABLE RAM AGAINST REQUIRED MEMORY FROM PFILE has been filed and fix will be released with AVDF 12.2BP4.
D] Not enough space for Oracle memory under /dev/shm
E] NUMA, Setting numa=off in the kernel for x86_64 machines.

A CASE CAUSED BY NUMA
For versions prior to 11gR2 (where NUMA is disabled by default), ensure the NUMA (Non Uniform Memory Architecture) feature is turned OFF unless explicitly required and tested, as there have been issues reported with NUMA enabled. Refer to Document 759565.1 for more details.Oracle 最佳实践中建议禁用NUMA,.On NUMA Enabled Systems, the value of vm.min_free_kbytes = n * 0.4% of total Physical Memory. Here ‘n’ is the number of NUMA nodes.MySQL采用了线程模式,对于NUMA特性的支持并不好,如果单机只运行一个MySQL实例,建议关闭NUMA

 

zzz ***Thu Jun 4 17:01:36 CST 2020
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
4 1 9903044 4558384 288372 4925348 0 0 54 16 0 0 2 1 97 0 0
1 0 9903044 4552780 288372 4925388 0 0 387 258 7637 7453 8 3 89 0 0
1 0 9903036 4554308 288376 4925508 32 0 5907 105 7707 8353 8 1 90 1 0<<<<<
zzz ***Thu Jun 4 17:01:41 CST 2020
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
6 0 9903032 4827400 288376 4657420 0 0 54 16 0 0 2 1 97 0 0
3 0 9903032 4819560 288376 4657424 0 0 35 182 7782 7220 8 3 89 0 0<<<<4706M
2 0 9903028 7826248 288380 4657556 0 0 51 382 6676 6921 7 2 91 0 0
zzz ***Thu Jun 4 17:01:46 CST 2020
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
6 0 9903028 19882220 288380 4648484 0 0 54 16 0 0 2 1 97 0 0<<<<<19449M
0 0 9903028 19915868 288380 4646208 0 0 19 18 6634 6836 2 3 94 0 0
0 0 9903028 19915364 288388 4646112 0 0 51 658 5707 6528 1 1 98 0 0

[root@anbob01 proc]# free -g
             total       used       free     shared    buffers     cached
Mem:            31         12         18          0          0          4
-/+ buffers/cache:          7         23
Swap:           15          9          6
[root@anbob01 proc]# free -g
             total       used       free     shared    buffers     cached
Mem:            31         22          8          0          0          4
-/+ buffers/cache:         17         13
Swap:           15          9          6


[oracle@anbob01 ~]$ sqlplus anbob/anbob@10.129.1.1:1521/JKDB

SQL*Plus: Release 11.2.0.4.0 Production on Thu Jun 4 18:17:39 2020

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

ERROR:
ORA-01034: ORACLE not available
ORA-27102: out of memory
Linux-x86_64 Error: 12: Cannot allocate memory
Additional information: 26
Additional information: 294919
Additional information: 1744830464
Process ID: 0
Session ID: 0 Serial number: 0

[root@anbob01 proc]# numactl --hardware
available: 8 nodes (0-7)
node 0 cpus: 0 1
node 0 size: 4095 MB
node 0 free: 26 MB
node 1 cpus: 2 3
node 1 size: 4096 MB
node 1 free: 7 MB
node 2 cpus: 4 5
node 2 size: 4096 MB
node 2 free: 576 MB
node 3 cpus: 6 7
node 3 size: 4096 MB
node 3 free: 2302 MB
node 4 cpus: 8 9
node 4 size: 4096 MB
node 4 free: 1899 MB
node 5 cpus: 10 11
node 5 size: 4096 MB
node 5 free: 7 MB <<<<<<<<<<<
node 6 cpus: 12 13
node 6 size: 4096 MB
node 6 free: 7 MB  <<<<<<<<<<<<<<<
node 7 cpus: 14 15
node 7 size: 4096 MB
node 7 free: 7 MB   <<<<<<<<<<
node distances:
node   0   1   2   3   4   5   6   7 
  0:  10  20  20  20  20  20  20  20 
  1:  20  10  20  20  20  20  20  20 
  2:  20  20  10  20  20  20  20  20 
  3:  20  20  20  10  20  20  20  20 
  4:  20  20  20  20  10  20  20  20 
  5:  20  20  20  20  20  10  20  20 
  6:  20  20  20  20  20  20  10  20 
  7:  20  20  20  20  20  20  20  10 

you can check zoneinfo also.

In the case, We found that it was caused by a single process of NBU using 12G of memory.

 

Refrences
Summary of Causes for ORA-27102 on Startup (Doc ID 1080317.1)

密码保护:特殊恢复: Oracle 19c REDO和UNDO 文件被删除

$
0
0

这是一篇受密码保护的文章,您需要提供访问密码:


Oracle 11g 12c 18c 19c .. IMPDP Always Creates Indexes with Degree 1

$
0
0

Indexes seem always created with parallel degree 1 during import as seen from a sqlfile. Even if the specified parallelism is greater than 1. like

1,   Version 11.1.0.7 11.2.0.1

# impdp  ....  parallel=16 sqlfile=xx.sql
The sql file shows content like:
CREATE INDEX "<SCHEMA_NAME>"."<INDEX_NAME>" ON
"<SCHEMA_NAME>"."<TABLE_NAME>" ("<COLUMN_NAME>")
...
PARALLEL 1 ; <<<<<

ALTER INDEX "<SCHEMA_NAME>"."<INDEX_NAME>" PARALLEL 16;
 
The database is version 11.2.0.2 or above where Bug 8604502 has been fixed.

2, Version 11.2.0.2 to 11.2.0.4 12.1

Indexes seem always created with parallel degree 1 during import as seen from a sqlfile.

The import job with SQLFILE parameter option cannot use multiple execution streams.
It always executed with parallel 1. Hence its showing the PARALLEL 1 in generated sqlfile and this is an expected behavior.

When you run the import with parallel value and without sqlfile option, then you can see the actual index creation statement with correct parallel value in DW(worker) traces.

3. Version 12.2.0.1 and later

From 12.2, DataPump import (impdp) will always use non-parallel index creation during import overriding parameter ‘parallel’ in command line or parameter file.

Oracle Development said

“General support for parallel import of most object type, including indexes, is a 12.2 feature, which led to study of parallel creation of individual indexes. What was found was that using parallel index creation was generally slower than non-parallel. That led to a decision to backport the change to not use parallel index creation.”

This behavior is only seen for index creation during impdp.

After creating the index, DataPump impdp job will use ‘ALTER INDEX … PARALLEL n’ to set index degree to the value of ‘parallel’ parameter issued in command line or parameter file.

4,  Parallel meatdata

The speed of Data Pump import is highly affected by the level of parallelism that can be achieved. Prior to 12.2, object metadata was imported serially. However, as of 12.2, parallelism is allowed for object metadata import of some objects, but only for specific situations. Parallel import of metadata is disallowed under the following circumstances:

When the TRANSPORT_TABLESPACES parameter is used
If the PARALLEL parameter is specified in conjunction with TRANSPORT_TABLESPACES, an ORA-39047 error results
When the TRANSPORT_DATAFILES parameter is used
If the PARALLEL parameter is specified in conjunction with TRANSPORT_DATAFILES, an ORA-39047 error results
When the TRANSPORTABLE parameter is used
Setting TRANSPORTABLE=ALWAYS and FULL=Y allows the PARALLEL parameter to be set in the same command line without error. However, there is some confusion here. This does not allow parallel import of object metadata. It only specifies parallel import of row data due the FULL=Y setting.
When the NETWORK_LINK parameter is used
A network mode job simultaneously exports from one database while importing into another, and code for the ordering and dependency work for metadata is not yet implemented for this case
When TRIGGERs, VIEWs, OBJECT_GRANTS, SEQUENCEs, CONSTRAINTs and REF_CONSTRAINTs are imported with parallel>1
Bug 27577866, Bug 28451919 and Bug 28539085 uncovered problems with importing these objects in parallel. The fix of first two bugs were superseded by the fix of Bug 28539085. The bug 28539085 was superseded by Bug 29613245
With Patch 29613245 in place, DataPump allows the following objects to be imported in parallel: SEQUENCE, VIEW, TRIGGER, OBJECT_GRANT, CONSTRAINT and REF_CONSTRAINT.

Stated conversely, we can only achieve parallelism when importing object metadata for situations when we are importing from a dump file and when we are not performing a transportable tablespace operation.

Implementing object metadata parallelism for TTS and NETWORK_LINK is being considered for future releases, but there are no firm plans for this as yet.

 

For a quick solution I’ve decided to perform import in two steps. Firstly I will import everything but indexes and in next step I will import just indexes.

A case of my previous migration ORACLE  db.  https://www.anbob.com/archives/1888.html

References

12.2 DataPump Import (IMPDP) Creates Index With Parallel=1 Even If Parallel>1 Is Set In Command Line or Parameter File (Doc ID 2296686.1)

Oracle 12c 19c Automatic terminal/kill session feature

$
0
0

数据库会话同样会占用数据库资源,如客户端异常断开在客户端成为一个dead session会永远存在,如果客户端没有断开也没有活动就是一个idle session, 如果这个idle session做了一些修改未提交,然后下班或去吃饭、上WC、开会等,这时就会堵塞其他人对相同的数据做修改,这类会话可以叫做idle blocker session. 在自治数据库的时代这些session 都可以被释放或者kill / teminal 终结掉,下面对不同的session如何被释放

A, 对于dead session

Dead connection detection(DCD)是Oracle网络功能,主要用于客户端关闭其系统电源或客户端计算机意外崩溃而又无法正常关闭Oracle数据库连接的环境。

如果客户端计算机因正确断开会话而突然崩溃,则这些会话锁定的资源将继续被锁定,从而导致环境性能问题。由于这种情况可以在任何环境中发生,因此Oracle网络服务提供了DCD的功能,可以在早期检测到这种情况,从而可以快速恢复锁定的资源。

为了解决这种情况并检测死连接,Oracle在网络会话(12c中的tcp)层引入了一个新概念。服务器进程将SQL * Net Probe数据包发送到客户端,以检查在sqlnet.expire_time参数指定的每个固定时间间隔内连接是否仍然可用。如果通过探测数据包的通信失败,则会返回错误,导致服务器进程退出。

如何启用DCD

要在您的环境中启用DCD。在sqlnet.ora文件中设置SQLNET.EXPIRE_TIME参数,然后重新启动侦听器或重新装入侦听器。

其中n是时间间隔(以分钟为单位),在此间隔之后发送DCD的探测数据包。

1. For pre-9.2.0.4 oracle clients you can actually see the null packet of 10 bytes been received every x minutes after this time has elapsed.

2. Oracle Clients version > 9.2.0.4 the DCD packet is sent to the client from the respective database server. The null packets are actually written to the socket , but they are read only when the client becomes active.

3. versions 12.1 and newer The new method of DCD uses the TCP KEEPALIVE socket option.

There are 3 parameters associated with an operating system’s TCP keepalive (These are Linux parameters, but other operating system have similarly named parameters)

TCP_KEEPALIVE_TIME (the amount of time until the first keepalive packet is sent)
TCP_KEEPCNT(the number of probes to send)
TCP_KEEPINTVL (the interval between keepalive packets)

The sqlnet.ora parameter SQLNET.EXPIRE_TIME now sets the socket option TCP_KEEPALIVE.

If sqlnet.expire_time=1 then TCP_KEEPALIVE will be set to 60 seconds.  The other parameters KEEPINTVL and KEEPCNT get set to 6 and 10 respectively (which is very reasonable).  This means that, at minimum, DCD can only be set to 2 minutes.

B. 对于 idel session

MAX_IDLE_TIME指定会话可以空闲的最大分钟数。 此后会话将自动终止。12.2 引入的新参数。这参数是整个db或CDB级,不可以alter session.  单位分钟,超过idle上限时间后会收到 ORA-03113错误。
SQL> @i

USERNAME             INST_NAME            HOST_NAME                  I# SID   SERIAL#  VERSION    STARTED  SPID       OPID  CPID            SADDR            PADDR
-------------------- -------------------- ------------------------- --- ----- -------- ---------- -------- ---------- ----- --------------- ---------------- ----------------
SYS                  PDB1-anbob19c        oel7db1                     1 390   34170    19.0.0.0.0 20200611 3874       33    3783            000000006AC81028 000000006B9069A8

SQL> alter system set max_idle_time=1;
SQL> @pd max_id
Show all parameters and session values from x$ksppi/x$ksppcv...

       NUM N_HEX NAME                                                     VALUE                          DESCRIPTION
---------- ----- -------------------------------------------------------- ------------------------------ ---------------------------------------------------------------------------------------------------
       480   1E0 max_idle_time                                            1                              maximum session idle time in minutes
       481   1E1 max_idle_blocker_time                                    0                              maximum idle time for a blocking session in minutes


SQL> set time on
19:46:51 SQL>
19:46:51 SQL> select * from dual;
select * from dual
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Process ID: 4298
Session ID: 449 Serial number: 3375

19:49:37 SQL>

C.  对于idle blocker session

如果一个idle session阻塞了其他会话,在19c 20c中可以使用参数max_idle_blocker_time 自动终止blocker, 当会话持有其他会话所需的资源时,该会话被视为阻塞会话. 如

1. 该会话持有另一个会话所需的锁。
2。 该会话是并行操作,并且其使用者组,PDB或数据库已达到其最大并行服务器限制或已排队的并行操作。
3.会话的PDB或数据库实例即将达到其SESSIONS或PROCESSES限制。
此参数与MAX_IDLE_TIME参数的不同之处在于,MAX_IDLE_TIME适用于所有会话(阻塞和非阻塞),而MAX_IDLE_BLOCKING_TIME仅适用于阻塞会话。  因此,为了使MAX_IDLE_BLOCKING_TIME有效,其限制必须小于MAX_IDLE_TIME限制。

SQL> @pd max_id
Show all parameters and session values from x$ksppi/x$ksppcv...

       NUM N_HEX NAME                                                     VALUE                          DESCRIPTION
---------- ----- -------------------------------------------------------- ------------------------------ ---------------------------------------------------------------------------------------------------
       480   1E0 max_idle_time                                            0                              maximum session idle time in minutes
       481   1E1 max_idle_blocker_time                                    1                              maximum idle time for a blocking session in minutes

-- connect new session1
USERNAME             INST_NAME            HOST_NAME                  I# SID   SERIAL#  VERSION    STARTED  SPID       OPID  CPID            SADDR            PADDR
-------------------- -------------------- ------------------------- --- ----- -------- ---------- -------- ---------- ----- --------------- ---------------- ----------------
ANBOB                PDB1-anbob19c        oel7db1                     1 456   25738    19.0.0.0.0 20200611 5466       52    5464            000000006AD259F8 000000006B91FFC8


SQL> set sqlp session1
session1set sqlp session1>
session1>
session1>create table t1(id int);
Table created.

session1>insert into t1 values(1);
1 row created.

session1>commit;
Commit complete.

session1>update t1 set id=10 where id=1;
1 row updated.

session1>

-- connect new session2
USERNAME             INST_NAME            HOST_NAME                  I# SID   SERIAL#  VERSION    STARTED  SPID       OPID  CPID            SADDR            PADDR
-------------------- -------------------- ------------------------- --- ----- -------- ---------- -------- ---------- ----- --------------- ---------------- ----------------
ANBOB                PDB1-anbob19c        oel7db1                     1 465   44769    19.0.0.0.0 20200611 5605       54    5603            000000006AD3C120 000000006B922A88


SQL> set sqlp session2>
session2>delete t where id=1;

0 rows deleted.

session2>delete t1 where id=1;
-- hang 

-- session 3
SQL> select * from dba_blockers;

HOLDING_SESSION     CON_ID
--------------- ----------
            456          3

SQL> select * from dba_waiters;

WAITING_SESSION WAITING_CON_ID HOLDING_SESSION HOLDING_CON_ID LOCK_TYPE                  MODE_HELD                                MODE_REQUESTED                             LOCK_ID1   LOCK_ID2
--------------- -------------- --------------- -------------- -------------------------- ---------------------------------------- ---------------------------------------- ---------- ----------
            465              3             456              3 Transaction                Exclusive                                Exclusive                                    589824        893


-- One minute later

# session 2 feedback delete complated

session2>delete t1 where id=1;
1 row deleted.

# session 1
session1>
session1>select * from t1;
select * from t1
       *
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Process ID: 5466
Session ID: 456 Serial number: 25738

当然可以使用resource manager 精准指定组或用户群配置idle session或idle blocker session.

— over

升级Oracle 19c经验: TTS 在使用datapump导matadata时EXCLUDE=STATISTICS 不启作用

$
0
0

在使用TTS迁移一个8T 的11.2.04到19c PDB时,发现expdp和imdp元数据过程中statistic占用了约1个多小时的时间, 但是我检查了实时步骤中确实有写exclude=statistics,猜测是参数没启作用,不能忍,需要确认下原因。

结果发现这是一个预期行为,因为在TTS metadata时statistics不是exclude的有效值。 这样即使在expdp metadata时或impdp时指定了exclude=statistics,同样在过程中看到statistics环节依旧花费大量的时间。

You can check this by running the following query as SYS:

select object_path, comments from transportable_export_objects where object_path like 'TRANSPORTABLE_EXPORT%STATIS%';

(传输表空间)TTS 时正确的在exclud statistics的语法时:
For tables: EXCLUDE=TABLE_STATISTICS
For Indexes: EXCLUDE=INDEX_STATISTICS
For tables and indexes: EXCLUDE=TABLE_STATISTICS, INDEX_STATISTICS

Oracle Internal BLOCK CHANGE TRACKING

$
0
0

Oracle RMAN was able to take incremental backups already in 9i. However, prior to introduction of Oracle 10g block change
tracking (BCT), RMAN had to scan the whole datafile to and filter out the blocks that were not changed since base incremental
backup and overhead or incremental backup was as high as full backup. Oracle 10g new feature, block change tracking,
minimizes number of blocks RMAN needs to read to a strict minimum. With block change tracking enabled RMAN accesses
on disk only blocks that were changed since the latest base incremental backup.

METHODS AND TOOLS USED

X$ tables provide wealth of information about Oracle internals and block change tracking in particular. The tables starting
with X$KRC provide easy access to block change tracking file. Those tables are not simply exposing memory regions. When
selecting from X$KRC tables, block change tracking file is accessed behind the scenes.
Extended SQL tracing using event 10046 was used extensively on foreground sessions as well as background processes such as
DBWx, LGWR, CKPT, and CTWR. It’s worth to mention here that DBWR doesn’t post “db file parallel write” wait event with
Async IO enabled (Oracle 10.2.0.2 on Linux 32 bit). Here is where strace utility comes handy and let researcher see all system
calls. In addition to strace, lsof utility can show open file descriptors for a given process.
To match content of X$KRC tables to the content of block change tracking file, hex dump or hex editor such as bvi is handy.
There are two events reserved for block change tracking tracing – 19774 and 19775. ORADEBUG utility is also very helpful.

Without BCT enabled or in cases when change tracking information cannot be used, RMAN has to read every block in the
datafile during incremental backup. Each block contains last system change number (SCN) of its last modification. The block is
copied only if its SCN is higher or equal to the base backup SCN. To be precise, the backup SCN is a checkpoint SCN that was made right before the backup. Since, usually, only handful of blocks is changed between incremental backups, RMAN  does a lot of useless work reading the blocks not required for backup. Block change tracking provides a way to identify the blocks required for backup without scanning the whole datafile. After that RMAN need only read blocks that are really required for this incremental backup. However, improvement in incremental backup requires some sacrifice during normal database operations. According to Oracle this performance overhead is supposed to be minimal. Nevertheless, by default change tracking is disabled. Oracle 10g introduces a new special background process – Check Tracking Writer (CTWR). This process takes care of logging information about changed blocks in block change tracking file. Lets start with a closer look at this file.

BLOCK CHANGE TRACKING FILE

BCT file is one per database. In case of RAC database, change tracking file is shared amongst all instances. Thus, BCT file
must be on shared storage. By default, BCT file is created in the location defined by parameter DB_CREATE_FILE_DEST
as Oracle managed file (OMF). If parameter is not defined then location is platform specific and, usually, somewhere within
ORACLE_HOME.
You can enable change tracking with the following statement:

SQL> ALTER DATABASE ENABLE BLOCK CHANGE TRACKING;

Alternatively, you can specify location of block change tracking file:

SQL> ALTER DATABASE ENABLE BLOCK CHANGE TRACKING USING FILE '/DB1/bct.ora';

To disable:

SQL> ALTER DATABASE DISABLE BLOCK CHANGE TRACKING;

View V$BLOCK_CHANGE_TRACKING can be queried to find out the status of change tracking in the database.

BCT FILE STRUCTURE

Block change tracking file consists of blocks. Each block has usual Oracle block format except that some of the bytes are not
used, for example, block SCN. Block format code for BCT file is 0x22 in hex which is the second byte of every block. The
first byte is the block type.
Below are the block types of BCT file with respective X$ views providing interface to access information in those blocks:
• 0x29 – BCT file header (X$KRCFH)
• 0x2B – unknown (X$KRCCDE)
• 0x2C – unknown (X$KRCCDS)
• 0x2F – datafile descriptors extent header
• 0x30 – datafile descriptor block (X$KRCFDE)
• 0x33 – bitmap extent header (X$KRCFBH)
• 0x34 – bitmap block (X$KRCBIT)
• 0x36 – probably, empty filler block

The block change tracking file block size is controlled by hidden parameter “_bct_file_block_size”. On Linux platform the BCT block size is by default 512 bytes. The default might be different on other platforms. Block size is stored in BCT file header as we will see later. Figure 1 depicts the sample structure of a block change tracking file and sequence of the blocks. This example is from Linux x86 32 bit platform.Note that the first block (or block zero if you wish) is often skipped just like with datafile but this might depend on the platform. There might be more than one chunk with datafile descriptors if they do not fit into one extent. The bulk of BCT file is occupied by bitmap extents. Bitmaps are the bread and butter of change tracking – bitmap blocks store bit flags for every block (every chunk to be precise) in Oracle database. This is what CTWR updates when blocks are changed and what RMAN reads to determine which blocks it needs to backup.

BCT FILE HEADER

Block type code is 0x29 Information form file header is presented via X$KRCFH table.
Most useful columns of X$KRCFH table:
• FHSWV – software version (10.2.0.2)
• FHCVN – BCT file compatibility version (10.0.0.0)
• FHDBI – DBID
• FHDBN – database name
• FHFSZ – BCT file size in blocks
• FHBSZ – block size in bytes (512)
• EXTBLKS – BCT file extent size in blocks (64 blocks = 32K)
• RSZ_NEWEXTCNT – BCT file increment size in extents ( 320 extents * 64 * 512 bytes = 10 MB)

There are many other columns that are not of so much practical interest or with less obvious purpose. For example,
RESETLOGS SCN, checkpoint SCN, number of version switches, the last allocated extent in BCT file and etc.

EXTENT MAP

The first 2176 blocks (including first unused block) have predefined layout. The rest of the BCT file is allocated in extents. On
Linux I observed extend size 32K or 64 blocks. It can be changed with hidden parameter “_bct_file_extent_size”. The first block
of an extent is extent header and the rest contain the data. View X$KRCEXT exposes extent map. Extent map only indicates
whether extent is used or not. X$KRCEXT view is very simple and there are two interesting columns:
BNO – extent header block number
USED – flag; 0 – unused and 1 – used
Note that there is not information about extent type.

DATAFILE DESCRIPTORS

Oracle allocates the whole extent for datafile descriptors. Header block has type 0x2F and other 63 blocks with type 0x30.
Information about datafile descriptors is externalized via X$KRCFDE fixed table.
One 512 byte block contains 4 datafile descriptors. Consequently, one extent with 63 useful blocks fits up to 252 descriptors.
Datafile descriptor extents are pre-allocated and formatted based on db_files init.ora parameter.
Below are the most important columns:

CTFBNO – change tracking file block number where descriptor is located
• FNO – absolute file number
• CHUNK – chunk size in database blocks (four 8K blocks for 32K chunk)
• CRESCN – datafile creation SCN
• CRETIME – datafile creation time
• CURR_LOWSCN – start SCN for the current version
• CURR_HIGHSCN – end SCN for the current version; for current version it’s set to max possible SCN – 248-1
• CURR_FIRST – header block number of the first bitmap extent for current version
• CURR_LAST – header block number of the first bitmap extent for current version
• CURR_EXTCNT – number of extents in the current version
• CURR_VERCNT – current version number
• CURR_VERTIME – current version time; time when the version started
• HIST_FIRST – header block number of the first bitmap extent for previous version
• HIST_LAST – header block number of the first bitmap extent for previous version
• HIST_EXTCNT – number of extents in the previous version
• HIST_VERCNT – current previous number
• HIST_VERTIME – previous version time
• OLDEST_LOW- start SCN for the oldest bitmap version available in BCT file

Oracle should have called the feature chunk change tracking because it keeps track of changes for the whole chunk and not
individual blocks. Chunk size seems is be default 32K but can be changed with underscore parameter “_bct_chunk_size”.
Consecutive blocks are joined in one chunk and it’s considered dirty if any of the blocks is changed. There are four 8K blocks
in one chunk, eight 4K blocks and one 32K block, for example. This approach, probably, simplifies implementation and
change tracking file sizing doesn’t depend on datafile block size. 32K seems to be chosen because this is the smallest chunk to
fit maximum Oracle block size on all platforms. It could be that on platforms with max block size less than 32, Oracle tracks
changes in smaller chunks, i.e. with higher granularity. Would be interesting to try setting chunk size to 16K and create
tablespace with 32K block size – interesting what error Oracle would throw.

BITMAP EXTENTS

Bitmap extent associated with a single datafile. One datafile has at least one bitmap extent for each bitmap version stored.
The first block is a bitmap extent header. Other 63 blocks contain bitmap. Each bit corresponds to a datafile chunk. When
chunk is not changes for particular version – the bit is 0. If any of the blocks in the chunk is changed, the bit is set to 1. So far
I mentioned bitmap version several times so it’s good place to describe it.

BITMAP VERSIONS

Oracle tracks which blocks (actually, chunks) are changed between two consecutive incremental backups of a datafile and not
only since the last backup. There are separate bitmaps covering each period between incremental backups. This timeframe and
associated bitmaps are called versions.

Every backup has checkpoint SCN associated with it. This is the SCN of the last checkpoint before the backup. Every change
tracking bitmap version has start SCN (or low SCN), which is equal to a checkpoint SCN of the previous backup. Every
version except current has end SCN (or high SCN) which is the checkpoint SCN of the next incremental backup. Current
version has end SCN set to maximum possible SCN value which is 248-1. Versions are associated with datafile and each
datafile has its own set of version even though they might be close to each other in terms of low and high SCN’s.
Every version has dedicated bitmaps and bitmap extents. We can now refine our understanding of bitmap – bits associated
with blocks that were changed between version’s low and high SCN’s are set to 1. Otherwise, they are zeroes.
Number of versions to keep is 8 by default and can be changed by hidden parameter “_bct_bitmaps_per_file”.
BITMAP EXTENT HEADER
Information from bitmap extent headers is externalized via X$KRCFBH fixed table. Some of its columns are described below:
• CTFBNO – header block number in change tracking file
• FNO – absolute file number of a datafile
• VERCNT – version number for bitmap in this extent
• VERTIME – timestamp when this version was started
• HIST_FIRST, HIST_LAST, HIST_EXTCNT, HIST_VERCNT, HIST_VERTIME – same as in X$KRCFDE
• LOWSCN – start SCN for the version
• HIGH – end SCN for the version
BITMAP BLOCK
Every 512 bytes block provides 488 bytes for bitmaps which represent 3904 bits and can, therefore, cover 3904 chunks. Recall that every chunk is 32K. One block can track changes for 122 MB of a datafile (3904 * 32K). The whole extent contains 63 bitmap blocks and covers up to 7686 MB of one datafile. Note that this is only for one version. If file is resized and doesn’t fit into existing number of extents, additional extents are allocated.
Bitmaps are externalized via X$KRCBIT fixed table. This X$ table contains row for every bit that is set to 1 and nothing for bits set to 0. This makes sense as only dirty chunks are of interests for backups. In other words, each row in X$KRCBIT represents a dirty chunk in one of the version.
X$KRCBIT has the following columns:
• CTFBNO – block number of extent header (X$KRCFBH)
• FNO – absolute file number of a datafile (X$KRCFBH)
• VERCNT – version number (X$KRCFBH)
• VERTIME – timestamp when this version was started (X$KRCFBH)
• BNO – first block number of a chunk in the datafile
• BCT – chunks size in datafile blocks (X$KRCFDE.CHUNK)
All the columns are carried over from the extent header or the datafile descriptor except the first block number of a chunk
that is inferred from bit offset.
BCT FILE SIZING
Minimal file size is about 11MB – 2176 initial blocks (1088K) + 320 extents (32K * 320 = 10 MB). 10 MB seems to be predefined increment for BCT file resizing. Out of those 320 extents, one is reserved for X$KRCCDR structure and at least one is reserved for datafile descriptors. Recall that one extent covers 252 files and they are pre-allocated based in init.ora parameter db_files. Thus, assuming that 8 versions for each datafile are kept, 318 extents can cover up to: 318 extents * 7686 MB / 8 version = 300 GB.
Note that in this case maximum number of existing datafiles can be up to: 318 extents / 8 version = 39 datafiles.

In real life, some datafiles are just few hundred megabytes and others sized in tens of megabytes so there is no simple formula
to fit all cases. Here is the complete statement that can be used to predict maximum BCT file size based in the current
database when all datafiles backed up with incremental backup at least 7 times:

SELECT((
(SELECT SUM(ceil(bytes /(7686 * 1024 * 1024))) * 8 bitmap_ext
FROM v$datafile) +
(SELECT ceil(VALUE / 252) file_descr_ext
FROM v$parameter
WHERE name = 'db_files') + 1)
* 32 + 1088) / 1024 bct_file_size_mb
FROM dual;

The statement is valid only for single instance. In RAC, every instance will use its own bitmap extents so number of bitmat extents should be multiplied by the number of RAC instance.

CHANGING BLOCKS
As I mentioned already, changes are tracked for 32K chunks of datafile so any block change in the chunk will render it dirty and bit must be set to 1. If the chunk was already dirty than nothing will change in the BCT file. Basically, if previous block SCN is more than corresponding to the datafile X$KRCFDE.CURR_LOWSCN.

DATAFILE INCREMENTAL BACKUP
When RMAN starts incremental backup of a datafile, it needs to create new version in change tracking file.
Current bitmap is marked as historical – XFLAG column of X$KRCFBH (bitmap headers) is set to 3 and HIGH column is set to the checkpoint SCN. Purge bitmaps that are older than 8 versions.
New extents are allocated and formatted. Bitmap extent header block is formatted but no version specific information is written. I.e.  querying X$KRCFBH we won’t see columns VERCNT, VERTIME, LOW, HIGH and etc. filled. This is because ofversion optimization – if blocks in datafile haven’t been changed between two incremental backup then new version is notcreated. Thus, version attributes are updated only when chunks are first marked as dirty.
Datafile descriptor is updated (X$KRCFDE) – attributes of current version and previous version are filled (CURR_% andHIST_% columns).
When some a chunk needs to be marked dirty for the first time after incremental backup, version information in bitmap extent
headers (X$KRCFBH) is initialized – XFLAGS is set to 2, CURR_VERCNT and CURR_VERTIME filled from datafile
descriptor. If file had no changes since last incremental backup – nothing is written to the bitmap.
There is no difference in handling different backup levels. Version is created in the same way whether its level 0, level 1 or
level 4 backup. By the way, it seems that Oracle 10g documentation officially mentions only support for levels 0 and 1.

PREDICT NUMBER OF BLOCKS READ AND BACKUP SIZE
Based on the bitmap content (X$KRCBIT) we can predict how many 32K chunks RMAN would read if incremental backup starts now. However, precise backup size prediction is only possible if database block size is 32K. Otherwise, a 32K chunk consists of multiple database blocks (2, 4, 8 or 16 depending on the block size). It’s quite possible that only one 8K block out of 4 in a chunk is changed. In this case RMAN will read the whole chunk but it needs to copy only one changed block. RMAN still can filter out blocks unchanged since last backup SCN based on block SCN just like it does for every block in case change tracking is not enabled.
The query below can be used to calculate how many kilobytes will be read by RMAN to backup datafile 7 based on last level 1
incremental backup (i.e. incremental level 1 or incremental cumulative level 2):

SELECT count(distinct bno) * 32
FROM x$krcbit b
WHERE b.fno = 7 AND b.vercnt >=
(SELECT MIN(ver) FROM
(SELECT curr_vercnt ver, curr_highscn high, curr_lowscn low
FROM x$krcfde WHERE fno = 7
UNION ALL
SELECT vercnt ver, high, low
FROM x$krcfbh WHERE fno = 7)
WHERE (SELECT MAX(bd.checkpoint_change#)
FROM v$backup_datafile bd
WHERE bd.file# = 7
AND bd.incremental_level <= 1) between low and high);

Running incremental backups for a while it’s possible to collect historical ration between number of blocks read and number and size of the backup. This would as well account for compression. Note that the query above is just an example and it has the following limitations:
• Chunk size is hard coded to 32K (could it vary on different platforms?)
• First block overhead is not accounted for
• No special case when required bitmap version is not available (purged) and the whole datafile must be read
• No case with backup optimization for level 0 (v$datafile_backup.used_optimization)
• No case when no data blocks in datafile is changed (no bitmap version but the first block must be backed up anyway)
• Only single datafile
• No accounting for unavailable base incremental backup

CHANGE TRACKING INSIDE ORACLE INSTANCE

There is a new special process in Oracle 10g – Change Tracking WRiter (CTWR). As name suggests the job of this process is to write to the block change tracking file. In normal Oracle operations there is no other process that is writing to it. RMAN shadow process reads from as well as writes to the change tracking file during incremental backups. Any process in Oracle can read from change tracking file. When user session queries X$KRC% views, it access BCT file behind the scenes. Shadow process itself accesses CTWR file without communications to CTWR process.
WHO DOES WHAT WHEN BLOCKS ARE CHANGED
Let’s see what happens when processes change blocks.
Every change to the datafile blocks generates redo entry. The process first produces change vector and than this change vector is applied to the block. The change is not applied directly to the block on disk except for direct path writes. The block is first loaded into buffer cache in SGA and then change vector is applied changing the block content. The block in the buffer cache is now considered dirty and later one of the Database WRiter processes (DBWx) will flash it to disk.
Neither of these processes is writing to the change tracking file to mark chunks in the bitmaps as dirty. This is the job of Checkpointer process (CTWR) but how does it know which blocks are changed? The process that updates the block in buffer cache is responsible for passing information to the CTWR about which blocks were changed. At the time the process puts redo entry in the log buffer, it also updates special buffer in the SGA that is used by change tracing writer process later to update the bitmaps. The mechanism of asynchronous updates to the BCT file is somewhat similar to log buffer but not entirely the same. It also doesn’t require flushing pending changes to the BCT file on commits so foreground processes won’t wait on something similar to “log file sync”.
When does CTWR update the BCT file? This happens usually during checkpoint. CKPT process signals to CTWR process to flush pending changes to the BCT file. CKPT is waiting for confirmation that CTWR received the instruction. However, CTWR doesn’t update bitmaps to the BCT file immediately after the signal from CKPT. Instead, it posts back to the CKPT.
As soon as CKPT gets the signal from CTWR, it updates datafile headers and controlfile so checkpoint is completed. CTWR in the meantime proceeds with updating bitmaps in the block change tracking file.
What happens with direct path writes when blocks are updated by shadow processes and written directly on disk? The process is similar. Direct path writes still generate some redo – block invalidations. When these redo entries are written to the log buffer, information is written to the change tracking buffer in SGA and CTWR process will update the bitmaps in BCT file later during the next “CTWR heartbeat” or checkpoint.
WHO DOES WHAT DURING RMAN INCREMENTAL BACKUP
When incremental datafile backup is started, RMAN shadow process in Oracle instance reads from the BCT file and then updates the BCT file header as well the block 2176 exposed via X$KRCCDR fixed table. The next step for RMAN is to signal to the CTWR process that new version needs to be created. At this point RMAN waits for CTWR to finish its business. CTWR creates the new version including allocation of new extents and purging old bitmaps as needed. After that, CTWR posts back to the RMAN shadow process. RMAN shadow process can now read all required past bitmaps and identify which chunks are required for backup. Finally, RMAN gets to its main part – reading dirty chunks and writing changed blocks to the backup piece.

CTWR WAIT EVENTS
CTWR process performs updates to the block 18 of the BCT file about every 20-60 seconds – I call it “CTWR heartbeat”.
During this heartbeat CTWR reads and writes block 18 posting sequentially two short wait events – “change tracking file synchronous read” and “change tracking file synchronous write”. When updating the BCT file, CTWR process also performs some IO against controlfile (“control file sequential read” and “control file parallel write”). Reads and writes to the BCT file itself are instrumented with “change tracking file synchronous read” and “change tracking file synchronous write”. However, updates of the bitmap blocks are NOT instrumented.

Several new enqueues related to block change tracking mechanism name like ‘enq: CT%’
HIDDEN PARAMETERS
• _bct_public_dba_buffer_size – total size of all public change tracking dba buffers, in
• _bct_initial_private_dba_buffer_size – initial number of entries in the private change tracking dba buffers

_bct_bitmaps_per_file – number of bitmaps to store for each datafile
• _bct_file_block_size – block size of change tracking file, in bytes
• _bct_file_extent_size – extent size of change tracking file, in bytes
• _bct_chunk_size – change tracking datafile chunk size, in bytes
• _bct_crash_reserve_size – change tracking reserved crash recovery SGA space, in bytes
• _bct_buffer_allocation_size – size of one change tracking buffer allocation, in bytes
• _bct_buffer_allocation_max – maximum size of all change tracking buffer allocations, in bytes
• _bct_buffer_allocation_min_extents – mininum number of extents to allocate per buffer allocation
• _bct_fixtab_file – change tracking file for fixed tables

There is a hidden parameter _bct_bitmaps_per_file set to 8 by default but no officially supported way to change number of versions.

BCT FILE corruption

if the BCT file gets corrupted due to whatever reason (bug, media corruption, human factor), it can be easily recreated online by disabling and re-enabling block change tracking. Again, don’t forget that all versioning information is lost in this case and next incremental backup will have to scan all blocks.

 

I’m going to start with a simple example.
First I’m going to turn on block change tracking, and then check the File# for my database files.

SQL> alter database enable block change tracking using file '/home/oracle/app/oracle/oradata/BSG18/bct.dbf';

SQL>
TABLESPACE_NAME                   FILE_ID
------------------------------ ----------
USERS                                   7
UNDOTBS1                                4
SYSTEM                                  1
SYSAUX                                  3


I am going to use 2 internal structures to walk through the BCT information. X$KRCFH - This structure contains the lowscn of the BCT file.  It is used by RMAN to determine                       the beginning SCN of the BCT file.  If no changes have been captured for a datafile,                       then RMAN knows at which point it can make that assumption. X$KRCFBH - This structure contains the versions of block changes available to use for backups.                          This structure contains a row for each datafile, and each version.                          This row points to a bitmap containing the block changes associated with the version.                           The key columns I am interested in to explain how it works are                                 FNO          - File number                                 VERCNT - Version number identifying the change records.                                                     This starts at 1 and increases over the life of the BCT file.                                  LOW        - Low SCN of the block changes. It is 0 for the first record                                                      indicating that it not complete                                  HIGH        - High SCN of the block changes. The last SCN number                                                       identifying the block changes. I am going to look the internal structure X$KRCFBH to view what's in the block change tracking.
select (select tablespace_name from dba_data_files where file_id=fno) tablespace_name,vercnt,to_char(vertime,'mm/dd/yy hh24:mi:ss') vertime,low, high from x$krcfbh
   where fno in (select file_id from dba_data_files);
SQL> SQL>   2
TABLESPACE     VERCNT VERTIME                  LOW       HIGH
---------- ---------- ----------------- ---------- ----------
SYSTEM              1 07/29/19 09:21:17          0          0
SYSAUX              1 07/29/19 09:21:18          0          0
UNDOTBS1            1 07/29/19 09:21:18          0          0
USERS               1 07/29/19 09:21:18          0          0

 I can see that there are entries for all my tablespaces, with a "version count" of 1 and low/high time of 0.  This is telling me that that no backups have been executed yet usingthe BCT. I am also going to look at X$KRCFH to see what the starting SCN is for the BCT file.
select lowscn from X$KRCFH;

LOWSCN
----------
8361930
Now I am going to execute a full backup and see what happens.

TABLESPACE     VERCNT VERTIME                  LOW       HIGH
---------- ---------- ----------------- ---------- ----------
SYSTEM              1 07/29/19 09:21:17          0    8362001
SYSAUX              1 07/29/19 09:21:18          0    8362001
UNDOTBS1            1 07/29/19 09:21:18          0    8362001
USERS               1 07/29/19 09:21:18          0    8362001


You can see that the high SCN set to 8362001.
Now I am going to look at the RMAN backup (USERS tablespace) to see what the Checkpoint SCN number was for the backup.

  File LV Type Ckp SCN    Ckp Time          Abs Fuz SCN Sparse Name
  ---- -- ---- ---------- ----------------- ----------- ------ ----
  7    0  Incr 8362002    07/29/19 09:46:59              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_users_fz01zl49_.dbf

Ahhh.. Now I can see how it fits together. The high SCN for the BCT file is SCN number right before the checkpoint taken with the backup. Now let's execute an incremental backup and see what happens.
TABLESPACE     VERCNT VERTIME                  LOW       HIGH
---------- ---------- ----------------- ---------- ----------
SYSAUX              1 07/29/19 09:21:18          0    8362001
SYSTEM              1 07/29/19 09:21:17          0    8362001
UNDOTBS1            1 07/29/19 09:21:18          0    8362001
USERS               1 07/29/19 09:21:18          0    8362001
SYSAUX              2 07/29/19 09:46:59    8362001    8363961
SYSTEM              2 07/29/19 09:46:59    8362001    8363961
UNDOTBS1            2 07/29/19 09:46:59    8362001    8363961

1    0  Incr 8362002    07/29/19 09:46:59              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_system_fz01xnqo_.dbf
1    1  Incr 8363962    07/29/19 10:04:00              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_system_fz01xnqo_.dbf
  
 By looking at the SYSTEM tablespace (file 1) I can see exactly what is happening with the BCT file. The first version marks the starting SCN prior to the first backup after creating the file. The second version marks the checkpoint SCN of the first backup (low), and the SCN prior to the second backup(high). These marks, LOW/HIGH SCN, can be used to identify the blocks that changed between the backups. Now I am going to perform a few more incremental backups with a few changes to the USERS tablespace and see what happens to the SYSTEM and USERS tablespaces versions. First here is my query. col tablespace_name format a10 select (select tablespace_name from dba_data_files where file_id=fno) tablespace_name,vercnt,to_char(vertime,'mm/dd/yy hh24:mi:ss') vertime,low, high from x$krcfbh    where fno in (select file_id from dba_data_files)     order by 1,2;
Now let’s see what what my backups look for File 1 (SYSTEM tablespace).
List of Backup Sets
===================
  File LV Type Ckp SCN    Ckp Time          Abs Fuz SCN Sparse Name
  ---- -- ---- ---------- ----------------- ----------- ------ ----
  1    0  Incr 8362002    07/29/19 09:46:59              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_system_fz01xnqo_.dbf
  1    1  Incr 8363962    07/29/19 10:04:00              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_system_fz01xnqo_.dbf
  1    1  Incr 8364371    07/29/19 10:13:21              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_system_fz01xnqo_.dbf
  1    1  Incr 8364482    07/29/19 10:15:19              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_system_fz01xnqo_.dbf
  1    1  Incr 8364637    07/29/19 10:16:38              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_system_fz01xnqo_.dbf
  1    1  Incr 8365075    07/29/19 10:26:23              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_system_fz01xnqo_.dbf
  1    1  Incr 8365373    07/29/19 10:29:13              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_system_fz01xnqo_.dbf
  1    1  Incr 8365578    07/29/19 10:30:21              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_system_fz01xnqo_.dbf
  1    1  Incr 8365763    07/29/19 10:30:58              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_system_fz01xnqo_.dbf

Let’s see what my backups look like for File 7 (USERS tablespace)

List of Backup Sets
===================
  File LV Type Ckp SCN    Ckp Time          Abs Fuz SCN Sparse Name
  ---- -- ---- ---------- ----------------- ----------- ------ ----
    7    0  Incr 8362002    07/29/19 09:46:59              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_users_fz01zl49_.dbf
    7    1  Incr 8363962    07/29/19 10:04:00              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_users_fz01zl49_.dbf
    7    1  Incr 8364371    07/29/19 10:13:21              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_users_fz01zl49_.dbf
    7    1  Incr 8364482    07/29/19 10:15:19              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_users_fz01zl49_.dbf
    7    1  Incr 8364637    07/29/19 10:16:38              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_users_fz01zl49_.dbf
    7    1  Incr 8365075    07/29/19 10:26:23              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_users_fz01zl49_.dbf
    7    1  Incr 8365373    07/29/19 10:29:13              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_users_fz01zl49_.dbf
    7    1  Incr 8365578    07/29/19 10:30:21              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_users_fz01zl49_.dbf
    7    1  Incr 8365763    07/29/19 10:30:58              NO    /home/oracle/app/oracle/oradata/BSG/datafile/o1_mf_users_fz01zl49_.dbf

 My backups for both of these match. They have the same Ckp SCN. Notice that I performed an Incremental level 0 backup, and then 8 Incremental level 1 backups. I actually performed both differential backups and cumulative backups but it didn't matter. Now let's look at the block change tracking file for these 2 tablespaces (system and users)
TABLESPACE     VERCNT VERTIME                  LOW       HIGH
---------- ---------- ----------------- ---------- ----------
SYSTEM              3 07/29/19 10:04:00    8363961    8364370
SYSTEM              4 07/29/19 10:13:21    8364370    8364481
SYSTEM              5 07/29/19 10:15:19    8364481    8364636
SYSTEM              6 07/29/19 10:16:38    8364636    8365074
SYSTEM              7 07/29/19 10:26:23    8365074    8365372
SYSTEM              8 07/29/19 10:29:13    8365372    8365577
SYSTEM              9 07/29/19 10:30:20    8365577    8365762
USERS               1 07/29/19 09:21:18          0    8362001
USERS               2 07/29/19 09:46:59    8362001    8364481
USERS               3 07/29/19 10:15:19    8364481    8364636

Very interesting.. Since I only made few changes to the users tablespace it has 3 versions, the oldest of which is the full backup.
The system tablespace has gone over 7 versions and it no longer has the original version from the level 0 backup.
Now let’s see if it used the BCT files for the backups.

     FILE# Creation Time     INCREMENTAL_LEVEL INCREMENTAL_CHANGE# CHECKPOINT_CHANGE# USED BCT
---------- ----------------- ----------------- ------------------- ------------------ ---
         1 07/29/19 09:47:24                 0                   0            8362002 YES
         1 07/29/19 10:04:02                 1             8362002            8363962 YES
         1 07/29/19 10:13:21                 1             8363962            8364371 YES
         1 07/29/19 10:15:19                 1             8364371            8364482 YES
         1 07/29/19 10:16:38                 1             8364482            8364637 YES
         1 07/29/19 10:26:24                 1             8364637            8365075 YES
         1 07/29/19 10:29:14                 1             8362002            8365373 YES
         1 07/29/19 10:30:22                 1             8362002            8365578 YES
         1 07/29/19 10:31:10                 1             8362002            8365763 NO

         7 07/29/19 09:47:00                 0                   0            8362002 YES
         7 07/29/19 10:04:02                 1             8362002            8363962 YES
         7 07/29/19 10:13:21                 1             8364300            8364371 YES
         7 07/29/19 10:15:19                 1             8364371            8364482 YES
         7 07/29/19 10:16:38                 1             8364482            8364637 YES
         7 07/29/19 10:26:24                 1             8364637            8365075 YES
         7 07/29/19 10:29:14                 1             8362002            8365373 YES
         7 07/29/19 10:30:22                 1             8362002            8365578 YES
         7 07/29/19 10:30:59                 1             8362002            8365763 YES

WOW…  Notice that the system tablespace (FILE #1) could not use the BCT file for the last backup, but the backup of the user tablespace (FILE #7) could because there were no changes between a few of the backups.

I also noticed the creation time of the RMAN backup, and the creation time of the BCT file record.  The new BCT file record (if changes occured) is created BEFORE the backup begins.

Finally, I can also see that the high SCN for users, 8364636,is older than the high SCN for system.

I am going to change users, and perform an incremental to see what happens to the LOW/HIGH scn in the next version.

TABLESPACE     VERCNT VERTIME                  LOW       HIGH
---------- ---------- ----------------- ---------- ----------
USERS               1 07/29/19 09:21:18          0    8362001
USERS               2 07/29/19 09:46:59    8362001    8364481
USERS               3 07/29/19 10:15:19    8364481    8364636
USERS               4 07/29/19 10:16:38    8364636    8366975

After the backup, there is no gap in the SCN numbers.  It created a new version that contained the changes between the previous version HIGH and the SCN of the incremental backup.

So what have we learned about the BCT ?

  1. The BCT has a structure that contains the starting SCN of changes captured.
  2. The BCT file creates version records associated with block changes before the backup begins.
  3. If no BCT changes are captured for a datafile, it is assumed that no changes occured since the starting SCN of the BCT file.
  4. The BCT file keeps a bit map of changes between 2 SCN numbers
  5. The BCT file keeps changes on a datafile level.
  6. If a datafile didn’t change between backups, it doesn’t create a new record in the BCT file.  It doesn’t matter if the backup is an incremental or full backup.
  7. By default 7 versions are kept for EACH DATAFILE. After 7 days of backups, some datafiles may still use the BCT if they haven’t changed.

References

Alex Gorbachev wrote a great paper and MOS note explaining it all.  ORACLE 10G BLOCK CHANGE TRACKING INSIDE OUT (Doc ID 1528510.1)

 

How to disable DRM? Oracle 10G, 11G, 12C-19C

$
0
0

DRM is driven by the following
1.) _gc_affinity_time = Time in minutes at which statistics will be evaluated (default = 10 mins)
2.) _gc_affinity_limit = # of times a node accesses a file/object (default = 50)
3.) _gc_affinity_minimum = minimum # of times per minute a file/object is accessed before affinity kicks in
(default = 600 per minute per cpu )
4.) _lm_drm_disable = LEVEL disable drm in different level( default=0 )
Level 4 disable read mostly
Level 5 disable DRM for all but undo
Level 7 disable DRM for all including undo.

Disable DRM in 12C and onward

Shutdown all nodes after setting _lm_drm_disable=7, and then startup them. To disable DRM, instead of _gc_policy_time, _lm_drm_disable should be used as it’s dynamic(had bug).

alter system set “_lm_drm_disable=7” scope=spfile sid=’*';

Disable DRM in 11g Release 1 and onward

alter system set “_gc_undo_affinity”=FALSE scope=spfile sid=’*';
alter system set “_gc_policy_time”=0 scope=spfile sid=’*';

Disable DRM  In 10g

alter system set “_gc_undo_affinity”=FALSE scope=spfile sid=’*';
alter system set “_gc_affinity_time”=0 scope=spfile sid=’*';

Note:
因为上面两个参数是静态,修改需要重启,通过修改下面两个动态参数也可以减少DRM 的效果:

10g

alter system set “_gc_affinity_limit”=250 sid=’*';
alter system set “_gc_affinity_minimum”=10485760 sid=’*';

11g

alter system set “_gc_policy_minimum”=10485760 sid=’*';

12c+

alter system set “_gc_policy_minimum”=10485760 sid=’*';

Note:
From 11g onwards, affinity management renamed to policy management. For example, x$object_affinity_statistics is renamed to x$object_policy_statistics. Similarly, initialization parameters are renamed too:Parameter _gc_affinity_limit is renamed to _gc_policy_limit; Parameter _gc_affinity_time is renamed to _gc_policy_time; A new view v$policy_history is introduced and all rows with policy_event = ‘initiate_affinity’ are associated with DRM events.

To disable DRM completely, set _gc_policy_limit and _gc_policy_minimum to much higher value, say 10Million. Setting the parameter _gc_policy_time to 0 will completely disable DRM, but that also means that you can not manually remaster objects

Viewing all 704 articles
Browse latest View live