昨晚業務系統導入資料并重建索引時一個會話突然停滯不前，用toad一看，一直在等待library cache lock。toad、oem中都看不到此鎖，會話每三秒啟動一次，但每次都是等待這個鎖。顯然，這和數據字典有關，應該是一個索引的數據字典中的記錄被鎖住了，導致無法重建。可是殺光了其他active的會話，問題仍然沒有得到解決，看來是某一個被殺死的會話持有該鎖，而會話尚未回滾完全，進程仍然吊死著。現在的問題就是找這個會話了。
首先想到的文檔就是oracle9i database reference了，找到附錄a，說明如下：
ｔhis event controls the concurrency between clients of the library cache. it acquires a lock on the object handle so that either:
one client can prevent other clients from accessing the same object
the client can maintain a dependency for a long time (for example, no other client can change the object)
this lock is also obtained to locate an object in the library cache.
wait time: 3 seconds (1 second for pmon)
parameters:
handle address
address of the object being loaded
lock address
address of the load lock being used. this is not the same thing as a latch or an enqueue, it is a state object.
mode
indicates the data pieces of the object which need to be loaded
namespace
see "namespace"
幾乎等于什么都沒說，不過lock address應該會有點用處。
轉而上網搜索解決方案，終于找到一篇metalink上的文檔：
doc id:
note:122793.1
subject: how to find the session holding a library cache lock
type: bulletin
status: published
content type: text/plain
creation date: 23-oct-2000
last revision date: 17-jul-2002
purpose
-------
in some situations it may happen your session is 'hanging' and is awaiting for
a 'library cache lock'. this document describes how to find the session that
in fact has the lock you are waiting for.
scope & application
-------------------
support analysts, dba's, ..
how to find the session holding a a library cache lock
------------------------------------------------------
common situations:
* a dml operation that is hanging because the table which is accessed is currently
undergoing changes (alter table). this may take quite a long time depending on
the size of the table and the type of the modification
(e.g. alter table x modify (col1 char(200) on thousands of records).
* the compilation of package will hang on library cache lock and library cache pin
if some users are executing any procedure/function defined in the same package.
in the first situation the v$lock view will show that the session doing the
'alter table' has an exclusive dml enqueue lock on the table object (lmode=6,
type=tm and id1 is the object_id of the table). the waiting session however does
not show up in v$lock yet so in an environment with a lot of concurrent sessions
the v$lock information is insufficient to track down the culprit blocking your
operation.
method 1: systemstate analysis
------------------------------
one way of finding the session blocking you is to analyze the system state dump.
using the systemstate event one can create a tracefile containing detailed
information on every oracle process. this information includes all the resources
held & requested by a specific process.
whilst an operation is hanging, open a new session and launch the following
statement:
alter session set events 'immediate trace name systemstate level 8';
oracle will now create a systemstate tracefile in your user_dump_dest directory.
get the pid (processid) of the 'hanging' session from the v$process by matching
paddr from v$session with addr from v$process:
select pid from v$process where addr=
(select paddr from v$session where sid=sid_of_hanging_session);
the systemstate dump contains a separate section with information for each
process. open the tracefile and do a search for 'process pid_from_select_stmt'.
in the process section look up the wait event by doing a search on 'waiting for'.
example output:
process 8:
----------------------------------------
so: 50050b08, type: 1, owner: 0, flag: init/-/-/0x00
(process) oracle pid=8, calls cur/top: 5007bf6c/5007bf6c, flag: (0) -
int error: 0, call error: 0, sess error: 0, txn error 0
(post info) last post received: 82 0 4
last post received-location: kslpsr
last process to post me: 5004ff08 1 2
last post sent: 0 0 13
last post sent-location: ksasnd
last process posted by me: 5004ff08 1 2
(latch info) wait_event=0 bits=0
process group: default, pseudo proc: 50058ac4
o/s info: user: daemon, term: pts/1, ospid: 15161
osd pid info: 15161
----------------------------------------
so: 5005f294, type: 3, owner: 50050b08, flag: init/-/-/0x00
(session) trans: 0, creator: 50050b08, flag: (41) usr/- bsy/-/-/-/-/-
did: 0001-0008-00000002, short-term did: 0000-0000-00000000
txn branch: 0
oct: 6, prv: 0, user: 41/lc
o/s info: user: daemon, term: pts/1, ospid: 15160, machine: goblin.forgotten.realms
program: [email protected] (tns v1-v3)
application name: sql*plus, hash value=3669949024
waiting for 'library cache lock' blocking sess=0x0 seq=253 wait_time=0
!>> handle address=5023ef9c, lock address=5019cad4, 10*mode+namespace=15
using the 'handle address' you can look up the process that is keeping a lock
on your resource by doing a search on the address within the same tracefile.
example output:
process 9:
----------------------------------------
so: 50050e08, type: 1, owner: 0, flag: init/-/-/0x00
(process) oracle pid=9, calls cur/top: 5007bbac/5007bbfc, flag: (0) -
int error: 0, call error: 0, sess error: 0, txn error 0
<cut> ....
----------------------------------------
so: 5019d5e4, type: 34, owner: 5015f65c, flag: init/-/-/0x00
!>> library object pin: pin=5019d5e4 handle=5023ef9c mode=x lock=0
user=5005fad4 session=5005fad4 count=1 mask=0511 savepoint=118218 flags=[00]
from the output we can see that the oracle process with pid 9 has an exclusive
lock on the object we are trying to access. using v$process and v$session we can
retrieve the sid,user,terminal,program,... for this process. the actual statement
that was launched by this session is also listed in the tracefile (statements and
other library cache objects are preceded by 'name=').
method 2: examine the x$kgllk table
-----------------------------------
the x$kgllk table (accessible only as sys/internal) contains all the
library object locks (both held & requested) for all sessions and
is more complete than the v$lock view although the column names don't
always reveal their meaning.
you can examine the locks requested (and held) by the waiting session
by looking up the session address (saddr) in v$session and doing the
following select:
select * from x$kgllk where kgllkses = 'saddr_from_v$session'
this will show you all the library locks held by this session where
kglnaobj contains the first 80 characters of the name of the object.
the value in kgllkhdl corresponds with the 'handle address' of the
object in method 1.
you will see that at least one lock for the session has kgllkreq > 0
which means this is a request for a lock (thus, the session is waiting).
if we now match the kgllkhdl with the handles of other sessions in
x$kgllk that should give us the address of the blocking session since
kgllkreq=0 for this session, meaning it has the lock.
select * from x$kgllk lock_a
where kgllkreq = 0
and exists (select lock_b.kgllkhdl from x$kgllk lock_b
where kgllkses = 'saddr_from_v$session' /* blocked session */
and lock_a.kgllkhdl = lock_b.kgllkhdl
and kgllkreq > 0);
if we look a bit further we can then again match kgllkses with saddr
in v$session to find further information on the blocking session:
select sid,username,terminal,program from v$session
where saddr in
(select kgllkses from x$kgllk lock_a
where kgllkreq = 0
and exists (select lock_b.kgllkhdl from x$kgllk lock_b
where kgllkses = 'saddr_from_v$session' /* blocked session */
and lock_a.kgllkhdl = lock_b.kgllkhdl
and kgllkreq > 0)
);
in the same way we can also find all the blocked sessions:
select sid,username,terminal,program from v$session
where saddr in
(select kgllkses from x$kgllk lock_a
where kgllkreq > 0
and exists (select lock_b.kgllkhdl from x$kgllk lock_b
where kgllkses = 'saddr_from_v$session' /* blocking session */
and lock_a.kgllkhdl = lock_b.kgllkhdl
and kgllkreq = 0)
);
related documents
-----------------
[note:1020008.6] script fully decoded locking script
[note:1054939.6] compilation of package is hanging on library cache lock
.
開頭說了半天廢話，后面給出了兩個解決方案，進行事件跟蹤或者查詢x$kgllk表，還是后者容易一點。搜尋半天，終于問到sysdna登錄的方法（權限低，沒密碼）。第一次用x$表，用sysdba手還有點發抖（那可是正式的業務系統），敲入命令發現運行了很久才反應，找出一個sid了，仔細一看，居然是一個后臺維護終端，程序是toad，狀態inactive。詢問相關人員，從未alter該表，看來是toad看表結構或者數據的時候出了問題，kill掉終于正常，索引很快建完。只是inactive的會話居然也持有鎖，真是奇怪。工具太好了，也是雙刃劍啊。
