淺談library cache裡的爭用(一)

oliseh發表於2016-03-29


library cache相關的爭用都出自於語句解析階段,我們今天要討論的是其中較為常見的一個場景,那就是兩個會話同時執行同一條SQL語句時到底會發生哪些爭用,對應的等待事件是什麼,隨著語句的不斷執行,這些等待事件是如何一步步演變的。

 

先準備一下實驗環境:

###建表

---connect ad/Uiop246!

create table t1 as select * from all_users;

 

###清空Share pool

---connect as sysdba

alter system flush shared_pool;

 

###實驗用到的script

/////////////////////////////////////

////    測試sql sql_exec.sql   ////

/////////////////////////////////////

declare

v_sql varchar2(1000);

int_cur number;

v_rows number;

v_sec number;

begin

select extract(second from to_timestamp(to_char(sysdate,'yyyymmdd hh24:mi:ss'),'yyyymmdd hh24:mi:ss')) into v_sec from dual;

while ( v_sec <> 30 ) loop

select extract(second from to_timestamp(to_char(sysdate,'yyyymmdd hh24:mi:ss'),'yyyymmdd hh24:mi:ss')) into v_sec from dual;

end loop;

dbms_output.put_line('begin_time:'||sysdate);

int_cur:=dbms_sql.open_cursor;

for i in 1..50000 loop

v_sql:='select * from t1 where user_id='||i;

dbms_sql.parse(int_cur,v_sql,dbms_sql.native);

v_rows:=dbms_sql.execute(int_cur);

end loop;

dbms_output.put_line('end_time:'||sysdate);

end;

/

 

/////////////////////////////////////

////  統計用:sess_1st.sql       ////

/////////////////////////////////////

define v_sid1=&1

define v_sid2=&2

drop table sess_stat_1st;

drop table sess_evt_1st;

create table sess_stat_1st as select sn.name,se.sid,se.value from v$sesstat se,v$statname sn where se.sid in (&v_sid1,&v_sid2) and se.statistic#=sn.statistic# and sn.name like '%parse%';

create table sess_evt_1st as select sid,event,total_waits from v$session_event where sid in (&v_sid1,&v_sid2) order by sid,event;

 

/////////////////////////////////////

////  統計用:sess_2nd.sql       ////

/////////////////////////////////////

define v_sid1=&1

define v_sid2=&2

drop table sess_stat_2nd;

drop table sess_evt_2nd;

create table sess_stat_2nd as select sn.name,se.sid,se.value from v$sesstat se,v$statname sn where se.sid in (&v_sid1,&v_sid2) and se.statistic#=sn.statistic# and sn.name like '%parse%';

create table sess_evt_2nd as select sid,event,total_waits from v$session_event where sid in (&v_sid1,&v_sid2) order by sid,event;

col event format a30

set linesize 120  pagesize 60

select nd.sid,nd.event,nd.total_waits-nvl(st.total_waits,0) diff_waits from sess_evt_1st st,sess_evt_2nd nd where st.sid(+)=nd.sid and st.event(+)=nd.event order by event,sid;

select nd.sid,nd.name,nd.value-nvl(st.value,0) diff_stats from sess_stat_2nd nd,sess_stat_1st st where st.name(+)=nd.name and nd.sid=st.sid(+);

 

實驗開始時我們要執行的語句還沒有在shared pool裡,也就是要從硬解析開始!

<<<< 1、硬解析的情況:當要執行的語句不在library cache裡的時候,兩個session同時執行相同的語句時必有一個session會進行硬解析 >>>>

###記錄session 1 & session 2的會話資訊

---session 1: ad/Uiop246!

set linesize 100

select a.sid,a.serial#,b.spid from v$session a,v$process b where a.paddr=b.addr and a.audsid=sys_context('userenv','sessionid');

       SID    SERIAL# SPID

---------- ---------- ------------------------

      3307       1987 6160700

 

---session 2: ad/Uiop246!

set linesize 100

select a.sid,a.serial#,b.spid from v$session a,v$process b where a.paddr=b.addr and a.audsid=sys_context('userenv','sessionid');

       SID    SERIAL# SPID

---------- ---------- ------------------------

      6605       1185 14549288

 

###session第一輪執行該語句

---session 3:先記錄session 1 & session 2的初始值等待事件及統計值

@sess_1st.sql 3307 6605

 

---session 1:

@sql_exec.sql

begin_time:20160327 16:42:30

end_time:20160327 16:43:32

 

---session 2:

@sql_exec.sql

begin_time:20160327 16:42:30

end_time:20160327 16:43:32

 

耗時顯示兩session硬解析耗時均為62

 

---session 3:顯示session 1 & session 2首次執行期間產生的等待事件及parse相關統計值

@sess_2nd.sql 3307 6605

       SID EVENT                          DIFF_WAITS

---------- ------------------------------ ----------

      3307 Disk file operations I/O                0

      6605 Disk file operations I/O                2

      3307 SQL*Net message from client             2

      6605 SQL*Net message from client             2

      3307 SQL*Net message to client               2

      6605 SQL*Net message to client               2

      3307 cursor: mutex X                         4

      3307 cursor: pin S                          49

      6605 cursor: pin S                          62

      3307 cursor: pin S wait on X              2206

      6605 cursor: pin S wait on X              2485

      3307 db file sequential read                 0

      6605 db file sequential read                 2

      3307 events in waitclass Other              39

      6605 events in waitclass Other             110

      3307 latch: shared pool                      1

      3307 library cache load lock               481

      6605 library cache load lock               474

      3307 library cache lock                    456

      6605 library cache lock                    477

      3307 library cache: mutex X               1723

      6605 library cache: mutex X               1470

 

       SID NAME                                                             DIFF_STATS

---------- ---------------------------------------------------------------- ----------

      6605 parse time cpu                                                         3048

      3307 parse time cpu                                                         3658

      6605 parse time elapsed                                                     6174

      3307 parse time elapsed                                                     6248

      6605 parse count (total)                                                   50016

      3307 parse count (total)                                                   50007

      6605 parse count (hard)                                                    22428

      3307 parse count (hard)                                                    27575

      6605 parse count (failures)                                                    0

      3307 parse count (failures)                                                    0

      6605 parse count (describe)                                                    0

      3307 parse count (describe)                                                    0

 

等待事件裡出現最多的是cursor: pin S wait on X,因為Session 1 & session 2執行的語句都是相同的,且是同一時刻發起,相同語句被兩個session同時執行的機率是很大的,這過程中必定有一個session執行的是硬解析,另一個執行的是軟解析;比如session 1正在硬解析語句select * from t1 where user_id=1,硬解析的時候對於子游標需要以獨佔模式持有cursor pin型別的mutex,這時session 2也要執行該語句,那麼它需要申請以共享模式在這個子游標上持有cursor pin型別的mutex,獨佔模式是排它的,所以session 2就等待在了"cursor: pin S wait on X"事件上,直到session 1完成硬解析,session 2才能獲得共享的cursor pinmutex。因此"cursor: pin S wait on X"的成因是:在session 1 & session 2同時執行相同的語句,其中一個session正在硬解析,另一個請求軟解析的session就會等待在"cursor: pin S wait on X"事件上。

再來看看解析有關的指標parse count (hard),兩個session分別承擔了2242827575次硬解析,硬解析的總和50003次,接近5W次,對於特定的一條SQL來說若在session 1裡進行了硬解析那麼在session 2必定是軟解析。

仔細思考一下parse count (hard)cursor: pin S wait on X其實有一定的內在聯絡:parse count (hard)值小的Sessioncursor: pin S wait on X等待次數往往會更多,在以上的輸出中sid6605的硬解析次數為22428,剩下的有27572次都是軟解析,之所以這27572次都能做到軟解析是因為其共享了sid:3307執行27575次硬解析後的結果,sid:6605執行軟解析申請共享模式cursor pin mutex過程中就會更容易被sid:3307硬解析時持有的獨佔cursor pin mutex所阻塞;反觀sid:3307的硬解析次數為27575,剩餘的22425次都是軟解析,軟解析次數較sid:3307要少,因此其遇到cursor: pin S wait on X的機率自然就小很多了,所以說硬解析次數少(或者說軟解析次數多)sessioncursor: pin S wait on X事件上的等待次數會比較多。

 

library cache locklibrary cache load lock事件對應的爭用物件主要集中在父、子游標以及表上,也是由於兩session併發所導致的爭用,只要有解析,不管是硬解析還是軟解析(軟軟解析除外),都會在執行過程中申請library cache lock資源,一旦申請的過程中與其它session發生爭搶,就會遇到這兩類等待

少量的cursor: pin S事件來自於sql_exec.sql指令碼里的” select extract(second from to_timestamp(to_char(sysdate,'yyyymmdd hh24:mi:ss'),'yyyymmdd hh24:mi:ss')) into v_sec from dual”語句。在此不多討論,後面會有介紹到cursor: pin S

至此,每條語句都被執行了至少兩遍,接下來就應該是軟解析了。

<<<<<<<<<<<< 2、軟解析的情況:當語句都被執行過一遍後,接下來的都是軟解析了 >>>>>>>>>>>>>

###確認Sql還在shared pool

select count(*) from v$sql where sql_text like 'select * from t1 where user_id=%';

  COUNT(*)

----------

     50001

    

###session第二輪執行該語句,因為都在shared pool裡所以這次應該都是軟解析    

---session 3:執行前先統計session 1 & session 2的當前值

@sess_1st.sql 3307 6605

 

---session 1:

@sql_exec.sql

begin_time:20160327 16:54:30

end_time:20160327 16:54:33

 

---session 2:

@sql_exec.sql

begin_time:20160327 16:54:30

end_time:20160327 16:54:33

 

---session 3: 顯示session 1 & session 2 第二輪執行期間產生的等待事件及parse相關統計值

@sess_2nd.sql 3307 6605

       SID EVENT                          DIFF_WAITS

---------- ------------------------------ ----------

      3307 Disk file operations I/O                0

      6605 Disk file operations I/O                0

      3307 SQL*Net message from client             2

      6605 SQL*Net message from client             2

      3307 SQL*Net message to client               2

      6605 SQL*Net message to client               2

      3307 cursor: mutex X                         0

      3307 cursor: pin S                          18

      6605 cursor: pin S                          18

      3307 cursor: pin S wait on X                 0

      6605 cursor: pin S wait on X                 0

      3307 db file sequential read                 0

      6605 db file sequential read                 0

      3307 events in waitclass Other               0

      6605 events in waitclass Other               0

      3307 latch: shared pool                      0

      3307 library cache load lock                 0

      6605 library cache load lock                 0

      3307 library cache lock                      0

      6605 library cache lock                      0

      3307 library cache: mutex X               9658   <---軟解析時library cache:mutex X事件居多

      6605 library cache: mutex X               8130

 

22 rows selected.

 

 

       SID NAME                                                             DIFF_STATS

---------- ---------------------------------------------------------------- ----------

      6605 parse time cpu                                                          114

      3307 parse time cpu                                                          140

      6605 parse time elapsed                                                      126

      3307 parse time elapsed                                                      128

      6605 parse count (total)                                                   50001

      3307 parse count (total)                                                   50002

      6605 parse count (hard)                                                        0

      3307 parse count (hard)                                                        0

      6605 parse count (failures)                                                    0

      3307 parse count (failures)                                                    0

      6605 parse count (describe)                                                    0

      3307 parse count (describe)                                                    0

 

這次parse count (hard)值為0,等待事件裡"library cache: mutex X"佔大頭,搜尋hash連結串列尋找父遊標控制程式碼的時候需要"library cache: mutex X"的保護,找到父遊標控制程式碼後訪問控制程式碼內容的時候需要"library cache: mutex X"的保護,同樣在訪問子游標控制程式碼內容的時候也需要申請"library cache: mutex X""X"表示以獨佔模式持有,所以兩個session同時執行特定的SQL,在尋找父遊標控制程式碼、訪問父遊標與子游標控制程式碼內容的時候是序列的,等前一個session釋放了"library cache: mutex X",後一個session才能申請"library cache: mutex X"

 

至此,每條語句至少被執行了四遍,下面我們等待軟軟解析的出現

<<<<<<<<<<<< 3、預料中的軟軟解析並未出現  >>>>>>>>>>>>>

###Session第三輪執行語句後的等待事件及parse統計值

---session 3:執行前先統計session 1 & session 2的當前值

@sess_1st.sql 3307 6605

 

---session 1:

@sql_exec.sql

begin_time:20160327 16:57:30

end_time:20160327 16:57:33

 

---session 2:

@sql_exec.sql

begin_time:20160327 16:57:30

end_time:20160327 16:57:33

 

---session 3: 顯示session 1 & session 2 第三輪執行期間產生的等待事件及parse相關統計值

@sess_2nd.sql 3307 6605

 

       SID EVENT                          DIFF_WAITS

---------- ------------------------------ ----------

      3307 Disk file operations I/O                0

      6605 Disk file operations I/O                0

      3307 SQL*Net message from client             2

      6605 SQL*Net message from client             2

      3307 SQL*Net message to client               2

      6605 SQL*Net message to client               2

      3307 cursor: mutex X                         0

      3307 cursor: pin S                           9

      6605 cursor: pin S                          14

      3307 cursor: pin S wait on X                 0

      6605 cursor: pin S wait on X                 0

      3307 db file sequential read                 0

      6605 db file sequential read                 0

      3307 events in waitclass Other               0

      6605 events in waitclass Other               0

      3307 latch: shared pool                      0

      3307 library cache load lock                 0

      6605 library cache load lock                 0

      3307 library cache lock                      0

      6605 library cache lock                      0

      3307 library cache: mutex X               7039

      6605 library cache: mutex X               5837

 

       SID NAME                                                             DIFF_STATS

---------- ---------------------------------------------------------------- ----------

      6605 parse time cpu                                                          119

      3307 parse time cpu                                                          130

      6605 parse time elapsed                                                      109

      3307 parse time elapsed                                                      107

      6605 parse count (total)                                                   50002

      3307 parse count (total)                                                   50002

      6605 parse count (hard)                                                        0

      3307 parse count (hard)                                                        0

      6605 parse count (failures)                                                    0

      3307 parse count (failures)                                                    0

      6605 parse count (describe)                                                    0

      3307 parse count (describe)                                                    0

 

 

至此已經執行了三輪,5Wsql裡的絕大部分都被解析了6次,執行了6次,等待事件和第二輪一樣以"library cache: mutex X"為主

 

SQL> select parse_calls,executions,count(1) from v$sql where sql_text like 'select * from t1 where user_id=%' group by parse_calls,executions;

 

PARSE_CALLS EXECUTIONS   COUNT(1)

----------- ---------- ----------

          4          6          3

          5          5         32

          5          6        586

          6          4          5

          6          5        770

          6          6      48604

         27         27          1

 

只要有解析,即便是軟解析也是要在物件上加lockpin的,lockpin前也要先獲取相應的mutex,因此亦會產生一定的開銷,解析1次執行n次的軟軟解析才是效率最高的,要實現軟軟解析必須要求session_cached_Cursor設定的足夠大,在我們這個資料庫裡Session_cached_cursor只設定成20,值太小所以無法緩衝更多的cursor以實現軟軟解析,大部分sqlparse_callsexecutions數量一樣,仍然是parse一次execute一次

SQL> show parameter session_Cached_cursor

 

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

session_cached_cursors               integer     20

     

<<<<<<<<<<<< 4、調大session_cached_cursor,軟解析的同時將cursor緩衝至PGA  >>>>>>>>>>>>>

下面我們分別在session 1session 2中把session_cached_cursors設定成50200

---session 1:

SQL> alter session set session_cached_cursors=50200;   

 

Session altered.

 

---session 2:

SQL> alter session set session_cached_cursors=50200;

 

Session altered.

 

###session第四輪執行該語句

---session 3:執行前先統計session 1 & session 2的當前值

@sess_1st.sql 3307 6605

 

---session 1:

@sql_exec.sql

begin_time:20160327 17:04:30

end_time:20160327 17:06:31

 

---session 2:

@sql_exec.sql

begin_time:20160327 17:04:30

end_time:20160327 17:06:30

我們發現把session_cached_cursor調大後,語句執行事件從前一次執行時的3秒增加到了120秒,這是因為需要把5W條的SQL依次快取進PGA,快取的這一次是比較慢的,後面再執行就很快了,正所謂磨刀不誤砍柴工

 

---session 3:

@sess_2nd.sql 3307 6605

       SID EVENT                          DIFF_WAITS

---------- ------------------------------ ----------

      3307 Disk file operations I/O                0

      6605 Disk file operations I/O                0

      3307 SQL*Net message from client             2

      6605 SQL*Net message from client             2

      3307 SQL*Net message to client               2

      6605 SQL*Net message to client               2

      3307 cursor: mutex X                         0

      3307 cursor: pin S                          62

      6605 cursor: pin S                          69

      3307 cursor: pin S wait on X                 0

      6605 cursor: pin S wait on X                 0

      3307 db file sequential read                 0

      6605 db file sequential read                 0

      3307 events in waitclass Other               0

      6605 events in waitclass Other               0

      3307 latch: shared pool                      0

      3307 library cache load lock                 0

      6605 library cache load lock                 0

      3307 library cache lock                      0

      6605 library cache lock                      0

      3307 library cache: mutex X                 15        <---library cache:mutex X次數少了

      6605 library cache: mutex X                 16

 

       SID NAME                                                             DIFF_STATS

---------- ---------------------------------------------------------------- ----------

      6605 parse time cpu                                                        11533

      3307 parse time cpu                                                        11535

      6605 parse time elapsed                                                    11611

      3307 parse time elapsed                                                    11701

      6605 parse count (total)                                                   49984

      3307 parse count (total)                                                   49984

      6605 parse count (hard)                                                        0

      3307 parse count (hard)                                                        0

      6605 parse count (failures)                                                    0

      3307 parse count (failures)                                                    0

      6605 parse count (describe)                                                    0

      3307 parse count (describe)                                                    0

 

 

v$open_cursor裡的cursor_typeOPEN-RECURSIVE,並不是我們所熟悉的"SESSION CURSOR CACHED",可能與我們使用dbms_sql包執行sql有關係

SQL> select sid,cursor_type,count(1) from v$open_cursor where sid in (3307,6605) group by sid,cursor_type;

 

       SID CURSOR_TYPE                                                        COUNT(1)

---------- ---------------------------------------------------------------- ----------

      3307 OPEN                                                                      1

      3307 OPEN-RECURSIVE                                                        50003

      3307 PL/SQL CURSOR CACHED                                                      1

      3307 SESSION CURSOR CACHED                                                     3

      6605 OPEN                                                                      1

      6605 OPEN-RECURSIVE                                                        50003

      6605 PL/SQL CURSOR CACHED                                                      1

      6605 SESSION CURSOR CACHED                                                     3

 

SQL> select parse_calls,executions,count(1) from v$sql where sql_text like 'select * from t1 where user_id=%' group by parse_calls,executions;

 

PARSE_CALLS EXECUTIONS   COUNT(1)

----------- ---------- ----------

          6          8         20

          7          7         32

          7          8        587

          8          6          5

          8          7        770

          8          8      48586  <---executions=parse_calls,說明還是軟解析

         27         27          1

 

第四輪的執行還是軟解析,整個過程中幾乎沒有明顯的等待事件,時間都耗在了cache遊標到PGA的過程中

 

<<<<<<<<<<<<  5、軟軟解析來了  >>>>>>>>>>>>>

###session第五輪執行該語句

@sess_1st.sql 3307 6605

 

---session 1:

@sql_exec.sql

begin_time:20160327 17:12:30

end_time:20160327 17:12:32

 

---session 2:

@sql_exec.sql

begin_time:20160327 17:12:30

end_time:20160327 17:12:32

 

---session 3:

@sess_2nd.sql 3307 6605     

       SID EVENT                          DIFF_WAITS

---------- ------------------------------ ----------

      3307 Disk file operations I/O                0

      6605 Disk file operations I/O                0

      3307 SQL*Net message from client             2

      6605 SQL*Net message from client             2

      3307 SQL*Net message to client               2

      6605 SQL*Net message to client               2

      3307 cursor: mutex X                         0

      3307 cursor: pin S                         478

      6605 cursor: pin S                         669

      3307 cursor: pin S wait on X                 0

      6605 cursor: pin S wait on X                 0

      3307 db file sequential read                 0

      6605 db file sequential read                 0

      3307 events in waitclass Other               0

      6605 events in waitclass Other               0

      3307 latch: shared pool                      0

      3307 library cache load lock                 0

      6605 library cache load lock                 0

      3307 library cache lock                      0

      6605 library cache lock                      0

      3307 library cache: mutex X                  2

      6605 library cache: mutex X                  0

 

       SID NAME                                                             DIFF_STATS

---------- ---------------------------------------------------------------- ----------

      6605 parse time cpu                                                           59

      3307 parse time cpu                                                           51

      6605 parse time elapsed                                                       73

      3307 parse time elapsed                                                       65

      6605 parse count (total)                                                       1    <---解析次數接近0

      3307 parse count (total)                                                       1

      6605 parse count (hard)                                                        0

      3307 parse count (hard)                                                        0

      6605 parse count (failures)                                                    0

      3307 parse count (failures)                                                    0

      6605 parse count (describe)                                                    0

      3307 parse count (describe)                                                    0

 

select parse_calls,executions,count(1) from v$sql where sql_text like 'select * from t1 where user_id=%' group by parse_calls,executions;

PARSE_CALLS EXECUTIONS   COUNT(1)

----------- ---------- ----------

          6          9          2

          6         10         18

          7          9         72

          7         10        547

          8          8         53

          8          9       3538

          8         10      45769   <---executions增加,parse_calls不變,說明已經是軟軟解析了

         10         10          1

         27         27          1

 

在第五輪的執行中,終於出現了軟軟解析,耗時僅2秒,由於子游標的執行計劃已經被緩衝到了PGA,所以語句執行期間只要以共享模式獲得"cursor: pin S"型別的mutex就能輕鬆訪問到PGA裡的執行計劃。

 

<<<<<<<<<<<<  6、往後的都是軟軟解析了  >>>>>>>>>>>>>

###session第六輪執行該語句

@sess_1st.sql 3307 6605

 

---session 1:

@sql_exec.sql

begin_time:20160327 17:17:30

end_time:20160327 17:17:32

 

---session 2:

@sql_exec.sql

begin_time:20160327 17:17:30

end_time:20160327 17:17:32

 

---session 3:

@sess_2nd.sql 3307 6605              

       SID EVENT                          DIFF_WAITS

---------- ------------------------------ ----------

      3307 Disk file operations I/O                0

      6605 Disk file operations I/O                0

      3307 SQL*Net message from client             2

      6605 SQL*Net message from client             2

      3307 SQL*Net message to client               2

      6605 SQL*Net message to client               2

      3307 cursor: mutex X                         0

      3307 cursor: pin S                         520

      6605 cursor: pin S                         597

      3307 cursor: pin S wait on X                 0

      6605 cursor: pin S wait on X                 0

      3307 db file sequential read                 0

      6605 db file sequential read                 0

      3307 events in waitclass Other               0

      6605 events in waitclass Other               0

      3307 latch: shared pool                      0

      3307 library cache load lock                 0

      6605 library cache load lock                 0

      3307 library cache lock                      0

      6605 library cache lock                      0

      3307 library cache: mutex X                  3

      6605 library cache: mutex X                  0

 

       SID NAME                                                             DIFF_STATS

---------- ---------------------------------------------------------------- ----------

      6605 parse time cpu                                                           69

      3307 parse time cpu                                                           52

      6605 parse time elapsed                                                       56

      3307 parse time elapsed                                                       56

      6605 parse count (total)                                                       1

      3307 parse count (total)                                                       1

      6605 parse count (hard)                                                        0

      3307 parse count (hard)                                                        0

      6605 parse count (failures)                                                    0

      3307 parse count (failures)                                                    0

      6605 parse count (describe)                                                    0

      3307 parse count (describe)                                                    0       

 

SQL> select parse_calls,executions,count(1) from v$sql where sql_text like 'select * from t1 where user_id=%' group by parse_calls,executions;

 

PARSE_CALLS EXECUTIONS   COUNT(1)

----------- ---------- ----------

          6         11          3

          6         12         17

          7         10          4

          7         11         86

          7         12        529

          8         10        253

          8         11       5174

          8         12      43933

         12         12          1

         27         27          1

 

總結一下:

輪次

Session 1 :

解析方式

Session 2 :

解析方式

主要

Waitevent

執行耗時

1

/軟解析

/硬解析

cursor: pin S wait on X

library cache: mutex X

library cache lock

library cache load lock

62s

2

軟解析

軟解析

library cache: mutex X

3s

3

軟解析

軟解析

library cache: mutex X

3s

4

軟解析

軟解析

無顯著等待

120s(cursor cachePGA的時間)

5

軟軟解析

軟軟解析

cursor: pin S

2s

6

軟軟解析

軟軟解析

cursor: pin S

2s


來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/53956/viewspace-2069886/,如需轉載,請註明出處,否則將追究法律責任。

相關文章