聊聊Dataguard的三種保護模式實驗(上)

realkid4發表於2015-11-03

 

Data GuardOracle高可用性HA的重要解決方案。針對不同的系統保護需求,DG提供了三種不同型別的保護模式(Protection Mode),分別為:最大保護(Maximum Protection)、最大可用(Maximum Availability)和最大效能(Maximum performance)。在實際應用場景下,我們需要根據不同的業務場景和資料可用性需求,來設定DG環境的保護型別。

 

1、三種保護模式Protection Mode

 

三種保護模式是DG的核心概念。DG本質上是一種基於Redo Log的資料同步機制。UndoRedoOracle早期奠定行業地位的核心技術。Undo負責記錄事務操作的前映象,而Redo負責記錄事務操作的後映象。在Oracle事務commit的動作中,寫入日誌檔案是一個一定需要完成的動作。寫入日誌檔案之後,即使立刻出現嚴重的例項終止事件,在重新啟動例項的時候也會進行例項恢復動作,將事務落實。

 

DG環境中,無論採用何種初始化方法,都是確保一個PrimaryStandby的初始化資料一致,之後Primary一端接收的事務型別操作,均會以歸檔日誌串列的方式傳遞到Standby端的standby redo log和歸檔日誌列表中,最後重複應用這些日誌,實現PrimaryStandby端一致。

 

PrimaryStandby是相互為備份的冗餘結構,Standby跟隨Primary的情況,反映了HA結構的可用性級別。理論上,最保險的策略是一個事務要保證在PrimaryStandby上都提交了,才返回給使用者說已經完成。這樣是可以保證主備庫完全一致的最保險做法。另一個極端情況,就是主庫“自顧自”進行事務處理,獨立將日誌進行傳輸,也不用管日誌是否傳輸到或者應用到。

 

針對不同的傳輸情況,DG區分為三種保護型別:

 

ü  最大可用模式Maximum Availability

 

在官方文件中,對這種模式的描述如下:

 

This protection mode provides the highest level of data protection that is possible without compromising the availability of a primary database. Transactions do not commit until all redo data needed to recover those transactions has been written to the online redo log and to the standby redo log on at least one synchronized standby database. If the primary database cannot write its redo stream to at least one synchronized standby database, it operates as if it were in maximum performance mode to preserve primary database availability until it is again able to write its redo stream to a synchronized standby database.

 

Maximum Availability模式下,事務只有在所有相關日誌都被傳輸到至少一個Standby端日誌的時候,才可以正式提交。但是,如果Primary在傳輸日誌的過程中,發現所有standby端都不能進行傳輸,模式會退化到最大效能模式(Maximum Performance)工作方式。應該說,Maximum Availability是一種自適應的保護模式,當出現問題的時候,DG會退而求其次,確保Primary主庫事務進行。

 

ü  最大效能模式(Maximum Performance

 

官方文件中介紹如下:“This protection mode provides the highest level of data protection that is possible without affecting the performance of a primary database. This is accomplished by allowing transactions to commit as soon as all redo data generated by those transactions has been written to the online log. Redo data is also written to one or more standby databases, but this is done asynchronously with respect to transaction commitment, so primary database performance is unaffected by delays in writing redo data to the standby database(s).

This protection mode offers slightly less data protection than maximum availability mode and has minimal impact on primary database performance.

This is the default protection mode.

 

最大效能模式是在不影響主庫工作情況下,可以提供的最高資料保護級別。當事務進行提交的時候,主庫不會去確認日誌是否寫入到備庫中,更不會確認是否被apply。這種方式下,主庫的工作效能是不會收到備庫提交應用的影響的。當然,這種保護模式會有一定的事務資料丟失,但是絕對不會出現資料誤提交的情況。

 

DG而言,最大效能模式是預設的保護模式。當我們完成了DG安裝之後,就自動進入了Maximum Performance模式。

 

ü  最大保護模式(Maximum Protection

 

最大保護模式在官方中的描述為:

 

This protection mode ensures that no data loss will occur if the primary database fails. To provide this level of protection, the redo data needed to recover a transaction must be written to both the online redo log and to the standby redo log on at least one synchronized standby database before the transaction commits. To ensure that data loss cannot occur, the primary database will shut down, rather than continue processing transactions, if it cannot write its redo stream to at least one synchronized standby database.

Transactions on the primary are considered protected as soon as Data Guard has written the redo data to persistent storage in a standby redo log file. Once that is done, acknowledgment is quickly made back to the primary database so that it can proceed to the next transaction. This minimizes the impact of synchronous transport on primary database throughput and response time. To fully benefit from complete Data Guard validation at the standby database, be sure to operate in real-time apply mode so that redo changes are applied to the standby database as fast as they are received. Data Guard signals any corruptions that are detected so that immediate corrective action can be taken.

 

最大保護模式是完全HA架構理想中的事務模式。如果Primary資料庫進行一個事務,連帶Standby資料庫也要同步進行操作。如果由於網路、執行模式等原因,Standby不能夠跟上主庫的操作,那麼主庫會放棄事務,並且強制停庫。

 

保護模式的三種和資料庫之間傳輸日誌的機制是密切相關的。主要體現是否同步傳輸Redo日誌和對日誌進行確認兩個方面。我們配置三種日誌模式,一定要以Log_Archive_Config引數配置為基礎。

 

Maximum Availability

Maximum Performance

Maximum Protection

AFFIRM

NOAFFIRM

AFFIRM

SYNC

ASYNC

SYNC

DB_UNIQUE_NAME

DB_UNIQUE_NAME

DB_UNIQUE_NAME

 

下面透過一系列的測試,來分析三種保護模式的工作行為方式。

 

2、環境介紹

 

筆者使用環境為Oracle 11gR2,具體版本為11.2.0.4。主備庫環境已經搭建完成,同步保護模式是採用預設方式。

 

主庫資訊:

 

 

SQL> select name, open_mode, database_role, protection_mode from v$database;

NAME      OPEN_MODE            DATABASE_ROLE    PROTECTION_MODE

--------- -------------------- ---------------- --------------------

VLIFE     READ WRITE           PRIMARY          MAXIMUM PERFORMANCE

 

SQL> select instance_name from v$instance;

INSTANCE_NAME

----------------

vlife

 

 

主庫與備庫連線方式,採用預設的非同步非確認方式。

 

 

SQL> select dest_id, dest_name, TRANSMIT_MODE, ASYNC_BLOCKS, AFFIRM TYPE, VALID_NOW, VALID_TYPE, VALID_ROLE, DB_UNIQUE_NAME from v$archive_dest where status<>'INACTIVE';

 

   DEST_ID DEST_NAME            TRANSMIT_MODE ASYNC_BLOCKS TYPE VALID_NOW        VALID_TYPE      VALID_ROLE   DB_UNIQUE_NAME

---------- -------------------- ------------- ------------ ---- ---------------- --------------- ------------ ---------------

         1 LOG_ARCHIVE_DEST_1   SYNCHRONOUS              0 NO   YES              ALL_LOGFILES    ALL_ROLES    NONE

         2 LOG_ARCHIVE_DEST_2   ASYNCHRONOUS         61440 NO   YES              ONLINE_LOGFILE  PRIMARY_ROLE vlifesb

 

 

此時,傳輸通道配置。

 

 

SQL> show parameter LOG_ARCHIVE_DEST_2;

 

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

log_archive_dest_2                   string      SERVICE=vlifesb valid_for=(online_logfiles,primary_role) db_unique_name=vlifesb

log_archive_dest_20                  string     

log_archive_dest_21                  string     

 

 

備庫資訊如下:

 

 

SQL> select name, open_mode, database_role, protection_mode from v$database;

 

NAME      OPEN_MODE            DATABASE_ROLE    PROTECTION_MODE

--------- -------------------- ---------------- --------------------

VLIFE     READ ONLY WITH APPLY PHYSICAL STANDBY MAXIMUM PERFORMANCE

 

SQL> select instance_name from v$instance;

 

INSTANCE_NAME

----------------

vlifesb

 

SQL> col dest_name for a20;

SQL> select dest_id, dest_name, TRANSMIT_MODE, ASYNC_BLOCKS, AFFIRM TYPE, VALID_NOW, VALID_TYPE, VALID_ROLE, DB_UNIQUE_NAME from v$archive_dest where status<>'INACTIVE';

 

   DEST_ID DEST_NAME            TRANSMIT_MODE ASYNC_BLOCKS TYPE VALID_NOW        VALID_TYPE      VALID_ROLE   DB_UNIQUE_NAME

---------- -------------------- ------------- ------------ ---- ---------------- --------------- ------------ ------------------------------

         1 LOG_ARCHIVE_DEST_1   SYNCHRONOUS              0 NO   YES              ALL_LOGFILES    ALL_ROLES    NONE

         2 LOG_ARCHIVE_DEST_2   ASYNCHRONOUS         61440 NO   WRONG VALID_TYPE ONLINE_LOGFILE  PRIMARY_ROLE vlife

        32 STANDBY_ARCHIVE_DEST SYNCHRONOUS              0 NO   YES              ALL_LOGFILES    ALL_ROLES    NONE

 

 

SQL> show parameter LOG_ARCHIVE_DEST_2;

 

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

log_archive_dest_2                   string      SERVICE=vlife valid_for=(online_logfiles,primary_role) db_unique_name=vlife

log_archive_dest_20                  string     

log_archive_dest_21                  string     

 

 

3、最大可用模式Maximum Availability Mode測試

 

當前從預設的最大效能切換到最大可用模式,首先需要滿足將日誌傳輸模式進行修改。

 

 

SQL> alter system set log_archive_dest_2='SERVICE=vlifesb sync affirm net_timeout=30 valid_for=(online_logfiles,primary_role) db_unique_name=vlifesb';

 

System altered

 

SQL> select dest_id, dest_name, TRANSMIT_MODE, ASYNC_BLOCKS, AFFIRM TYPE, VALID_NOW, VALID_TYPE, VALID_ROLE, DB_UNIQUE_NAME, NET_TIMEOUT from v$archive_dest where status<>'INACTIVE';

 

   DEST_ID DEST_NAME            TRANSMIT_MODE ASYNC_BLOCKS TYPE VALID_NOW        VALID_TYPE      VALID_ROLE   DB_UNIQUE_NAME                 NET_TIMEOUT

---------- -------------------- ------------- ------------ ---- ---------------- --------------- ------------ ------------------------------ -----------

         1 LOG_ARCHIVE_DEST_1   SYNCHRONOUS              0 NO   YES              ALL_LOGFILES    ALL_ROLES    NONE                                     0

         2 LOG_ARCHIVE_DEST_2   PARALLELSYNC             0 YES  YES              ONLINE_LOGFILE  PRIMARY_ROLE vlifesb                                 30

 

 

此時,將保護模式使用alter database進行設定。

 

 

SQL> alter database set standby database to maximize availability;

Database altered

 

SQL> select name, open_mode, database_role, protection_mode from v$database;

 

NAME      OPEN_MODE            DATABASE_ROLE    PROTECTION_MODE

--------- -------------------- ---------------- --------------------

VLIFE     READ WRITE           PRIMARY          MAXIMUM AVAILABILITY

 

 

在切換動作的時候,主庫日誌情況如下:

 

 

Wed Oct 21 15:13:48 2015

alter database set standby database to maximize availability

 

Completed: alter database set standby database to maximize availability

 

Wed Oct 21 15:13:49 2015

Destination LOG_ARCHIVE_DEST_2 is UNSYNCHRONIZED –發現沒有同步,需要補充。

******************************************************************

LGWR: Setting 'active' archival for destination LOG_ARCHIVE_DEST_2

******************************************************************

Wed Oct 21 15:13:49 2015

NSS2 started with pid=34, OS id=9186

LGWR: Standby redo logfile selected to archive thread 1 sequence 82

LGWR: Standby redo logfile selected for thread 1 sequence 82 for destination LOG_ARCHIVE_DEST_2

Thread 1 advanced to log sequence 82 (LGWR switch)

  Current log# 3 seq# 82 mem# 0: /u01/app/oracle/oradata/VLIFE/onlinelog/o1_mf_3_c1kb1c24_.log

  Current log# 3 seq# 82 mem# 1: /u01/app/oracle/fast_recovery_area/VLIFE/onlinelog/o1_mf_3_c1kb1c43_.log

Wed Oct 21 15:13:53 2015

Archived Log entry 104 added for thread 1 sequence 81 ID 0xfad4f44b dest 1:

Wed Oct 21 15:13:54 2015

ARC3: Archive log rejected (thread 1 sequence 81) at host 'vlifesb'

FAL[server, ARC3]: FAL archive failed, see trace file.

ARCH: FAL archive failed. Archiver continuing

ORACLE Instance vlife - Archival Error. Archiver continuing.

Wed Oct 21 15:14:42 2015

Destination LOG_ARCHIVE_DEST_2 is SYNCHRONIZED

LGWR: Standby redo logfile selected to archive thread 1 sequence 83

LGWR: Standby redo logfile selected for thread 1 sequence 83 for destination LOG_ARCHIVE_DEST_2

Thread 1 advanced to log sequence 83 (LGWR switch)

  Current log# 1 seq# 83 mem# 0: /u01/app/oracle/oradata/VLIFE/onlinelog/o1_mf_1_c1kb19q4_.log

  Current log# 1 seq# 83 mem# 1: /u01/app/oracle/fast_recovery_area/VLIFE/onlinelog/o1_mf_1_c1kb19sb_.log

Wed Oct 21 15:14:42 2015

Archived Log entry 107 added for thread 1 sequence 82 ID 0xfad4f44b dest 1:

 

 

Primary端,在進行切換之後,Oracle發現傳輸狀態不是同步情況。於是自動加快進行日誌傳輸和同步動作。在Standby端,也可以看到後續追趕動作。

 

 

SQL> select name, open_mode, database_role, protection_mode from v$database;

 

NAME      OPEN_MODE            DATABASE_ROLE    PROTECTION_MODE

--------- -------------------- ---------------- --------------------

VLIFE     READ ONLY WITH APPLY PHYSICAL STANDBY MAXIMUM AVAILABILITY

 

 

Standby端上的日誌追趕動作。

 

 

Wed Oct 21 08:27:05 2015

Primary database is in MAXIMUM PERFORMANCE mode

Re-archiving standby log 4 thread 1 sequence 80

Wed Oct 21 08:27:05 2015

Media Recovery Waiting for thread 1 sequence 81

RFS[14]: Assigned to RFS process 31500

RFS[14]: Selected log 5 for thread 1 sequence 81 dbid -87496857 branch 892734889

Wed Oct 21 08:27:05 2015

Archived Log entry 76 added for thread 1 sequence 80 ID 0xfad4f44b dest 1:

Recovery of Online Redo Log: Thread 1 Group 5 Seq 81 Reading mem 0

  Mem# 0: /u01/app/oracle/oradata/VLIFESB/onlinelog/o1_mf_5_c265gqd8_.log

  Mem# 1: /u01/app/oracle/fast_recovery_area/VLIFESB/onlinelog/o1_mf_5_c265gqj0_.log

Wed Oct 21 15:13:52 2015

Primary database is in MAXIMUM AVAILABILITY mode

Changing standby controlfile to MAXIMUM AVAILABILITY mode

Changing standby controlfile to RESYNCHRONIZATION level

Standby controlfile consistent with primary

RFS[15]: Assigned to RFS process 969

RFS[15]: Selected log 4 for thread 1 sequence 82 dbid -87496857 branch 892734889

Wed Oct 21 15:13:53 2015

Archived Log entry 77 added for thread 1 sequence 81 ID 0xfad4f44b dest 1:

Wed Oct 21 15:13:53 2015

Media Recovery Waiting for thread 1 sequence 82 (in transit)

Recovery of Online Redo Log: Thread 1 Group 4 Seq 82 Reading mem 0

  Mem# 0: /u01/app/oracle/oradata/VLIFESB/onlinelog/o1_mf_4_c265gc9q_.log

  Mem# 1: /u01/app/oracle/fast_recovery_area/VLIFESB/onlinelog/o1_mf_4_c265gcfk_.log

Wed Oct 21 15:14:41 2015

Archived Log entry 78 added for thread 1 sequence 82 ID 0xfad4f44b dest 1:

Wed Oct 21 15:14:41 2015

Media Recovery Waiting for thread 1 sequence 83

Wed Oct 21 15:14:42 2015

Primary database is in MAXIMUM AVAILABILITY mode

Changing standby controlfile to MAXIMUM AVAILABILITY level

Standby controlfile consistent with primary

RFS[16]: Assigned to RFS process 976

RFS[16]: Selected log 4 for thread 1 sequence 83 dbid -87496857 branch 892734889

Recovery of Online Redo Log: Thread 1 Group 4 Seq 83 Reading mem 0

  Mem# 0: /u01/app/oracle/oradata/VLIFESB/onlinelog/o1_mf_4_c265gc9q_.log

  Mem# 1: /u01/app/oracle/fast_recovery_area/VLIFESB/onlinelog/o1_mf_4_c265gcfk_.log

 

 

此時,兩個庫由於網路通暢,同步狀態正常,同步測試正常。

 

 

Maximium Availiablity模式下使用)

--主庫下

SQL> create table t_m as select * from dba_objects where rownum<10;

Table created

 

--Standby

SQL> select count(*) from t_m;

 

  COUNT(*)

----------

         9

 

 

如果此時中斷應用日誌,Standby情況如下:

 

 

SQL> alter database recover managed standby database cancel;

Database altered

 

SQL> select name, open_mode, database_role, protection_mode from v$database;

 

NAME      OPEN_MODE            DATABASE_ROLE    PROTECTION_MODE

--------- -------------------- ---------------- --------------------

VLIFE     READ ONLY            PHYSICAL STANDBY MAXIMUM AVAILABILITY

 

 

日誌情況如下:

 

 

Wed Oct 21 15:20:49 2015

alter database recover managed standby database cancel

 

Wed Oct 21 15:20:49 2015

MRP0: Background Media Recovery cancelled with status 16037

Errors in file /u01/app/oracle/diag/rdbms/vlifesb/vlifesb/trace/vlifesb_pr00_17539.trc:

ORA-16037: user requested cancel of managed recovery operation

Managed Standby Recovery not using Real Time Apply

Recovery interrupted!

Recovered data files to a consistent state at change 1692263

Wed Oct 21 15:20:49 2015

MRP0: Background Media Recovery process shutdown (vlifesb)

Managed Standby Recovery Canceled (vlifesb)

Completed: alter database recover managed standby database cancel

 

 

如果此時再中斷監聽器,中斷連線。此時資料庫不能做到實時同步。

 

 

[oracle@vLIFE-URE-OT-DB-STANDBY ~]$ lsnrctl stop

 

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 21-OCT-2015 15:24:17

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

 

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1521)))

The command completed successfully

 

--主庫

***********************************************************************

Fatal NI connect error 12541, connecting to:

 (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=172.16.19.90)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=vlifesb)(CID=(PROGRAM=oracle)(HOST=vLIFE-URE-OT-DB-PRIMARY)(USER=oracle))))

 

  VERSION INFORMATION:

        TNS for Linux: Version 11.2.0.4.0 - Production

        TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production

  Time: 21-OCT-2015 15:24:38

  Tracing not turned on.

  Tns error struct:

    ns main err code: 12541

   

TNS-12541: TNS:no listener

    ns secondary err code: 12560

    nt main err code: 511

   

TNS-00511: No listener

    nt secondary err code: 111

    nt OS err code: 0

Error 12541 received logging on to the standby

Check whether the listener is up and running.

PING[ARC2]: Heartbeat failed to connect to standby 'vlifesb'. Error is 12541.

 

 

強制日誌切換。

 

 

SQL> alter system switch logfile;

System altered

 

SQL> alter system switch logfile;

System altered

 

SQL> alter system switch logfile;

System altered

 

 

SQL> select * from v$archive_dest_status;

 

   DEST_ID DEST_NAME            STATUS    TYPE           DATABASE_MODE   RECOVERY_MODE           PROTECTION_MODE      DESTINATION                                                                      STANDBY_LOGFILE_COUNT STANDBY_LOGFILE_ACTIVE ARCHIVED_THREAD# ARCHIVED_SEQ# APPLIED_THREAD# APPLIED_SEQ# ERROR                                                                            SRL DB_UNIQUE_NAME                 SYNCHRONIZATION_STATUS SYNCHRONIZED GAP_STATUS

---------- -------------------- --------- -------------- --------------- ----------------------- -------------------- -------------------------------------------------------------------------------- --------------------- ---------------------- ---------------- ------------- --------------- ------------ -------------------------------------------------------------------------------- --- ------------------------------ ---------------------- ------------ ------------------------

         1 LOG_ARCHIVE_DEST_1   VALID     LOCAL          OPEN            IDLE                    MAXIMUM PERFORMANCE  /u01/app/oracle/product/11.2.0/dbhome_1/dbs/arch                                                     0                      0                1            85               0            0                                                                                  NO  NONE                           CHECK CONFIGURATION    NO          

         2 LOG_ARCHIVE_DEST_2   ERROR     PHYSICAL       OPEN_READ-ONLY  IDLE                    RESYNCHRONIZATION    vlifesb                                                                                              3                      0                1            82               1           82 ORA-12541: TNS: ???à?????ò                                                       YES vlifesb                        CHECK CONNECTIVITY     NO           RESOLVABLE GAP

 

 

SQL> select recid, name, sequence# from v$archived_log where sequence#>82;

 

     RECID NAME                                                                              SEQUENCE#

---------- -------------------------------------------------------------------------------- ----------

       108 vlifesb                                                                                  83

       109 /u01/app/oracle/fast_recovery_area/VLIFE/archivelog/2015_10_21/o1_mf_1_83_c2ghkz         83

       110 /u01/app/oracle/fast_recovery_area/VLIFE/archivelog/2015_10_21/o1_mf_1_84_c2ghl0         84

       111 /u01/app/oracle/fast_recovery_area/VLIFE/archivelog/2015_10_21/o1_mf_1_85_c2ghl4         85

 

 

standby端,歸檔日誌就沒有傳輸到。

 

 

SQL> select recid, name, sequence# from v$archived_log where sequence#>82;

 

     RECID NAME                                                                              SEQUENCE#

---------- -------------------------------------------------------------------------------- ----------

        79 /u01/app/oracle/fast_recovery_area/VLIFESB/archivelog/2015_10_21/o1_mf_1_83_c2gh         83

 

SQL> select group#, dbid, archived from v$standby_log;

 

    GROUP# DBID                                     ARCHIVED

---------- ---------------------------------------- --------

         4 UNASSIGNED                               NO

         5 UNASSIGNED                               NO

         6 UNASSIGNED                               YES

 

 

此時,PrimaryStandby的連線明顯被中斷,日誌不能傳送,也就達不到同步確認的設定要求。但是此時,Primary還是可以進行事務操作。

 

 

 

(事務可以進行)

SQL> insert into t_m select * from dba_objects where rownum<10;

9 rows inserted

 

SQL> commit;

Commit complete

 

 

此時,如果恢復兩者連線,啟動監聽器和日誌應用。

 

 

[oracle@vLIFE-URE-OT-DB-STANDBY ~]$ lsnrctl start

 

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 21-OCT-2015 15:51:46

 

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

 

Starting /u01/app/oracle/product/11.2.0/dbhome_1/bin/tnslsnr: please wait...

 

TNSLSNR for Linux: Version 11.2.0.4.0 - Production

System parameter file is /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora

Log messages written to

(篇幅原因,有省略…….

 

SQL> alter database recover managed standby database using current logfile disconnect from session;

Database altered

 

 

之後主庫和從庫日誌上進行歸檔日誌傳輸和後續同步動作,篇幅原因,日誌資訊省略。

 

從上面實驗中,我們可以看到最大可用性模式的核心即使“可用”。所謂可用,即使保證PrimaryStandby整體的可用。如果在日誌傳輸通路順暢,兩者之間會維持嚴格的同步關係,行為類似於最大保護模式。但是,如果連線或者同步動作不能滿足要求,DG是不會終止例項執行,而是退而求其次,進行一種類似最大效能模式的工作方式。


來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/17203031/viewspace-1821550/,如需轉載,請註明出處,否則將追究法律責任。

相關文章