環境介紹:
OS:Centos 6.4 64bit
Database:PostgreSQL9.4
Memory:2G
CPU:1核
下載安裝:
在pgfoundry下載pgfincore-v1.1.1.tar.gz,,將原始碼解壓到資料庫原始碼下的contrib下。不要在其github上下載,目前應該有一些bug,最新版本為1.1.1,1.1.2在我試用的時候發現大部分函式無法使用。
其次在我編譯時一直出現如下錯誤:
1 2 3 4 5 6 |
[postgres@localhost pgfincore-1.1.1]$ make /bin/sh: pg_config: command not found make: pg_config: Command not found cp pgfincore.sql pgfincore--1.1.1.sql cp: cannot create regular file `pgfincore--1.1.1.sql': Permission denied make: *** [pgfincore--1.1.1.sql] Erro<code>r 1 |
所以在此我修改了一下Makefile檔案,內容如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# contrib/pgfincore/Makefile MODULE_big = pgfincore OBJS = pgfincore.o EXTENSION = pgfincore DATA = pgfincore--1.1.1.sql pgfincore--unpackaged--1.1.1.sql ifdef USE_PGXS PG_CONFIG = pg_config PGXS := $(shell $(PG_CONFIG) --pgxs) include $(PGXS) else subdir = contrib/pgfincore top_builddir = ../.. include $(top_builddir)/src/Makefile.global include $(top_srcdir)/contrib/contrib-global.mk |
現在進行make,即可:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
[postgres@localhost pgfincore-1.1.1]$ make gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -O2 -fpic -I. -I. -I../../src/include -D_GNU_SOURCE -c -o pgfincore.o pgfincore.c pgfincore.c: In function ‘pgsysconf’: pgfincore.c:227: warning: implicit declaration of function ‘heap_form_tuple’ pgfincore.c:227: warning: assignment makes pointer from integer without a cast pgfincore.c: In function ‘pgfadvise’: pgfincore.c:477: warning: assignment makes pointer from integer without a cast pgfincore.c: In function ‘pgfadvise_loader’: pgfincore.c:710: warning: assignment makes pointer from integer without a cast pgfincore.c: In function ‘pgfincore’: pgfincore.c:1016: warning: assignment makes pointer from integer without a cast gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -g -O2 -fpic -shared -o pgfincore.so pgfincore.o -L../../src/port -L../../src/common -Wl,--as-needed -Wl,-rpath,'/opt/hg3.0/lib',--enable-new-dtags [postgres@localhost pgfincore-1.1.1]$ make install /bin/mkdir -p '/opt/hg3.0/lib/postgresql' /bin/mkdir -p '/opt/hg3.0/share/postgresql/extension' /bin/mkdir -p '/opt/hg3.0/share/postgresql/extension' /usr/bin/install -c -m 755 pgfincore.so '/opt/hg3.0/lib/postgresql/pgfincore.so' /usr/bin/install -c -m 644 pgfincore.control '/opt/hg3.0/share/postgresql/extension/' /usr/bin/install -c -m 644 pgfincore--1.1.1.sql pgfincore--unpackaged--1.1.1.sql '/opt/hg3.0/share/postgresql/extension/' [postgres@localhost pgfincore-1.1.1]$ |
實驗記錄:
1、安裝:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
[postgres@localhost bin]$ ./psql psql (9.4.5) Type "help" for help. postgres=# create extension pgfincore ; CREATE EXTENSION postgres=# select * from pg_extension ; extname | extowner | extnamespace | extrelocatable | extversion | extconfig | extcondition -----------+----------+--------------+----------------+------------+-----------+ -------------- plpgsql | 10 | 11 | f | 1.0 | | pgfincore | 10 | 2200 | t | 1.1.1 | | (2 rows) postgres=# |
2、準備工作:記錄比較使用pgfincore前後的效能差,使用外掛pg_stat_statments記錄每條sql的使用時間。
其次將shared_buffer改為16mb,減少其對pgfincore的影響。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
postgres=# create extension pg_stat_statements ; CREATE EXTENSION postgres=# select * from pg_extension ; extname | extowner | extnamespace | extrelocatable | extversion | ex tconfig | extcondition --------------------+----------+--------------+----------------+------------+--- --------+-------------- plpgsql | 10 | 11 | f | 1.0 | | pgfincore | 10 | 2200 | t | 1.1.1 | | pg_stat_statements | 10 | 2200 | t | 1.2 | | (3 rows) |
3、建立表testmem,分別在使用和不適用pgfincore情況下進行select, update, delete的操作,並對比執行時間,在這裡為了更好的去作比較,建立三張表testmem1,testmem2,testmem3,分別進行三種操作進行對比。並插入相同的資料。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
[postgres@localhost bin]$ ./psql psql (9.4.5) Type "help" for help. postgres=# d No relations found. postgres=# create table testmem1(t1 int, t2 text, t3 text, t4 bigint, t5 text, t6 text, t7 varchar, t8 char(22), t9 numeric);CREATE TABLE postgres=# create table testmem2(t1 int, t2 text, t3 text, t4 bigint, t5 text, t6 text, t7 varchar, t8 char(22), t9 numeric); CREATE TABLE postgres=# create table testmem3(t1 int, t2 text, t3 text, t4 bigint, t5 text, t6 text, t7 varchar, t8 char(22), t9 numeric); CREATE TABLE postgres=# insert into testmem1 select generate_series(1,999999), 'wangshuo'||generate_series(1,999999)::text||random()::text, generate_series(1,999999)::text||'sure',generate_series(1,999999), random()::text,random()::text,random()::text,'liuyuanyuan', generate_series(1,999999); INSERT 0 999999 postgres=# select pg_relation_size('testmem1')/1024/1024||'MB'; ?column? ---------- 166MB (1 row) postgres=# insert into testmem2 select generate_series(1,999999), 'wangshuo'||generate_series(1,999999)::text||random()::text, generate_series(1,999999)::text||'sure',generate_series(1,999999), random()::text,random()::text,random()::text,'liuyuanyuan', generate_series(1,999999); INSERT 0 999999 postgres=# insert into testmem3 select generate_series(1,999999), 'wangshuo'||generate_series(1,999999)::text||random()::text, generate_series(1,999999)::text||'sure',generate_series(1,999999), random()::text,random()::text,random()::text,'liuyuanyuan', generate_series(1,999999); INSERT 0 999999 postgres=# select pg_relation_size('testmem2')/1024/1024||'MB'; ?column? ---------- 166MB (1 row) postgres=# select pg_relation_size('testmem3')/1024/1024||'MB'; ?column? ---------- 166MB (1 row) postgres=# |
三種操作效能對比:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
postgres=# select * from testmem1 where t1 1208; t1 | t2 | t3 | t4 | t5 | t6 | t7 | t8 | t9 --------+------------------------------------+------------+--------+------------ ----------+----------------------+----------------------+----------------------- -+-------- 1209 | wangshuo12090.964191045146435 | 1209sure | 1209 | 0.075409890 152514 | 0.468206159770489 | 0.692714618053287 | liuyuanyuan | 1209 1210 | wangshuo12100.652063825167716 | 1210sure | 1210 | 0.026932121 720165 | 0.802233531605452 | 0.706556385848671 | liuyuanyuan | 1210 1211 | wangshuo12110.724962076637894 | 1211sure | 1211 | 0.732285636 477172 | 0.816857317462564 | 0.868489125277847 | liuyuanyuan | 1211 1212 | wangshuo12120.0478062951005995 | 1212sure | 1212 | 0.190716865 006834 | 0.898683389648795 | 0.537546805106103 | liuyuanyuan | 1212 1213 | wangshuo12130.582098880317062 | 1213sure | 1213 | 0.161297460 086644 | 0.395338968373835 | 0.684920639265329 | liuyuanyuan | 1213 1214 | wangshuo12140.53120110463351 | 1214sure | 1214 | 0.253457680 810243 | 0.428548218682408 | 0.671272001229227 | liuyuanyuan postgres=# update testmem2 set t7 = random()::text where t1 1208; UPDATE 788223 postgres=# delete from testmem3 where t1 1208; DELETE 788223 |
使用pgfincore前:通過查詢表pg_stat_statments檢視三種操作時間:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
postgres=# SELECT query, calls, total_time, rows, 100.0 * shared_blks_hit / nullif(shared_blks_hit + shared_blks_read, 0) AS hit_percent FROM pg_stat_statements ORDER BY total_time DESC; -[ RECORD 1 ]------------------------------------------------------ query | update testmem2 set t7 = random()::text where t1 ?; calls | 1 total_time | 14303.53 rows | 788223 hit_percent | 97.7162513440240383 -[ RECORD 2 ]------------------------------------------------------ query | delete from testmem3 where t1 ?; calls | 1 total_time | 5256.305 rows | 788223 hit_percent | 97.4004941321803582 -[ RECORD 6 ]------------------------------------------------------ query | select * from testmem1 where t1 ?; calls | 1 total_time | 2397.866 rows | 788223 hit_percent | 0.15039714245429336843 |
使用pgfincore後:通過查詢表pg_stat_statments檢視三種操作時間(重啟機器後,重新建庫,重新建表、插資料):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
SELECT query, calls, total_time, rows, 100.0 * shared_blks_hit / nullif(shared_blks_hit + shared_blks_read, 0) AS hit_percent FROM pg_stat_statements ORDER BY total_time DESC; -[ RECORD 1 ]------------------------------------------------------ query | update testmem2 set t7 = random()::text where t1 ?; calls | 1 total_time | 6800.237 rows | 788223 hit_percent | 97.6908892215405358 -[ RECORD 2 ]------------------------------------------------------ query | delete from testmem3 where t1 ?; calls | 1 total_time | 3295.312 rows | 788223 hit_percent | 97.3715873996294009 -[ RECORD 3 ]------------------------------------------------------ query | select * from testmem1 where t1 ?; calls | 1 total_time | 891.002 rows | 788223 hit_percent | 0.000000000000000000000000 |
明顯看出當執行select時速度提升明顯,其他的時間提升並不明顯,這是因為資料還需要進行IO操作,所以在這裡速度提升應該僅僅是讀入的時間節省掉了。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 |
postgres=# select pgfadvise_willneed('testmem1'); -[ RECORD 1 ]------+------------------------------------- pgfadvise_willneed | (base/13003/16384,4096,42554,279311) postgres=# select * from pgfincore('testmem1'); -[ RECORD 1 ]-+----------------- relpath | base/13003/16384 segment | 0 os_page_size | 4096 rel_os_pages | 42554 pages_mem | 42554 group_mem | 1 os_pages_free | 279317 databit | postgres=# select count(*) from testmem1 where t1 12; -[ RECORD 1 ] count | 99986 postgres=# insert into testmem1 select generate_series(1,9999), 'wangshuo'||generate_series(1,9999)::text||random()::text, generate_series(1,9999)::text||'sure',generate_series(1,9999), random()::text,random()::text,random()::text,'liuyuanyuan', generate_series(1,9999); INSERT 0 9999 postgres=# select * from pgfincore('testmem1');-[ RECORD 1 ]-+----------------- relpath | base/13003/16384 segment | 0 os_page_size | 4096 rel_os_pages | 42980 pages_mem | 42980 group_mem | 1 os_pages_free | 277433 databit | postgres=# select pg_relation_size('testmem1'); -[ RECORD 1 ]----+---------- pg_relation_size | 176046080 postgres=# select 42980*4096; -[ RECORD 1 ]------- ?column? | 176046080 postgres=# insert into testmem1 select generate_series(1,999999), 'wangshuo'||generate_series(1,999999)::text||random()::text, generate_series(1,999999)::text||'sure',generate_series(1,999999), random()::text,random()::text,random()::text,'liuyuanyuan', generate_series(1,999999); INSERT 0 999999 postgres=# select * from pgfincore('testmem1'); -[ RECORD 1 ]-+----------------- relpath | base/13003/16384 segment | 0 os_page_size | 4096 rel_os_pages | 85532 pages_mem | 6188 group_mem | 22 os_pages_free | 403050 databit | postgres=# |
上面這個實驗驗證了willneed會將小部分新加入的資料載入到cache中,能夠快速的定位查詢,但是當插入資料量較大時,就不會及時的寫入到cache中了。
總結:pgfincore目前適用於資料量非常大、更新不頻繁、更新量較小的表去載入到快取中,有助於提高應用效率。
外掛函式介紹:
這裡會對函式進行介紹,並對所有函式進行操作實驗。
1、pgsysconf:
這個函式輸出OS block的大小(os_page_size),OS中剩餘的page數(os_pages_free)和OS擁有的page總數(os_total_pages)。
1 2 3 4 5 |
postgres=# select * from pgsysconf(); os_page_size | os_pages_free | os_total_pages --------------+---------------+---------------- 4096 | 167445 | 476688 (1 row) |
2、pgsysconf_pretty:
這個函式和上面的一樣,僅僅是輸出更易懂些。
1 2 3 4 5 |
postgres=# select * from pgsysconf_pretty(); os_page_size | os_pages_free | os_total_pages --------------+---------------+---------------- 4096 bytes | 654 MB | 1862 MB (1 row) |
3、pgfadvise_willneed:
這個函式是
輸出檔名(relpath),OS block大小(os_page_size),物件佔用系統page數(rel_os_pages),OS剩餘的page數(os_pages_free)。
1 2 3 4 5 |
postgres=# select * from pgfadvise_willneed('testmem1'); relpath | os_page_size | rel_os_pages | os_pages_free ------------------+--------------+--------------+--------------- base/13003/16384 | 4096 | 42554 | 167294 (1 row) |
4、pgfadvise_dontneed:
這個函式對當前物件設定dontneed標記。dontneed標記的意思就是當作業系統需要釋放記憶體時優先釋放標記為dontneed的pages。輸出的意義和上面一致,就不多做介紹了。
1 2 3 4 5 |
postgres=# select * from pgfadvise_dontneed('testmem1'); relpath | os_page_size | rel_os_pages | os_pages_free ------------------+--------------+--------------+--------------- base/13003/16384 | 4096 | 42554 | 332798 (1 row) |
5、pgfadvise_NORMAL、pgfadvise_SEQUENTIAL、pgfadvise_RANDOM:
和pgfadvise_dontneed大同小異,分別將物件標記為normal、sequential、random,意思按照字面意思理解就可以了。
其實pgfadvise是一個單獨的函式,引數是relname,fork,action,含義分別為物件名、檔案分支名以及行為id,上面的3、4、5函式都是輸入不同引數而設定的函式,其中willneed、dontneed、normal、sequential、random的值分別是10、20、30、40、50,且預設fork為main,即表檔案。
這裡的pgfadvise主要呼叫了Linux下的函式posix_fadvise,標記值也是posix_fadvise所需要的。
6、pgfincore:
這個函式是是提供物件在作業系統快取中的資訊的。
它分為三個函式,引數分別為(relname, fork, getdatabit),(relname, getdatabit),(relname),三個引數意思為物件名,程式名(這個地方預設是main),是否要顯示databit(很長,注意顯示),第一個函式需要全部輸入,第二個函式預設fork為main,第三個函式預設fork為main,getdatabit為false。
它輸出的是檔案位置及名稱(relpath),檔案順序(segment),OS page或block大小(os_page_size),物件佔用系統快取需要的頁面個數(rel_os_pages),物件已經佔用快取頁面個數(pages_mem),在快取中連續的頁面組的個數(group_mem),OS剩餘的page數(os_pages_free),載入資訊的點陣圖(databit)。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
postgres=# select * from pgfincore('testmem1', 'main', false); -[ RECORD 1 ]-+----------------- relpath | base/13003/16384 segment | 0 os_page_size | 4096 rel_os_pages | 42554 pages_mem | 2 group_mem | 1 os_pages_free | 325475 databit | postgres=# select * from pgfincore('testmem1', false); -[ RECORD 1 ]-+----------------- relpath | base/13003/16384 segment | 0 os_page_size | 4096 rel_os_pages | 42554 pages_mem | 2 group_mem | 1 os_pages_free | 325475 databit | postgres=# select * from pgfincore('testmem1'); -[ RECORD 1 ]-+----------------- relpath | base/13003/16384 segment | 0 os_page_size | 4096 rel_os_pages | 42554 pages_mem | 2 group_mem | 1 os_pages_free | 325475 databit | |
7、pgfadvise_loader:
這個函式可以對頁面快取直接進行兩方面的作用。因此,它能通過頁面的點陣圖在快取中來對頁面進行載入或解除安裝。
它分為兩個函式和上邊的類似,就是設定預設值,的輸入引數是(relname, fork, segment, load, unload, databit)和(relname, segment, load, unload, databit),分別是物件名,檔案分支名,檔案序號,是否載入,是否解除安裝,點陣圖資訊。第二個函式預設fork為main。
它輸出的是物理檔名及path(relpath),OS page或block大小(os_page_size), OS中剩餘的page數(os_pages_free),載入的page數(pages_load),解除安裝的page數(pages_unload)。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
postgres=# select * from pgfincore('testmem1'); -[ RECORD 1 ]-+----------------- relpath | base/13003/16384 segment | 0 os_page_size | 4096 rel_os_pages | 42554 pages_mem | 0 group_mem | 0 os_pages_free | 331290 databit | postgres=# select * from pgfadvise_loader('testmem1', 0, true, true, B'1100'); -[ RECORD 1 ]--+----------------- relpath | base/13003/16384 os_page_size | 4096 os_pages_free | 331294 pages_loaded | 2 pages_unloaded | 2 postgres=# select * from pgfincore('testmem1'); -[ RECORD 1 ]-+----------------- relpath | base/13003/16384 segment | 0 os_page_size | 4096 rel_os_pages | 42554 pages_mem | 2 group_mem | 1 os_pages_free | 331290 databit | |