Redis最大clients數研究
背景
Redis的預設 maxclients 的大小是 10000
之前有同事問過, 應用伺服器比較多的情況下 這個值會比較高.
當時設定到了 15000
最近突然想, redis 的最大值能夠大多少.
昨天驗證了 nginx的一些情況, 今天想繼續驗證redis的情況
結論
Redis 可以超過10萬個以上的clients
他並不會收到 64k個埠號的限制.
因為跟nginx一樣, 限制是基於TCP四元組來的.
但是clients 多了之後, 壓測試CPU的效能消耗是非常高的.
當使用 8萬個client進行壓測時.
發現 rps最高合計到了 12萬.
單獨使用 一個機器的2萬個client進行壓測時
發現 rps 可以到 7萬多.
但是同時 8萬個 clients 時 CPU 長時間 100% 平均響應時間 300ms
如果是 2萬個 clients 時 CPU 大概是在 60% 平均響應時間:70ms
引數配置
maxcliets 120000
requirepass Somepassword
daemonize yes
多個機器進行驗證
將 redis-benchmark 放到 測試機器的 /usr/bin 目錄下
然後執行命令:
redis-benchmark -h 172.24.110.186 -p 6379 -a MyPassword -n 1000000 -c 30000 -d 256 -q
然後檢視redsi的client資訊
[root@ALLDB redis]# redis-cli -a MyPassword info clients
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Clients
connected_clients:115749
cluster_connections:0
maxclients:120000
client_recent_max_input_buffer:20480
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0
也就是說可以超過 65536 的限制.
但是效能可能會下降比較厲害
預設引數設定
sysctl -w net.ipv4.ip_local_port_range="15000 64000"
sysctl -w net.ipv4.tcp_max_tw_buckets=5000
sysctl -w net.ipv4.tcp_tw_reuse=1
降低極限進行測試
四臺機器同事進行測試
redis-benchmark -h 172.24.110.186 -p 6379 -a MyPassword -n 1000000000 -c 20000 -d 256 -q
CPU使用情況
top - 09:25:44 up 1 day, 16:26, 1 user, load average: 0.23, 0.15, 0.10
Threads: 6 total, 1 running, 5 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.6 us, 9.6 sy, 0.0 ni, 72.0 id, 0.0 wa, 0.0 hi, 16.8 si, 0.0 st
KiB Mem : 64757916 total, 1879036 free, 25873224 used, 37005656 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 30984120 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25572 root 20 0 1671136 445244 2224 R 99.9 0.7 1:20.88 redis-server
25573 root 20 0 1671136 445244 2224 S 0.0 0.7 0:00.00 bio_close_file
25574 root 20 0 1671136 445244 2224 S 0.0 0.7 0:00.00 bio_aof
25575 root 20 0 1671136 445244 2224 S 0.0 0.7 0:00.00 bio_lazy_free
25576 root 20 0 1671136 445244 2224 S 0.0 0.7 0:00.09 jemalloc_bg_thd
25577 root 20 0 1671136 445244 2224 S 0.0 0.7 0:00.00 jemalloc_bg_thd
網路連線情況:
netstat -ano |grep 6379 |awk '{print $6}' |sort |uniq -c
78601 ESTABLISHED
2 LISTEN
四臺伺服器 8萬個client的壓測資料
PING_INLINE: rps=28660.0 (overall: 32883.0) avg_msec=308.951 (overall: 307.907)
PING_INLINE: rps=40567.5 (overall: 32775.8) avg_msec=387.327 (overall: 312.077)
PING_INLINE: rps=52361.5 (overall: 32663.5) avg_msec=177.727 (overall: 312.621)
PING_INLINE: rps=73640.3 (overall: 32782.0) avg_msec=69.453 (overall: 308.377)
如果設定到一臺機器上面只有 20000個clients 時
PING_INLINE: rps=72059.9 (overall: 72334.7) avg_msec=70.371 (overall: 70.676)
一個壓力機器時的情況:
[root@ALLDB redis]# top -Hp 25572 -bn 1
top - 09:43:03 up 1 day, 16:43, 1 user, load average: 0.22, 0.66, 0.50
Threads: 6 total, 1 running, 5 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.5 us, 4.2 sy, 0.0 ni, 86.6 id, 0.0 wa, 0.0 hi, 6.7 si, 0.0 st
KiB Mem : 64757916 total, 2043384 free, 25663996 used, 37050536 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 31193388 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25572 root 20 0 1943520 490168 2396 R 53.3 0.8 12:24.22 redis-server
25573 root 20 0 1943520 490168 2396 S 0.0 0.8 0:00.00 bio_close_file
25574 root 20 0 1943520 490168 2396 S 0.0 0.8 0:00.00 bio_aof
25575 root 20 0 1943520 490168 2396 S 0.0 0.8 0:00.00 bio_lazy_free
25576 root 20 0 1943520 490168 2396 S 0.0 0.8 0:00.39 jemalloc_bg_thd
25577 root 20 0 1943520 490168 2396 S 0.0 0.8 0:00.00 jemalloc_bg_thd
Redis的整體監控情況
# Server
redis_version:7.2.5
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:14927caef46752a5
redis_mode:standalone
os:Linux 3.10.0-1160.el7.x86_64 x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:4.8.5
process_id:25572
process_supervised:no
run_id:215cc7957b47242f0590b1600f862f275391696e
tcp_port:6379
server_time_usec:1718501727403450
uptime_in_seconds:1276
uptime_in_days:0
hz:500
configured_hz:10
lru_clock:7225695
executable:/redis/./redis-server
config_file:/redis/redis.conf
io_threads_active:0
listener0:name=tcp,bind=*,bind=-::*,port=6379
如上是基本資訊.
# Clients
connected_clients:78602
cluster_connections:0
maxclients:120000
client_recent_max_input_buffer:20480
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0
這些事 client的基本資訊
# Memory
used_memory:1670916880
used_memory_human:1.56G
used_memory_rss:776372224
used_memory_rss_human:740.41M
used_memory_peak:1692626856
used_memory_peak_human:1.58G
used_memory_peak_perc:98.72%
used_memory_overhead:900971680
used_memory_startup:7141008
used_memory_dataset:769945200
used_memory_dataset_perc:46.28%
allocator_allocated:1673267288
allocator_active:1673748480
allocator_resident:1703129088
total_system_memory:66312105984
total_system_memory_human:61.76G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:481192
allocator_rss_ratio:1.02
allocator_rss_bytes:29380608
rss_overhead_ratio:0.46
rss_overhead_bytes:-926756864
mem_fragmentation_ratio:0.46
mem_fragmentation_bytes:-896829440
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:893830192
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.3.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
預設沒有驅逐策略, 其實是需要修改一下的.
# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1718501626
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:1
rdb_current_bgsave_time_sec:-1
rdb_saves:9
rdb_last_cow_size:84361216
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:5
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:1094144
total_commands_processed:43051884
instantaneous_ops_per_sec:96746
total_net_input_bytes:2287393296
total_net_output_bytes:815541572
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:592.46
instantaneous_output_kbps:651.64
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:34
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:2010123
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:10069
total_forks:9
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:43654130
total_writes_processed:42597790
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:997287
reply_buffer_expands:0
eventloop_cycles:6032609
eventloop_duration_sum:337303802
eventloop_duration_cmd_sum:10437452
instantaneous_eventloop_cycles_per_sec:10870
instantaneous_eventloop_duration_usec:124601
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:4a04cf5a7396791ae7361b7fdec40bc906b24d71
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:287.111681
used_cpu_user:87.725018
used_cpu_sys_children:0.715498
used_cpu_user_children:2.248074
used_cpu_sys_main_thread:286.880444
used_cpu_user_main_thread:87.613899
# Modules
# Errorstats
# Cluster
cluster_enabled:0
# Keyspace