問大家一個(gè)關(guān)于(twemproxy)nutcracker-0.2.4無法自動(dòng)刪除節(jié)點(diǎn)的問題？

zghover

想用twemproxy(nutcracker-0.2.4) 搭建一個(gè)redis集群，但發(fā)現(xiàn)無法自動(dòng)刪除節(jié)點(diǎn)。

版本：
nutcracker-0.2.4

配置文件：
dds:  
  listen: 0.0.0.0:10000         #使用哪個(gè)端口啟動(dòng)Twemproxy  
  redis: true                   #是否是Redis的proxy  
  hash: fnv1a_64                #指定具體的hash函數(shù)  
  distribution: ketama          #具體的hash算法  
  auto_eject_hosts: true        #是否在結(jié)點(diǎn)無法響應(yīng)的時(shí)候臨時(shí)摘除結(jié)點(diǎn)
  timeout: 400                  #超時(shí)時(shí)間（毫秒）  
  server_retry_timeout: 1000    #重試的時(shí)間（毫秒）  
  server_failure_limit: 1       #結(jié)點(diǎn)故障多少次就算摘除掉  
  servers:                      #下面表示所有的Redis節(jié)點(diǎn)（IP:端口號(hào):權(quán)重）  
   - 192.168.98.91:6379:10
    - 192.168.98.93:6379:20
    - 192.168.98.99:6379:60

用-v 11啟動(dòng)服務(wù)器，一遍查看日志。

用一個(gè)客戶端，每隔0.1秒發(fā)送一個(gè)set請(qǐng)求，每個(gè)請(qǐng)求的key是隨機(jī)生成的，開始的時(shí)候一切正常，能分配到3臺(tái)機(jī)器上。
后來我在192.168.98.99上kill掉redis服務(wù)。
發(fā)現(xiàn)日志如下：


31694 [Fri Nov 29 18:49:41 2013] nc_response.c:120 s 10 active 0 is done
31695 [Fri Nov 29 18:49:41 2013] nc_core.c:207 close s 10 '192.168.98.99:6379' on event 0001 eof 1 done 1 rb 244 sb 1291
31696 [Fri Nov 29 18:49:41 2013] nc_stats.c:1125 metric 'server_connections' in pool 0 server 2
31697 [Fri Nov 29 18:49:41 2013] nc_stats.c:1157 decr field 'server_connections' to 0
31698 [Fri Nov 29 18:49:41 2013] nc_stats.c:1125 metric 'server_eof' in pool 0 server 2
31699 [Fri Nov 29 18:49:41 2013] nc_stats.c:1142 incr field 'server_eof' to 1
31700 [Fri Nov 29 18:49:41 2013] nc_server.c:266 server '192.168.98.99:6379:60' failure count 1 limit 1
31701 [Fri Nov 29 18:49:41 2013] nc_server.c:281 update pool 0 'dds' to delete server '192.168.98.99:6379:60' for next 1 secs
31702 [Fri Nov 29 18:49:41 2013] nc_stats.c:1039 metric 'server_ejects' in pool 0
31703 [Fri Nov 29 18:49:41 2013] nc_stats.c:1056 incr field 'server_ejects' to 1
31704 [Fri Nov 29 18:49:41 2013] nc_ketama.c:122 2 of 3 servers are live for pool 0 'dds'
31705 [Fri Nov 29 18:49:41 2013] nc_ketama.c:168 192.168.98.91:6379:6379 weight 10 of 30 pct 0.33333 points per server 104
31706 [Fri Nov 29 18:49:41 2013] nc_ketama.c:168 192.168.98.93:6379:6379 weight 20 of 30 pct 0.66667 points per server 212
31707 [Fri Nov 29 18:49:41 2013] nc_ketama.c:210 updated pool 0 'dds' with 2 of 3 servers live in 13 slots and 316 active points in 3680 slots
31708 [Fri Nov 29 18:49:41 2013] nc_server.c:63 unref conn 0x8c8b4e8 owner 0x8c7ecb0 from '192.168.98.99:6379:60'
31709 [Fri Nov 29 18:49:41 2013] nc_connection.c:267 put conn 0x8c8b4e8
31710 [Fri Nov 29 18:49:41 2013] nc_stats.c:996 skip swap of current 0x8c830f8 shadow 0x8c8f700 as aggregator is busy
31711 [Fri Nov 29 18:49:41 2013] nc_core.c:285 event 0001 on c 8
31712 [Fri Nov 29 18:49:41 2013] nc_message.c:300 get msg 0x8c87248 id 1010 request 1 owner sd 8
31713 [Fri Nov 29 18:49:41 2013] nc_mbuf.c:99 get mbuf 0x8c8b320
31714 [Fri Nov 29 18:49:41 2013] nc_mbuf.c:182 insert mbuf 0x8c8b320 len 0
31715 [Fri Nov 29 18:49:41 2013] nc_connection.c:307 recv on sd 8 35 of 16360

說明twemproxy已經(jīng)更新了ketama（一致性hash算法）的節(jié)點(diǎn)。

客戶端繼續(xù)發(fā)送請(qǐng)求，卻不時(shí)的出現(xiàn)Connection refused的錯(cuò)誤，再查看日志，發(fā)現(xiàn)192.168.98.99又被添加入ketama的節(jié)點(diǎn)中，日志如下：

33598 [Fri Nov 29 18:49:42 2013] nc_ketama.c:122 3 of 3 servers are live for pool 0 'dds'
33599 [Fri Nov 29 18:49:42 2013] nc_ketama.c:168 192.168.98.91:6379:6379 weight 10 of 90 pct 0.11111 points per server 52
33600 [Fri Nov 29 18:49:42 2013] nc_ketama.c:168 192.168.98.93:6379:6379 weight 20 of 90 pct 0.22222 points per server 104
33601 [Fri Nov 29 18:49:42 2013] nc_ketama.c:168 192.168.98.99:6379:6379 weight 60 of 90 pct 0.66667 points per server 320
33602 [Fri Nov 29 18:49:42 2013] nc_ketama.c:210 updated pool 0 'dds' with 3 of 3 servers live in 13 slots and 476 active points in 3680 slots


所有會(huì)有以下錯(cuò)誤：


33978 [Fri Nov 29 18:49:42 2013] nc_server.c:642 key '測(cè)試9' on dist 0 maps to server '192.168.98.99:6379:60'
33979 [Fri Nov 29 18:49:42 2013] nc_util.c:225 malloc(176) at 0x8c8b5a0 @ nc_connection.c:100
33980 [Fri Nov 29 18:49:42 2013] nc_server.c:44 ref conn 0x8c8b5a0 owner 0x8c7ecb0 into '192.168.98.99:6379:60
33981 [Fri Nov 29 18:49:42 2013] nc_connection.c:208 get conn 0x8c8b5a0 client 0
33982 [Fri Nov 29 18:49:42 2013] nc_server.c:452 connect to server '192.168.98.99:6379:60'
33983 [Fri Nov 29 18:49:42 2013] nc_server.c:494 connecting on s 11 to server '192.168.98.99:6379:60'
33984 [Fri Nov 29 18:49:42 2013] nc_message.c:165 insert msg 1082 into tmo rbt with expiry of 400 msec
33985 [Fri Nov 29 18:49:42 2013] nc_stats.c:1125 metric 'in_queue' in pool 0 server 2
33986 [Fri Nov 29 18:49:42 2013] nc_stats.c:1142 incr field 'in_queue' to 1
33987 [Fri Nov 29 18:49:42 2013] nc_stats.c:1125 metric 'in_queue_bytes' in pool 0 server 2
33988 [Fri Nov 29 18:49:42 2013] nc_stats.c:1172 incr by field 'in_queue_bytes' to 33
33989 [Fri Nov 29 18:49:42 2013] nc_stats.c:1125 metric 'requests' in pool 0 server 2
33990 [Fri Nov 29 18:49:42 2013] nc_stats.c:1142 incr field 'requests' to 46
33991 [Fri Nov 29 18:49:42 2013] nc_stats.c:1125 metric 'request_bytes' in pool 0 server 2
33992 [Fri Nov 29 18:49:42 2013] nc_stats.c:1172 incr by field 'request_bytes' to 1324
33993 [Fri Nov 29 18:49:42 2013] nc_request.c:498 forward from c 8 to s 11 req 1082 len 33 type 49 with key '測(cè)試9'
33994 [Fri Nov 29 18:49:42 2013] nc_stats.c:996 skip swap of current 0x8c830f8 shadow 0x8c8f700 as aggregator is busy
33995 [Fri Nov 29 18:49:42 2013] nc_core.c:285 event 0019 on s 11
33996 [Fri Nov 29 18:49:42 2013] nc_core.c:207 close s 11 '192.168.98.99:6379' on event 0019 eof 0 done 0 rb 0 sb 0: Connection refused
33997 [Fri Nov 29 18:49:42 2013] nc_stats.c:1125 metric 'server_err' in pool 0 server 2
33998 [Fri Nov 29 18:49:42 2013] nc_stats.c:1142 incr field 'server_err' to 1
33999 [Fri Nov 29 18:49:42 2013] nc_stats.c:1125 metric 'in_queue' in pool 0 server 2
34000 [Fri Nov 29 18:49:42 2013] nc_stats.c:1157 decr field 'in_queue' to 0
34001 [Fri Nov 29 18:49:42 2013] nc_stats.c:1125 metric 'in_queue_bytes' in pool 0 server 2
34002 [Fri Nov 29 18:49:42 2013] nc_stats.c:1187 decr by field 'in_queue_bytes' to 0
34003 [Fri Nov 29 18:49:42 2013] nc_server.c:376 close s 11 schedule error for req 1082 len 33 type 49 from c 8: Connection refused
34004 [Fri Nov 29 18:49:42 2013] nc_server.c:266 server '192.168.98.99:6379:60' failure count 1 limit 1
34005 [Fri Nov 29 18:49:42 2013] nc_server.c:281 update pool 0 'dds' to delete server '192.168.98.99:6379:60' for next 1 secs
34006 [Fri Nov 29 18:49:42 2013] nc_stats.c:1039 metric 'server_ejects' in pool 0
34007 [Fri Nov 29 18:49:42 2013] nc_stats.c:1056 incr field 'server_ejects' to 2
34008 [Fri Nov 29 18:49:42 2013] nc_ketama.c:122 2 of 3 servers are live for pool 0 'dds'
34009 [Fri Nov 29 18:49:42 2013] nc_ketama.c:168 192.168.98.91:6379:6379 weight 10 of 30 pct 0.33333 points per server 104
34010 [Fri Nov 29 18:49:42 2013] nc_ketama.c:168 192.168.98.93:6379:6379 weight 20 of 30 pct 0.66667 points per server 212
34011 [Fri Nov 29 18:49:42 2013] nc_ketama.c:210 updated pool 0 'dds' with 2 of 3 servers live in 13 slots and 316 active points in 3680 slots
34012 [Fri Nov 29 18:49:42 2013] nc_server.c:63 unref conn 0x8c8b5a0 owner 0x8c7ecb0 from '192.168.98.99:6379:60'

此時(shí)又檢查到192.168.98.99錯(cuò)誤，又把它從ketama中刪除掉，但又來又被加上了，然后一直這樣下去。

請(qǐng)教大家，這是怎么回事？多謝。

zghover

對(duì)該軟件做了一些修改，若有節(jié)點(diǎn)死掉，會(huì)重構(gòu)一致性hash算法(ketama)，也就實(shí)現(xiàn)了自動(dòng)刪除節(jié)點(diǎn)功能。
但若該節(jié)點(diǎn)自動(dòng)恢復(fù)后，如何把該節(jié)點(diǎn)又添加到hash算法中，還在研究中。