求助：RHEL HA節(jié)點(diǎn)2運(yùn)行幾分鐘后就被節(jié)點(diǎn)1 fence重啟

hp_star

fence使用的是fence_ipmilan設(shè)備，兩臺(tái)機(jī)器的心跳網(wǎng)卡和fence設(shè)備在同一個(gè)網(wǎng)絡(luò)中。

以下是/etc/hosts的內(nèi)容：
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

########### product IP ##########
192.168.10.20   vip
192.168.10.21   db1
192.168.10.22   db2

########### heartbeat IP of cluster #############
192.168.20.1    node1
192.168.20.2    node2

########### iLO IP of nodes ###########
192.168.20.3    node1ilo
192.168.20.4    node2ilo

節(jié)點(diǎn)1的Message日志：
Jul 14 22:48:27 db1 corosync[4336]:   [TOTEM ] A processor failed, forming new configuration.
Jul 14 22:48:29 db1 corosync[4336]:   [QUORUM] Members[1]: 1
Jul 14 22:48:29 db1 corosync[4336]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Jul 14 22:48:29 db1 kernel: dlm: closing connection to node 2
Jul 14 22:48:29 db1 corosync[4336]:   [CPG   ] chosen downlist: sender r(0) ip(192.168.20.1) ; members(old:2 left:1)
Jul 14 22:48:29 db1 corosync[4336]:   [MAIN  ] Completed service synchronization, ready to provide service.
Jul 14 22:48:29 db1 fenced[4387]: fence node2 success

以下是節(jié)點(diǎn)1上的fenced.log部分內(nèi)容：
Jul 14 21:46:28 fenced fencing node node2
Jul 14 21:46:43 fenced fence node2 success
Jul 14 21:54:35 fenced fencing node node2
Jul 14 21:54:51 fenced fence node2 success
Jul 14 22:02:41 fenced fencing node node2
Jul 14 22:02:57 fenced fence node2 success
Jul 14 22:45:40 fenced fenced 3.0.12.1 started
Jul 14 22:48:06 fenced fencing node node2
Jul 14 22:48:07 fenced fence node2 dev 0.0 agent fence_ipmilan result: error from agent
Jul 14 22:48:07 fenced fence node2 failed
Jul 14 22:48:10 fenced fencing node node2
Jul 14 22:48:29 fenced fence node2 success
Jul 14 22:48:29 fenced telling cman to remove nodeid 2 from cluster
Jul 14 22:55:06 fenced fencing node node2
Jul 14 22:55:22 fenced fence node2 success
Jul 14 23:02:02 fenced fencing node node2
Jul 14 23:02:18 fenced fence node2 success
Jul 14 23:08:59 fenced fencing node node2

我現(xiàn)在懷疑兩個(gè)方面：
1是心跳網(wǎng)絡(luò)關(guān)閉了廣播
2是兩個(gè)節(jié)點(diǎn)的系統(tǒng)時(shí)間不一致

請(qǐng)問(wèn)第一點(diǎn)如何測(cè)試一下，是用 ping xxx.xxx.xxx.255命令嗎？
第二點(diǎn)會(huì)導(dǎo)致fencing嗎？

hp_star

抱歉問(wèn)題還是存在，再描述一下問(wèn)題：

兩臺(tái)機(jī)器的cman和rgmanager 服務(wù)都是開(kāi)機(jī)自動(dòng)啟動(dòng)。
沒(méi)有配quorum disk，fence使用的fence_ipmi

在集群正常運(yùn)行時(shí)直接執(zhí)行shutdown -ry 0重啟，服務(wù)可以切換到另一個(gè)節(jié)點(diǎn)，但是等這個(gè)節(jié)點(diǎn)啟動(dòng)后過(guò)幾分鐘就會(huì)被正常運(yùn)行的節(jié)點(diǎn)fence，所以它就一直在重啟》被fence》重啟的循環(huán)。

不過(guò)發(fā)現(xiàn)重啟前先將cman服務(wù)和rgmanager都停掉，或執(zhí)行ccs -h host --stop，再重啟，系統(tǒng)起來(lái)后就不會(huì)被fence。

以下是我測(cè)試multicast address的結(jié)果，我好像是網(wǎng)絡(luò)的問(wèn)題：

[root@node2 ~]# tcpdump -i eth2 host 239.192.145.113
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes
16:56:23.356670 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
16:56:34.784205 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
16:56:46.211659 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
16:56:57.639938 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
16:57:22.270343 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 75
16:57:22.270936 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 318
16:57:22.271781 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 414
16:57:22.273190 IP node2.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 414
16:57:22.291402 IP node2.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 75
16:57:22.293069 IP node2.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 446
16:57:22.294785 IP node2.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 550
16:57:22.295187 IP node2.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 414
16:57:22.304720 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 550
16:57:22.305397 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 454
16:57:22.515179 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
16:57:28.722255 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 75
16:57:28.722681 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 171
16:57:28.774519 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 75
16:57:28.774794 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 201
16:57:28.775175 IP node2.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 201
16:57:28.775665 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 313
16:57:28.776023 IP node2.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 467
16:57:28.776521 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 201
16:57:28.987090 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
16:57:40.419938 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
16:57:51.847535 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
16:58:03.275718 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
16:58:14.703557 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
16:58:26.131173 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
16:58:37.558770 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
16:58:48.986406 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119

16:59:00.414809 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
16:59:11.842508 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
16:59:23.270324 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
16:59:34.698492 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
16:59:46.126171 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
16:59:57.553608 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
17:00:08.980664 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
17:00:20.411082 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
17:00:31.844305 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
17:00:43.277512 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
17:00:54.711376 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
17:01:06.144642 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
17:01:17.577841 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
17:01:29.010987 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
17:01:40.444306 IP node1.hpoms-dps-lstn > 239.192.145.113.netsupport: UDP, length 119
17:01:51.877522 IP node1.hpoms-dp

hp_star

問(wèn)題已經(jīng)解決。

是rc.local在搞鬼。
[root@xsl-rms-database1 ~]# cat /etc/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

/etc/init.d/network restart
touch /var/lock/subsys/local

開(kāi)機(jī)cman啟動(dòng)后，rc.local重啟所有網(wǎng)卡，造成心跳斷了一段時(shí)間（打開(kāi)20-30秒），當(dāng)然會(huì)發(fā)生fence了。