服務(wù)器掛了。給看下具體那出問(wèn)題了。

baixianglu

2機(jī) message日志：

1機(jī) message日志：
Sep 26 03:10:10 XZ-XJ-AAA01 syslogd: line 31: unknown priority name "err                                           @loghost"Sep 26 03:10:10 XZ-XJ-AAA01 syslogd: line 32: unknown priority name "err        @59.43.50.100"Sep 26 03:10:10 XZ-XJ-AAA01 syslogd: line 33: unknown priority name "err        @59.43.50.102"Sep 26 03:10:10 XZ-XJ-AAA01 syslogd: line 34: unknown priority name "info       @59.43.50.100"Sep 26 03:10:10 XZ-XJ-AAA01 syslogd: line 35: unknown priority name "info       @59.43.50.102"Sep 26 08:37:56 XZ-XJ-AAA01 su: [ID 810491 auth.crit] 'su root' failed for xznoc on /dev/pts/32Sep 27 10:06:28 XZ-XJ-AAA01 su: [ID 810491 auth.crit] 'su root' failed for xznoc on /dev/pts/24Sep 27 10:09:04 XZ-XJ-AAA01 last message repeated 3 timesSep 27 10:38:28 XZ-XJ-AAA01 e1000g: [ID 801725 kern.info] NOTICE: pciex8086,105e - e1000g[1] : link downSep 27 10:38:28 XZ-XJ-AAA01 e1000g: [ID 801725 kern.info] NOTICE: pciex8086,105e - e1000g[2] : link downSep 27 10:38:30 XZ-XJ-AAA01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 0 (e1000g1) node 1 in troubleSep 27 10:38:30 XZ-XJ-AAA01 llt: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 1 (e1000g2) node 1 in troubleSep 27 10:38:36 XZ-XJ-AAA01 llt: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (e1000g1) node 1 inactive 8 sec (11736191Sep 27 10:38:36 XZ-XJ-AAA01 llt: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (e1000g2) node 1 inactive 8 sec (116911477)Sep 27 10:38:37 XZ-XJ-AAA01 llt: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (e1000g1) node 1 inactive 9 sec (11736191

2主機(jī)vcs日志：

AAA01主機(jī)vcs 日志27日故障前沒(méi)有日志，26日日志如下：
2013/09/26 17:50:52 VCS ERROR V-16-2-13067 (XZ-XJ-AAA02) Agent is calling clean for resource(solid_app) because the resource became OFFLINE unexpectedly, on its own.
2013/09/26 17:50:52 VCS INFO V-16-2-13068 (XZ-XJ-AAA02) Resource(solid_app) - clean completed successfully.
2013/09/26 17:50:52 VCS INFO V-16-1-10307 Resource solid_app (Owner: Unspecified, Group: solid) is offline on XZ-XJ-AAA02 (Not initiated by VCS)
2013/09/26 17:50:52 VCS NOTICE V-16-1-10300 Initiating Offline of Resource solid_ip (Owner: Unspecified, Group: solid) on System XZ-XJ-AAA02
2013/09/26 17:50:52 VCS NOTICE V-16-1-10300 Initiating Offline of Resource solid_mount (Owner: Unspecified, Group: solid) on System XZ-XJ-AAA02
2013/09/26 17:51:58 VCS ERROR V-16-2-13027 (XZ-XJ-AAA02) Resource(nms_net) - monitor procedure did not complete within the expected time.
2013/09/26 19:20:19 VCS INFO V-16-1-53504 VCS Engine Alive message!!
2013/09/26 23:20:20 VCS INFO V-16-1-53504 VCS Engine Alive message!!

27日故障前2主機(jī)上數(shù)據(jù)庫(kù)日志有報(bào)錯(cuò)：
從8點(diǎn)59分開始有creation failed報(bào)錯(cuò)，數(shù)據(jù)庫(kù)開始出現(xiàn)問(wèn)題：
Fri Sep 27 08:59:52 2013
ksvcreate: Process(q001) creation failed
Fri Sep 27 09:00:04 2013
Process startup failed, error stack:
Fri Sep 27 09:00:04 2013
Errors in file /export/home/oracle/ora10g/admin/ora10g/bdump/ora10g_psp0_6501.trc:
ORA-27300: OS system dependent operation:pipe failed with status: 24
ORA-27301: OS failure message: Too many open files
ORA-27302: failure occurred at: skgpspawn2
Fri Sep 27 09:00:05 2013
Process q001 died, see its trace file
Fri Sep 27 09:00:05 2013
ksvcreate: Process(q001) creation failed
Fri Sep 27 09:00:17 2013
Process startup failed, error stack:
Fri Sep 27 09:00:17 2013
Errors in file /export/home/oracle/ora10g/admin/ora10g/bdump/ora10g_psp0_6501.trc:
ORA-27300: OS system dependent operation:pipe failed with status: 24
ORA-27301: OS failure message: Too many open files
ORA-27302: failure occurred at: skgpspawn2
Fri Sep 27 09:00:18 2013
Process q001 died, see its trace file
Fri Sep 27 09:00:18 2013
ksvcreate: Process(q001) creation failed
Fri Sep 27 09:00:30 2013
Process startup failed, error stack:
Fri Sep 27 09:00:30 2013
Errors in file /export/home/oracle/ora10g/admin/ora10g/bdump/ora10g_psp0_6501.trc:
ORA-27300: OS system dependent operation:pipe failed with status: 24
ORA-27301: OS failure message: Too many open files
ORA-27302: failure occurred at: skgpspawn2
Fri Sep 27 09:00:31 2013
Process q001 died, see its trace file
Fri Sep 27 09:00:31 2013
ksvcreate: Process(q001) creation failed
Fri Sep 27 09:00:34 2013
Process startup failed, error stack:



附帶explorer

東方蜘蛛

2013/09/26 17:48:52 VCS WARNING V-16-1-10023 Agent NIC not sending alive messages since Thu Sep 26 17:46:36 2013
2013/09/26 17:48:52 VCS WARNING V-16-1-53025 Agent NIC has faulted; ipm connection was lost; restarting the agent
2013/09/26 17:49:12 VCS ERROR V-16-1-10015 Cannot start /opt/VRTSvcs/bin/NIC/NICAgent please check file
2013/09/26 17:49:12 VCS WARNING V-16-1-10485 Excessive delay between successive calls to GAB heartbeat (20 seconds)
2013/09/26 17:50:52 VCS ERROR V-16-2-13067 (XZ-XJ-AAA02) Agent is calling clean for resource(solid_app) because the resource became OFFLINE unexpectedly, on its own.
檢查下主機(jī)的網(wǎng)絡(luò)連接情況、交換機(jī)情況。。。。。VCS不熟。。。。