cluster下 包無法正常切換

weixiaoma0

cluster下 包無法正常切換
跪求解，什么問題，怎么分析日志

# ls
ora_start.sh   ora_stop.sh    oracle.asc     oracle.sh      oracle.sh.log
# cat oracle.sh.log

        ########### Node "mq1": Starting package at Tue Aug 19 18:58:39 EAT 2008 ###########
Aug 19 18:58:39 - Node "mq1": Activating volume group vgmqm with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "vgmqm" has been successfully changed.
Aug 19 18:58:40 - Node "mq1": Activating volume group vgoradata with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "vgoradata" has been successfully changed.
Aug 19 18:58:40 - Node "mq1": Checking filesystems:
   /dev/vgmqm/lvol1
   /dev/vgoradata/lvol1
fsck: /etc/default/fs is used for determining the file system type
file system is clean - log replay is not required
fsck: /etc/default/fs is used for determining the file system type
file system is clean - log replay is not required
Aug 19 18:58:40 - Node "mq1": Mounting /dev/vgmqm/lvol1 at /var/mqm
Aug 19 18:58:40 - Node "mq1": Mounting /dev/vgoradata/lvol1 at /ora_data
Aug 19 18:58:40 - Node "mq1": Adding IP address 172.16.2.194 to subnet 172.16.2.0
ttytype: couldn't open /dev/tty for reading
stty: : Not a typewriter
(c)Copyright 1983-2006 Hewlett-Packard Development Company, L.P.
(c)Copyright 1979, 1980, 1983, 1985-1993 The Regents of the Univ. of California
(c)Copyright 1980, 1984, 1986 Novell, Inc.
(c)Copyright 1986-2000 Sun Microsystems, Inc.
(c)Copyright 1985, 1986, 1988 Massachusetts Institute of Technology
(c)Copyright 1989-1993  The Open Software Foundation, Inc.
(c)Copyright 1990 Motorola, Inc.
(c)Copyright 1990, 1991, 1992 Cornell University
(c)Copyright 1989-1991 The University of Maryland
(c)Copyright 1988 Carnegie Mellon University
(c)Copyright 1991-2006 Mentat Inc.
(c)Copyright 1996 Morning Star Technologies, Inc.
(c)Copyright 1996 Progressive Systems, Inc.

Confidential computer software. Valid license from HP required for
possession, use or copying.  Consistent with FAR 12.211 and 12.212,
Commercial Computer Software, Computer Software Documentation, and
Technical Data for Commercial Items are licensed to the U.S. Government
under vendor's standard commercial license.


SQL*Plus: Release 10.2.0.3.0 - Production on 星期二 8月 19 18:58:43 2008

Copyright (c) 1982, 2006, Oracle.  All Rights Reserved.

SQL> SQL> Connected to an idle instance.
SQL> ORACLE instance started.

Total System Global Area 1610612736 bytes
Fixed Size                  2045752 bytes
Variable Size             385878216 bytes
Database Buffers         1207959552 bytes
Redo Buffers               14729216 bytes
Database mounted.
# cd ../oracle1
# ls
ora_start.sh   ora_stop.sh    oracle.asc     oracle.sh      oracle.sh.log
# cat oracle.sh.log

        ########### Node "mq1": Starting package at Mon Jun  9 13:24:51 EAT 2008 ###########
Jun  9 13:24:51 - Node "mq1": Activating volume group vgoradata with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "vgoradata" has been successfully changed.
Jun  9 13:24:51 - Node "mq1": Checking filesystems:
   /dev/vgoradata/lvol1
   /dev/vgoradata/lvol2
fsck: /etc/default/fs is used for determining the file system type
file system is clean - log replay is not required
fsck: /etc/default/fs is used for determining the file system type
file system is clean - log replay is not required
Jun  9 13:24:51 - Node "mq1": Mounting /dev/vgoradata/lvol1 at /oradata01
Jun  9 13:24:52 - Node "mq1": Mounting /dev/vgoradata/lvol2 at /oradata02
Jun  9 13:24:52 - Node "mq1": Adding IP address 172.16.2.194 to subnet 172.16.2.0
.ICEauthority
.TTauthority
.Xauthority
.dt
.dtprofile
.firefox-license
.fonts.cache-1
.mozilla
.profile
.rhosts
.sh_history
.ssh
.sw
bin
cdrom
dev
etc
home
lib
lost+found
net
opt
oracle
oradata01
oradata02
sbin
stand
tmp
tmpho
usr
var

        ########### Node "mq1": Package start completed at Mon Jun  9 13:24:52 EAT 2008 ###########

        ########### Node "mq1": Halting package at Mon Jun  9 13:26:21 EAT 2008 ###########
.ICEauthority
.TTauthority
.Xauthority
.dt
.dtprofile
.firefox-license
.fonts.cache-1
.mozilla
.profile
.rhosts
.sh_history
.ssh
.sw
bin
cdrom
dev
etc
home
lib
lost+found
net
opt
oracle
oradata01
oradata02
sbin
stand
tmp
tmpho
usr
var
Jun  9 13:26:21 - Node "mq1": Remove IP address 172.16.2.194 from subnet 172.16.2.0
Jun  9 13:26:21 - Node "mq1": Unmounting filesystem on /dev/vgoradata/lvol2
Jun  9 13:26:21 - Node "mq1": Unmounting filesystem on /dev/vgoradata/lvol1
Jun  9 13:26:21 - Node "mq1": Deactivating volume group vgoradata
Deactivated volume group in Exclusive Mode.
Volume group "vgoradata" has been successfully changed.

        ########### Node "mq1": Package halt completed at Mon Jun  9 13:26:24 EAT 2008 ###########

        ########### Node "mq1": Starting package at Mon Jun  9 14:13:43 EAT 2008 ###########
Jun  9 14:13:43 - Node "mq1": Activating volume group vgoradata with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "vgoradata" has been successfully changed.
Jun  9 14:13:43 - Node "mq1": Checking filesystems:
   /dev/vgoradata/lvol1
   /dev/vgoradata/lvol2
fsck: /etc/default/fs is used for determining the file system type
file system is clean - log replay is not required
fsck: /etc/default/fs is used for determining the file system type
file system is clean - log replay is not required
Jun  9 14:13:43 - Node "mq1": Mounting /dev/vgoradata/lvol1 at /oradata01
Jun  9 14:13:43 - Node "mq1": Mounting /dev/vgoradata/lvol2 at /oradata02
Jun  9 14:13:44 - Node "mq1": Adding IP address 172.16.2.194 to subnet 172.16.2.0
.ICEauthority
.TTauthority
.Xauthority
.dt
.dtprofile
.firefox-license
.fonts.cache-1
.mozilla
.profile
.rhosts
.sh_history
.ssh
.sw
bin
cdrom
dev
etc
home
lib
lost+found
net
opt
oracle
oradata01
oradata02
sbin
stand
tmp
tmpho
usr
var

        ########### Node "mq1": Package start completed at Mon Jun  9 14:13:44 EAT 2008 ###########

        ########### Node "mq1": Halting package at Mon Jun  9 16:44:26 EAT 2008 ###########
ttytype: couldn't open /dev/tty for reading
stty: : Not a typewriter
(c)Copyright 1983-2006 Hewlett-Packard Development Company, L.P.
(c)Copyright 1979, 1980, 1983, 1985-1993 The Regents of the Univ. of California
(c)Copyright 1980, 1984, 1986 Novell, Inc.
(c)Copyright 1986-2000 Sun Microsystems, Inc.
(c)Copyright 1985, 1986, 1988 Massachusetts Institute of Technology
(c)Copyright 1989-1993  The Open Software Foundation, Inc.
(c)Copyright 1990 Motorola, Inc.
(c)Copyright 1990, 1991, 1992 Cornell University
(c)Copyright 1989-1991 The University of Maryland
(c)Copyright 1988 Carnegie Mellon University
(c)Copyright 1991-2006 Mentat Inc.
(c)Copyright 1996 Morning Star Technologies, Inc.
(c)Copyright 1996 Progressive Systems, Inc.

Confidential computer software. Valid license from HP required for
possession, use or copying.  Consistent with FAR 12.211 and 12.212,
Commercial Computer Software, Computer Software Documentation, and
Technical Data for Commercial Items are licensed to the U.S. Government
under vendor's standard commercial license.

Processing Database instance "mbcenter": log file /ora_app/ora10g/product/10.2.0//shutdown.log
logout
Jun  9 16:44:26 - Node "mq1": Remove IP address 172.16.2.194 from subnet 172.16.2.0
Jun  9 16:44:27 - Node "mq1": Unmounting filesystem on /dev/vgoradata/lvol1
Jun  9 16:44:28 - Node "mq1": Deactivating volume group vgoradata
Deactivated volume group in Exclusive Mode.
Volume group "vgoradata" has been successfully changed.

        ########### Node "mq1": Package halt completed at Mon Jun  9 16:44:30 EAT 2008 ###########

        ########### Node "mq1": Starting package at Mon Jun  9 17:19:31 EAT 2008 ###########
Jun  9 17:19:31 - Node "mq1": Activating volume group vgoradata with exclusive option.
vgchange: Activation mode requested for the volume group "/dev/vgoradata" conflicts with configured mode.
        ERROR:  Function activate_volume_group
        ERROR:  Failed to activate vgoradata
Jun  9 17:19:31 - Node "mq1": Deactivating volume group vgoradata
Volume group "vgoradata" has been successfully changed.

        ########### Node "mq1": Package start failed at Mon Jun  9 17:19:34 EAT 2008 ###########

        ########### Node "mq1": Starting package at Mon Jun  9 17:20:19 EAT 2008 ###########
Jun  9 17:20:19 - Node "mq1": Activating volume group vgoradata with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "vgoradata" has been successfully changed.
Jun  9 17:20:20 - Node "mq1": Checking filesystems:
   /dev/vgoradata/lvol1
   /dev/vgoradata/lvol2
fsck: /etc/default/fs is used for determining the file system type
file system is clean - log replay is not required
fsck: /etc/default/fs is used for determining the file system type
file system is clean - log replay is not required
Jun  9 17:20:20 - Node "mq1": Mounting /dev/vgoradata/lvol1 at /var/mqm
Jun  9 17:20:21 - Node "mq1": Mounting /dev/vgoradata/lvol2 at /oradata01
Jun  9 17:20:21 - Node "mq1": Adding IP address 172.16.2.194 to subnet 172.16.2.0
ttytype: couldn't open /dev/tty for reading
stty: : Not a typewriter
(c)Copyright 1983-2006 Hewlett-Packard Development Company, L.P.
(c)Copyright 1979, 1980, 1983, 1985-1993 The Regents of the Univ. of California
(c)Copyright 1980, 1984, 1986 Novell, Inc.
(c)Copyright 1986-2000 Sun Microsystems, Inc.
(c)Copyright 1985, 1986, 1988 Massachusetts Institute of Technology
(c)Copyright 1989-1993  The Open Software Foundation, Inc.
(c)Copyright 1990 Motorola, Inc.
(c)Copyright 1990, 1991, 1992 Cornell University
(c)Copyright 1989-1991 The University of Maryland
(c)Copyright 1988 Carnegie Mellon University
(c)Copyright 1991-2006 Mentat Inc.
(c)Copyright 1996 Morning Star Technologies, Inc.
(c)Copyright 1996 Progressive Systems, Inc.

Confidential computer software. Valid license from HP required for
possession, use or copying.  Consistent with FAR 12.211 and 12.212,
Commercial Computer Software, Computer Software Documentation, and
Technical Data for Commercial Items are licensed to the U.S. Government
under vendor's standard commercial license.

Processing Database instance "mbcenter": log file /ora_app/ora10g/product/10.2.0//startup.log
logout

        ########### Node "mq1": Package start completed at Mon Jun  9 17:20:25 EAT 2008 ###########

     

lbseraph

1. 時間沒錯吧？日志里怎么是2008年的？
2. 上面日志看到8月19日的應(yīng)該是到啟Oracle那邊了吧，那后面的應(yīng)該看Oracle的日志吧？cmviewcl -v看一下package的狀態(tài)是什么樣的？

weixiaoma0

# cmviewcl -v

CLUSTER        STATUS      
cluster1       up           
  
  NODE           STATUS       STATE        
  mq1            up           running      
   
    Cluster_Lock_LVM:
    VOLUME_GROUP          PHYSICAL_VOLUME       STATUS              
    /dev/vglock           /dev/dsk/c4t0d5       up                  
   
    Network_Parameters:
    INTERFACE    STATUS           PATH                NAME         
    PRIMARY      up               0/3/0/0/0/0/2/0/0/  lan4         
    PRIMARY      up               0/1/1/0             lan0         
    STANDBY      up               0/3/0/0/0/0/4/0/0/  lan6         
    STANDBY      up               0/2/2/0             lan2         

    PACKAGE        STATUS        STATE         AUTO_RUN     NODE        
    oracle         up            running       enabled      mq1         
      
      Policy_Parameters:
      POLICY_NAME     CONFIGURED_VALUE
      Failover        configured_node
      Failback        manual
      
      Script_Parameters:
      ITEM       STATUS   MAX_RESTARTS  RESTARTS NAME
      Service    unknown  0             0        service1
      Subnet     up                              172.16.2.0
      
      Node_Switching_Parameters:
      NODE_TYPE    STATUS       SWITCHING    NAME
      Primary      up           enabled      mq1 (current)
      Alternate    up           enabled      mq2
  
  NODE           STATUS       STATE        
  mq2            up           running      
   
    Cluster_Lock_LVM:
    VOLUME_GROUP          PHYSICAL_VOLUME       STATUS              
    /dev/vglock           /dev/dsk/c8t0d5       up                  
   
    Network_Parameters:
    INTERFACE    STATUS           PATH                NAME         
    PRIMARY      up               0/3/0/0/0/0/2/0/0/  lan4         
    PRIMARY      up               0/1/1/0             lan0         
    STANDBY      up               0/3/0/0/0/0/4/0/0/  lan6         
    STANDBY      up               0/2/2/0             lan2  

weixiaoma0

回復(fù) 2# lbseraph

# cmviewcl -v  雙機應(yīng)該沒有問題啊

CLUSTER        STATUS      
cluster1       up           
  
  NODE           STATUS       STATE        
  mq1            up           running      
   
    Cluster_Lock_LVM:
    VOLUME_GROUP          PHYSICAL_VOLUME       STATUS              
    /dev/vglock           /dev/dsk/c4t0d5       up                  
   
    Network_Parameters:
    INTERFACE    STATUS           PATH                NAME         
    PRIMARY      up               0/3/0/0/0/0/2/0/0/  lan4         
    PRIMARY      up               0/1/1/0             lan0         
    STANDBY      up               0/3/0/0/0/0/4/0/0/  lan6         
    STANDBY      up               0/2/2/0             lan2         

    PACKAGE        STATUS        STATE         AUTO_RUN     NODE        
    oracle         up            running       enabled      mq1         
      
      Policy_Parameters:
      POLICY_NAME     CONFIGURED_VALUE
      Failover        configured_node
      Failback        manual
      
      Script_Parameters:
      ITEM       STATUS   MAX_RESTARTS  RESTARTS NAME
      Service    unknown  0             0        service1
      Subnet     up                              172.16.2.0
      
      Node_Switching_Parameters:
      NODE_TYPE    STATUS       SWITCHING    NAME
      Primary      up           enabled      mq1 (current)
      Alternate    up           enabled      mq2
  
  NODE           STATUS       STATE        
  mq2            up           running      
   
    Cluster_Lock_LVM:
    VOLUME_GROUP          PHYSICAL_VOLUME       STATUS              
    /dev/vglock           /dev/dsk/c8t0d5       up                  
   
    Network_Parameters:
    INTERFACE    STATUS           PATH                NAME         
    PRIMARY      up               0/3/0/0/0/0/2/0/0/  lan4         
    PRIMARY      up               0/1/1/0             lan0         
    STANDBY      up               0/3/0/0/0/0/4/0/0/  lan6         
    STANDBY      up               0/2/2/0             lan2  


   

lbseraph

回復(fù) 4# weixiaoma0

那就是說雙機切包機制正常，下一步你應(yīng)該看應(yīng)用的日志來查你的問題了。
   

weixiaoma0

回復(fù) 5# lbseraph

嗯，但是包切換的話，另一個節(jié)點不行，如果VG有問題的話，會在oracle.sh.log會顯示嗎？

我從那邊下手找故障點呢？謝謝�。�！
   

lbseraph

如果是VG有問題導致沒法mount LV之類，你連package都沒法啟動起來的。但你現(xiàn)在cmviewcl的輸出可以看到這個package已經(jīng)是up and running的狀態(tài)了。如果包沒起來，package的log可以看出VG的問題（如果有的話，比如無法激活VG之類）。