實例無法用CRS啟停，重新刪除注冊也報錯，懷疑是OCR的問題，請教一下解決方法

qujie1030

生產(chǎn)環(huán)境:aix 5.3+ha5.3+oracle rac 10.2.0.3，之前RAC運行正常，現(xiàn)在出現(xiàn)srvctl無法啟停數(shù)據(jù)庫的情況。麻煩高手指點迷津。
oraserver1:/Orahome$crs_stat -t -v
Name           Type           R/RA   F/FT   Target    State     Host        
----------------------------------------------------------------------
ora....R1.lsnr application    0/5    0/0    ONLINE    ONLINE    oraserver1  
ora....er1.gsd application    0/5    0/0    ONLINE    ONLINE    oraserver1  
ora....er1.ons application    0/3    0/0    ONLINE    ONLINE    oraserver1  
ora....er1.vip application    0/0    0/0    ONLINE    ONLINE    oraserver1  
ora....R2.lsnr application    0/5    0/0    ONLINE    ONLINE    oraserver2  
ora....er2.gsd application    0/5    0/0    ONLINE    ONLINE    oraserver2  
ora....er2.ons application    0/3    0/0    ONLINE    ONLINE    oraserver2  
ora....er2.vip application    0/0    0/0    ONLINE    ONLINE    oraserver2  
ora.racdb.db   application    0/1    0/1    ONLINE    ONLINE    oraserver1  
ora....acdb.cs application    0/1    0/1    ONLINE    OFFLINE               
ora....db1.srv application    0/1    0/0    ONLINE    OFFLINE               
ora....db2.srv application    0/1    0/0    ONLINE    OFFLINE               
ora....b1.inst application    0/1    0/0    ONLINE    ONLINE    oraserver1  
ora....b2.inst application    0/1    0/0    ONLINE    OFFLINE    oraserver2  

ora....b2.inst顯示OFFLINE，sqlplus手動啟動實例之后還是顯示offline，輸入crs_start -all后，ora....b2.inst變成ONLINE，ora....acdb.cs application  ，ora....db1.srv application ora....db2.srv 仍是OFFLINE
oraserver1:/Orahome$crs_start -all
Attempting to start `ora.racdb.racdb.cs` on member `oraserver1`
Attempting to start `ora.racdb.racdb.racdb1.srv` on member `oraserver1`
Attempting to start `ora.racdb.racdb2.inst` on member `oraserver2`
Attempting to start `ora.racdb.racdb.racdb2.srv` on member `oraserver2`
Start of `ora.racdb.racdb.racdb1.srv` on member `oraserver1` failed.
oraserver2 : CRS-1019: Resource ora.racdb.racdb.racdb1.srv (application) cannot run on oraserver2


Start of `ora.racdb.racdb.racdb2.srv` on member `oraserver2` failed.
oraserver1 : CRS-1019: Resource ora.racdb.racdb.racdb2.srv (application) cannot run on oraserver1


Start of `ora.racdb.racdb2.inst` on member `oraserver2` succeeded.
Start of `ora.racdb.racdb.cs` on member `oraserver1` failed.
Attempting to start `ora.racdb.racdb.cs` on member `oraserver2`
Start of `ora.racdb.racdb.cs` on member `oraserver2` failed.
CRS-1006: No more members to consider

CRS-0215: Could not start resource 'ora.racdb.racdb.cs'.

CRS-0215: Could not start resource 'ora.racdb.racdb.racdb1.srv'.

CRS-0215: Could not start resource 'ora.racdb.racdb.racdb2.srv'.

oraserver1:/Orahome$crs_stat -t -v
Name           Type           R/RA   F/FT   Target    State     Host        
----------------------------------------------------------------------
ora....R1.lsnr application    0/5    0/0    ONLINE    ONLINE    oraserver1  
ora....er1.gsd application    0/5    0/0    ONLINE    ONLINE    oraserver1  
ora....er1.ons application    0/3    0/0    ONLINE    ONLINE    oraserver1  
ora....er1.vip application    0/0    0/0    ONLINE    ONLINE    oraserver1  
ora....R2.lsnr application    0/5    0/0    ONLINE    ONLINE    oraserver2  
ora....er2.gsd application    0/5    0/0    ONLINE    ONLINE    oraserver2  
ora....er2.ons application    0/3    0/0    ONLINE    ONLINE    oraserver2  
ora....er2.vip application    0/0    0/0    ONLINE    ONLINE    oraserver2  
ora.racdb.db   application    0/1    0/1    ONLINE    ONLINE    oraserver1  
ora....acdb.cs application    0/1    0/1    ONLINE    OFFLINE               
ora....db1.srv application    0/1    0/0    ONLINE    OFFLINE               
ora....db2.srv application    0/1    0/0    ONLINE    OFFLINE               
ora....b1.inst application    0/1    0/0    ONLINE    ONLINE    oraserver1  
ora....b2.inst application    0/1    0/0    ONLINE    ONLINE    oraserver2  

于是想手動將實例從crs中刪除后再重新添加，結(jié)果報錯如下：
oraserver1:/Orahome$srvctl remove instance -f -d racdb -i racdb1
PRKO-2007 : Invalid instance name: racdb1
oraserver1:/Orahome$srvctl remove instance -f -d racdb -i racdb2
PRKO-2007 : Invalid instance name: racdb2
oraserver1:/Orahome$srvctl remove database -d racdb
Remove the database racdb? (y/[n]) y
PRKR-1001 : cluster database racdb does not exist
PRKO-2005 : Application error: Failure in getting Cluster Database Configuration for: racdb

強(qiáng)制從crs中取消注冊：
oraserver1:/Orahome$crs_unregister ora.racdb.racdb1.inst
oraserver1:/Orahome$crs_unregister ora.racdb.racdb2.inst
oraserver1:/Orahome$crs_unregister ora.racdb.db

用srvctl重新添加資源報錯如下：
oraserver1:/Orahome/crs/bin#./srvctl add database -d racdb -o /Orahome/product/10g
PRKR-1002 : cluster database racdb already exists
oraserver1:/Orahome/crs/bin#./srvctl add instance -d racdb -i racdb1 -n oraserver1
PRKR-1008 : adding of instance racdb1 on node oraserver1 to cluster database racdb failed.
CRS-0213: Could not register resource 'ora.racdb.racdb1.inst'.
PRKO-2010 : Error in adding instance to node: oraserver1

oraserver1:/Orahome/crs/bin#./srvctl add instance -d racdb -i racdb1 -n oraserver1
PRKR-1008 : adding of instance racdb1 on node oraserver1 to cluster database racdb failed.
CRS-0213: Could not register resource 'ora.racdb.racdb1.inst'.
PRKO-2010 : Error in adding instance to node: oraserver1

停機(jī)時間有限，實在沒招只能用系統(tǒng)之前自動備份的OCR去恢復(fù)原狀。
oraserver1:/Orahome$srvctl config database -d racdb -a
PRKR-1001 : cluster database racdb does not exist
PRKO-2005 : Application error: Failure in getting Cluster Database Configuration for: racdb

實例配置信息也看不到，我認(rèn)為是OCR出現(xiàn)問題，打算下個停機(jī)時間重建OCR。

jinguzhong

HACMP起來了沒？邏輯卷什么的都正常嗎？

qujie1030

HA和邏輯卷都是正常的，應(yīng)該和HA沒關(guān)系。從sqlplus進(jìn)去可以手動啟節(jié)點2上的實例，監(jiān)聽也是好的，rac可以正常訪問，就是srvctl管理不了實例，crs里的srv資源起不來。

qujie1030

貼段crsd.log

2012-02-27 16:54:17.148: [  CRSRES][12139]32Attempting to start `ora.racdb.racdb.racdb1.srv` on member `oraserver1`
2012-02-27 16:54:18.181: [  CRSRES][11882]32Attempting to start `ora.racdb.racdb.cs` on member `oraserver1`
2012-02-27 16:54:20.463: [  CRSRES][11369]32startRunnable: setting CLI values
2012-02-27 16:54:20.493: [  CRSRES][11369]32Attempting to start `ora.racdb.racdb2.inst` on member `oraserver2`
2012-02-27 16:54:20.958: [  OCRSRV][4131]th_select_handler: Failed to retrieve procctx from ht. constr = [408261616] retval lht [-27] Signal CV.
2012-02-27 16:54:21.118: [  OCRSRV][4131]th_select_handler: Failed to retrieve procctx from ht. constr = [409352912] retval lht [-27] Signal CV.
2012-02-27 16:54:21.219: [  CRSRES][12396]32startRunnable: setting CLI values
2012-02-27 16:54:21.257: [  CRSRES][12396]32Attempting to start `ora.racdb.racdb.racdb2.srv` on member `oraserver2`
[  OCRAPI][7472]a_check_permission_int: Other doesn't have permission
2012-02-27 16:54:21.823: [  OCRSRV][7472]proas_lookup_cache: User has no premission to operate on key [DATABASE.DATABASES.racdb]
2012-02-27 16:54:21.835: [  CRSAPP][12396]32StartResource error for ora.racdb.racdb.racdb2.srv error code = 1
2012-02-27 16:54:23.519: [  OCRSRV][4131]th_select_handler: Failed to retrieve procctx from ht. constr = [408261616] retval lht [-27] Signal CV.
2012-02-27 16:54:23.860: [  CRSRES][11369]32Start of `ora.racdb.racdb2.inst` on member `oraserver2` succeeded.
2012-02-27 16:54:29.022: [  OCRSRV][4131]th_select_handler: Failed to retrieve procctx from ht. constr = [383960624] retval lht [-27] Signal CV.
[  OCRAPI][7472]a_check_permission_int: Other doesn't have permission
2012-02-27 16:54:29.072: [  OCRSRV][7472]proas_lookup_cache: User has no premission to operate on key [DATABASE.DATABASES.racdb]
2012-02-27 16:54:42.581: [  CRSRES][12396]32Start of `ora.racdb.racdb.racdb2.srv` on member `oraserver2` failed.
2012-02-27 16:54:43.494: [  CRSRES][12396]32oraserver1 : CRS-1019: Resource ora.racdb.racdb.racdb2.srv (application) cannot run on oraserver1


2012-02-27 16:54:44.990: [  CRSRES][11398]32startRunnable: setting CLI values
2012-02-27 16:54:45.320: [  CRSAPP][11398]32StartResource error for ora.racdb.racdb.racdb2.srv error code = 1
2012-02-27 16:54:46.824: [  CRSRES][12139]32Start of `ora.racdb.racdb.racdb1.srv` on member `oraserver1` failed.
2012-02-27 16:54:47.598: [  CRSRES][12139]32oraserver2 : CRS-1019: Resource ora.racdb.racdb.racdb1.srv (application) cannot run on oraserver2


2012-02-27 16:54:49.709: [  CRSRES][11882]32Start of `ora.racdb.racdb.cs` on member `oraserver1` failed.
2012-02-27 16:54:50.728: [  CRSRES][11882]32startRunnable: setting CLI values
2012-02-27 16:54:50.745: [  CRSRES][11882]32Attempting to start `ora.racdb.racdb.cs` on member `oraserver2`
[  OCRAPI][7472]a_check_permission_int: Other doesn't have permission
2012-02-27 16:54:51.080: [  OCRSRV][7472]proas_lookup_cache: User has no premission to operate on key [DATABASE.DATABASES.racdb]
2012-02-27 16:54:51.979: [  CRSRES][12689]32Attempting to start `ora.racdb.racdb.racdb1.srv` on member `oraserver1`
2012-02-27 16:54:52.374: [  CRSRES][12689]32Start of `ora.racdb.racdb.racdb1.srv` on member `oraserver1` failed.
2012-02-27 16:54:52.804: [  CRSRES][13203]32startRunnable: setting CLI values
2012-02-27 16:54:52.812: [  CRSRES][13203]32Attempting to start `ora.racdb.racdb.racdb2.srv` on member `oraserver2`
2012-02-27 16:54:53.029: [  CRSAPP][13203]32StartResource error for ora.racdb.racdb.racdb2.srv error code = 1
2012-02-27 16:54:53.216: [  CRSRES][12689]32oraserver2 : CRS-1019: Resource ora.racdb.racdb.racdb1.srv (application) cannot run on oraserver2


2012-02-27 16:54:53.229: [  CRSRES][13203]32Start of `ora.racdb.racdb.racdb2.srv` on member `oraserver2` failed.
2012-02-27 16:54:54.454: [  CRSRES][13203]32oraserver1 : CRS-1019: Resource ora.racdb.racdb.racdb2.srv (application) cannot run on oraserver1


2012-02-27 16:54:55.124: [  CRSAPP][11882]32StartResource error for ora.racdb.racdb.cs error code = 1
2012-02-27 16:54:55.328: [  CRSRES][11882]32Start of `ora.racdb.racdb.cs` on member `oraserver2` failed.
2012-02-27 16:54:56.124: [  CRSRES][11882]32CRS-1006: No more members to consider

346279055

在節(jié)點上手動挺一下數(shù)據(jù)庫試試吧

qujie1030

手動啟停都沒問題，就是用srvctl管理不了，我就想讓數(shù)據(jù)庫能正常用srvctl管理。