Tuesday, January 18, 2011

Exchange 2010 DR test failed and recovery DAG




On our Exchange 2010 DR test, we lost all the nodes in the cluster. Below the clear up steps which we remove the old setting and re-add the nodes, database back to DAG:


Get DAG info
Get-DAG shows node 1/3 are stopped node and node 2 is started but missing node4
Remove DAG
Remove-DatabaseAvailabilityGroupServer -Identity DAG01 -ConfigurationOnly:$TRUE -MailboxServer MBP01
WARNING: The operation wasn't successful because an error was encountered. You may find more details in log file  "C:\ExchangeSetupLogs\DagTasks\dagtask_2010-11-15_14-15-28.839_remove-databaseavailabiltygroupserver.log".
Mailbox server 'MBP01' cannot be removed from the database availability group because mailbox database 'DB01' has  multiple copies. Use Remove-MailboxDatabaseCopy either to remove the copy from this server or to remove the copies from other servers in the database availability group.
    + CategoryInfo          : InvalidArgument: (:) [Remove-DatabaseAvailabilityGroupServer], RemoveDagServer...icatedE
   xception
    + FullyQualifiedErrorId : C26FF955,Microsoft.Exchange.Management.SystemConfigurationTasks.RemoveDatabaseAvailabili
   tyGroupServer

[PS] C:\Windows\system32>Get-MailboxDatabaseCopyStatus *\MBP01

Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                                              Length    Length                             State       
----                                          ------          --------- ----------- --------------------   ------------
DB01\MBP01                              Dismounted      0         0                                  Failed     
DB02\MBP01                              Failed          0         0           11/12/2010 11:50:57 AM Failed     
DB03\MBP01                              Failed          0         0           11/12/2010 11:47:36 AM Failed     
DB07\MBP01                              Failed          0         0           11/12/2010 11:47:23 AM Failed     

Remove DB
Get-MailboxDatabaseCopyStatus *\MBP01

Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                                              Length    Length                             State      
----                                          ------          --------- ----------- --------------------   ------------
DB01\MBP01                              Dismounted      0         0                                  Failed     
DB02\MBP01                              Failed          0         0           11/12/2010 11:50:57 AM Failed     
DB03\MBP01                              Failed          0         0           11/12/2010 11:47:36 AM Failed     
DB07\MBP01                              Failed          0         0           11/12/2010 11:47:23 AM Failed     

Get-MailboxDatabaseCopyStatus *\MBP01 | Remove-MailboxDatabaseCopy

Get-MailboxDatabaseCopyStatus *\MBP03 | Remove-MailboxDatabaseCopy

Get-MailboxDatabaseCopyStatus *\MBP04 | Remove-MailboxDatabaseCopy

Get-MailboxDatabaseCopyStatus *\MBP02 | Remove-MailboxDatabaseCopy

Example log:

The database "DB01" is currently hosted on server "MBP01". Use Move-ActiveMailboxDatabase to move the active copy  of the database to a different server. You can use the Remove-MailboxDatabase task if this is the only copy.Confirm

Remove Server
Remove-DatabaseAvailabilityGroupServer -Identity DAG01 -ConfigurationOnly:$TRUE -MailboxServer MBP01

Remove-DatabaseAvailabilityGroupServer -Identity DAG01 -ConfigurationOnly:$TRUE -MailboxServer MBP02

Remove-DatabaseAvailabilityGroupServer -Identity DAG01 -ConfigurationOnly:$TRUE -MailboxServer MBP03

Remove-DatabaseAvailabilityGroupServer -Identity DAG01 -ConfigurationOnly:$TRUE -MailboxServer MBP04

Confirm
Are you sure you want to perform this action?
Removing Mailbox server "MBP01" from database availability group "DAG01".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): a
Clear up cluster
On each nodes:
cluster node /force
Attempting to clean up node '' ...
Clean up successfully completed.
Disable DAG computer account
In AD computer OU:
·         Disable tag01 computer object
·         Assign “Exchange trusted subsystem” full control to the object:


Remove failover cluster role

On each node:

servermanagercmd -r failover-clustering

Servermanagercmd.exe is deprecated, and is not guaranteed to be supported in fut
ure releases of Windows. We recommend that you use the Windows PowerShell cmdlets that are available for Server Manager.

Start Removal...
[Removal] Succeeded: [Failover Clustering] Failover Clustering.

Success: Removal succeeded.
Add servers back to DAG
Add-DatabaseAvailabilityGroupServer -Identity DAG01 -MailboxServer MBP01
Add-DatabaseAvailabilityGroupServer -Identity DAG01 -MailboxServer MBP02
Add-DatabaseAvailabilityGroupServer -Identity DAG01 -MailboxServer MBP03
Add-DatabaseAvailabilityGroupServer -Identity DAG01 -MailboxServer MBP04

Test Cluster
Move cluster resource group to each nodes:
cluster group "cluster group"  /moveto:mbp02

Group                Node            Status
-------------------- --------------- ------
cluster group        MBP02        Online


Get info
Get-MailboxDatabaseCopyStatus *
Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                                              Length    Length                             State      
----                                          ------          --------- ----------- --------------------   ------------
DB03\MBP02                              Mounted         0         0                                  Healthy    
DB04\MBP02                              Dismounted      0         0                                  Failed     
DB07\MBP04                              Mounted         0         0                                  Healthy    
DB08\MBP04                              Dismounted      0         0                                  Failed     
DB02\MBP04                              Dismounted      0         0                                  Failed     
DB06\MBP04                              Dismounted      0         0                                  Failed     
DB05\MBP03                              Mounted         0         0                                  Healthy    
DB01\MBP01                              Dismounted      0         0                                  Failed     
DR_DB08\UMP01                           Mounted         0         0                                  Healthy    
drdb01\UMP01                            Mounted         0         0                                  Healthy    
Mailbox Database 1150652460\UMP01          Dismounted      0         0                                  Failed     
dr_db02\UMP01                           Mounted         0         0                                  Healthy    
dr_db03\UMP01                           Mounted         0         0                                  Healthy    
dr_db04\UMP01                           Mounted         0         0                                  Healthy    
DR_DB07\UMP01                           Mounted         0         0                                  Healthy    
dr_db05\UMP01                           Mounted         0         0                                  Healthy    
DR_DB06\UMP01                           Mounted         0         0                                  Healthy    

Remove old database
·         Clean up all move request
Get-MoveRequest | Remove-MoveRequest
·         Move users to DR databae
Get-MailboxDatabase | Mount-Database
Get-Mailbox -Database DB08 | New-MoveRequest -TargetDatabase dr_db08
Get-MoveRequestStatistics
Get-MoveRequest | Remove-MoveRequest
·         Mount public folder database :
Get-PublicFolderDatabase | Mount-Database
·         Format all the database drives on each node
Create DB
Create the “Logs” folder in the each database folder and run commands:


new-mailboxdatabase -Server 'MBP01' -Name 'DB02' -EdbFilePath 'D:\MPS\DB02\DB02.EDB' -LogFolderPath 'D:\MPS\DB02\LOGS'

new-mailboxdatabase -Server 'MBP02' -Name 'DB03' -EdbFilePath 'D:\MPS\DB03\DB03.EDB' -LogFolderPath 'D:\MPS\DB03\LOGS'

new-mailboxdatabase -Server 'MBP02' -Name 'DB04' -EdbFilePath 'D:\MPS\DB04\DB04.EDB' -LogFolderPath 'D:\MPS\DB04\LOGS'

new-mailboxdatabase -Server 'MBP03' -Name 'DB05' -EdbFilePath 'D:\MPS\DB05\DB05.EDB' -LogFolderPath 'D:\MPS\DB05\LOGS'

new-mailboxdatabase -Server 'MBP03' -Name 'DB06' -EdbFilePath 'D:\MPS\DB06\DB06.EDB' -LogFolderPath 'D:\MPS\DB06\LOGS'

new-mailboxdatabase -Server 'MBP04' -Name 'DB07' -EdbFilePath 'D:\MPS\DB07\DB07.EDB' -LogFolderPath 'D:\MPS\DB07\LOGS'

new-mailboxdatabase -Server 'MBP04' -Name 'DB08' -EdbFilePath 'D:\MPS\DB08\DB08.EDB' -LogFolderPath 'D:\MPS\DB07\LOGS'

Mount DB
Get-MailboxDatabase | Mount-Database -Force

Add DB copy
Add-MailboxDatabaseCopy DB01 -MailboxServer mbp02
Add-MailboxDatabaseCopy DB02 -MailboxServer mbp04
Add-MailboxDatabaseCopy DB03 -MailboxServer mbp01
Add-MailboxDatabaseCopy DB04 -MailboxServer mbp03
Add-MailboxDatabaseCopy DB05 -MailboxServer mbp02
Add-MailboxDatabaseCopy DB07 -MailboxServer mbp01 -DomainController dc04
Troubleshooting
Get-MailboxDatabaseCopyStatus * | where{ $_.status -like "fail*"} |Suspend-MailboxDatabaseCopy

Get-MailboxDatabaseCopyStatus * | Suspend-MailboxDatabaseCopy
Get-MailboxDatabaseCopyStatus * | Update-MailboxDatabaseCopy –DeleteExistingFiles

No comments: