We just upgrade the EV to 9 and starting to archive the Ms exchange 2010. After 2 weeks, we found the performance of the EV server is really slow and found this error in the event log:
From the technote: TECH129173,
Various errors may be logged as a result, including errors in the following event categories:
Agent Client Broker
Archive Task
Exchange Provisioning Task
Journal Task
Public Folder Task
Retrieval Task
Agent Client Broker
Archive Task
Exchange Provisioning Task
Journal Task
Public Folder Task
Retrieval Task
Solution
It is possible to exceed the size of the Outlook 2007 credential cache, as the Outlook 2007 cache can hold details of a maximum of 32 objects (MAPI connections). This limit has been raised in a recent Microsoft Outlook hot fix that can be obtained using the following link:
So we drop the concurrent threads from 5 to 3:
For three days, it runs no problem but the error comes back after 4th day. Checked domain controller, exchange CAs server, no errors from all of them
We ran the dtrage:
Here is the one when the error comes up:
[7280] | (ArchiveTask) | <7500> | EV:M | CArchivingAgent::PostProcessArchivedItemEx - BuildFromSessionAndLocation succeeded. |
[7280] | (ArchiveTask) | <7500> | EV:H | :CArchivingAgent::PostProcessArchivedItem() |Getting the value of the createShortcut flag from the Shortcut message | |
[7280] | (ArchiveTask) | <6376> | EV:M | AuditInterface::SUCCESS_AUDIT (with IShortcutAccessor) |
[7280] | (ArchiveTask) | <6376> | EV:M | CEVAudit::WillPerformAudit Determine whether to audit or not. |
[7280] | (ArchiveTask) | <6376> | EV:M | CEVAudit::WillPerformAudit Auditing is off, current status '1' |
[7280] | (ArchiveTask) | <6376> | EV:M | CArchivingAgent::PostProcessArchivedItemEx - Getting Policy Information |
[7280] | (ArchiveTask) | <6376> | EV:H | :CArchivingAgent::PostProcessArchivedItem() |Attempting to delete the body of the shortcut | |
[7280] | (ArchiveTask) | <7500> | EV:M | CArchivingAgent::PostProcessArchivedItemEx - Shortcut properties checked and valid |
[7280] | (ArchiveTask) | <6376> | EV:M | RemoveTransactionIDFromMsg : Property found and removed. [0x0]. |
[7280] | (ArchiveTask) | <7500> | EV:M | CAA::SUP() - bsSSiteeEntryId = 0000000038A1BB1005E5101AA1BB08002B2A56C20000454D534D44422E444C4C00000000000000001B55FA20AA6611CD9BC800AA002FC45A090000006F75746C6F6F6B2E6E612E646972656374656E657267792E636F7270002F6F3D4445434F52502F6F753D46697273742041646D696E6973747261746976652047726F75702F636E3D526563697069656E74732F636E3D4A58696F6E6700 |
[7280] | (ArchiveTask) | <7500> | EV:M | CPolicyTargetGroupCache::GetUsersPolicyTargetGroup - Found entry for /o=DOMAINNAME/ou=First Administrative Group/cn=Recipients/cn=User1 in cache |
[7280] | (ArchiveTask) | <7500> | EV:M | CPolicyTargetGroupCache::GetUsersPolicyTargetGroup - User [/o=DOMAINNAME/ou=First Administrative Group/cn=Recipients/cn=User1] maps to Policy [14358B2E0B7FE9C4FAF244D1DBA136DF41012p00DEArchiveSite] [Whole Exchange Organization] |
[7280] | (ArchiveTask) | <7500> | EV:M | EPC::GP - Returning Policy : [Default Exchange Mailbox Policy][1015A028970BBB847B47E9A0823A1A4CD1012700DEArchiveSite] |
[7280] | (ArchiveTask) | <7500> | EV:M | CAA::GUP() - /o=DOMAINNAME/ou=First Administrative Group/cn=Recipients/cn=User1 is using Policy : Default Exchange Mailbox Policy |
[7280] | (ArchiveTask) | <7500> | EV:M | CArchivingAgent::PostProcessArchivedItemEx - Setting item to shortcut. |
[7280] | (ArchiveTask) | <6980> | EV:M | CMailboxHelper::CPAS() - GetMsgServiceTable |
[7280] | (ArchiveTask) | <6980> | EV:M | CMailboxHelper::CPAS() - QueryAllRows |
[7280] | (ArchiveTask) | <6980> | EV:M | CMailboxHelper::CPAS() - ConfigureMsgService |
[7280] | (ArchiveTask) | <6980> | EV:M | ConfigureMsgServiceAttempt - Trying ConfigureMsgService using PR_PROFILE_UNRESOLVED_SERVER=[SITEMBP03], Attempt=[1] |
[7280] | (ArchiveTask) | <6980> | EV:M | ConfigureMsgServiceAttempt - ConfigureMsgService failed with error [0x8007000E] |
[7280] | (ArchiveTask) | <6376> | EV:M | AddProps: Processing: (null) |
[7280] | (ArchiveTask) | <6376> | EV:M | AddProps: Using 1 additional shortcut properties |
Here is the one show it has no problem to connect to GC or Domain controller:
[7280] | (ArchiveTask) | <7288> | EV:M | CDirecSiteyVaultObject::GetAttributeListFromType Attribute list for type ExchangeServerEntry : ExchangeServerEntryId,SiteEntryId,ExchangeComputer,ExchangeServerADSPath,ExchangeLegacyDN,ExchangeVersion,ExchangeServicePack,ExchangeServerIdentity,ExchangeServerFQDN,ExchangeServerGCOverride,OrganizationEntryId,ManualAdminSecurityDesc,OverrideFlags,DefVaultSSiteeEntryId,DefIndexingServiceEntryId,ExchServerVersion,ADExchSvrDN,ExtendedConfigurationData |
[7280] | (ArchiveTask) | <7288> | EV:M | CBaseDirecSiteyServiceWrapper::CreateDirecSiteyService() - Entry [m_nNumTries = 40] |
[7280] | (ArchiveTask) | <7288> | EV:L | CBaseDirecSiteyServiceWrapper::CreateDirecSiteyService() - Successfully communicated with an EV DirecSitey Service on the local machine |
[7280] | (ArchiveTask) | <7288> | EV:M | CDirecSiteyVaultObject::GetAttributeListFromType Attribute list for type Organization : OrganizationEntryId,SiteEntryId,DomainName,GCOverride |
[7280] | (ArchiveTask) | <7288> | EV:M | AC::GESV() - Using FQDN from Organization Entry [DOMAINNAME] |
[7280] | (ArchiveTask) | <7288> | EV:M | FindExchServerDn - no search base passed in so look one up. |
[7280] | (ArchiveTask) | <7288> | EV:M | GetADDomainControllerEx - using FQDN DOMAINNAME |
[7280] | (ArchiveTask) | <7288> | EV:M | GetADDomainControllerEx - DsGetDcName dwResult==[1355] |
[7280] | (ArchiveTask) | <7288> | EV:M | GetADDomainControllerEx - 2nd DsGetDcName dwResult==[0] |
[7280] | (ArchiveTask) | <7288> | EV:M | GetADDomainControllerEx - returning | DC Name==[\\SITEDC03.na.domainFQDN.local],| Search Format==[GC://SITEDC03.na.domainFQDN.local],| GC Found==[True] |
[7280] | (ArchiveTask) | <7288> | EV:M | FindExchServerDn - successfully found GC. |
[7280] | (ArchiveTask) | <7288> | EV:M | FindExchServerDn - Bind to Search Base: GC://SITEDC03.na.domainFQDN.local |
[7280] | (ArchiveTask) | <7288> | EV:M | Execute search |
[7280] | (ArchiveTask) | <7288> | EV:M | FindExchServerDn - Exchange Server Found. DN: CN=SITEMBP03,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=EXCHAGNEORGNAME,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=directenergy,DC=corp |
[7280] | (ArchiveTask) | <7288> | EV:M | AC::GESV() - Returning values|ExchServerDn = CN=SITEMBP03,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=EXCHAGNEORGNAME,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=directenergy,DC=corp|ExchSearchPath = GC://SITEDC03.na.domainFQDN.local|ExchFQDN = DOMAINNAME |
We went back check the event log again, we found there were event 2244 and 3007 before the event 3411. Here is the event 2244 and 3411:
The users are different but the email subject was the same. Here is the result from dTrace:
10:38:28.379 | [7204] | (ArchiveTask) | <3728> | CMAPISaveset::get__SavesetUnicode (Entry) |
10:38:28.379 | [7204] | (ArchiveTask) | <3728> | CSavesetOnIStg2::IsSavesetUnicode (Entry) |
10:38:28.379 | [7204] | (ArchiveTask) | <3728> | CSavesetOnIStg2::IsSavesetUnicode (Exit). hr=Success [0] |
10:38:28.379 | [7204] | (ArchiveTask) | <3728> | CMAPISaveset::get__SavesetUnicode (Exit). hr=Success [0] |
10:38:28.379 | [7204] | (ArchiveTask) | <3728> | CMsgSess::OpenMsg: Opening session |
10:38:28.379 | [7204] | (ArchiveTask) | <3728> | CMsgSess::OpenMsg:Opening IMsg on IStg, flags 0x80000000 |
10:38:28.379 | [7204] | (ArchiveTask) | <3728> | CMsgSess::OpenMsg: Message open |
10:38:28.379 | [7204] | (ArchiveTask) | <3728> | CMAPISaveset::OpenSharedMessageOnIMsg (Exit). hr=<0x8004011b> |
10:38:28.379 | [7204] | (ArchiveTask) | <3728> | CMAPISaveset::OpenSharedMessage (Exit). hr=Success [0] |
10:38:28.379 | [7204] | (ArchiveTask) | <3728> | CMAPISaveset::_GetWritableMAPIOBJECT (Exit). hr=Success [0] |
10:38:28.379 | [7204] | (ArchiveTask) | <3728> | CMAPISaveset2::_GetWritableMAPIOBJECT _com_error exception. hr=<0x8004011b> |
10:38:28.379 | [7204] | (ArchiveTask) | <3728> | CMAPISaveset2::_GetWritableMAPIOBJECT (Exit). hr=<0x8004011b> |
10:38:28.379 | [7204] | (ArchiveTask) | <3728> | HRXEX fn trace :|Source : .\ArchivingAgent.cpp (CArchivingAgent::PI_BuildSaveset) [lines {15643,15650,15657,15661,15665,15667,15678,15683,15708}] built Aug 6 19:31:14 2010|Description: Call error 0x8004011b |
So we run the export command to export the message to PST file and opened it from outlook:
New-MailboxExportRequest -mailbox user1 -ContentFilter {(SUBJECT -LIKE "*SYSTEM OUTAGE NOTIFICATION*)} -filepath C:\export.pst
From the PST file, we found that the message has more than 2000 recipients in the too field. It is no problem to open the message in the outlook but it hung the outlook when we want to save the message in msg format from the outlook.
When we killed outlook.exe:
C:\Program Files\Support Tools>taskkill /fi "imagename eq outlook.exe"
SUCCESS: The process with PID 4424 has been terminated.
We have this error message:
From the perfmon, we see when we want to save the message, the CPU and memory went to high:
And the outlook process is using high CPU and high page out. The nonpage byte and free PTE is the same:
From the Microsoft KB 948074, http://support.microsoft.com/kb/948074. It is the KB for outlook 2003 but the registry key works for outlook 2007:
HKEY_CURRENT_USER\Software\Microsoft\Office\12.0\Outlook\Options\Mail
RecipientMax DWORD Value 4096
After changed the vault, the 2244 and 3007 stopped and the event 3411 stopped too. It is more likely that the Outlook hung the EV archive task and cause error.
No comments:
Post a Comment