Exchange Logs not truncated after Full Backup

Exchange logs are not truncated after a full back-up using Netbackup 7.6 resulting in the Logs directory being full and failed and suspended copies on passive members of the DAG. This situation exists even though the active database is mounted. Mail will not flow for the active databases whose logs volume is full. 

CAUSE:

After a Full backup has been successfully completed (Status 0), Netbackup API sends Exchange a message that the backup is complete. Once Exchange receives this notification it truncates the logs that have been backed up. The truncation does not occur because the API did not communicate properly with MSExchange or the backup did not have a ‘Status 0’. It is the responsibility of Exchange to truncate the backup. Other possible causes are Backupuser is not as admin on the Exchange server and not the user associated the Netbackup services.

Before executing the following step, you must stop DAG replication to the passive copies. 

Step 1.

Manually delete old logs only after confirming that they are older than the database checkpoint file E##.chk. You can eyeball the dates to determine which log to delete or you can run the following Powershell command:

C:\Program Files\Microsoft\Exchange Server\V14\Bin>eseutil /mk "I:\Program Files\Exchange Server\Managers\e04.chk"

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server

Version 14.03

Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating FILE DUMP mode...

      Checkpoint file: I:\Program Files\Exchange Server\Managers\e04.chk

      LastFullBackupCheckpoint: (0x0,0,0)

      Checkpoint: (0x17A37A,481,9D)

      FullBackup: (0x16CE13,8,16)

      FullBackup time: 07/19/2014 03:01:48

      IncBackup: (0x0,0,0)

      IncBackup time: 00/00/1900 00:00:00

      Signature: Create time:03/03/2012 18:28:35 Rand:4867917 Computer:

      Env (CircLog,Session,Opentbl,VerPage,Cursors,LogBufs,LogFile,Buffers)

          (    off,   1627,  81350,  16384,  81350,   2048,   2048, 619035)

Operation completed successfully in 0.31 seconds.

Note the highlighted number. Then look for this file: E0417a37a.log and delete all logs files older than this file.

This would free up necessary space on the transaction log volume to get mail flowing again. However, this does not mean that the logs may not fill up again.

Step 2:

This step allows you to update the CatalogData folder which is found in the same location as the database you are working on. Take the following steps on databases whose logs are not being truncated.

In a DAG, log truncation is delayed until all necessary log files are replicated to passive databases. The Replication service will delete the backed up log files both from both nodes that contain active and passive copies. It then verifies that the to-be-deleted log files have successfully been applied to the copy database and both active database and the database copies checkpoint has passed the log files to be deleted.

1.      Disable the Content Index for the database - Set-MailboxDatabase <Database> -IndexEnabled $False

2.      Stop the Microsoft Exchange Search (The Microsoft Search Indexer Server is a dependent and will also stop) on nodes with passive database

3.      Rename the catalog folder on all nodes including the node with the active database

4.      Re-enable the Content Index for database - Set-MailboxDatabase <Database> -IndexEnabled $True

A new CatalogData folder should automatically be created.

5.      Start Microsoft Search Service on the passive nodes

6.      Update the Catalog on the passive nodes – Update-MailboxDatabaseCopy <Database> -CatalogOnly

After completing these two steps run a full backup to test the truncation of the logs. If the logs are not truncated, uninstall and reinstall Netbackup on all nodes.

As a last resort:

1. Delete the DAG configuration
2. Execute the PS script 'ResetSearchIndex.ps1 -force mailboxservername'
3. Recreate the DAG