HANA Integration with Commvault

Commvault software provides a simplified end-to-end backup and recovery solution for both large-scale single node and multi-node SAP HANA environments. It uses the SAP Backint interface program to back up the data directly to the attached media. In case of a data loss due to hardware failure, natural disaster, accidental deletion, or corruption of data, you can restore the backed up data and the log files directly from the media using the SAP Backint interface program. This reduces complexity, human intervention and the risk of data loss.

With this approach, each dSource snapshot becomes a small metadata structure that is recorded in the Delphix Engine (eliminating storage of the actual HANA backup files themselves in Delphix). The metadata captures a recovery timestamp and "points" to the corresponding backups and logs which are stored outside of Delphix in the third-party backup tool. During a dSource snapshot, Delphix will capture metadata which includes:

  • the source instance SID 
  • the source database tenant which is being virtualized by Delphix Engine. 
  • a specific timestamp for which the snapshot is being defined

The specific dSource snapshot timestamp will be used by Delphix Engine to request recovery by HANA to that timestamp (ie provision of a ROOT VDB). During provision or refresh of the specific VDB, Delphix will communicate with HANA via HANA Backint format commands. It is important to point out that HANA will not reliably recover to the precise, requested timestamp. HANA will instead recover to a nearby, but different achieved timestamp, which will also be tracked in VDB snapshot metadata after recovery completes. 

Furthermore, in scenarios where Commvault does not yet have the necessary logs up to and beyond the requested timestamp, HANA will report successful recovery to a timestamp that it cannot reach, and the achieved timestamp will fall substantially short. Even worse, if multiple RECOVERY requests are made for the exact same requested timestamp, and the log contents in Commvault are different each time, HANA will report success and will have recovered each attempt to a different achieved timestamp. This is a major problem for Delphix because it is critical that every VDB which is recovered from a single dSource snapshot (which has its own static requested recovery timestamp) must result in identical, consistent data every time. For this reason, this proposed solution is using an OFFSET to minimize the risk that Commvault lacks all necessary logs to recover up to/beyond the requested timestamp. In our internal testing, once Commvault has logs beyond point of the requested timestamp, it will yield a consistent achieved timestamp in recovery for each RECOVERY attempt, therefore providing the consistency in resultant VDBs. As stated before, the value of the achieved timestamp will typically fall short of the requested timestamp; that is a HANA characteristic outside of Delphix's control.

The offset will be a field presented to the user at the time of dSource creation. The default value of this will be set to 0 and can be modified. This value should be based upon the lag that the user sees for the archive logs getting generated and then getting transferred to the Commvault system. During the dSource creation, this value will be subtracted from the current UTC time and the timestamp value will then be saved as a part of the metadata on the Delphix Engine. This value will then be used during the VDB creation in the recovery command. For eg:- If the current UTC time value is 2020-01-01 13:00:00 and the OFFSET value passed is 30 mins, then the timestamp saved during the VDB creation for this snapshot will be 2020-01-01 12:30:00. This value will be retrieved during the VDB creation and will be passed in the recovery command. Recovery will then be attempted for this timestamp value.

As a part of the VDB creation process, below fields will be saved as a part of the VDB metadata:-

  1. Execution timestamp :- Real-time Timestamp on the target VM at which RECOVER DATABASE command was issued.
  2. Recovery timestamp :-  Timestamp specified by the plugin in RECOVER DATABASE UNTIL command - inherited from the Origin Timestamp field of the parent dSource snapshot
  3. Achieved timestamp :- Timestamp reported by HANA in backup.log at the conclusion of the specific RECOVER DATABASE UNTIL command.
  4. Recovery Info :- This will contain the recovery string as reported by HANA in the backup.log file once the recovery gets completed

This information will be available to the user by a python tool which will be provided to the user apart from the plugin. For more information regarding the tool, please refer HANA 2.0 Plugin Tools

HANA 2.0 Plugin with Commvault Support

The external Backup ingestion feature is extended by adding support for Commvault backups. Source Backups are taken and managed by Commvault, a third-party backup-and-recovery solution that is certified for us with SAP HANA. Provisioning and recovery of a VDB from a dSource snapshot will invoke backups of data and archive logs which is performed outside of Delphix using the Commvault integrated backup solution. Below are some Commvault terminologies and their description in details which is referenced and used with the HANA 2.0 Plugin.

Architecture: Below is the architecture for HANA 2.0 plugin integration with Commvault. Details are mentioned in the pointers below

HANA Integration with Commvault

Commvault software provides a simplified end-to-end backup and recovery solution for both large-scale single node and multi-node SAP HANA environments. It uses the SAP Backint interface program to back up the data directly to the attached media. In case of a data loss due to hardware failure, natural disaster, accidental deletion, or corruption of data, you can restore the backed up data and the log files directly from the media using the SAP Backint interface program. This reduces complexity, human intervention and the risk of data loss.

With this approach, each dSource snapshot becomes a small metadata structure that is recorded in the Delphix Engine (eliminating storage of the actual HANA backup files themselves in Delphix). The metadata captures a recovery timestamp and "points" to the corresponding backups and logs which are stored outside of Delphix in the third-party backup tool. During a dSource snapshot, Delphix will capture metadata which includes:

  • the source instance SID 
  • the source database tenant which is being virtualized by Delphix Engine. 
  • a specific timestamp for which the snapshot is being defined

The specific dSource snapshot timestamp will be used by Delphix Engine to request recovery by HANA to that timestamp (ie provision of a ROOT VDB). During provision or refresh of the specific VDB, Delphix will communicate with HANA via HANA Backint format commands. It is important to point out that HANA will not reliably recover to the precise, requested timestamp. HANA will instead recover to a nearby, but different achieved timestamp, which will also be tracked in VDB snapshot metadata after recovery completes. 

Furthermore, in scenarios where Commvault does not yet have the necessary logs up to and beyond the requested timestamp, HANA will report successful recovery to a timestamp that it cannot reach, and the achieved timestamp will fall substantially short. Even worse, if multiple RECOVERY requests are made for the exact same requested timestamp, and the log contents in Commvault are different each time, HANA will report success and will have recovered each attempt to a different achieved timestamp. This is a major problem for Delphix because it is critical that every VDB which is recovered from a single dSource snapshot (which has its own static requested recovery timestamp) must result in identical, consistent data every time. For this reason, this proposed solution is using an OFFSET to minimize the risk that Commvault lacks all necessary logs to recover up to/beyond the requested timestamp. In our internal testing, once Commvault has logs beyond point of the requested timestamp, it will yield a consistent achieved timestamp in recovery for each RECOVERY attempt, therefore providing the consistency in resultant VDBs. As stated before, the value of the achieved timestamp will typically fall short of the requested timestamp; that is a HANA characteristic outside of Delphix's control.

The offset will be a field presented to the user at the time of dSource creation. The default value of this will be set to 0 and can be modified. This value should be based upon the lag that the user sees for the archive logs getting generated and then getting transferred to the Commvault system. During the dSource creation, this value will be subtracted from the current UTC time and the timestamp value will then be saved as a part of the metadata on the Delphix Engine. This value will then be used during the VDB creation in the recovery command. For eg:- If the current UTC time value is 2020-01-01 13:00:00 and the OFFSET value passed is 30 mins, then the timestamp saved during the VDB creation for this snapshot will be 2020-01-01 12:30:00. This value will be retrieved during the VDB creation and will be passed in the recovery command. Recovery will then be attempted for this timestamp value.

As a part of the VDB creation process, below fields will be saved as a part of the VDB metadata:-

  1. Execution timestamp :- Real-time Timestamp on the target VM at which RECOVER DATABASE command was issued.
  2. Recovery timestamp :-  Timestamp specified by the plugin in RECOVER DATABASE UNTIL command - inherited from the Origin Timestamp field of the parent dSource snapshot
  3. Achieved timestamp :- Timestamp reported by HANA in backup.log at the conclusion of the specific RECOVER DATABASE UNTIL command.
  4. Recovery Info :- This will contain the recovery string as reported by HANA in the backup.log file once the recovery gets completed

This information will be available to the user by a python tool which will be provided to the user apart from the plugin. For more information regarding the tool, please refer HANA 2.0 Plugin Tools

HANA 2.0 Plugin with Commvault Support

The external Backup ingestion feature is extended by adding support for Commvault backups. Source Backups are taken and managed by Commvault, a third-party backup-and-recovery solution that is certified for us with SAP HANA. Provisioning and recovery of a VDB from a dSource snapshot will invoke backups of data and archive logs which is performed outside of Delphix using the Commvault integrated backup solution. Below are some Commvault terminologies and their description in details which is referenced and used with the HANA 2.0 Plugin.

Architecture: Below is the architecture for HANA 2.0 plugin integration with Commvault. Details are mentioned in the pointers below

  • Terminologies: Below are short descriptions of terminology related to Commvault and Delphix which is referenced in this Plugin architecture.
    • Commvault related Terminology:
      • Commserve: The CommServe host is the central management component of the CommCell environment. It coordinates and executes all CommCell operations. There can be only one CommServe host in a CommCell environment. The CommServe software can be installed in physical, virtual, and clustered environments. 
      • Commcell Console: The CommCell Console is the central management user interface for managing the CommCell environment - monitoring and controlling active jobs, and viewing events related to all activities. The CommCell Console allows centralized and decentralized organizations to manage all data movement activities through a single, common interface.
        image2019-11-8_13-23-16.png
      • Client: A client is a logical grouping of the software agents that facilitate the protection, management, and movement of data associated with the client.
      • hdbbackint: It is an interface program that allows the SAP HANA Studio or the hdbsql command to communicate with SAP HANA using streams (or pipes) to perform backup and restore.
      • Backint: Backint for SAP HANA is an API that enables 3rd party tool vendors to directly connect their backup agents to the SAP HANA database. Backups are transferred via pipe from the SAP HANA database to the 3rd party backup agent, which runs on the SAP HANA database server and then sends the backups to the 3rd party backup server.
    • Delphix related Terminology: 
      • Commvault Based Ingestion Flag: This is an option available on Plugin UI and selecting (or checking the checkbox) this option means the user is using Delphix Commvault/Backint approach.

      • SAP HANA Source Tenant Database Name: This is visible on plugin UI as an input field. The customer will provide the source Tenant Database name from which Delphix will create virtual copies.
      • RPO offset: RPO offset is a customer-tunable number of minutes which customer can optionally modify to control calculation of a specific dSource snapshot. The default value in minutes would be 0. The offset value will be a positive quantity of minutes; that offset value will be deducted from the current timestamp at the time that a dSource snapshot is created. This value can, however, be changed from the UI.
    • Recovery with Commvault: All the backups will be stored on Commvault for the customer. While recovering, the plugin will issue a command using backint for recovery and will specify the time till which to recover. The key point to remember here is that because of the lag between the data transfer to the commvault system, the actual recovery time will be less than the time which was used in the recovery command. Once the logs reach the commvault server, the recovery will happen to a consistent time for a dSource snapshot.