Introduction

Linking is the process by which the Delphix Engine establishes a relationship with a data source. Linking is initiated when a Delphix user wants to import and manage copies of data from a discovered data source.

After linking a data source, the Delphix Engine can import data periodically from the data source and manage it as it evolves over time. A data source is called a linked source after it has undergone the linking process. The Delphix Engine tracks linked sources as dSources. A dSource is an object that outlines how data should be imported from a data source and managed on the Delphix Engine.

The standard process for importing data from a linked source is called SnapSync. After linking a data source, an initial SnapSync is performed to create a copy data on the Delphix Engine. Any following data imports are called incremental SnapSyncs and typically do not require that all data from a data source be imported again: incremental SnapSyncs will identify the subset of data that has been modified since the last SnapSync and apply only these data deltas to the copy of the data on the Delphix Engine.

For some data platforms, the linking process and SnapSync require a staging environment. A staging environment is an environment suitable for facilitating resource-intensive portions of the linking process and SnapSync. If a data platform requires a staging environment for importing data, a staging environment will be selected during the linking process. After selecting a staging environment, the environment hosting the data source is referred to as the primary environment. A staging environment is often separate from a primary environment, but it does not have to be (a primary environment can be its own staging environment).

Glossary

Direct Sources vs. Staging Sources

A toolkit may use one of two strategies to coordinate linking to a data source.

Syncing with direct sources is done by using rsync (on Unix platforms) or robocopy (on Windows). There are only very limited opportunities for the toolkit to customize this syncing process, and so this strategy is only recommended for very simple toolkits that really just need to copy data in as-is.

Syncing with linked sources is more complex. For linked sources, the Delphix Engine will mount a storage area on the staging environment (using NFS on Unix and iSCSI on Windows). The toolkit will be in charge of getting data from the source onto this storage. For example, some DBMS systems will have a "backup" or "consistent data dump" command. The toolkit could direct such a command to deposit its data onto the mounted storage area. Most toolkits will use this strategy due to its increased customizability and flexibility.

Choosing the correct strategy for your toolkit's use case is an important decision. This choice will ultimately inform the layout and management of data imported into the Delphix Engine.

Linked Source Definitions

A toolkit's linkedSourceDefinition augments a data platform's dSource object and dictates how SnapSync is implemented.

A linkedSourceDefinition has the following parts:

• Type – A type that dictates which syncing strategy is to be used (direct or linked).
• Hooks – A series of Lua scripts used to manage a linked source and assist the SnapSync process. The hooks specification changes with the type of linkedSourceDefinition chosen.
• Parameters – A specification of metadata required to manage a linked source and assist the SnapSync process.

The type of linkedSourceDefinition and the parameters list will dictate what metadata is queried for during linking. This metadata is attached to the dSource object when it is created and can be modified after linking.

Direct Source Definition

Direct toolkits are appropriate for syncing a manageable number of medium-sized files that don't change often, and which may be copied as-is, with no modification or augmentation. Note that rsync and robocopy will not efficiently import data from data sources with large files, with lots and lots of files, or with rapidly changing files.

If you elect to use direct sources in your toolkit, your linkedSourceDefinition  must be declared as type ToolkitLinkedDirectSource.

A direct toolkit's source config objects will each include a path field that dictates the location of its data sources. The toolkit's preSnapshot hook must tell the Delphix Engine when and from where to sync.

A direct toolkit's Lua hooks must be put into the toolkit's direct subdirectory. 

You can customize this process by implementing SnapSync hooks:

More information about the use of direct sources can be found in Build a Direct Toolkit.

Staged Source Definition

If you elect to use staged sources, your toolkit will be responsible for all details of copying data from the primary environment onto the Delphix-managed storage located on the staging environment. This storage will be mounted on the staging environment using NFS (on Unix) or iSCSI (on Windows).

If you elect to use staged sources in your toolkit, your linkedSourceDefinition  must be declared as type ToolkitLinkedStagedSource.

You can customize this process by implementing linking and SnapSync hooks:

Linking Delphix DB

The following is a walkthrough of how to implement linking for the fictional database platform, Delphix DB.

Fill Out the Linked Source Definition

Delphix DB, like most databases, stores its data in large data files and rapidly changing log files. Delphix DB databases can handle hundreds of transactions every second and vary in size from a couple megabytes to many terabytes. Data with this profile is impossible to efficiently import using a file-level syncing tool like rsync. Delphix DB offers both native replication and backup tools that can be used as better ways means of importing data. Therefore, we will want to use staged sources rather than direct sources. 

The Delphix DB toolkit will have a linkedSourceDefinition of type ToolkitLinkedStagedSource. Because we have specified a staged source, Delphix will mount storage to a specified staging environment at a specified location; our linking implementation will be responsible for manipulating the Delphix DB backup and replication tools to usher data onto the Delphix Engine through this mount.

Our linkedSourceDefinition will also augment the Delphix DB dSource object with two fields: stagingDbName and stagingPort. The user will be required to fill in values for these two fields when linking a dSource. These values will parameterize interactions with the Delphix DB tools throughout the linking and syncing processes.

{
	"type": "ToolkitLinkedStagedSource",
    "parameters": {
		"type": "object",
		"properties": {
			"stagingDbName": {
	    		"type": "string",
				"prettyName": "Staging DB Name",
				"description": "The name of the staging database to create.",
				"default": "stagingDB"
			},
			"stagingPort": {
				"type": "integer",
				"prettyName": "Staging Port",
				"description": "The port of the staging database to create.",
				"default": 1234
			}
		}
	}
}

Supply Lua Script for Initial Sync

At the root of your toolkit, create a directory named staged (if this directory does not already exist).

Inside the staged directory, create a file named resync.lua. This file will contain Lua code responsible for performing the initial SnapSync.

Prior to running resync.lua, the Delphix Engine will mount storage at the specified path on the specified staging environment. All resync.lua logic can assume it has access to this mount through the staging environment.

The body of our resync.lua implementation will begin by connecting to the Delphix DB instance being linked (the primary database) and taking a backup. This backup will be placed in a known location that the staging environment can later access.

The installPath field on the Delphix DB repository object will point us at the binaries required to perform the backup. The dbName field on the Delphix DB sourceConfig object will point use at the database instance being linked. See Build A Toolkit: Discovering Data Sources and Dependencies for more information about the repository and sourceConfig objects.

Our Lua script will then connect to the staging environment and create a fresh Delphix DB instance (the staging database). The staging database will be configured to store its data inside the mount from the Delphix Engine: the dataDirectory field on the source object will contain the path to this mount. The staging database will also be configured with the name and port specified on the dSource object: the parameters object will contain these values. See How to Write Lua Hooks for more information about the source and parameters objects.

We will initiate the staging database with data from the backup of the primary database. By loading data from a backup, we reduce the amount of data that will later need to be replicated to the staging database over the network.

Last, we will configure the primary database to act as a master that replicates itself to the staging database as a slave. As replication runs, the staging database will write the data it receives onto the Delphix Engine through the mount. The staging database will not be too far behind the primary database since the staging database was just created from a recent backup.

At the end of resync.lua, we will leave the staging database running on top of the mounted storage. The staging database will continue to receive replicated data so that the Delphix Engine's copy of the data is only slightly behind the data source itself.

Note that this staging database is not guaranteed to stay up-to-date with the data source. The staging database may become out-of-sync if it is taken offline during environment maintenance or if a network outage disrupts replication. An out-of-sync Delphix DB slave may not be able to catch up with its master if the master's change rate is sufficiently high. If this happens, the resync process can manually be rerun from the Delphix CLI to rebuild the slave from a recent backup.

While running resync.lua, if any of the RunBash operations exit non-zero, Lua execution will stop and an error will be displayed. A user can restart a failed resync by initiating another SnapSync.

-- Get IP address of the staging environment. This address will be used to configure replication from the data source.
ip = RunBash {
	command 		= "echo \"\\\"$(hostname -i)\\\"\" > $DLPX_OUTPUT_FILE",
	environment 	= source.stagingEnvironment,
	user 			= source.stagingEnvironmentUser,
	host 			= source.stagingHost,
	variables 		= {},
	outputSchema 	= { type = "string" }
}
 
-- Initiate a map of "environment variable name" -> "environment variable value".
-- This map draws from both the result of a Bash operation and the linkedSourceDefinition parameters.
envMap = {
	DELPHIXDB 		= repository.installPath,
	PRIMARYDBNAME 	= sourceConfig.dbName,
	STAGINGDBNAME 	= parameters.stagingDbName,
	STAGINGPORT		= parameters.stagingPort,
    STAGINGDIR      = source.dataDirectory,
	STAGINGHOSTIP 	= ip
}
 
--
-- Begin body of resync logic.
--
 
-- Step 1:
-- Generate a backup of the primary Delphix DB instance at a known location.
-- Note that this Bash logic is running on the primary environment (as opposed to the staging environment).
RunBash {
	command 		= "$DELPHIXDB backup $PRIMARYDBNAME -o /tmp/backup.tar.gz",
	environment 	= source.environment,
	user 			= source.environmentUser,
	host 			= source.host,
	variables 		= envMap
}
 
-- Step 2:
-- Create a fresh Delphix DB instance to later be configured as a slave to the primary database.
-- Note that this Bash logic is running on the staging environment.
RunBash {
	command 		= "$DELPHIXDB new $STAGINGDBNAME --port=$STAGINGPORT --dataDir=$STAGINGDIR",
	environment 	= source.stagingEnvironment,
	user 			= source.stagingEnvironmentUser,
	host 			= source.stagingHost,
	variables 		= envMap
}
 
-- Step 3:
-- Initiate staging database by loading data from the backup.
RunBash {
	command 		= "$DELPHIXDB restore $STAGINGDBNAME /tmp/backup.tar.gz",
	environment 	= source.stagingEnvironment,
	user 			= source.stagingEnvironmentUser,
	host 			= source.stagingHost,
	variables 		= envMap
}
 
-- Step 4:
-- Start up the staging database.
RunBash {
	command 		= "$DELPHIXDB start $STAGINGDBNAME",
	environment 	= source.stagingEnvironment,
	user 			= source.stagingEnvironmentUser,
	host 			= source.stagingHost,
	variables 		= envMap
}
 
-- Step 5:
-- Configure the primary database as a master. This command will also configure the staging database as a slave.
RunBash {
	command 		= "$DELPHIXDB add-slave $PRIMARYDBNAME $STAGINGHOSTIP",
	environment 	= source.environment,
	user			 = source.environmentUser,
	host 			= source.host,
	variables 		= envMap
}
 
-- Step 6:
-- Start replication of data from the primary environment to the staging environment.
RunBash {
	command 		= "$DELPHIXDB start-repl $STAGINGDBNAME",
	environment 	= source.environment,
	user 			= source.environmentUser,
	host 			= source.host,
	variables 		= envMap
}

Supply Lua Script for SnapSync

At the root of your toolkit, create a directory named staged (if this directory does not already exist).

Inside the staged directory, create files named preSnapshot.lua and postSnapshot.lua. These files will contain Lua code responsible for manipulating the primary and staging environments during a SnapSync.

Both initial and incremental SnapSync for staged toolkits will simply involve snapshotting the mounted storage. Snapshots are atomic and will thus record a crash-consistent view of the staging database data files and logs. The snapshotted data files and logs will eventually be copied and recovered during the provisioning process.

To guarantee the data being snapshotted is fully consistent on disk, we will augment SnapSync with a "preSnapshot.lua" implementation that temporarily halts Delphix DB replication and flushes all write buffers on the staging database. Our implementation of  "postSnapshot.lua" will resume replication after a snapshot has successfully been recorded.

envMap = {
	DELPHIXDB 		= repository.installPath,
	PRIMARYDBNAME 	= sourceConfig.dbName,
	STAGINGDBNAME 	= parameters.stagingDbName,
	STAGINGPORT		= parameters.stagingPort
}
 
RunBash {
	command 		= "$DELPHIXDB stop-repl $STAGINGDBNAME",
	environment 	= source.environment,
	user 			= source.environmentUser,
	host 			= source.host,
	variables 		= envMap
}
 
RunBash {
	command 		= "$DELPHIXDB flush $STAGINGDBNAME",
	environment 	= source.stagingEnvironment,
	user 			= source.stagingEnvironmentUser,
	host 			= source.stagingHost,
	variables 		= envMap
}

envMap = {
	DELPHIXDB 		= repository.installPath,
	PRIMARYDBNAME 	= sourceConfig.dbName,
	STAGINGDBNAME 	= parameters.stagingDbName,
	STAGINGPORT		= parameters.stagingPort
}
 
RunBash {
	command 		= "$DELPHIXDB start-repl $STAGINGDBNAME",
	environment 	= source.environment,
	user 			= source.environmentUser,
	host 			= source.host,
	variables 		= envMap
}

Supply Lua Scripts for Managing Staging Environment

At the root of your toolkit, create a directory named staged and another directory named resources (if these directories do not already exist).

Inside the staged directory, create files named startStaging.lua, stopStaging.lua, and status.lua. These files will contain Lua code responsible for monitoring and manipulating the replication process.

Inside the resources directory, create a file named query_staging_status.sh. This file will contain Bash logic to determine that status of the Delphix DB. In status.lua, we will simply coordinate the execution of this Bash logic by referencing it via the resources object. See How to Write Lua Hooks for more information about the resources object.

Enable/Disable

Start by implementing startStaging.lua and stopStaging.lua so that Delphix DB dSources can properly enabled and disabled.

An enabled dSource indicates that the Delphix Engine is actively collecting data from the associated linked source. For the Delphix DB data platform, an enabled dSource should indicate that the staging database is receiving and applying replicated writes from the linked source.

A disabled dSource indicates that the Delphix Engine is not collecting data from the associated linked source. For example, a dSource is often disabled prior to doing maintenance on the staging environment. For the Delphix DB data platform, a disabled dSource should indicate that the staging database is shut down, replication is disabled on the primary database, and the mounted storage can be safely unmounted.

Below we implement startStaging.lua, which is run when a disabled dSource is enabled. We also implement stopStaging.lua, which is run when an enabled dSource is disabled.

envMap = {
	DELPHIXDB 		= repository.installPath,
	PRIMARYDBNAME 	= sourceConfig.dbName,
	STAGINGDBNAME 	= parameters.stagingDbName,
	STAGINGPORT		= parameters.stagingPort
}
 
RunBash {
	command 		= "$DELPHIXDB start-repl $STAGINGDBNAME",
	environment 	= source.environment,
	user 			= source.environmentUser,
	host 			= source.host,
	variables 		= envMap
}
RunBash {
	command 		= resources["start_staging_database.sh"],
	environment 	= source.stagingEnvironment,
	user 			= source.stagingEnvironmentUser,
	host 			= source.stagingHost,
	variables 		= envMap
}

envMap = {
	DELPHIXDB 		= repository.installPath,
	PRIMARYDBNAME 	= sourceConfig.dbName,
	STAGINGDBNAME 	= parameters.stagingDbName,
	STAGINGPORT		= parameters.stagingPort
}
RunBash {
	command 		= "$DELPHIXDB stop-repl $STAGINGDBNAME",
	environment 	= source.environment,
	user 			= source.environmentUser,
	host 			= source.host,
	variables 		= envMap
}
RunBash {
	command 		= "$DELPHIXDB stop $STAGINGDBNAME",
	environment 	= source.stagingEnvironment,
	user 			= source.stagingEnvironmentUser,
	host 			= source.stagingHost,
	variables 		= envMap
}

Status

Next, implement status.lua so that Delphix DB staging databases can properly monitored.

status.lua will be executed periodically by the Delphix Engine to ensure that the data import process is healthy. For the Delphix DB data platform, the data import process is considered healthy if the staging database is receiving and applying replicated writes from the linked source.

When our Lua runs, the query_staging_status.sh Bash logic will be executed on the staging environment. The logic will use a CLI tool to check the health of the staging database. At the end of the script, we will write a string to $DLPX_OUTPUT_FILE to pass the status back to Lua.

After the Bash logic finishes executing, the Delphix Engine will validate the data written to $DLPX_OUTPUT_FILE matches the outputSchema defined in Lua. If the data is malformed, a fault will be posted against the dSource with information about the malformed data. If the data is well formed, the string will become a Lua object which can further be manipulated in status.lua.

In our case, we will simply return the string from query_staging_status.sh to the Delphix Engine.

After the Lua script finishes executing, the Delphix Engine will validate the returned data indicated the dSource is either "ACTIVE" or "INACTIVE". If the data is malformed, a fault will be posted against the dSource with information about the malformed data. If the data is well formed, the status of the dSource will be updated in the Delphix Management application and CLI.

envMap = {
	DELPHIXDB 		= repository.installPath,
	PRIMARYDBNAME 	= sourceConfig.dbName,
	STAGINGDBNAME 	= parameters.stagingDbName,
	STAGINGPORT		= parameters.stagingPort
}
status = RunBash {
	command 		= resources["query_staging_status.sh"],
	environment 	= source.stagingEnvironment,
	user 			= source.stagingEnvironmentUser,
	host 			= source.stagingHost,
	variables 		= envMap,
	outputSchema 	= { type = "string" }
}
return status

# Check if the output of status contains the string "replication-running"
status=$($DELPHIXDB status $STAGINGDBNAME)
if [[ $status == *"replication-running"* ]]
then
	echo "\"ACTIVE\"" > $DLPX_OUTPUT_FILE
else
	echo "\"INACTIVE\"" > $DLPX_OUTPUT_FILE
fi