This page provides definitions of major concepts.
Components of the Delphix Dynamic Data Platform
|Delphix Dynamic Data Platform|
The combined suite of products for virtualizing, masking, and working with your data.
|Delphix Management||Formerly called the Admin App. The interface for Delphix administrators to adjust settings and policies.|
|Delphix Self-Service||Formerly called Jet Stream. Enables application development teams to get data on demand.|
|Delphix Reporting||Formerly called Mission Control. A centralized reporting and auditing appliance for overseeing multiple Delphix deployments.|
|Delphix Masking||The Masking capability of the Delphix Dynamic Data Platform represents an automated approach to protecting non-production environments, replacing confidential information such as social security numbers, patient records, and credit card information with fictitious, yet realistic data.|
Terms for Using the Delphix Engine
Ways to Access the Delphix Engine
|Application Programming Interface (API)|
A method by which you can access a Delphix Engine programmatically.
|Command Line Interface (CLI)||A method by which you can access a Delphix Engine using SSH, which supports input of text commands.|
|Graphical User Interface (GUI)||A method by which you can access a Delphix Engine using a web browser.|
Delphix Virtualization Concepts
|Blocks or data blocks||Physical chunks of data that contain Relational Database Management System (RDBMS) rows. RDBMS databases are made up of files containing data blocks, which are backed up and restored with traditional database tools or virtualized with Delphix.|
A generic term for Delphix virtualized objects from which you can create virtual databases (VDBs).
A dataset can be a dSource, VDB, or vFiles.
The system, typically an RDBMS, that feeds information to the Delphix Engine, and from which virtual objects are derived. A data source can be a database, an application, or a set of unstructured files.
Not to be confused with a dSource, which is a virtualized, compressed duplicate of this database. (See below.)
Object groups are arbitrary collections of dSources or VDBs used for organization.
Note: Objects cannot be moved between groups.
|Delphix Connector||A service that runs on a Windows proxy host and enables communication between the Delphix Engine and the Windows Target Environment where it is installed.|
Is an object on the Delphix Engine that is a virtualized representation of a database that the Delphix Engine uses to create and update virtual copies of your database. As a virtualized representation of your source data, it cannot be managed, manipulated, or examined by database tools. Because dSources are simply source data, you must provision a VDB in order to distribute/clone/test the data being pulled in. VDBs can also later be refreshed from the same or other points in time synced from the dSource.
Connection details, host information, and data source information for Delphix to communicate with a host or cluster.
|Hooks||Delphix initiated calls to external scripts used to automate tasks, primarily on VDBs and dSources.|
|Host||The physical or logical machine that contains database instances. A host can be distinguished from an environment because the host has a physical reference point in its IP address. For example, you can specify a host (by referring to its host name or IP address) that a Delphix Environment points to.|
A standalone program which validates that host machines are configured correctly before the Delphix Engine uses them for syncing from data sources or provisioning VDBs.
HostChecker is a Delphix script that you should run before adding any Environment to your Delphix Engine. It is available to download from the HostChecker subfolder at download.delphix.com.
|LogSync||This feature which enables the ingestion and retention of more granular (log-based) source change data. This more granular change data allows for VDB point-in-time provision, refresh, or rollback.|
Copies of the source Delphix Engine information on the target Delphix Engine, which can include objects such as dSources, VDBs and vFiles. Replicas preserve object relationships and naming nomenclatures. Replicas do noot copy platform specific configurations such as users, permissions etc.
You can replicate datasets between two instances of the Delphix Engine. Replication consists of a profile-replica pair. It is configured on the source Delphix Engine and copies specified objects to a target Delphix Engine. The source engine then sends incremental updates manually or according to a schedule. In addition, you can provision VDBs from replicated objects, allowing for geographical distribution of data and remote provisioning.
|Replication profile||Replication configuration on the source Delphix Engine. Formerly called "Replication spec."|
Snapshots represent the state of a dataset at a specific moment in time. They accumulate over time from policies or are generated by manual creation. Snapshots allow you to choose a point in time from which to provision, refresh, or rollback.
If you have LogSync enabled, you can provision refresh or rollback from a point in time between the snapshots endpoints.
Note: Point in time provisioning is also dependent on log retention.
The standard process for importing data from a linked source into the Delphix Engine. An initial SnapSync is performed to create a dSource on the Delphix Engine. Incremental SnapSyncs are performed to provide additional points in time to the dSource on the Delphix Engine.
The original (either physical or virtual) database. It is commonly a production database, although it could be any database that the user designates as a source. Delphix creates a dSource from the source database.
|Source Environment||An environment from which the Delphix Engine can capture data.|
|Staging Environment||An environment suitable for facilitating resource-intensive portions of the linking process and SnapSync.|
|Target Environment||An environment on which the Delphix Engine can create VDBs.|
|Timeflow||The collection of Snapshots created by SnapSync policies or, in the case of SQL Server, the pre-provisioning process. When you provision or refresh a VDB, you pick a point in the Timeflow of a dSource or another VDB from which to provision.|
|Unstructured files||Data stored in a filesystem that is NOT usually accessed by a DBMS or similar software. Unstructured files can consist of anything from a simple directory to the root of a complex application like Oracle E-Business Suite. It is a dataset that is treated as simply a directory tree full of files. Like with other data types, you can configure a dSource to sync periodically with a set of unstructured files external to the Delphix Engine. Virtualized unstructured files are called vFiles (see below).|
|Validated Sync||The process that runs on a staging database within a Staging Environment, and which executes either before a snapshot is taken (SQL Server) or after a snapshot is taken (Oracle).|
Virtual unstructured files. A virtual copy of data files created and managed by Delphix. vFiles are fully functional read/write copies of the original unstructured files source. You can mount vFiles across one target environment or several.
|vFiles (Empty)||Creating an Empty vFiles places an initially-empty mount on Target Environments. You can then create data directly on Delphix. This is useful when you have no existing files to copy into the Delphix Engine, but you do have files which you will generate, track, and copy with vFiles. For more information see Creating Empty vFiles from the Delphix Engine.|
|Virtual Database (VDB)||A database provisioned from either a dSource or another VDB which is a full read/write copy of the source data. A VDB is created and managed by the Delphix Engine.|
|Virtual dataset||Comprehensive term that includes VDBs and vFiles.|
|V2P||Virtual to Physical. This refers to the process of moving a VDB to a physical database, for example in a disaster recovery situation.|
The terms below describe actions you can perform on a Delphix Engine.
|Link||The process of establishing a relationship between a data source and the Delphix Engine. After linking a data source, the Delphix Engine can import data periodically and manage it as it evolves over time. In the GUI, synonymous with "Add dSource."|
|Mask||Masking replaces sensitive data with fictitious data in non-prod environments (such as VDBs). It provides realistic data with which to work while reducing security risks. For more details about masking, see Masking Terms below.|
|Migrating a VDB||Moving a VDB to a new Target Environment.|
|Provision||Create a new VDB from a dSource or VDB.|
Refreshing a VDB will re-provision it from the dSource. As with the normal provisioning process, you can choose to refresh the VDB from a Snapshot or a specific point in time. Refreshing a VDB will delete any changes that have been made to the VDB prior to the refresh operation; you are essentially resetting it to the state you select during the refresh process.
Rewinding a VDB rolls it back to a previous point in its Timeflow. The VDB will no longer contain changes that occurred after the rewind point. This also creates an alternative Timeflow.
Delphix Virtualization Users and Privileges
|Object||User Privileges||Group Privileges|
Access statistics on the dSource, VDB, or Snapshot such as usage, history, and space consumption
Access statistics on all dSources, VDBs, or Snapshots in the group such as usage, history, and space consumption
|delphix_admin||Manages all data objects: dSources, virtual databases (VDBs), users, groups, and related policies and resources, all collectively referred to as the Delphix Engine "domain." The delphix_admin user manages the Delphix Engine domain using either the browser-based Delphix Management application or the Command Line Interface (CLI).||Has privileges over all objects and users.|
|sysadmin user||Can perform typical system administration duties such as: modifying NTP, SNMP, SMTP settings; managing storage; downloading support logs for the Delphix Engine; and performing upgrades and patches. The sysadmin user launches the initial Delphix Setup configuration application and has access to the Command Line Interface (CLI).||Has privileges for storage, upgrades, network etc.|
Types of Notification
|Event||Completion of some action in the Delphix Engine. Examples include user-initiated tasks such as snapshots or VDB provisioning, policy-based tasks, and background monitoring and maintenance tasks.|
Caused by a single event on a Delphix Engine. Also known as a System Event, and viewable through the System Event Viewer. Examples include warnings on source/target environment settings, recoverable errors, or incorrect connection settings.
Alert Levels: Informational, Warning, Critical
A persistent event on a Delphix Engine that remains until the issue is resolved. The fault may be marked resolved automatically or require that it be resolved manually. Selecting to Ignore a fault will also ignore future faults of that exact type against the same object.
System faults describe states and configurations that may negatively impact the functionality of the Delphix Engine and which can only be resolved through active user intervention.
Examples: Delphix Engine storage failure, Communication failures between the Delphix Engine and a source or target environment/host
Fault Levels: Warning, Critical
Delphix Self-Service (Jet Stream) Terms
Has full access to all report data and can configure and administer Delphix Self-Service. Additionally, can use the Delphix Engine to:
A logical reference to a point in time on a branch. You can use it as a point from which to fork new branches. It can also be the target of policies – for example, you can arrange to keep this bookmark for two years.
Bookmarks are a way to mark and name a particular moment of data on a timeline. You can restore the active branch's timeline to the moment of data marked with a bookmark. You can also share bookmarks with other Delphix Self-Service users, which allows them to restore their own active branches to the moment of data in your container. The data represented by a bookmark is protected and will not be deleted until the bookmark is deleted.
Branches are task-specific groupings you can create within a data container. A branch is used to track a logical task, and contains a timeline of the historical data for that task. As you work within your data container, you can create more branches over time to run or complete separate tasks.
Branches represent a logical sequence of activity, separate from the underlying data lineage. This is the main concept introduced in the core engine and forms the basis of many higher level primitives. Branches:
|Branch group / target group||A collection of multiple Branches that are treated as a single entity. The system can determine compatibility automatically, or a template can be used to create more complex orchestration.|
|Branch timeline||A dynamic point-in-time interface for user actions within the Branch. Common activities include re-setting Data Sources to run a test, refreshing the Data Container with the most current source data, and bookmarking data to share or track interesting moments of time along the branch timeline.|
Consists of one or more Data Sources, such as databases, application binaries, or other application data. Allows users to:
|Data template||Created by the Delphix Administrator, data templates consist of the data sources users need in order to manage their data playground and their testing and/or development environments. Data templates serve as the parent for a set of data containers that the administrator assigns to Delphix Self-Service users. Additionally, data templates enforce the boundaries for how data is shared. Data can only be shared directly with other users whose containers were created from the same parent data template.|
|Data User||Delphix Self-Service data users have access to production data provided in a data container. The data container provides these users with a playground in which to work with data using the Self-Service Toolbar.|
Delphix Reporting (Mission Control) Terms
Admin users have full access to all report data and can configure the Delphix Reporting appliance. (They can also assign auditor users a set of tags to restrict which report data they can view. There is no default auditor account. The first Delphix Administrator will need to create the auditor users and will be responsible for creating their User IDs and Passwords.)
An Admin user can:
|Auditor user||Auditor users can only view report data.|
Reports present aggregated data across all connected Delphix Engine .
Interactive reports such as Storage Breakdown and History display interactive graphical representations of historical and current storage usage across all Delphix Engines you are monitoring. These visualizations of storage and disk capacity enable you to analyze and mediate storage across Delphix Engines from multiple perspectives.
|Tagging||You can tag Delphix Engines in Delphix Reporting with a set of arbitrary text strings. You can then filter reports to show only data from Delphix Engines with a certain tag. You can also use tags to restrict auditor users so that they can only view data from Delphix Engines with that tag.|
Delphix Masking Engine Terms
|Application||The IT assets (programs, data, processes) that support a business function. For example, if a bank offers payroll services to its clients, there would be an application in its IT division to support that business.|
|Connector||Where the Delphix Engine stores JDBC database connection information. Builds a connection between the source database and the Delphix interface for data masking.|
|Domain||The Domain represents the correlation between various sensitive data categories and the masking algorithm which will be applied to them|
|EngineSync||The ability to coordinate the use of masking algorithms across multiple Delphix Engines.|
|Masking Environment||Defines a collection of masking constructs (connectors, rule sets / inventories, and jobs) that support masking for a given application environment. In order to mask databases and files within the Delphix Engine, you first need to create an environment in which the Delphix Engine will store the connection information and masking rules for those data stores. An Environment can contain multiple database connections and multiple file connections. Environments are connected to applications for informational purposes.|
|In-place masking||"Mask data in place" refers to updating a database with masked data. This includes reading data from the table defined in the Rule Set, masking the data with the Delphix Engine, and updating the tables with the masked data.|
|Inventory||The Delphix Engine automatically stores the masking rules for each sensitive column in the repository database in the Environment's "inventory." When you select a table to mask, its columns will appear and you can select them for masking. Afterwards, you can edit the columns with an appropriate algorithm required for masking.|
|Masked VDB||A virtual database with masked data.|
|On-the-fly masking||With on-the-fly masking, you specify the source of the information to be masked, and where the masked data will be loaded. On-the-fly masking is an Extract Transform Load (ETL) process.|
|Profile data||A way to identify the location of Non-Public Information (NPI) or sensitive data if you are unsure of what data needs to be masked in the first place. Profiling data is not necessary when you have already identified the sensitive data you need to mask.|
|Rule Set||A Rule Set points to a collection of tables or flat files that Delphix uses for masking data. The Rule Set allows you to identify, select, and configure which tables you need to mask. For those tables that do not have a primary key defined, you can define a logical key with a combination of columns (or ROWID for Oracle database).|
|Selective Data Distribution (SDD)||Permits the distribution of masked data between Delphix Engines. The sources received on a target Delphix Engine do not include and secure the original parent source, thereby making the original source inaccessible from the target.|
|Secure Lookup||The most commonly used type of algorithm. It is easy to generate and works with different languages. This algorithm replaces real, sensitive data with fictional data, it is possible that it will create repeating data patterns, known as “collisions.” For example, the names “Tom” and “Peter” could both be masked as “Matt.” Because names and addresses naturally recur in real data, this mimics an actual Data Set. However, if you want the Delphix Engine to mask all data into unique outputs, you should use segmented mapping, described below.|
|Segmented Mapping||Produces no overlaps or repetitions in the masked data. You can mask up to a maximum of 36 values using segmented mapping. You might use this method if you need columns with unique values, such as Social Security Numbers, Primary Key columns, or foreign key columns. You can set the algorithm to produce alphanumeric results or only numbers.|
Allows you to state what values will replace the original data. There will be no collisions in the masked data, because it always matches the same input to the same output. For example “David” will always become “Ragu,” and “Melissa” will always become “Jasmine.” The algorithm checks whether an input has already been mapped; if so, the algorithm changes the data to its designated output. You can use a Mapping Algorithm on any set of values, of any length, but you must know how many values you plan to mask.Note: When you use a Mapping Aalgorithm, you cannot mask more than one table at a time. You must mask tables serially.
|Binary Lookup||Replaces images that appear in object columns. For example, if a bank has an object column that stores images of checks, you can use a binary lookup algorithm to mask those images. The Delphix Engine cannot change data within images themselves, such as the names on X-rays or driver’s licenses. However, you can replace all such images with a new, fictional image. This fictional image is provided by the owner of the original data.|
The only type of algorithm that allows you to reverse its masking. For example, you can use a tokenization algorithm to mask data before you send it to an external vendor for analysis. The vendor can then identify accounts that need attention without having any access to the original, sensitive data. Once you have the vendor’s feedback, you can reverse the masking and take action on the appropriate accounts.
Like mapping, a tokenization algorithm creates a unique token for each input such as “David” or “Melissa.” The Delphix Engine stores both the token and the original so that you can reverse masking later.
|Min Max||Values that are extremely high or low in certain categories allow viewers to infer someone’s identity, even if their name has been masked. For example, a salary of $1M suggests a company’s CEO, and some age ranges suggest higher insurance risk. You can use a Min Max algorithm to move all values of this kind into the midrange.|
|Data Cleansing||Data Cleansing does not perform any masking. Instead, it standardizes varied spellings, misspellings, and abbreviations for the same name. For example, “Ariz,” “Az,” and “Arizona” can all be cleansed to “AZ.”|
|Free Text Redaction|
Helps remove sensitive data that appears in free-text columns such as “Notes.” This type of algorithm requires some expertise to use, because you must set it to recognize sensitive data within a block of text.
One challenge is that individual words might not be sensitive on their own, but together they can be. The algorithm uses profiler sets to determine what information it needs to mask. You can decide which expressions the algorithm uses to search for material such as addresses. For example, you can set the algorithm to look for “St,” “Cir,” “Blvd,” and other words that suggest an address. You can also use pattern matching to identify potentially sensitive information. For example, a number that takes the form 123-45-6789 is likely to be a Social Security Number.
You can use a free text redaction algorithm to show or hide information by displaying either a “black list” or a “white list.”