Overview
Beginning with 5.3.0.0 release, Delphix engines have had the ability to forward events and metrics to a Splunk host using Fluentd. In 6.0.13.0 Delphix has extended this capability for use with other monitoring packages (ELK Stack, Datadog, etc.). Forwarding to Splunk can still be configured out of the box by providing about the Splunk host and other parmaters. Fluentd plugins provide a mechanism to customize the Delphix engine's forwarding capabilities for output to other data consumers.
The Fluentd plugin is a tar file with two components. A fluentd configuration file template is required. Optionally, any gem files that are needed to execute the directives in the configuration file can be included. Once the plugin is uploaded, the configuration occurs by collecting user input for the variables required by the template. The user interfaces adapts based on the variables required by the configuration template.
The current implementation is limited to one data consumer at a time. Only one plugin, in addition to the default SplunkHec plugin, can be uploaded. This is an intended simplification for the initial Fluentd release version.
The current implementation is limited to one data consumer at a time. Only one plugin, in addition to the default SplunkHec plugin, can be uploaded. This is an intended simplification for the initial Fluentd release version.
Technical Details
Architectural diagram
GUI, API or CLI changes (if any)
APIs with a corresponding CLI for uploading plugins and Fluentd configuration have been added. The service/fluentd/plugins API displays the plugins that are available.
service fluentd plugins> ls Objects REFERENCE PLUGIN GEMS ATTRIBUTEDEFINITIONS SCHEMADEFINITION FLUENTD_PLUGIN-1 splunkHec … … … Operations requestUploadToken
The splunkHec plugin appears by default and cannot be deleted. Additionally, the requestUploadToken operation provides a token needed to upload a plugin via the data/upload API. If a second plugin is uploaded, it will appear in the list. Before a third plugin could be uploaded, the second must be selected and deleted.
The following curl commands illustrate the use of the API to upload a plugin
curl -s -X POST -k --data @- http://<engine-name>/resources/json/delphix/session -c ~/cookies.txt -H "Content-Type: application/json" <<EOF
{
"type": "APISession",
"version": {
"type": "APIVersion",
"major": 1,
"minor": 4,
"micro": 3
}
}
EOF
curl -s -X POST -k --data @- http://<engine-name>/resources/json/delphix/login -b ~/cookies.txt -c ~/cookies2.txt -H "Content-Type: application/json" <<EOF
{
"type": "LoginRequest",
"username": "sysadmin",
"password": "sysadmin",
"target": "SYSTEM"
}
EOF
curl -s -X POST -k --data @- http://<engine-name>/resources/json/delphix/service/fluentd/plugins/requestUploadToken -b ~/cookies2.txt -H "Content-Type: application/json" <<EOF
Returns the token e.g: {"type":"OKResult","status":"OK","result":{"type":"FileUploadResult","url":"/resources/json/delphix/data/upload","token":"59346df5-3ecd-4ced-afbf-97acefa156dc"},"job":null,"action":null}
curl -b ~/cookies2.txt -X POST -F "file=@/Users/blewis/ws/far-dev/splunk_host_port.far" -F "token=<token>" http:/<engine-name>/resources/json/delphix/data/upload
The service/fluentd/plugins API displays the tokens that are available.
An example of creating a configuration with the default splunkHec plugin
service fluentd configuration> create service fluentd configuration create *> set plugin=splunkHec; set enabled=true; edit attributes service fluentd configuration create attributes *> add; set name=hec_host; set value=http://vmname-splunkhost.delphix.com; back service fluentd configuration create attributes *> add; set name=hec_port; set value=8088; back; service fluentd configuration create attributes *> add; set type=FluentdSecretAttribute; service fluentd configuration create attributes 2 *> set name=hec_token; set secretValue=bb75c032-bdea-4c19-b152-d158cf13e019; back service fluentd configuration create attributes *> add; set name=eventsIndex; set value=delphix_events; back; service fluentd configuration create attributes *> add; set name=protocol; set value=https; back service fluentd configuration create attributes *> add; set name=metricsPushFrequency; set value=60; back service fluentd configuration create attributes *> add; set name=metricsIndex; set value=delphix_metrics; back service fluentd configuration create attributes *> add; set name=eventsPushFrequency; set value=60; back; service fluentd configuration create attributes *> ls Properties 0: type: FluentdRegularAttribute (*) name: hec_host (*) value: http://vmname-splunkhost.delphix.com (*) 1: type: FluentdRegularAttribute (*) name: hec_port (*) value: 8088 (*) 2: type: FluentdSecretAttribute (*) name: hec_token (*) secretValue: ******** (*) 3: type: FluentdRegularAttribute (*) name: eventsIndex (*) value: delphix_events (*) 4: type: FluentdRegularAttribute (*) name: protocol (*) value: https (*) 5: type: FluentdRegularAttribute (*) name: metricsPushFrequency (*) value: 60 (*) 6: type: FluentdRegularAttribute (*) name: metricsIndex (*) value: delphix_metrics (*) 7: type: FluentdRegularAttribute (*) name: eventsPushFrequency (*) value: 60 (*) ## Use the "add" command to add an element to this array. service fluentd configuration create attributes *> back service fluentd configuration create *> ls Properties type: FluentdConfig name: (unset) attributes: [ ... ] (*) enabled: true (*) plugin: splunkHec (*) service fluentd configuration create *> commit
Creating Default splunkHec Configuration From GUI
- Login as a sysadmin user and select Fluentd Configuration under the Preferences menu.
- The splunkHec plugin appears in the dropdown by default.
- Additional plugins can be added by using the + button on the right.
- The new plugin uploaded can be viewed in the dropdown menu and selected to add a configuration.
- The delete button on the right can be used to delete a selected plugin.
- The name of fields for configuration would be the same as those provided in the uploaded plugin file.
- The configuration can be saved by clicking the save button on the bottom.
- When loading an existing configuration on UI, the secret fields are masked.
Implementation
Secret attributes have been introduced in order to protect private data such as passwords and tokens. In the CLI and API, they are identified by the FluentdSecretAttribute type. In plugins, attribute names in the config file template beginning with _secret indicate secret data.
Known issues
Fluentd runs in a systemd container to limit negative side-effects from a bad plugin. More security can be added to this container by running as a non-root user, in addition to limiting the size of the log and buffer files it can write.