Snowflake

Snowflake database data sink

Properties

Properties supported in this sink are shown below ( * indicates required fields )
Property
Description
Name *
Name of the data sink
Description
Description of the data sink
Processing Mode
Select for batch and un-select for streaming. If 'Batch' is selected the value of the switch is set to true. If 'Streaming' is selected the value of the switch is set to false.Default: true
Connection *
Pre-defined Snowflake connection
Database *
Database in the Snowflake account that you are attempting to accessExample: EMPLOYEES
Schema *
Schema within the database that you are attempting to access
Table *
Table to write to
Warehouse
The default virtual warehouse to use for the session after connecting
Role
The default security role to use for the session after connectingExample: ACCOUNTADMIN
Select Fields / Columns
Comma separated list of fields / columns to select from inputs to the sinkExample: id, name, city, state, zipDefault: *
Output Mode
The values should be one of Append, Overwrite, ErrorIfExists or IgnoreDefault: ErrorIfExists
Timezone
The time zone to be used by Snowflake
spark: Use the time zone from Sparksnowflake: Use the current time zone for Snowflakesf_default: Use the default time zone for the Snowflake user who is connectingUse a specific time zoneExample: snowflake, sf_default, America/New_YorkDefault: spark
Compress
If set to on, the data passed between Snowflake and Spark is compressedDefault: on
Parallelism
The size of the thread pool to use for data uploads and downloads between Snowflake and SparkExample: 8Default: 4
Purge
If this is set to on, then the connector deletes temporary files created when transferring from Spark to Snowflake via external data transfer.Default: off
Preactions
A semicolon-separated list of SQL commands that are executed before data is transferred between Spark and Snowflake.Example: Select * from EMPLOYEES where ACTIVE = true
Postactions
A semicolon-separated list of SQL commands that are executed after data is transferred between Spark and Snowflake.Example: Select * from EMPLOYEES where ACTIVE = false
Truncate Columns
If set to on (default), a COPY command automatically truncates text strings that exceed the target column length. If set to off, the command produces an error if a loaded string exceeds the target column length.Default: on
Truncate Table
This parameter controls whether Snowflake retains the schema of a Snowflake target table when overwriting that table.Default: off
Continue on Error
This variable controls whether the COPY command aborts if the user enters invalid data (for example, invalid JSON format for a variant data type column).Default: off
Auto Pushdown
This parameter controls whether automatic query pushdown is enabledDefault: off
Keep Column Case
When writing a table from Spark to Snowflake, the Spark connector defaults to shifting the letters in column names to uppercase, unless the column names are in double quotes.Default: off