Properties supported in this sink are shown below ( * indicates required fields )
Property
Description
Name *
Name of the data sink
Description
Description of the data sink
Processing Mode
Select for batch and un-select for streaming. If 'Batch' is selected the value of the switch is set to true. If 'Streaming' is selected the value of the switch is set to false.Default: true
Select Fields / Columns
Comma separated list of fields / columns to select from inputs to the sinkExample: id, name, city, state, zipDefault: *
Path*
This path where the file is locatedExample: s3://[bucketpath],hdfs://[URL]Default:
Output Mode
If mode is batch mode, the values should be either of Append, Overwrite, ErrorIfExists or Ignore. If streaming mode, the values should be append, complete or update.Default: ErrorIfExists
Checkpoint Location
Path to checkpoint file locationExample: hdfs://hdfs_location,s3a://s3_bucket/location
SQL to Execute on Each Partition
SQL to execute in streaming modeExample: locationDefault:
Partition By
Comma separated column names to partition byExample: year, month, day
Part Files Per Partition
Number of part files to write per partition column. WARNING: Setting this value may degrade performance drastically. It may also increase memory and CPU resource usageExample: 10,20,2000
Record Name
Top level record name in write resultDefault: topLevelRecord
Record Namespace
Record namespace in write resultDefault:
Compression
Compression option allows to specify a compression codec used in write. Currently supported codecs are uncompressed, snappy, deflate, bzip2, xz and zstandardDefault: snappy