MongoDB

MongoDB database data source

Properties

Properties supported in this source are shown below ( * indicates required fields )
Property
Description
Name *
Name of the data source
Description
Description of the data source
Connection *
Pre-defined MongoDB connection
Database *
Database to connect.Example: customerdb
Collection *
Collection data to fetchExample: products
Schema
Source schema to assist during the design of the pipeline
Select Fields / Columns
Comma separated list of fields / column names to select from sourceDefault: *
Filter Expression
SQL where clause for filtering recordsExample: date = '2022-01-01',year=22 and month = 6 and day = 2
Distinct Values
Select rows with distinct column valuesDefault: false
Partitioner
The partitioner full class name.Example: com.mongodb.spark.sql.connector.read.partitioner.SamplePartitionerDefault: com.mongodb.spark.sql.connector.read.partitioner.SamplePartitioner
Keep Alive (ms)
The length of time to keep a MongoClient available for sharing.Example: 100,000Default: 5,000
Sample Size
The number of documents to sample from the collection when inferring the schemaExample: 1,000Default: 1,000
Normalize Column Names
Normalizes column names by replacing special characters ,;{}()&/\n\t= and space with the given stringExample: _