XML

XML file data source
PropertiesProperties supported in this source are shown below ( * indicates required fields )
Property
Description
Name * 
Name of the data source
Description
Description of the data source
Processing Mode
Select for batch and un-select for streaming. If 'Batch' is selected the value of the switch is set to true. If 'Streaming' is selected the value of the switch is set to false.﻿﻿﻿Default: true
Infer Schema
Attempts to infer an appropriate type for each resulting column, like a boolean, numeric or date type. Otherwise, all resulting columns are of string type﻿﻿﻿Default: false
File Location Path * 
Path to where the file or folder is located. You can use standard globbing expressions﻿﻿﻿Example: s3://datamorph-data/studentdata/2012.xml,s3://datamorph-data/studentdata/2012.xml,s3://datamorph-data/studentdata/2012,s3://datamorph-data/studentdata/.xml,s3://datamorph-data//{12/31,01/01}.xml
Row Validation XSD Path
Path to an XSD file that is used to validate the XML for each row individually. Rows that fail to validate are treated like parse errors as above. The XSD does not otherwise affect the schema provided, or inferred﻿﻿﻿Example: s3://datamorph/xsds/studentdatarow.xsd
Schema
Source schema to assist during the design of the pipeline
Row Tag
The row tag treat as a row. For example, in this XML <books> <book><book/> ...<books/>, the value would be book﻿﻿﻿Example: ROW﻿Default: ROW
Root Tag
The root tag to treat as the root. For example, in this XML <books> <book><book/> ...<books/>, the appropriate value would be books
Filename Column
Adds the absolute path of the file being read as a new column with the provided name﻿﻿﻿Example: file_name
Select Fields / Columns
Comma separated list of fields / column names to select from source﻿﻿﻿Default: *
Filter Expression
SQL where clause for filtering records﻿﻿﻿Example: date = '2022-01-01',year=22 and month = 6 and day = 2
Distinct Values
Select rows with distinct column values﻿﻿﻿Default: false
Path Glob Filter
Optional glob pattern to only include files with paths matching the pattern. The syntax follows org.apache.hadoop.fs.GlobFilter. It does not change the behavior of partition discovery.
Recursive File Lookup
Recursively load files and it disables partition inferring. If your folder structure is partitioned with columnName=value (Eg. processDate=2022-01026), then using the recursive option WILL NOT read the partitions correctly.﻿﻿﻿Default: false
Base Schema Directory Path
Path to base directory / location of an XSD file that is used to validate the XML for each row individually. This is needed when there are multiple schemas using includes within a master XSD
Ignore Namespace
If selected, namespaces prefixes on XML elements and attributes are ignored. Tags <abc:author> and <def:author> would, for example, be treated as if both are just <author>. Note that, at the moment, namespaces cannot be ignored on the rowTag element, only its children. Note that XML parsing is in general not namespace-aware even if set to false﻿﻿﻿Default: false
Sampling Ratio
Sampling ratio for inferring schema (0.0 ~ 1)﻿﻿﻿Example: 0.5﻿Default: 1
Exclude Attribute
Select to exclude attributes in elements﻿﻿﻿Default: false
Whitespaces as Null
Select to treat whitespaces as a null value﻿﻿﻿Default: false
Mode
The mode for dealing with corrupt records during parsing﻿﻿﻿Default: PERMISSIVE
Column Name Of Corrupt Record
The name of new field where malformed strings are stored﻿﻿﻿Default: _corrupt_record
Attribute Prefix
The prefix for attributes so that we can differentiate attributes and elements. This will be the prefix for field names﻿﻿﻿Default: _
Value Tag
The tag used for the value when there are attributes in the element having no child﻿﻿﻿Default: _VALUE
Character Set
The character set﻿﻿﻿Example: UTF-8,UTF-16,ISO-8859-1﻿Default: UTF-8
Ignore Surrounding Spaces
Defines whether or not surrounding whitespaces from values being read should be skipped﻿﻿﻿Default: false
Wildcard Column Name
Name of a column existing in the provided schema which is interpreted as a 'wildcard'. It must have type string or array of strings. It will match any XML child element that is not otherwise matched by the schema. The XML of the child becomes the string value of the column. If an array, then all unmatched elements will be returned as an array of strings. As its name implies, it is meant to emulate XSD's xs:any type.﻿﻿﻿Example:
Timestamp Format
Specifies an additional timestamp format that will be tried when parsing values as TimestampType columns. The format is specified as described in DateTimeFormatter. Defaults to try several formats, including ISO_INSTANT, including variations with offset timezones or no timezone﻿﻿﻿Default: UTC
Date Format
Specifies an additional timestamp format that will be tried when parsing values as DateType columns. The format is specified as described in DateTimeFormatter﻿﻿﻿Default: ISO_DATE
Normalize Column Names
Normalizes column names by replacing special characters ,;{}()&/\n\t= and space with the given string﻿﻿﻿Example: _
Ignore Corrupt Files
If selected, jobs will continue to run when encountering corrupted files and the contents that have been read will still be returned
Ignore Missing Files
Select to ignore missing files while reading data from files
Modified Before
An optional timestamp to only include files with modification times occurring before the specified Time. The provided timestamp must be in the following form: YYYY-MM-DDTHH:mm:ss﻿﻿﻿Example: 2020-06-01T13:00:00
Modified After
An optional timestamp to only include files with modification times occurring after the specified Time. The provided timestamp must be in the following form: YYYY-MM-DDTHH:mm:ss﻿﻿﻿Example: 2020-06-01T13:00:00
Watermark Field Name
Field name to be used as watermark. If unspecified in streaming mode, the default field name is 'tempWatermark'.﻿﻿﻿Example: myConsumerWatermark﻿Default: tempWatermark
Watermark Value
Watermark value setting﻿﻿﻿Example: 10 seconds,2 minutes
 ﻿
﻿
﻿﻿﻿