Options¶
Bases: FileDFReadOptions, GenericOptions
Options for FileDFReader.
Added in 0.9.0
Examples:
Note
You can pass any value supported by Spark,
even if it is not mentioned in this documentation. Option names should be in camelCase!
The set of supported options depends on Spark version.
from onetl.file import FileDFReader
options = FileDFReader.Options(recursive=True)
recursive = Field(default=None, alias='recursiveFileLookup')
class-attribute
instance-attribute
¶
If True, perform recursive file lookup.
Warning
This disables partition inferring using file paths.
Warning
Can be used only in Spark 3+. See SPARK-27990.
apply_to_reader(reader)
¶
Apply provided format to pyspark.sql.DataFrameReader.
Returns:
-
DataFrameReader–With options applied