Skip to content

FileDF Writer

Bases: FrozenModel

Allows you to write Spark DataFrame as files in a target path of specified file connection with parameters. support hooks

Parameters:

  • connection (BaseFileDFConnection) –

    File DataFrame connection. See File DF Connections section.

  • format (BaseWritableFileFormat) –

    File format to write.

  • target_path (PathLike | str) –

    Directory path to write data to.

  • options (FileDFWriterOptions) –

    Common writing options.

Examples:

from onetl.connection import SparkLocalFS
from onetl.file import FileDFWriter
from onetl.file.format import CSV

local_fs = SparkLocalFS(spark=spark)

writer = FileDFWriter(
    connection=local_fs,
    format=CSV(delimiter=","),
    target_path="/path/to/directory",
)

from onetl.connection import SparkLocalFS
from onetl.file import FileDFWriter
from onetl.file.format import CSV

csv = CSV(delimiter=",")
local_fs = SparkLocalFS(spark=spark)

writer = FileDFWriter(
    connection=local_fs,
    format=csv,
    target_path="/path/to/directory",
    options=FileDFWriter.Options(if_exists="replace_entire_directory"),
)

run(df)

Method for writing DataFrame as files. support hooks

Note

Method does support only batching DataFrames.

Parameters:

  • df (DataFrame) –

    Spark dataframe

Examples:

Write df to target:

writer.run(df)