Skip to content

File Uploader Options

Bases: GenericOptions

File uploading options.

Added in 0.3.0

Examples:

from onetl.file import FileUploader

options = FileUploader.Options(
    if_exists="replace_entire_directory",
    delete_local=True,
    workers=4,
)

if_exists = Field(default=(FileExistBehavior.ERROR), alias=(avoid_alias('mode'))) class-attribute instance-attribute

How to handle existing files in the target directory.

Possible values:

  • error (default) - mark file as failed
  • ignore - mark file as skipped
  • replace_file - replace existing file with a new one
  • replace_entire_directory - delete local directory content before downloading files

Changed in 0.9.0

Renamed mode → if_exists

delete_local = False class-attribute instance-attribute

If True, remove local file after successful download.

If download failed, file will left intact.

Added in 0.2.0

Changed in 0.3.0

Move FileUploader.delete_local to FileUploaderOptions

workers = Field(default=1, ge=1) class-attribute instance-attribute

Number of workers to create for parallel file upload.

1 (default) means files will me uploaded sequentially. 2 or more means files will be uploaded in parallel workers.

Recommended value is min(32, os.cpu_count() + 4), e.g. 5.

Added in 0.8.1