Skip to content

File Downloader Options

Bases: GenericOptions

File downloading options.

Added in 0.3.0

Examples:

from onetl.file import FileDownloader

options = FileDownloader.Options(
    if_exists="replace_entire_directory",
    delete_source=True,
    workers=4,
)

delete_source = False class-attribute instance-attribute

If True, remove source file after successful download.

If download failed, file will left intact.

Added in 0.2.0

Changed in 0.3.0

Move FileUploader.delete_local to FileUploaderOptions

if_exists = Field(default=(FileExistBehavior.ERROR), alias=(avoid_alias('mode'))) class-attribute instance-attribute

How to handle existing files in the local directory.

Possible values:

  • error (default) - mark file as failed
  • ignore - mark file as skipped
  • replace_file - replace existing file with a new one
  • replace_entire_directory - delete local directory content before downloading files

Changed in 0.9.0

Renamed mode → if_exists

workers = Field(default=1, ge=1) class-attribute instance-attribute

Number of workers to create for parallel file download.

1 (default) means files will me downloaded sequentially. 2 or more means files will be downloaded in parallel workers.

Recommended value is min(32, os.cpu_count() + 4), e.g. 5.

Added in 0.8.1