Skip to content

S3 Warehouse

Bases: IcebergWarehouse, FrozenModel

Iceberg S3 Warehouse. support hooks

Added in 0.15.0

Note

This warehouse uses a dedicated Iceberg S3 client to access the warehouse data. It does not rely on SparkS3 implementation.

Parameters:

  • path (str) –

    Warehouse path

  • host (str) –

    S3 endpoint hostname

  • port (int) –

    S3 endpoint port

  • protocol (Literal['http', 'https'], default: "https" ) –

    Protocol to use for S3 connections

  • bucket (str) –

    S3 bucket name

  • path_style_access (bool, default: False ) –

    Whether to use path-style access

  • access_key (str) –

    S3 access key

  • secret_key (str) –

    S3 secret key

  • session_token (str) –

    S3 session token for temporary credentials

  • region (str) –

    S3 region

  • extra (Dict[str, str], default: {} ) –

    Additional S3 configuration parameters

Examples:

from onetl.connection import Iceberg

warehouse = Iceberg.S3Warehouse(
    path="/warehouse",
    host="s3.domain.com",
    protocol="http",
    bucket="my-bucket",
    region="us-east-1",
    path_style_access=True,
    access_key="my_access_key",
    secret_key="my_secret_key",
)

get_config()

Return flat dict with warehouse configuration.

get_packages(package_version) classmethod

Get package names to be downloaded by Spark. support hooks

See Maven package index for all available packages.

Parameters:

  • package_version (str) –

    Iceberg package version in format major.minor.patch.

Returns:

  • list[str] –

    List of Maven coordinates.

Examples:

from onetl.connection import Iceberg

# Note: Iceberg 1.10.0 requires Java 11+
Iceberg.S3Warehouse.get_packages(package_version="1.10.0")