Skip to content

onETL Docs

File DataFrame Connections

onETL Docs

About onETL
Concepts
Contribution Guide
QuickStart
Logging
Plugins
Security
Connections
Connections
- DB connections
  DB connections
  - Clickhouse
    
    Clickhouse
    
    Connection
    
    Prerequisites
    
    Execute
    
    Read
    
    Write
    
    SQL
    
    Types
  - Greenplum
    
    Greenplum
    
    Connection
    
    Prerequisites
    
    Execute
    
    Read
    
    Write
    
    Types
  - Hive
    
    Hive
    
    Connection
    
    Prerequisites
    
    Execute
    
    Read
    
    Write
    
    SQL
    
    Slots
  - Iceberg
    
    Iceberg
    
    Connection
    
    Prerequisites
    
    Execute
    
    Read
    
    Write
    
    SQL
    
    Basic Auth
    
    Bearer Auth
    
    OAuth2 Client Credentials
    
    Catalog REST
    
    Catalog Filesystem
    
    Warehouse S3
    
    Warehouse Filesystem
    
    Warehouse Delegated
  - Kafka
    
    Kafka
    
    Connection
    
    Prerequisites
    
    Read
    
    Write
    
    Auth
    
    Basic Auth
    
    Kerberos Auth
    
    Scram Auth
    
    protocol
    
    Plaintext protocol
    
    SSL protocol
    
    Slots
    
    Troubleshooting
  - MongoDB
    
    MongoDB
    
    Connection
    
    Prerequisites
    
    Read
    
    Write
    
    Pipelines
    
    Types
  - MSSQL
    
    MSSQL
    
    Connection
    
    Prerequisites
    
    Execute
    
    Read
    
    Write
    
    SQL
    
    Types
  - MySQL
    
    MySQL
    
    Connection
    
    Prerequisites
    
    Execute
    
    Read
    
    Write
    
    SQL
    
    Types
  - Oracle
    
    Oracle
    
    Connection
    
    Prerequisites
    
    Execute
    
    Read
    
    Write
    
    SQL
    
    Types
  - Postgres
    
    Postgres
    
    Connection
    
    Prerequisites
    
    Execute
    
    Read
    
    Write
    
    SQL
    
    Types
- File connection
  File connection
  - FTP
  - SFTP
  - FTPS
  - S3
  - Samba
  - WebDAV
  - HDFS
    
    HDFS
    
    Connection
    
    Slots
- File as DataFrame
  File as DataFrame
  - Base
  - Local files
  - HDFS Files
    
    HDFS Files
    
    Connection
    
    Prerequisites
    
    Slots
  - S3 Files
    
    S3 Files
    
    Connection
    
    Prerequisites
    
    Troubleshooting
DB
DB
- DBReader
- DBWriter
File
File
- File Downloader
  File Downloader
- File Uploader
  File Uploader
- File Mover
  File Mover
- File Filters
  File Filters
- File Limits
  File Limits
File as DataFrame
File as DataFrame
- Reader
  Reader
  - File DataFrame Reader
  - Options
- Writer
  Writer
  - File DataFrame Writer
  - Options
- File Formats
  File Formats
  - Base
  - Avro
  - CSV
  - Excel
  - JSON
  - JSONLines
  - ORC
  - Parquet
  - XML
Hooks
Hooks
HWM
HWM
- YAML HWM Store
Install
Install
- Full
- Minimal install
- Files
- Spark
- Kerberos
Strategy
Strategy
Troubleshooting
Troubleshooting
- Spark
Changelog
Changelog
- 0.15.0
- 0.14.1
- 0.14.0
- 0.13.5
- 0.13.4
- 0.13.3
- 0.13.1
- 0.13.0
- 0.12.5
- 0.12.4
- 0.12.3
- 0.12.2
- 0.12.1
- 0.12.0
- 0.11.2
- 0.11.1
- 0.11.0
- 0.10.2
- 0.10.1
- 0.10.0
- 0.9.5
- 0.9.4
- 0.9.3
- 0.9.2
- 0.9.1
- 0.9.0
- 0.8.1
- 0.8.0
- 0.7.2
- 0.7.1
- 0.7.0

Table of contents

For developers

File DataFrame Connections¶

Spark LocalFS
Spark HDFS
Spark S3

For developers¶

Base interface

Made with Material for MkDocs