Plugins¶

version added 0.6.0

What are plugins?¶

Terms¶

Plugin - some Python package which implements some extra functionality for onETL, like hooks
Plugin autoimport - onETL behavior which allows to automatically import this package if it contains proper metadata (entry_points)

Features¶

Plugins mechanism allows to:

Automatically register hooks which can alter onETL behavior
Automatically register new classes, like HWM type, HWM stores and so on

Limitations¶

Unlike other projects (like Airflow 1.x), plugins does not inject imported classes or functions to onetl.* namespace. Users should import classes from the plugin package explicitly to avoid name collisions.

How to implement plugin?¶

Create a Python package some-plugin with a file some_plugin/setup.py:

# some_plugin/setup.py
from setuptools import setup

setup(
    # if you want to import something from onETL, add it to requirements list
    install_requires=["onetl"],
    entry_points={
        # this key enables plugins autoimport functionality
        "onetl.plugins": [
            "some-plugin-name=some_plugin.module",  # automatically import all module content
            "some-plugin-class=some_plugin.module.internals:MyClass",  # import a specific class
            "some-plugin-function=some_plugin.module.internals:my_function",  # import a specific function
        ],
    },
)

See setuptools documentation for entry_points

How plugins are imported?¶

User should install a package implementing the plugin:

pip install some-package

Then user should import something from onetl module or its submodules:

import onetl
from onetl.connection import SomeConnection

# and so on

This import automatically executes something like:

import some_plugin.module
from some_plugin.module.internals import MyClass
from some_plugin.module.internals import my_function

If specific module/class/function uses some registration capabilities of onETL, like @hook decorator, it will be executed during this import.

How to enable/disable plugins?¶

version added 0.7.0

Disable/enable all plugins¶

By default plugins are enabled.

To disabled them, you can set environment variable ONETL_PLUGINS_ENABLED to false BEFORE importing onETL. This will disable all plugins autoimport.

But user is still be able to explicitly import some_plugin.module, executing all decorators and registration capabilities of onETL.

Disable a specific plugin (blacklist)¶

If some plugin is failing during import, you can disable it by setting up environment variable ONETL_PLUGINS_BLACKLIST=some-failing-plugin. Multiple plugin names could be passed with , as delimiter.

Again, this environment variable should be set BEFORE importing onETL.

Disable all plugins except a specific one (whitelist)¶

You can also disable all plugins except a specific one by setting up environment variable ONETL_PLUGINS_WHITELIST=some-not-failing-plugin. Multiple plugin names could be passed with , as delimiter.

Again, this environment variable should be set BEFORE importing onETL.

If both whitelist and blacklist environment variables are set, blacklist has a higher priority.

How to see logs of the plugins mechanism?¶

Plugins registration emits logs with INFO level:

import logging

logging.basicConfig(level=logging.INFO)

INFO   |onETL| Found 2 plugins
INFO   |onETL| Loading plugin 'my-plugin'
INFO   |onETL| Skipping plugin 'failing' because it is in a blacklist

More detailed logs are emitted with DEBUG level, to make output less verbose:

import logging

logging.basicConfig(level=logging.DEBUG)

DEBUG  |onETL| Searching for plugins with group 'onetl.plugins'
DEBUG  |Plugins| Plugins whitelist: []
DEBUG  |Plugins| Plugins blacklist: ['failing-plugin']
INFO   |Plugins| Found 2 plugins
INFO   |onETL| Loading plugin (1/2):
DEBUG            name: 'my-plugin'
DEBUG            package: 'my-package'
DEBUG            version: '0.1.0'
DEBUG            importing: 'my_package.my_module:MyClass'
DEBUG  |onETL| Successfully loaded plugin 'my-plugin'
DEBUG            source: '/usr/lib/python3.11/site-packages/my_package/my_module/my_class.py'
INFO   |onETL| Skipping plugin 'failing' because it is in a blacklist