Skip to content

Executing statements in MySQL

Warning

Methods below read all the rows returned from DB to Spark driver memory, and then convert them to DataFrame.

Do NOT use them to read large amounts of data. Use DBReader or MySQL.sql instead.

How to

There are 2 ways to execute some statement in MySQL

Use MySQL.fetch

Use this method to perform some SELECT query which returns small number or rows, like reading MySQL config, or reading data from some reference table. Method returns Spark DataFrame.

Method accepts MySQL.FetchOptions.

Warning

Please take into account MySQL types.

Syntax support for MySQL.fetch

This method supports any query syntax supported by MySQL, like:

  • ✅︎ SELECT ... FROM ...
  • ✅︎ WITH alias AS (...) SELECT ...
  • ✅︎ SELECT func(arg1, arg2) or {?= call func(arg1, arg2)} - special syntax for calling function
  • ✅︎ SHOW ...
  • ❌ SET ...; SELECT ...; - multiple statements not supported

Examples in MySQL.fetch

from onetl.connection import MySQL

mysql = MySQL(...)

df = mysql.fetch(
    "SELECT value FROM some.reference_table WHERE key = 'some_constant'",
    options=MySQL.FetchOptions(queryTimeout=10),
)
mysql.close()
value = df.collect()[0][0]  # get value from first row and first column

Use MySQL.execute

Use this method to execute DDL and DML operations. Each method call runs operation in a separated transaction, and then commits it.

Method accepts MySQL.ExecuteOptions.

Syntax support in MySQL.execute

This method supports any query syntax supported by MySQL, like:

  • ✅︎ CREATE TABLE ..., CREATE VIEW ..., and so on
  • ✅︎ ALTER ...
  • ✅︎ INSERT INTO ... SELECT ..., UPDATE ..., DELETE ..., and so on
  • ✅︎ DROP TABLE ..., DROP VIEW ..., and so on
  • ✅︎ CALL procedure(arg1, arg2) ... or {call procedure(arg1, arg2)} - special syntax for calling procedure
  • ✅︎ other statements not mentioned here
  • ❌ SET ...; SELECT ...; - multiple statements not supported

Examples for MySQL.execute

from onetl.connection import MySQL

mysql = MySQL(...)

mysql.execute("DROP TABLE schema.table")
mysql.execute(
    """
    CREATE TABLE schema.table (
        id bigint,
        key text,
        value float
    )
    ENGINE = InnoDB
    """,
    options=MySQL.ExecuteOptions(queryTimeout=10),
)

Options

MySQLFetchOptions

Bases: JDBCFetchOptions

__doc__ = JDBCFetchOptions.__doc__.replace('SomeDB', 'MySQL') class-attribute instance-attribute

_forward_refs_updated = False class-attribute

fetchsize = None class-attribute instance-attribute

How many rows to fetch per round trip.

Tuning this option can influence performance of reading.

Warning

Default value depends on driver. For example, Oracle has default fetchsize=10.

query_timeout = Field(default=None, alias='queryTimeout') class-attribute instance-attribute

The number of seconds the driver will wait for a statement to execute. Zero means there is no limit.

This option depends on driver implementation, some drivers can check the timeout of each query instead of an entire JDBC batch.

Config

__init__(**kwargs)

_check_options_allowed(values)

_forward_refs() classmethod

_get_matching_options(values, matches) classmethod

_strip_prefix(key, prefix) staticmethod

_strip_prefixes(values)

_warn_unknown_options(values)

parse(options) classmethod

If a parameter inherited from the ReadOptions class was passed, then it will be returned unchanged. If a Dict object was passed it will be converted to ReadOptions.

Otherwise, an exception will be raised

MySQLExecuteOptions

Bases: JDBCExecuteOptions

__doc__ = JDBCExecuteOptions.__doc__.replace('SomeDB', 'MySQL') class-attribute instance-attribute

_forward_refs_updated = False class-attribute

fetchsize = None class-attribute instance-attribute

How many rows to fetch per round trip.

Tuning this option can influence performance of reading.

Warning

Default value depends on driver. For example, Oracle has default fetchsize=10.

query_timeout = Field(default=None, alias='queryTimeout') class-attribute instance-attribute

The number of seconds the driver will wait for a statement to execute. Zero means there is no limit.

This option depends on driver implementation, some drivers can check the timeout of each query instead of an entire JDBC batch.

Config

__init__(**kwargs)

_check_options_allowed(values)

_forward_refs() classmethod

_get_matching_options(values, matches) classmethod

_strip_prefix(key, prefix) staticmethod

_strip_prefixes(values)

_warn_unknown_options(values)

parse(options) classmethod

If a parameter inherited from the ReadOptions class was passed, then it will be returned unchanged. If a Dict object was passed it will be converted to ReadOptions.

Otherwise, an exception will be raised