Plugins and Configuration

The general concept of abstract interfaces allows users to create functionality in terms of the interface, separating the concerns of usage and implementation. This tooling is intended to enhance that concept by providing a straightforward way to expose and discover implementations of an interface, as well as factory functionality to create instances of an implementation from a JSON-compliant configuration structure. We provide two mixin classes and a number of utility functions to achieve this. While the two mixin classes function independently and can be utilized on their own, they have been designed such that their combination is symbiotic.

The `Pluggable` Mixin

Motivation: We want to be able to define interfaces to generic concepts and structures that higher-level functionality can be defined around without strictly catering themselves to any particular implementation. We additionally want to allow freedom in implementation variety without much adding to the implementation burden.

In SMQTK-Core, this is addressed via the Pluggable abstract mixin class:

import abc
from smqtk_core.plugin import Pluggable

class MyInterface(Pluggable):
    @abc.abstractmethod
    def my_behavior(self, x: str) -> int:
        """My fancy behavior."""

class NumLetters(MyInterface):
    def my_behavior(self, x: str) -> int:
        return len(x)

class IntCast(MyInterface):
    def my_behavior(self, x: str) -> int:
        return int(x)

if __name__ == "__main__":
    # Discover currently available implementations and print out their names
    impl_types = MyInterface.get_impls()
    print("MyInterface implementations:")
    for t in impl_types:
        print(f"- {t.__name__}")

Running the above in a .py file would then output:

MyInterface implementations:
- NumLetters
- IntCast

This is of course a naive example where implementations are defined right next to the interface. This is not a requirement. Implementations may be spread out across other sub-modules within a package, or even in other packages. In the below section, Plugin Discovery Methods, and in the example Creating an Interface and Exposing Implementations, we will show how to expose implementations of a plugin-enabled interface.

Interfaces vs. Implementations

Classes that inherit from the Pluggable mixin are considered either plugin implementations, or further pluggable interfaces, depending on whether they fully implement abstract methods or not, respectively.

Plugin Discovery Methods

SMQTK-Core’s plugin discovery via the get_impls() method currently allows for finding a plugin implementations in 3 ways:

sub-classes of an interface type defined in the current runtime.
within python modules listed in the environment variable specified by YourInterface.PLUGIN_ENV_VAR. (default SMQTK-Core environment variable name is SMQTK_PLUGIN_PATH, which is defined in Pluggable.PLUGIN_ENV_VAR).
within python modules specified under the entry point extensions namespace defined by YourInterface.PLUGIN_NAMESPACE (default SMQTK-Core extension namespace is smqtk_plugins, which is defined in Pluggable.PLUGIN_NAMESPACE).

When exposing interface implementations, it is generally recommended to use a package’s entry point extensions (3rd bullet above).

The `Configurable` Mixin

Motivation: We want generic helpers to enable serializable configuration for classes while minimally impacting standard class development.

SMQTK-Core provides the Configurable mixin class as well as other helper utility functions in smqtk_core.configuration for generating class instances from configurations. These use python’s inspect module to determine constructor parameterization and default configurations.

Currently this module uses the JSON-serializable format as the basis for input and output configuration dictionaries as a means of defining a relatively simple playing field for communication. Serialization and deserialization is detached from these configuration utilities so tools may make their own decisions there. Python dictionaries are used as a medium in between serialization and configuration input/output.

Classes that inherit from Configurable do need to at a minimum implement the get_config() instance method. This is due to currently lacking the ability to introspect the connection between constructor parameters and how those values are retained in the class. See this method’s doc-string for more details.

The Convenient Combination: `Plugfigurable`

It will likely be desirable to utilize both the Pluggable and Configurable mixins when constructing your own new interfaces. To facilitate this, and to reduce excessive typing, we provide the Plugfigurable helper class. This class does not add or change any functionality. It is merely a convenience to indicate the multiply inherit from both mixin types. Also regarding multiple in inheritance, either Pluggable' nor :class:.Configurable` define a __init__ method, so

Examples

Creating an Interface and Exposing Implementations

In this example, we will show:

A simple interface definition that inherits from both Pluggable and Configurable via the convenient Plugfigurable class.
A sample implementation that is defined in a different module.
The subsequent exposure of that implementation via a python package’s entry point extensions.

Let’s start with the example interface definition which, let’s say, is defined in the MyPackage.interface module of our hypothetical MyPackage package:

# File: MyPackage/interface.py
import abc
from smqtk_core import Plugfigurable

class MyInterface(Plugfigurable):
    """
    A new interface that transitively inherits from Pluggable and Configurable.
    """

    @abc.abstractmethod
    def my_behavior(self, x: str) -> int:
        """My fancy behavior."""

Then, in another package module, MyPackage.implementation, let’s say we define the following implementation. This implementation will need to define all parent class abstract methods in order for the class to satisfy the definition of an “implementation” (see Interfaces vs. Implementations).

# File: MyPackage/implementation.py
from MyPackage.interface import MyInterface
from typing import Any, Dict

class MyImplementation(MyInterface):

    def __init__(self, paramA: int = 1, paramB: int = 2):
        """Implementation constructor."""
        self.a = paramA
        self.b = paramB

    # Abstract method from the Configurable mixin.
    def get_config(self) -> Dict[str, Any]:
        # As per Configurable documentation, this should return the same
        # non-self keys as the constructor.
        return {
            "paramA": self.a,
            "paramB": self.b,
        }

    # Abstract method from MyInterface
    def my_behavior(self, x: str) -> int:
        """My fancy implementation."""
        ...

Lastly, our implementation should be exposed via our package’s entrypoint metadata, using the “smqtk_plugins” namespace. This namespace value is derived from the base Pluggable mixin’s PLUGIN_NAMESPACE class property. Entry point metadata may be specified for a package either via the setuptools.setup() function, the setup.cfg file, or, when using poetry, a [tool.poetry.plugins."..."] section in the pyproject.toml file. This is illustrated in the following:

setuptools.setup() function

from setuptools import setup
setup(
    entry_points={
        "smqtk_plugins": [
            "unique_key = MyPackage.implementation",
            ...
        ]
    }
)

The setup.cfg file

[options.entry_points]
smqtk_plugins =
    unique_key = MyPackage.implementation
    ...

with Poetry in the pyproject.toml file

[tool.poetry.plugins."smqtk_plugins"]
"my_plugins" = "MyPackage.implementation"
...

Now, this implementation will show up as an available implementation of the interface class:

>>> from MyPackage.interface import MyInterface
>>> MyInterface.get_impls()
{<class 'MyPackage.implementation.MyImplementation'>}

The MyImplementation class above should also be all set for configuration because it defines the one required abstract method get_config() and because it’s constructor is only anticipating JSON-compliant data-types. If more complicated types are desired by the constructor, that is completely OK! In such cases, additional methods would need to be overridden as defined in the smqtk_core.configuration module.

Supporting configuration with more complicated constructors

In the above example, only very simple, already-JSON-compliant data types were utilized in the constructor. This not intended to imply that there is a restriction in constructor specification. Instead, the Configurable mixin provides overridable methods to insert conversion to and from JSON-compliant dictionaries, to provide a class “default” configuration as well as for factory-generation of a class instance from an input configuration.

Instead of the above implementation, let us consider a slightly more complex implementation of MyInterface defined above. In this new implementation we show the implementation of two additional class methods from the Configurable mixin: Configurable.get_default_config() and Configurable.from_config(). These allow us to customize the how we maintain JSON-compliance.

from MyPackage.interface import MyInterface
from typing import Any, Dict, Type, TypeVar
from datetime import datetime


C = TypeVar("C", bound="DateContainer")


class DateContainer (MyInterface):
    """
    This example implementation takes in datetime instances as constructor
    parameters, one of which has a default.
    Both parameters in this example require a `datetime` instance value at
    construction time, but since `b_date` has a default value, it is not
    strictly required that an input configuration provide a value for the
    `b_date` parameter since there is a default to draw upon.
    """

    def __init__(
        self,
        a_date: datetime,
        b_date: datetime = datetime.utcfromtimestamp(0),
    ):
        self.a_date = a_date
        self.b_date = b_date

    # NEW FROM PREVIOUS EXAMPLE
    # Abstract method from the Configurable mixin.
    @classmethod
    def get_default_config(cls) -> Dict[str, Any]:
        # Utilize the mixin-class implementation to introspect our
        # constructor and make a parameter-to-value dictionary.
        cfg = super().get_default_config()
        # We are ourself, so we know that cfg['a_date'] has a default value
        # and it will be a datetime instance.
        cfg['b_date'] = datetime_to_str(cfg['b_date'])
        # We know that `a_date` is not given a default, so its "default"
        # value of None, which is JSON-compliant, is left alone.
        return cfg

    # NEW FROM PREVIOUS EXAMPLE
    # Abstract method from the Configurable mixin.
    @classmethod
    def from_config(
        cls: Type[C],
        config_dict: Dict,
        merge_default: bool = True
    ) -> C:
        # Following the example found in the Configurable.from_config
        # doc-string.
        config_dict = dict(config_dict)
        # Convert required input data into the constructor-expected types.
        # This implementation will expectedly error if the expected input
        # is missing.
        config_dict['a_date'] = str_to_datetime(config_dict['a_date'])
        # b_date might not be there because there's a default that can fill
        # in.
        b_date = config_dict.get('b_date', None)
        if b_date is not None:
          config_dict['b_date'] = str_to_datetime(b_date)
        return super().from_config(config_dict, merge_default=merge_default)

    # Abstract method from the Configurable mixin.
    def get_config(self) -> Dict[str, Any]:
        # This now matches the same complex-to-JSON conversion as the
        # `get_default_config`. We show the use of a helper function to
        # reduce code duplication.
        return {
            "a_date": datetime_to_str(self.date),
        }

    # Abstract method from MyInterface
    def my_behavior(self, x: str) -> int:
        """My fancy implementation."""
        ...


def datetime_to_str(dt: datetime) -> str:
    """ Local helper function for config conversion from datetime. """
    # This conversion may be arbitrary to the level of detail that this
    # local implementation considers important. We choose to use strings
    # here as an example, but there's nothing special that requires that
    # other than JSON type compliance.
    return str(dt)


def str_to_datetime(s: str) -> datetime:
    """ Local helper function for config conversion into datetime """
    # Reverse of above converter.
    if '.' in s:  # has decimal seconds
        return datetime.strptime(s, "%Y-%m-%d %H:%M:%S.%f")
    return datetime.strptime(s, "%Y-%m-%d %H:%M:%S")

The above then allows us to create instances of this instance with the JSON config:

>>> inst = DateContainer.from_config({
...   "a_date": "2021-01-01 00:00:00.123123"
...   "b_date": "1970-01-01 00:00:01",
... })
>>> str(inst.a_date)
'2021-01-01 00:00:00.123123'
>>> str(inst.b_date)
'1970-01-01 00:00:01'

Or even just:

>>> inst = DateContainer.from_config({
...   "a_date": "2021-01-01 00:00:00.123123"
... })
>>> str(inst.a_date)
'2021-01-01 00:00:00.123123'
>>> str(inst.b_date)
'1970-01-01 00:00:00'

While such inline usage is less likely to be called so directly as opposed to just calling the constructor, this form is useful when constructing directly from a deserialized configuration:

>>> # Maybe received this from a web request!
>>> from_file = '{"a_date": "2021-01-01 00:00:00.123123", "b_date": "1970-01-01 00:00:01"}'=
>>> import json
>>> inst = DateContainer.from_config(json.loads(from_file))
>>> str(inst.a_date)
'2021-01-01 00:00:00.123123'
>>> str(inst.b_date)
'1970-01-01 00:00:01'

Instances may also be “cloned” by creating a new instance from the current configuration output from another instance using the Configurable.get_config() instance method:

>>> # assume and `inst` from above
>>> inst2 = DataContainer.from_config(inst.get_config())
>>> assert inst.a_date == inst2.a_date
>>> assert inst.b_date == inst2.b_date

This is again a little silly in the demonstration context because we know what instance type and attributes are to construct a second one, however if the concrete instance is not known at runtime, this may be an alternate means of constructing a duplicate instance (at least “duplicate” in regards to construction).

Multiple Implementation Choices

One usage mode of smqtk_core.configuration configuration is when a configuration slot may be comprised of one of multiple Configurable-implementing choices. Also found in the smqtk_core.configuration module are additional helper functions for navigating this use-case:

make_default_config()

to_config_dict()

from_config_dict()

These methods utilize JSON-compliant dictionaries to represent configurations that follow the schema:

{
    "type": "object",
    "properties": {
        "type": {"type": "string"},
    },
    "additionalProperties": {"type": "object"},
}

Getting Defaults from a group of types

When programmatically creating a multiple-choice configuration structure for output, the make_default_config() function may be convenient to use. This takes in some Iterable of Configurable-inheriting types and returns a JSON-compliant dictionary. This may be useful, for example, when a higher-order tool wants to programmatically generate a default configuration for itself for serialization or for some other interface.

This function may be called with an independently generated Iterable of inputs, however this also melds well with the:meth:.Pluggable.get_impls class method. For example, let us use the pluggable MyInterface class defined above:

>>> from smqtk_core.configuration import make_default_config
>>> cfg_dict = make_default_config(MyInterface.get_impls())
>>> cfg_dict
{
    "type": None,
    "__main__.MyImplementation": {
        "paramA": 1,
        "paramB": 2
    },
    "__main__.DateContainer": {
        "a_date": None,
        "b_date": "1970-01-01 00:00:00"
    }
}

See the method documentation for additional details.

Factory constructing from configuration

The opposite of above, the from_config_dict() will take a configuration multiple-choice dictionary structure and “hydrate” an instance of the configured type with the configured parameters (if it’s available, of course). This function again takes an Iterable of Configurable-inheriting types which again melds well with the Pluggable.get_impls class method where applicable.

For example, if we take the “default” configuration output above, change the "type" value to refer to the “MyImplementation” type and pass it to this function, we get a fully constructed instance:

>>> from smqtk_core.configuration import from_config_dict
>>> # Let's modify the config away from default values.
>>> cfg_dict['__main__.MyImplementation'] = {'paramA': 77, 'paramB': 444}
>>> inst = from_config_dict(cfg_dict, MyInterface.get_impls())
>>> assert inst.a == 77
>>> assert inst.b == 444

See the method documentation for additional details.

Help with writing unit tests for Configurable-implementing types

When creating new implementations of things that includes Configurable functionality it is often good to include tests that make sure the expected configuration capabilities operate as they should. We include a helper function to lower the cost of adding such a test: configuration_test_helper(). This will exercise certain runtime-assumptions that are not strictly required to define and construct Configurable-inheriting types.

For an example, let’s assume we are writing a unit test for the above-defined MyImplementation class:

>>> from smqtk_core.configuration import configuration_test_helper
>>>
>>> class TestMyImplementation:
...     def test_config(self):
...         inst = MyImplementation(paramA=77, paramB=444)
...         for i in configuration_test_helper(inst):
...             # Checking that yielded instance properties are as expected
...             assert i.a == 77
...             assert i.b == 444

See the method documentation for additional details.

Module References

`smqtk_core`

class smqtk_core.Plugfigurable[source]: When you don’t want to have to constantly inherit from two mixin classes all the time, we provide this as a convenience that descends from both mixin classes: Pluggable and Configurable.

`smqtk_core.plugin`

Helper functions and mixin interface for implementing class type discovery, filtering and a convenience mixin class.

This package provides a number of discover_via_… functions that return sets of type instances as found by the method described by that function.

These methods may be composed to create a pool of types that may be then filtered via the filter_plugin_types function to those types that are specifically “plugin types” for the given interface class. See the is_valid_plugin function documentation for what it means to be a “plugin” of an interface type.

While the above are defined in fairly general terms, the Pluggable class type defined last here is a mixin class that utilizes all of the above in a manner specific manner for the purposes of SMQTK. This mixin class defines the class-method get_impls() that will return currently discoverable plugins underneath the type it was called on. This discovery will follow the values of the PLUGIN_ENV_VAR and PLUGIN_NAMESPACE class variables defined in the interface class you are calling get_impls() from, using inherited values if not immediately specified.

Because these plugin semantics are pretty low level and commonly utilized, logging can be extremely verbose. Logging in this module, while still exists, is set to emit only at log level 1 or lower (“trace”).

NOTE: The type annotations for discover_via_subclasses and filter_plugin_types are currently set to the broad Type annotation. Ideally these should use Type[T] instead, but there is currently a known issue with mypy where it aggressively assumes that an annotated type must be constructable, so it emits an error when the functions are called with an abstract interface_type. When this is resolved in mypy these annotations should be updated.

exception smqtk_core.plugin.NotAModuleError[source]: Exception for when the discover_via_entrypoint_extensions function found an entrypoint that was not a module specification.

class smqtk_core.plugin.Pluggable[source]

Interface for classes that have plugin implementations.

classmethod get_impls() → Set[Type[P]][source]

Discover and return a set of classes that implement the calling class.

See the various discover_via_*() functions in this module for more details on the logic of how implementing classes (aka “plugins”) are discovered.

The class-level variables PLUGIN_ENV_VAR and PLUGIN_NAMESPACE may be overridden to change what environment and entry-point extension are looked for, respectively.

Returns: Set of discovered class types that are considered “valid” plugins of this type. See is_valid_plugin() for what we define a “valid” type to be relative to this class.

classmethod is_usable() → bool[source]

Check whether this class is available for use.

Since certain plugin implementations may require additional dependencies that may not yet be available on the system, or other runtime conditions, this method may be overridden to check for those and return a boolean saying if the implementation is available for usable. When this method returns True, the class is declaring that it should be constructable and usable in the current environment.

By default, this method will return True unless a subclass overrides this class-method with their specific logic.

NOTES:

This should be a class method
When an implementation is deemed not usable, this should emit a
(user) warning, or some other kind of logging, detailing why the implementation is not available for use.

Returns: Boolean determination of whether this implementation is usable in the current environment.
Return type: bool

smqtk_core.plugin.discover_via_entrypoint_extensions(entrypoint_ns: str) → Set[Type][source]

Discover and return types defined in modules exposed through the entry-point extensions defined for the given namespace by installed python packages.

Other installed python packages may define one or more extensions for a namespace, as specified by ns, in their “setup.py”. This should be a single or list of extensions that specify modules within the installed package where plugins for export are implemented.

Currently, this method only accepts extensions that export a module as opposed to specifications of a specific attribute in a module. This is due to other methods of type discovery not necessarily honoring the selectivity that specific attribute specification provides (Looking at you __subclasses__…).

For example, as a single specification string:

...
entry_points={
    "smqtk_plugins": "my_package = my_package.plugins"
]
...

Or in list form of multiple specification strings:

...
entry_points = {
    "smqtk_plugins": [
        "my_package_mode_1 = my_package.mode_1.plugins",
        "my_package_mode_2 = my_package.mode_2.plugins",
    ]
}
...

Parameters: entrypoint_ns – The name of the entry-point mapping in to look for extensions under.
Returns: Set of discovered types from the modules and class types specified in the extensions under the specified entry-point.

smqtk_core.plugin.discover_via_env_var(env_var: str) → Set[Type][source]

Discover and return types specified in python-importable modules specified in the given environment variable.

We expect the given environment variable to define zero or more python module paths from which to yield all contained type definitions (i.e. things that descent from type). If there is an empty path element, it is skipped (e.g. “foo::bar:baz” will only attempt importing foo, bar and baz modules).

These python module paths should be separated with the same separator as would be used in the PYTHONPATH environment variable specification.

If a module defines no class types, then no types are included from that source for return.

An expected use-case for this discovery method is for modules that are not installed but otherwise accessible via the python search path. E.g. local modules, modules accessible through PYTHONPATH search path modification, modules accessible through sys.path modification.

Any errors raised from attempting to import a module are propagated upward.

Parameters: env_var – The name of the environment variable to read from.
Raises: ModuleNotFoundError – When one or more module paths specified in the given environment variable are not importable.
Returns: Set of discovered types from the modules specified in the environment variable’s contents.

smqtk_core.plugin.discover_via_subclasses(interface_type: Type) → Set[Type][source]

Utilize the __subclasses__ to discover nested subclasses for a given interface type.

This approach will be able to observe any implementations that have been defined, anywhere at all, at the point of invocation, which can circumvent efforts towards specificity that other discovery methods may provide. For example, discover_via_entrypoint_extensions may return a single type that was specifically exported from a module whereas this method will, called afterwards, yield all the other types defined in that entry-point-imported module.

The use of this discovery method may also result in different returns depending on the import state at the time of invocation. E.g. further imports may increase the quantity of returns from this function.

This function uses depth-first-search when traversing subclass tree.

Reference:: https://docs.python.org/3/library/stdtypes.html#class.__subclasses__
NOTE: subclasses are retained via weak-references, so if a normal condition: is exposing types from something that otherwise raised an exception or if a local definition is leaking, apparently an import gc; gc.collect() wipes out the return as long as it’s not referenced, of course as long as its reference is not retained by something.

Parameters: interface_type – The interface type to recursively find subclasses under.
Returns: Set of recursive subclass types under interface_type.

smqtk_core.plugin.filter_plugin_types(interface_type: Type, candidate_pool: Collection[Type]) → Set[Type][source]

Filter the given set of types to those that are “plugins” of the given interface type.

See the documentation for is_valid_plugin() for what we define a “plugin type” to be relative to the given interface_type.

We consider that there may be duplicate type instances in the given candidate pool. Due to this we will consider an instance of a type only once and return a set type to contain the validated types.

Parameters

interface_type – The parent type to filter on.
candidate_pool – Some iterable of types from which to collect interface type plugins from.

Returns

Set of types that are considered “plugins” of the interface types following the above listed rules.

smqtk_core.plugin.is_valid_plugin(cls: Type, interface_type: Type) → bool[source]

Determine if a class type is a valid candidate for plugin discovery.

In particular, the class type cls must satisfy several conditions:

It must not literally be the given interface type.

It must be a strict subtype of interface_type.

It must not be an abstract class. (i.e. no lingering abstract methods or properties if the abc.ABCMeta metaclass has been used).

If the cls is a subclass of Pluggable, it must report as usable via its is_usable() class method.

Logging for this function, when enabled can be very verbose, and is only active with a logging level of 1 or lower.

Parameters

cls – The class type whose validity is being tested
interface_type – The base class under consideration

Returns

True if the class is a valid candidate for discovery, and False otherwise.

Return type

bool

`smqtk_core.configuration`

Helper interface and functions for generalized object configuration, to and from JSON-compliant dictionaries.

While this interface and utility methods should be general enough to add JSON-compliant dictionary-based configuration to any object, this was created in mind with the SMQTK plugin module.

Standard configuration dictionaries should be JSON compliant take the following general format:

{
    "type": "one-of-the-keys-below",
    "ClassName1": {
        "param1": "val1",
        "param2": "val2"
    },
    "ClassName2": {
        "p1": 4.5,
        "p2": null
    }
}

The “type” key is considered a special key that should always be present and it specifies one of the other keys within the same dictionary. Each other key in the dictionary should be the name of a Configurable inheriting class type. Usually, the classes named within a block inherit from a common interface and the “type” value denotes a selection of a specific sub-class for use, though this is not required property of these constructs.

class smqtk_core.configuration.Configurable[source]

Interface for objects that should be configurable via a configuration dictionary consisting of JSON types.

classmethod from_config(config_dict: Dict, merge_default: bool = True) → C[source]

Instantiate a new instance of this class given the configuration JSON-compliant dictionary encapsulating initialization arguments.

This base method is adequate without modification when a class’s constructor argument types are JSON-compliant. If one or more are not, however, this method then needs to be overridden in order to convert from a JSON-compliant stand-in into the more complex object the constructor requires. It is recommended that when complex types are used they also inherit from the Configurable in order to hopefully make easier the conversion to and from JSON-compliant stand-ins.

When this method does need to be overridden, this usually looks like the following pattern:

D = TypeVar("D", bound="MyClass")

class MyClass (Configurable):

    @classmethod
    def from_config(
        cls: Type[D],
        config_dict: Dict,
        merge_default: bool = True
    ) -> D:
        # Perform a shallow copy of the input ``config_dict`` which
        # is important to maintain idempotency.
        config_dict = dict(config_dict)

        # Optionally guarantee default values are present in the
        # configuration dictionary.  This is useful when the
        # configuration dictionary input is partial and the logic
        # contained here wants to use config parameters that may
        # have defaults defined in the constructor.
        if merge_default:
            config_dict = merge_dict(cls.get_default_config(),
                                     config_dict)

        #
        # Perform any overriding of `config_dict` values here.
        #

        # Create and return an instance using the super method.
        return super().from_config(config_dict,
                                   merge_default=merge_default)

Note on type annotations: When defining a sub-class of configurable and override this class method, we will need to defined a new TypeVar that is bound at the new class type. This is because super requires a type to be given that descends from the implementing type. If C is used as defined in this interface module, which is upper-bounded on the base Configurable class, the type analysis will see that we are attempting to invoke super with a type that may not strictly descend from the implementing type (MyClass in the example above), and cause an error during type analysis.

Parameters

config_dict (dict) – JSON compliant dictionary encapsulating a configuration.
merge_default (bool) – Merge the given configuration on top of the default provided by get_default_config.

Returns

Constructed instance from the provided config.

abstract get_config() → Dict[str, Any][source]

Return a JSON-compliant dictionary that could be passed to this class’s from_config method to produce an instance with identical configuration.

In the most cases, this involves naming the keys of the dictionary based on the initialization argument names as if it were to be passed to the constructor via dictionary expansion. In some cases, where it doesn’t make sense to store some object constructor parameters are expected to be supplied at as configuration values (i.e. must be supplied at runtime), this method’s returned dictionary may leave those parameters out. In such cases, the object’s from_config class-method would also take additional positional arguments to fill in for the parameters that this returned configuration lacks.

Returns: JSON type compliant configuration dictionary.
Return type: dict

classmethod get_default_config() → Dict[str, Any][source]

Generate and return a default configuration dictionary for this class. This will be primarily used for generating what the configuration dictionary would look like for this class without instantiating it.

By default, we observe what this class’s constructor takes as arguments, turning those argument names into configuration dictionary keys. If any of those arguments have defaults, we will add those values into the configuration dictionary appropriately. The dictionary returned should only contain JSON compliant value types.

It is not be guaranteed that the configuration dictionary returned from this method is valid for construction of an instance of this class.

Returns: Default configuration dictionary for the class.
Return type: dict

>>> # noinspection PyUnresolvedReferences
>>> class SimpleConfig(Configurable):
...     def __init__(self, a=1, b='foo'):
...         self.a = a
...         self.b = b
...     def get_config(self):
...         return {'a': self.a, 'b': self.b}
>>> self = SimpleConfig()
>>> config = self.get_default_config()
>>> assert config == {'a': 1, 'b': 'foo'}

smqtk_core.configuration.cls_conf_from_config_dict(config: Dict, type_iter: Iterable[Type[T]]) → Tuple[Type[T], Dict][source]

Helper function for getting the appropriate type and configuration sub-dictionary based on the provided “standard” SMQTK configuration dictionary format (see above module documentation).

Parameters

config – Configuration dictionary to draw from.
type_iter – An iterable of class types to select from.

Raises

ValueError –

This may be raised if:

type field not present in config.
type field set to None
type field did not match any available configuration in the given config.
Type field did not specify any implementation key.

Returns

Appropriate class type from type_iter that matches the configured type as well as the sub-dictionary from the configuration. From this return, type.from_config(config) should be callable.

smqtk_core.configuration.cls_conf_to_config_dict(cls: Type, conf: Dict) → Dict[source]

Helper function for creating the appropriate “standard” smqtk configuration dictionary given a Configurable-implementing class and a configuration for that class.

This very simple function simply arranges a semantic class key and an associated dictionary into a normal pattern used for configuration in SMQTK:

>>> class SomeClass (object):

… pass >>> cls_conf_to_config_dict(SomeClass, {0: 0, ‘a’: ‘b’}) == { … ‘type’: ‘smqtk_core.configuration.SomeClass’, … ‘smqtk_core.configuration.SomeClass’: {0: 0, ‘a’: ‘b’} … } True

Parameters

cls (type[Configurable]) – A class type implementing the Configurable interface.
conf (dict) – SMQTK standard type-optioned configuration dictionary for the given class and dictionary pair.

Returns

“Standard” SMQTK JSON-compliant configuration dictionary

Return type

dict

smqtk_core.configuration.configuration_test_helper(inst: C, config_ignored_params: Union[Set, FrozenSet] = frozenset({}), from_config_args: Sequence = ()) → Tuple[C, C, C][source]

Helper function for testing the get_default_config/from_config/get_config methods for class types that in part implement the Configurable mixin class. This function also tests that inst’s parent class type’s get_default_config returns a dictionary whose keys’ match the constructor’s inspected parameters (except “self” of course).

This constructs 3 additional instances based on the given instance following the pattern:

inst-1  ->  inst-2  ->  inst-3
        ->  inst-4

This refers to inst-2 and inst-4 being constructed from the config from inst, and inst-3 being constructed from the config of inst-2. The equivalence of each instance’s config is cross-checked with the other instances. This is intended to check that a configuration yields the same class configurations and that the config does not get mutated by nested instance construction.

This function uses assert calls to check for consistency.

We return all instances constructed in case the caller wants to make additional instance integrity checks.

Parameters

inst (Configurable) – Configurable-mixin inheriting class to test.
config_ignored_params (set[str]) – Set of parameter names in the instance type’s constructor that are ignored by get_default_config and from_config. This is empty by default.
from_config_args (tuple) – Optional additional positional arguments to the input inst.from_config method after the configuration dictionary.

Returns

Instance 2, 3, and 4 as described above.

Return type

(Configurable,Configurable,Configurable)

smqtk_core.configuration.from_config_dict(config: Dict, type_iter: Iterable[Type[C]], *args: Any) → C[source]

Helper function for instantiating an instance of a class given the configuration dictionary config from available types provided by type_iter via the Configurable interface’s from_config class-method.

args are additionally positional arguments to be passed to the type’s from_config method on return.

Example: >>> class SimpleConfig(Configurable): … def __init__(self, a=1, b=’foo’): … self.a = a … self.b = b … def get_config(self): … return {‘a’: self.a, ‘b’: self.b} >>> example_config = { … ‘type’: ‘smqtk_core.configuration.SimpleConfig’, … ‘smqtk_core.configuration.SimpleConfig’: { … “a”: 3, … “b”: “baz” … }, … } >>> inst = from_config_dict(example_config, {SimpleConfig}) >>> isinstance(inst, SimpleConfig) True >>> inst.a == 3 True >>> inst.b == “baz” True

Raises

ValueError –
This may be raised if:
- type field not present in config.
- type field set to None
- type field did not match any available configuration in the given config.
- Type field did not specify any implementation key.
AssertionError – This may be raised if the class specified as the configuration type, is present in the given type_iter but is not a subclass of the Configurable interface.
TypeError – Insufficient/incorrect initialization parameters were specified for the specified type’s constructor.

Parameters

config – Configuration dictionary to draw from.
type_iter – An iterable of class types to select from.
args (object) – Other positional arguments to pass to the configured class’ from_config class method.

Returns

Instance of the configured class type as specified in config and as available in type_iter.

smqtk_core.configuration.make_default_config(configurable_iter: Iterable[Type[C]]) → Dict[str, Union[None, str, Dict]][source]

Generated default configuration dictionary for the given iterable of Configurable-inheriting types.

For example, assuming the following simple class that descends from Configurable, we would expect the following behavior:

>>> # noinspection PyAbstractClass
>>> class ExampleConfigurableType (Configurable):
...     def __init__(self, a, b):
...        ''' Dummy constructor '''
>>> make_default_config([ExampleConfigurableType]) == {
...     'type': None,
...     'smqtk_core.configuration.ExampleConfigurableType': {
...         'a': None,
...         'b': None,
...     }
... }
True

Note that technically ExampleConfigurableType is still abstract as it does not implement get_config. The above call to make_default_config still functions because we only use the get_default_config class method and do not instantiate any types given to this function. While functionally acceptable, it is generally not recommended to draw configurations from abstract classes.

The "type" returned is None because we explicitly do not make any decisions about an appropriate default type. Additionally, this value stays None even when there is just one choice as we do not assume that is a valid choice as well as do not assume that the default configuration for that choice is valid for construction. This serves to cause the user to explicitly check that the multiple-choice configuration is set to point to a choice as well as that the choice is properly configured.

Parameters: configurable_iter – An iterable of class types class types that sub-class Configurable.
Returns: Base configuration dictionary with an empty type field, and containing the types and initialization parameter specification for all implementation types available from the provided getter method.

smqtk_core.configuration.to_config_dict(c_inst: Configurable) → Dict[source]

Helper function that transforms the configuration dictionary retrieved from configurable_inst into the “standard” SMQTK configuration dictionary format (see above module documentation).

For example, with a simple Configurable derived class:

>>> class SimpleConfig(Configurable):
...     def __init__(self, a=1, b='foo'):
...         self.a = a
...         self.b = b
...     def get_config(self):
...         return {'a': self.a, 'b': self.b}
>>> e = SimpleConfig(a=2, b="bar")
>>> to_config_dict(e) == {
...     "type": "smqtk_core.configuration.SimpleConfig",
...     "smqtk_core.configuration.SimpleConfig": {
...         "a": 2,
...         "b": "bar"
...     }
... }
True

Parameters: c_inst (Configurable) – Instance of a class type that subclasses the Configurable interface.
Returns: Standard format configuration dictionary.
Return type: dict

Plugins and Configuration

The Pluggable Mixin

Interfaces vs. Implementations

Plugin Discovery Methods

The Configurable Mixin

The Convenient Combination: Plugfigurable

Examples

Creating an Interface and Exposing Implementations

Supporting configuration with more complicated constructors

Multiple Implementation Choices

Getting Defaults from a group of types

Factory constructing from configuration

Help with writing unit tests for Configurable-implementing types

Module References

smqtk_core

smqtk_core.plugin

smqtk_core.configuration

The `Pluggable` Mixin

The `Configurable` Mixin

The Convenient Combination: `Plugfigurable`

`smqtk_core`

`smqtk_core.plugin`

`smqtk_core.configuration`