Creating Custom Functions

This section will explain all concepts needed for creating and distributing a Custom Function. A Custom Function is a Python file that implements an API so that Modelon Impact can use it.

Implementing the Custom Function API

Four (two required and two optional) functions need to be implemented to create a Custom Function:

• signature()
• configuration() (Optional)
• default_options() (Optional)
• run()

signature() - specifies the name of the Custom Function and parameters that the user can set. It returns a dictionary with keys:

Required parameters

• name - specifies the name of the Custom Function to appear in Modelon Impact.

Optional parameters

• parameters - list of parameters to be set by the user via the simulation browser. Each parameter is specified by a dictionary in the format:
  • name - the name of the parameter to appear in Modelon Impact
  • type - the type of the parameter: String, Number, Boolean or Enumeration
  • values - A list specifying values to choose the parameter from (applicable only for parameters of type Enumeration).
  • description - description of the parameter that appears when hovering over it
  • defaultValue - default value of the parameter
• version - the version number of the Custom Function
• description - description of what the Custom Function does
• can_initialize_from - Indicates whether to enable initializing from previous results for this Custom Function. Defaults to False if not entered.

An example of the signature() function, as implemented in the example linearize.py is shown below.

def signature():
    """
    The signature function specifying how the Custom Function is presented to the user
    in Modelon Impact.
    :return: A dictionary on the format:
                "version" - The version number of the Custom Function
                "name" - The name of the Custom Function to appear in Modelon Impact
                "description" - A description of what the Custom Function does.
                "parameters" - A list of parameters to be set by the user via the
                Modelon Impact simulation browser (optional).
                Each parameter is specified by a dictionary on the format (all
                optional):
                   "name" - the name of the parameter to appear in Modelon Impact
                   "type" - The type of the parameter: String, Number, Boolean or
                   Enumeration
                   "description" - A description of the parameter
                   "defaultValue" - The default value of the parameter
    """
    return {
        "version": "0.0.1",
        "name": "Linearize",
        "description": "Linearize the model and compute its state space representation"
        "(matrices A, B, C and D).",
        "parameters": [
            {
                "name": "t_linearize",
                "type": "Number",
                "description": "Time (in seconds) at which to perform linearization."
                "To linearize at initialization,"
                "set t=0.",
                "defaultValue": 1,
            },
            {
                "name": "print_to_log",
                "type": "Boolean",
                "description": "Linearized model statistics are printed in the log, "
                "if this option is set to True",
                "defaultValue": True,
            },
        ],
    }

configuration() - defines the configuration of the Custom Function.

Optional configurations

• result_type - The format of the result file. Possible values are "mat" and "csv". Default: "csv".
• requires_compilation - A boolean to skip compilation of the model when running the Custom Function. Default: True.

An example of the configuration() function is shown below.

def configuration():
    return {
        "result_type": "mat",
        'requires_compilation': True
    }

default_options() - defines default options that should be used when calling the Custom Function.

Optional configurations

• compiler - Key value pairs for default compiler options when compiling the FMU to be used by the Custom Function. Default: {}
• simulation - Key value pairs for defult simulation options that the Custom Function should be called with. Default: {}
• solver - Key value pairs for defult solver options that the Custom Function should be called with. Default: {}

An example of the default_options() function is shown below.

def default_options():
    return {
        "simulation": {
            "ncp": 500,
            "dynamic_diagnostics": False,
        }
    }

run() - defines the computation of the Custom Function.

Optional parameters

• get_fmu - is a function returning an FMU object for the model on which the Custom Function is applied.
• environment - dictionary specifying environment variables:
  • result_folder_path – path to the folder where the result is to be saved
  • result_file_name – the name of the file where the results are to be saved
  • workspace_id - The workspace ID
  • case_id - The case ID
  • experiment_id - The experiment ID
  • log_file_name - The name of the log file
• upload_custom_artifact - Function for uploading a custom artifact to the storage. It takes arguments:
  • artifact_id: The artifact ID.
  • local_file_path: Path to the local custom artifact to upload. Returns the route for downloading the artifact.
• parametrization - A dictionary containing the parameter values set in Modelon Impact experiment mode.
• initialize_from - A dictionary-like object containing a previous result to use for initialization.
• options - A dictionary containing the simulation-specific options set by the user. The options dictionary contains the following keys - simulationOptions, solverOptions and simulationLogLevel.

Thereafter follows one argument for each of the parameters of the Custom Function. In the example linearize.py, this means there is a third and fourth argument t_linearize and print_to_log.

def run(
    get_fmu,
    environment,
    parametrization,
    upload_custom_artifact,
    t_linearize,
    print_to_log,
):
    """
    The run function, defining the operation or computation of the custom function.
    :param get_fmu: A function returning an FMU object for the model the custom
    function is applied on, with applied
    non-structural parameters as set in Modelon Impact.
    :param environment: A dictionary specifying environment variables:
                          "result_folder_path" - The path to the folder where the
                          result is to be saved
                          "result_file_name" - The name of the file where the
                          results are to be saved
                          "workspace_id" - The workspace ID
                          "case_id" - The case ID
                          "experiment_id" - The experiment ID
                          "log_file_name" - The name of the log file
    :param upload_custom_artifact: Function for uploading a custom artifact to the
    storage. Takes arguments:
                          "artifact_id": The artifact ID.
                          "local_file_path": Path to the local custom artifact to upload
            Returns the route for downloading the artifact.
    :param parametrization: The parametrization of the model as set in Modelon Impact
    experiment mode.
    :param t_linearize: Time (in seconds) at which to perform linearization.
    :param print_to_log: Toggle weather the linearized model statistics should be shown
                          in the simulation log.
    """

    # In this case, the linearization is packaged into a separate function. This
    # enables to use it outside of Modelon Impact and thereby also makes
    # it convenient to test.
    model = LinearizeModel(get_fmu())
    return linearize(
        model,
        environment,
        upload_custom_artifact,
        t_linearize,
        print_to_log,
        parametrization,
    )

Supported packages and modules

The Python environment that Custom Functions run in has many useful packages that can be used. However, because of the nature of Python it is possible to import any module existing in the environment, even from secondary dependencies or internals not meant for consumers of the package. Using these types of packages or modules means that the Custom Function might stop working with future updates. This section will list all packages supported to use in Custom Functions. Authors of Custom Functions should make sure they are only making imports from packages listed here and furthermore check the documentation for that package on what modules and behavior are documented.

The packages can be split into three categories, computation and analysis, resource interfacing and report creation.

Computation and analysis

Packages for analysis that can be used for getting insights on the model output data.

NumPy - The fundamental package for scientific computing with Python

SciPy - Fundamental algorithms for scientific computing with Python

Pandas - A fast, powerful, flexible and easy to use data analysis and manipulation tool

Note

Pandas have a lot of optional dependencies. These dependencies are used for interfacing external formats or adding additional functionality. The Pandas made available to Custom Functions do not have all these optional dependencies installed, meaning some functionality of Pandas will not work or is not guaranteed to work in future versions. Pandas documentation for optional dependencies can be read here. Supported optional functionality for Pandas used by Custom Functions is:

• matplotlib - Plotting
• SciPy - Miscellaneous statistical functions
• openpyxl - Reading / writing for xlsx files

This means a lot of functions such as pandas.read_html (pandas.DataFrame.to_html can still be used), pandas.read_xml, pandas.read_sql, ... are not supported.

OPTIMICA Compiler Toolkit - The calculation engine used by Modelon Impact. It includes pyfmi, oct.steadystate and pyjmi.

Note

casadi and pymodelica have limited use right now as there is no API to get access to libraries in the workspace, which is needed for doing a compilation.

Warning

Trying to access files on disk in Modelon Impacts internal storage directly is strongly discouraged. Where Modelon Impact stores its files will change in future versions and relying on certain files existing at a path means that the Custom Function might break after any update. Instead use provided APIs to access data stored in Modelon Impact.

Resource interfacing

Some Custom Function needs data stored in Modelon Impact to make decisions. For example, information about the current experiment/case. This section lists packages and modules for doing this.

mimp.api.storage - API for data stored in Modelon Impact. Includes mimp.api.storage.result.read_result, mimp.api.storage.StorageApi and mimp.api.storage.FilesystemStore.

Note

These are deprecated and it is instead recommended that you use the Modelon Impact Client, see below.

Modelon Impact Client - Client library for easy scripting against Modelon Impact

Warning

It is only supported to use the client for reading application data. It is not supported to upload new data or start compilations/simulations with the client in a Custom Function. In the future we might loosen this restriction and allow the upload of certain data.

Report creation

Creating reports often requires a Custom Function to generate plots, tables, or text. Typically, these will be in HTML or some image format. The packages listed here are to help with generation of these artifacts.

Matplotlib - A comprehensive library for creating static, animated, and interactive visualizations

seaborn - A data visualization library with a high-level interface for drawing attractive and informative statistical graphics

Plotly - A graphing library for making interactive, publication-quality graphs

Pandas - Can be used for representing tables that can be converted to HTML

Note

See the note in the section above about limitations when using Pandas.

Dominate - A library for creating and manipulating HTML documents using an elegant DOM API

Miscellaneous

This section contains modules that do not fit into any clear use cases.

mimp.api - Additional convenience methods used in Custom Functions:

• mimp.api.options, functions for handling options
• mimp.api.custom_functions.defaults, functions for default options for built-in Custom Functions
• mimp.api.custom_functions.resulthandlers, result handlers used for simulation

The Python Standard Library - The standard library that comes bundled with Python

Interacting with Modelon Impact

A Custom Function can interact with Modelon Impact in three ways:

1. Return result
2. Write to log
3. Upload an artifact

Returning a result

Instructions to return the result to Modelon Impact:

1. In the run() function, write the result to a csv-file. See Result format for information on the format.
2. Save the csv-file in the path specified by the environment["result_folder_path"]. The name of the file should be set equal to the value of the environment["result_file_name"].

If the result is stored as a dictionary with variable names as keys and result lists as values, it can conveniently be printed to file using the Python package pandas:

# Write the result to a csv file in the prescribed path
csv_file_path = os.path.join(environment["result_folder_path"], environment["result_file_name"])
df = pandas.DataFrame(data=my_result_dictionary)
df.to_csv(csv_file_path, index=False)

Note: Result from a dynamic simulation in pyfmi comes in arrays rather than lists, so it needs to be converted.

# Converting pyfmi result format to a dict where each value is a list
return_res=dict((name,res[name].tolist())) for name in res.keys())

Result format

The result file format is CSV (comma-separated values).

• First row - names of the variables.
• Following rows - values of the variable for a specific time. For parameters and constants, which do not change over time, the same value is repeated in each row.

Example of a csv result file is seen below. k is a parameter, x is a variable, and a is an array.

time,k,x,a[1],a[2]
0   ,2,1, 3  ,4
1   ,2,3, 4  ,5

Writing to the log

Writing to the log is done by adding print statements in the run() function.

print('Print to log')

Custom Functions can use the simulationLogLevel in the options argument to the run() function to decide dynamically how verbose the writing to log should be.

Uploading an artifact

A custom artifact is a file that is created in the run() function and is uploaded to a workspace.

Follow the instructions below to upload a custom artifact:

1. Create the artifact in a temporary folder
2. Upload the artifact using upload_custom_artifact(). upload_custom_artifact() is a parameter of the run() function.

Example of creating and uploading a .mat-file is seen below.

# Write result to a .mat file that can be imported in MATLAB
# First write the result to a temporary directory
temp_dir = tempfile.mkdtemp()
temp_mat_file = os.path.join(temp_dir, "result.mat")
scipy.io.savemat(temp_mat_file, ss)

# Now upload the result to the server as a custom artifact
artifact_id = "ABCD"
artifact_route = upload_custom_artifact(artifact_id, temp_mat_file)

# Finally add a link for downloading the Artifact in the simulation log
print('Artifact can be downloaded from @artifact[here]({})'.format(artifact_route))

Access and download a custom artifact

Getting access to custom artifacts in the web UI requires Custom Functions to create a link in the simulation log. The example above creates this link with the final print statement. A user can then access and download a custom artifact directly by clicking on the link available in the log.

Using the Python Client, you can iterate through all custom artifacts and download them or list their IDs (artifact_id in the code above). It is also possible to access a specific custom artifact directly using one of those IDs.

Distributing your custom function

To enable other users to install your Custom Function it is recommended to add it to a Project and distribute it. To create a Custom Function in your own project:

1. Open the project for editing in VSCode.

2. Create subdirectories Resources/CustomFunctions (note that names are case sensitive).

3. Double check that .impact/project.json file in your project contains a reference to this directory. This should be like:

       {
       "relpath": "Resources/CustomFunctions",
       "contentType": "CUSTOM_FUNCTIONS",
       "defaultDisabled": false,
       "id": "5fc1a0703df74195ae8bc2c52af432ba"
       },
   

4. Create a Python file to develop the Custom Function code in the Resources/CustomFunctions subdirectory.

5. Implement the Custom Function interface, see Custom Function API.

The new function will become available in the workspaces which contain the project after the user interface has been reloaded.

Developing

When you are making changes to your Custom Function it can be useful to view it in the web UI. To see those latest code changes in the UI, reload the web page (press F5). If your custom function is not showing up in the web UI this is most likely because Modelon Impact fails to load it. During development it might therefore be useful to view the server logs for any issues related to loading Custom Functions.

CloudDesktop

The log files in /home/jovyan/impact/logs are useful for any Custom Function-related issues.

The log files in C:/Users/<user_name>/impact/logs are useful for any Custom Function-related issues.