Creating Custom Functions

This section will explain all concepts needed for creating and distributing a Custom Function. A Custom Function is a Python file that implements an API so that Modelon Impact can use it.

Implementing the Custom Function API

Modelon Impact utilizes four callbacks when processing a Custom function:

• signature() specifies the name and user settable parameters of the custom function.
• configuration() (Optional) defines the properties of the custom function affecting the server side execution.
• default_options() (Optional) defines the default compiler and simulation options expected by the custom function.
• run() is the main simulation execution entry point of the custom function.

The signature function

signature() - Specifies the name of the Custom Function and parameters that the user can set. The function is expected to returns a dictionary with keys:

Required

• name - Specifies the name of the Custom Function to appear in Modelon Impact.

Optional

• parameters - Specifies the list of parameters expected by the custom function. Those parameters can be specified by the user in Modelon Impact UX. They are then submitted into simulation backend via REST API and provided as Python variables to the run method of the Custom function. Each parameter is specified by a dictionary in the format:
  • name - The name of the parameter to appear in Modelon Impact.
  • type - Data type of the custom function parameter. Supported types are:
    • String - Value is a string.
    • Number - Value is real or integer.
    • Boolean - Value is true or false.
    • Enumeration - Values should specify the array of allowed choices, value is a string.
    • CaseResult - Custom function in Python will receive a case result reference as a CaseResult object (see example below). User interface for the parameter will contain case result browser. REST API will transfer a string of the form /.
    • ExperimentResult - Custom function in Python will receive a experiment result reference as a ExperimentResult object (see example below). User interface for the parameter will contain experiment result browser. REST API will transfer a string of the form .
    • FileURI - Custom function in Python will receive a URI reference as a FilePath object (see example below). User interface for the parameter will contain either a Modelica resources file selector or a Custom artifacts file selector. REST API will transfer a string of the form <scheme>://<netloc>/<path> pointing to an existing file, where supported schemes are modelica and impact-artifact.
    • VariableNames - Value is a list of strings where each string is a name of a model variable.
  • values - A list specifying values to choose the parameter from (applicable only for parameters of type Enumeration).
  • description - Description of the parameter that appears when hovering over it.
  • defaultValue - Default value of the parameter.
  • optional - If set to True, the parameter is considered optional and user may skip specifying a value for it. The defaultValue attribute can be used to specify a default value for the parameter to use if the parameter lacks a user specified value. If both defaultValue and user input value are not specified, the parameter is set to None. Defaults to False if not entered.
  • filter - Filters files that fulfill the given pattern defined by 'text1 (.ext1);;text2 (.ext2);' to show only files with fileextension .ext1 or .ext2 and displaying a description text 'text1' and 'text2', respectively. The parameter is only applicable for FileURI and follows the syntax of the Modelica loadSelector annotation.
• kind - Defautl is 'EXECUTOR'. Can also be 'ORCHESTRATOR', see orchestration for details.
• version - The version number of the Custom Function.
• description - Description of the custom function to be shown as a tool-tip.
• can_initialize_from - Indicates whether to enable initializing from previous results for this Custom Function. Defaults to False if not entered.

An example of the signature() function is shown below. Corresponding definition of a run() function is presented below.

def signature():
    return {
        "version": "0.0.1",
        "name": "My custom function",
        "description": "Custom function does cool stuff.",
        "can_initialize_from": False,
        "parameters": [
            {
                "name": "number_parameter",
                "type": "Number",
                "description": "A numeric parameter.",
                "defaultValue": 1,
            },
            {
                "name": "bool_param",
                "type": "Boolean",
                "description": "A boolean parameter.",
                "defaultValue": True,
            },
            {
                "name": "string_param",
                "type": "String",
                "description": "A string parameter.",
                "defaultValue": "",
            },
            {
                "name": "enum_param",
                "type": "Enumeration",
                "description": "An enum parameter.",
                "defaultValue": 1,
                "values": ["1", "2", "3"],
            },
            {
                "name": "file_uri_param",
                "type": "FileURI",
                "filter": "Text files (*.txt);;MATLAB MAT-files (*.mat)",
                "description": "A data file",
                "optional": True,
            },
            {
                "name": "experiment_result_param",
                "type": "ExperimentResult",
                "description": "An experiment result (can contain many cases)",
                "optional": True,
            },
            {
                "name": "case_result_param",
                "type": "CaseResult",
                "description": "A case result.",
                "optional": True,
            },
            {
                "name": "variable_names_param",
                "type": "VariableNames",
                "description": "List of variables",
                "defaultValue": [],
            }
        ],
    }

The configuration function

configuration() - Defines the configuration of the Custom Function.

Optional configurations

• result_type - The format of the result file. Possible values are "mat" and "csv". Default: "csv".
• requires_compilation - A boolean indicating if the custom function requires compiled FMU. If set to True, the run() function can utilize get_fmu argument. Default: True.
• requires_library_paths - A boolean to toggle listing of absolute paths for Modelica libraries, a workspace is configured with. Default: False. If set to True, the library path list can be accessed from library_paths parameter in the run function.

An example of the configuration() function is shown below.

def configuration():
    return {
        "result_type": "mat",
        "requires_compilation": True,
        "requires_library_paths": True
    }

The default options function

default_options() - defines the default options that should be used when calling the Custom Function.

Optional configurations

• compiler - Key value pairs for default compiler options when compiling the FMU to be used by the Custom Function. Default: {}.
• simulation - Key value pairs for defult simulation options that the Custom Function should be called with. Default: {}.
• solver - Key value pairs for defult solver options that the Custom Function should be called with. Default: {}.

An example of the default_options() function is shown below.

def default_options():
    return {
        "simulation": {
            "ncp": 500,
            "dynamic_diagnostics": False,
        }
    }

The run function

run() function defines the computation of the Custom Function.

Optional parameters

• get_fmu - A function returning an FMU object for the model on which the Custom Function is applied.For this parameter to be available, the requires_compilation configuration boolean needs to be True.
• environment - Dictionary specifying environment variables:
  • result_folder_path - Path to the folder where the result is to be saved.
  • result_file_name - The name of the file where the results are to be saved.
  • class_path - The Modelica class path for the model the custom function was called upon.
  • workspace_id - The workspace ID where the custom function is called.
  • experiment_id - The experiment ID for which the custom function is called.
  • case_id - The case ID for which the custom function is called.
  • log_file_name - The name of the log file to write.
• upload_custom_artifact - Function for uploading a custom artifact to the storage. It takes arguments:
• artifact_id: The artifact ID (string).
• local_file_path: Path to the local custom artifact to upload. Returns the route for downloading the artifact that can be logged.
• parametrization - A dictionary containing only the non-structural parameter values set in Experiment mode. These parameters need be set on the FMU object (obtained using get_fmu function) or similar for the custom function to reflect this modification.
• initialize_from - A variable name to value mapping object containing a previous result to use for initialization.
• initialize_from_meta - A Python dataclass containing two attributes 'experiment_id' and 'case_id' of the case, the current custom function execution was initialized from. This may be used instead of initialize_from in case more information is required for the initialization.
• options - A dictionary containing the simulation-specific options set by the user. The options dictionary contains the following keys - simulationOptions, solverOptions and simulationLogLevel.
• library_paths - A list of absolute paths for Modelica libraries, a workspace is configured with. For this parameter to be available, the requires_library_paths configuration boolean needs to be True.

Thereafter follows one argument for each of the parameters of the Custom Function.

The example below corresponds to the parameters as defined in the signature() function example above:

• number_parameter: A numeric parameter will contain a number.
• bool_param: A boolean parameter will contain a boolean value.
• string_param: A string parameter will contain a string.
• enum_param: An enum parameter will contain a string (out of the specified values).
• file_uri_param: Value is a FilePath class instance with the following methods:
  • download_as: Depending on the URI schema, returns either the custom artifact file name or the Modelica resource file name.
  • exists: Returns True if the path specified in the URI schema exists on disk.
  • get_URI: Return the URI string passed via REST API for this parameter.
  • get_locally: Return the absolute path to the file specified in the URI on the executor.
• experiment_result_param: Value is ExperimentResultData dataclass with experiment_id attribute.
• case_result_param: Value is CaseResultData dataclass with attributes experiment_id and case_id.
• variable_names_param: Value is a list of strings where each string is a name of a model variable.

from modelon.impact.client import Client

def run(
    get_fmu,
    environment,
    parametrization,
    upload_custom_artifact,
    number_parameter,
    bool_param,
    string_param,
    enum_param,
    file_uri_param,
    experiment_result_param,
    case_result_param,
    variable_names_param,
):
    # Use Modelon Impact Python client library to access the workspace.
    client = Client()
    workspace_id = environment['workspace_id']
    workspace = client.get_workspace(workspace_id)

    print(number_parameter)
    # Outputs the numeric(int/float) value

    print(bool_param)
    # Outputs the boolean value

    print(string_param)
    # Outputs the string value

    print(enum_param)
    # Outputs selected enumeration value.

    print(file_uri_param.get_locally())
    # Outputs the absolute path to the custom artifact/modelica resource
    print(file_uri_param.exists())
    # Outputs True/False based on if the path exists
    print(file_uri_param.get_URI())
    # Outputs the URI string set by the user
    print(file_uri_param.download_as)
    # Outputs the custom artifact/modelica resource file name.

    print(experiment_result_param.experiment_id)
    # Outputs the experiment ID.
    experiment = workspace.get_experiment(experiment_result_param.experiment_id)
    print(experiment.label)
    # Outputs the label for the experiment.

    print(case_result_param.experiment_id, case_result_param.case_id)
    # Outputs the experiment ID and case ID.
    case = workspace.get_experiment(case_result_param.experiment_id).get_case(case_result_param.case_id)
    print(case.meta.label)
    # Outputs the label for the case
    custom_artifacts = case.get_artifacts()
    print(custom_artifacts)
    # Outputs the list of custom artifacts class objects


    print(variable_names_param)
    # Outputs the list of variable names.

Supported packages and modules

The Python environment that Custom Functions run in has many useful packages that can be used. However, because of the nature of Python it is possible to import any module existing in the environment, even from secondary dependencies or internals not meant for consumers of the package. Using these types of packages or modules means that the Custom Function might stop working with future updates. This section will list all packages supported to use in Custom Functions. Authors of Custom Functions should make sure they are only making imports from packages listed here and furthermore check the documentation for that package on what modules and behavior are documented.

The packages can be split into three categories, computation and analysis, resource interfacing and report creation.

Computation and analysis

Packages for analysis that can be used for getting insights on the model output data.

NumPy - The fundamental package for scientific computing with Python

SciPy - Fundamental algorithms for scientific computing with Python

Pandas - A fast, powerful, flexible and easy to use data analysis and manipulation tool

Note

Pandas have a lot of optional dependencies. These dependencies are used for interfacing external formats or adding additional functionality. The Pandas made available to Custom Functions do not have all these optional dependencies installed, meaning some functionality of Pandas will not work or is not guaranteed to work in future versions. Pandas documentation for optional dependencies can be read here. Supported optional functionality for Pandas used by Custom Functions is:

• matplotlib - Plotting
• SciPy - Miscellaneous statistical functions
• openpyxl - Reading / writing for xlsx files

This means a lot of functions such as pandas.read_html (pandas.DataFrame.to_html can still be used), pandas.read_xml, pandas.read_sql, ... are not supported.

OPTIMICA Compiler Toolkit - The calculation engine used by Modelon Impact. It includes pyfmi, oct.steadystate and pyjmi.

Note

Working with CasADi and pymodelica packages may require access to the Modelica libraries in the workspace. The list of libraries can be accessed from library_paths run parameter. For this parameter to be available, the requires_library_paths configuration boolean needs to be True. The Modelica class path for the model, the custom function was called upon can be found in the environment dict as environment[class_path].

from pyjmi import transfer_optimization_problem

def configuration():
    return {
        "requires_compilation": True,
        "requires_library_paths": True
    }

def run(environment, library_paths):
    class_path = environment["class_path"]
    myModel = transfer_optimization_problem(
        class_path, library_paths
        )

Warning

Trying to access files on disk in Modelon Impacts internal storage directly is strongly discouraged. Where Modelon Impact stores its files will change in future versions and relying on certain files existing at a path means that the Custom Function might break after any update. Instead use provided APIs to access data stored in Modelon Impact.

Resource interfacing

Some Custom Function needs data stored in Modelon Impact to make decisions. For example, information about the current experiment/case. This section lists packages and modules for doing this.

mimp.api.storage - API for data stored in Modelon Impact. Includes mimp.api.storage.result.read_result, mimp.api.storage.StorageApi and mimp.api.storage.FilesystemStore.

Note

These are deprecated and it is instead recommended that you use the Modelon Impact Client, see below.

Modelon Impact Client - Client library for easy scripting against Modelon Impact

Warning

It is only supported to use the client for reading application data. It is not supported to upload new data or start compilations/simulations with the client in a Custom Function. In the future we might loosen this restriction and allow the upload of certain data.

Report creation

Creating reports often requires a Custom Function to generate plots, tables, or text. Typically, these will be in HTML or some image format. The packages listed here are to help with generation of these artifacts.

Matplotlib - A comprehensive library for creating static, animated, and interactive visualizations

seaborn - A data visualization library with a high-level interface for drawing attractive and informative statistical graphics

Plotly - A graphing library for making interactive, publication-quality graphs

Pandas - Can be used for representing tables that can be converted to HTML

Note

See the note in the section above about limitations when using Pandas.

Dominate - A library for creating and manipulating HTML documents using an elegant DOM API

Miscellaneous

This section contains modules that do not fit into any clear use cases.

mimp.api - Additional convenience methods used in Custom Functions:

• mimp.api.options, functions for handling options
• mimp.api.custom_functions.defaults, functions for default options for built-in Custom Functions
• mimp.api.custom_functions.resulthandlers, result handlers used for simulation

The Python Standard Library - The standard library that comes bundled with Python

Interacting with Modelon Impact

A Custom Function can interact with Modelon Impact in three ways:

1. Return result
2. Write to log
3. Upload an artifact

Returning a result

Instructions to return the result to Modelon Impact:

1. In the run() function, write the result to a csv-file. See Result format for information on the format.
2. Save the csv-file in the path specified by the environment["result_folder_path"]. The name of the file should be set equal to the value of the environment["result_file_name"].

If the result is stored as a dictionary with variable names as keys and result lists as values, it can conveniently be printed to file using the Python package pandas:

# Write the result to a csv file in the prescribed path
csv_file_path = os.path.join(environment["result_folder_path"], environment["result_file_name"])
df = pandas.DataFrame(data=my_result_dictionary)
df.to_csv(csv_file_path, index=False)

Note: Result from a dynamic simulation in pyfmi comes in arrays rather than lists, so it needs to be converted.

# Converting pyfmi result format to a dict where each value is a list
return_res=dict((name,res[name].tolist())) for name in res.keys())

Result format

The result file format is CSV (comma-separated values).

• First row - names of the variables.
• Following rows - values of the variable for a specific time. For parameters and constants, which do not change over time, the same value is repeated in each row.

Example of a csv result file is seen below. k is a parameter, x is a variable, and a is an array.

time,k,x,a[1],a[2]
0   ,2,1, 3  ,4
1   ,2,3, 4  ,5

Writing to the log

Writing to the log is done by adding print statements in the run() function.

print('Print to log')

Custom Functions can use the simulationLogLevel in the options argument to the run() function to decide dynamically how verbose the writing to log should be.

Uploading an artifact

A custom artifact is a file that is created in the run() function and is uploaded to a workspace.

Follow the instructions below to upload a custom artifact:

1. Create the artifact in a temporary folder
2. Upload the artifact using upload_custom_artifact(). upload_custom_artifact() is a parameter of the run() function.

Example of creating and uploading a .mat-file is seen below.

# Write result to a .mat file that can be imported in MATLAB
# First write the result to a temporary directory
temp_dir = tempfile.mkdtemp()
temp_mat_file = os.path.join(temp_dir, "result.mat")
scipy.io.savemat(temp_mat_file, ss)

# Now upload the result to the server as a custom artifact
artifact_id = "ABCD"
artifact_route = upload_custom_artifact(artifact_id, temp_mat_file)

# Finally add a link for downloading the Artifact in the simulation log
print('Artifact can be downloaded from @artifact[here]({})'.format(artifact_route))

Access and download a custom artifact

Getting access to custom artifacts in the web UI requires Custom Functions to create a link in the simulation log. The example above creates this link with the final print statement. A user can then access and download a custom artifact directly by clicking on the link available in the log.

Using the Python Client, you can iterate through all custom artifacts and download them or list their IDs (artifact_id in the code above). It is also possible to access a specific custom artifact directly using one of those IDs.

Distributing your custom function

To enable other users to install your Custom Function it is recommended to add it to a Project and distribute it. To create a Custom Function in your own project:

1. Open the project for editing in VSCode.

2. Create subdirectories Resources/CustomFunctions (note that names are case sensitive).

3. Double check that .impact/project.json file in your project contains a reference to this directory. This should be like:

       {
       "relpath": "Resources/CustomFunctions",
       "contentType": "CUSTOM_FUNCTIONS",
       "defaultDisabled": false,
       "id": "5fc1a0703df74195ae8bc2c52af432ba"
       },
   

4. Create a Python file to develop the Custom Function code in the Resources/CustomFunctions subdirectory.

5. Implement the Custom Function interface, see Custom Function API.

The new function will become available in the workspaces which contain the project after the user interface has been reloaded.

Developing

When you are making changes to your Custom Function it can be useful to view it in the web UI. To see those latest code changes in the UI, reload the web page (press F5). If your custom function is not showing up in the web UI this is most likely because Modelon Impact fails to load it. During development it might therefore be useful to view the server logs for any issues related to loading Custom Functions.

CloudDesktop

The log files in /home/jovyan/impact/logs are useful for any Custom Function-related issues.

The log files in C:/Users/<user_name>/impact/logs are useful for any Custom Function-related issues.