A Cloud Model Storage/Repository: The modelix model-server

API doc | Repository | buildfile | Artifacts: Nexus GitHub Packages

The model-server component provides a storage for models.

Bringing models to the cloud requires an alternative to the file system (and git) based storage of models. Modelix implements a data structure that allows replication between all MPS instances connected to the same model-server. It is very similar to the git storage format, but instead of files it stores nodes in its key-value data store.

Real-Time Collaboration and Operational Transformation

To support real-time collaboration, in addition to the snapshot of a version, the model-server also stores the operations that were applied on the previous version to produce the new version. Conflicts between concurrent modification are resolved using operational transformation (OT). The usually hard part of OT is to guarantee convergence, meaning that all clients end up with the same result independent of the order in which they receive changes. This problem becomes trivial, because versions are identified by a hash over all the content. Any divergence will result in a different hash which can easily be detected.

The remaining problem is what to do when a divergence is detected. From git we know that switching between versions is fast, because it only has to apply the difference between these versions. The same is true here: Fixing a divergence is as easy as switching to the correct version. If a client does not have any local changes it does not even need to apply the change operations itself - it can just switch to the new version. So in the rare case of a divergence fixing it is nothing special and is an operation that happens all the time.

MPS and the model-server

While this component is independent of MPS, it is possible to replicates the data of the MPS internal module repository into a running model-server. This can be achieved using the modelix MPS plugin: One can connect an MPS instances to a model-server, upload the current module repository, and keep it synchronized. This can also be a local MPS instance without a web editor - both options are supported at the same time.

Development

To reformat and add license header to all files run:

../gradlew spotlessApply

APIs

Valid keys are keys starting with the PROTECTED_PREFIX ($$$).

For details please refer to the OpenAPI documentation under doc/model-server.json.

JSON web tokens (JWT) based authorization

By default, the model server doesn’t check any permissions. To enable it you can specify the following environment variables.

Variable Description

MODELIX_PERMISSION_CHECKS_ENABLED

By default, permission checking is enabled when an algorithm for the JWT signature or a MODELIX_JWK_URI is configured. This variable can be set explicitly to true or false to avoid security issues by a misconfigured algorithm.

MODELIX_GENERATE_FAKE_JWT

By default, if no signature algorithm and no MODELIX_JWK_URI is configured, a token is generated for all requests with the identity unit-tests@example.com and no permissions. This option can be set to true or false to enable/disable this behaviour explicitly.

MODELIX_JWT_SIGNATURE_HMAC512_KEY MODELIX_JWT_SIGNATURE_HMAC384_KEY MODELIX_JWT_SIGNATURE_HMAC256_KEY

If JWT tokens are signed with the HMAC algorithm, this variable specifies the secret key.

The org.modelix:authorization library provides a utility function createModelixAccessToken for creating valid tokens signed with this algorithm,

MODELIX_JWT_SIGNATURE_HMAC512_KEY_FILE MODELIX_JWT_SIGNATURE_HMAC384_KEY_FILE MODELIX_JWT_SIGNATURE_HMAC256_KEY_FILE

Instead of specifying the key directly via MODELIX_JWT_SIGNATURE_HMAC512_KEY a path to a file containing the key can be specified. This is useful when keys are stored in kubernetes secrets.

MODELIX_JWK_URI

If keys are created and signed by some OpenID connect server the public keys are provided via HTTP. Here you can specify the URI of the key set. Only RSA (256, 384 and 512) keys are currently supported.

MODELIX_JWK_KEY_ID

Optional key ID that can be used together with MODELIX_JWK_URI. If specified, it ensures that only tokens that use the specified key are valid. If not specified, a token can use any RSA (256, 384 and 512) key provided by MODELIX_JWK_URI.

KEYCLOAK_BASE_URL KEYCLOAK_REALM

Legacy variables for the keycloak based authorization used by the Modelix Helm charts for workspaces.

The permissions claim of the token is expected to list directly granted permission. Additional granted permissions may be derived from that list. For example, an model-server/admin permission gives access to all repositories and administrative actions. repository/someRepository/admin on the other hand, gives only permission to all actions on the repository "someRepository".

A detailed list of all permissions can be found on the model server at /permissions.