Skip to main content

Storage

  • A file system storage with a directory structure that can be used for data persistence. It can be written to and read from the functions of SideApp, Trigger, and Pipeline.
  • Resources on the Circuit will reset and disappear when the Canvas is deployed, but the storage will remain intact.

Lifecycle​

  • The Circuit resource, which is deployed from the Canvas, is refreshed with each deployment.
  • Storage exists at the project level of the StudioApp and can be persisted as long as the components of the project or lower layers are not deleted. lifecycle

Specifications​

Overview of Functionality​

  • Utilizes persistent volumes in Kubernetes and is stored on Amazon EBS.
  • Storage volumes are mounted on containers running on each node.

Supported Nodes​

  • SideApp
  • Trigger

Capacity​

  • Maximum of 100GB per project (namespace).
  • Capacity limits may be affected by the disk space set at the cluster level.

Mapping of Nodes and Storage​

  • The Storage service is linked with each node using namespace-id, workspace-id, and the node's id.
  • Changing the workspace-id or node's id may lead to incorrect mapping, so please be cautious.
  • If a Workspace or a node tied to Storage is deleted, re-linking is possible if the namespace-id, workspace-id, and node's id remain the same.
Limitations

The currently offered Storage functionality only has basic features as an initial version and has limitations. Future developments are planned to address these.

  • It is not possible to connect from other nodes to read and write.
  • Deletion of the created Storage is not permitted.
  • Usage quantity cannot be verified.
  • Backup functionality is not provided. If necessary, please handle by exporting to external storage such as AWS S3.

Usage​

Adding Storage Volumes​

  • You can add a volume by selecting SideApp and Trigger, and then clicking the '+' in the right-side menu. how-to-setup

Setting the Path​

  • You can specify any path for mounting to the container image.

Sample​

  • Sample Canvas
    • *Right-click the link to download the file.
  • In this sample, a container image that can execute Python is run on the SideApp node, and the storage volume is mounted.
  • The container has the following structure, and since the Storage path on the Canvas is specified as /tmp, it will be mounted as follows:
/
├── app
│   ├── data
│   ├── output
│   ├── req
│   └── src
├── tmp // Folder added by Storage
...
  • The following Python code is executed using the Injection feature of Storage.
  • This code creates test-file.json in the tmp folder set in Storage and increments the numbers within the JSON file by 1.
  • If the Storage volume is set correctly, it can be seen that the number increases by 1 each time it is deployed.
import os
import json 

# Define the file path
json_path = '/tmp/test-file.json'

# Check if the JSON file already exists and has content
# If not, write the initial data into the JSON file 
if not os.path.exists(json_path) or os.stat(json_path).st_size == 0:
    with open(json_path, 'w') as outputfile:
        json.dump({"data": 0}, outputfile)

# Read and print the contents of the JSON file
with open(json_path, 'r') as outputfile:
    json_object = json.load(outputfile)
print(json_object)

# Increment each value in the JSON object by 1
incremented_json_object = {k: v+1 for k, v in json_object.items() if isinstance(v, int)}

# Write the modified JSON object back to the file
with open(json_path ,  "w") as outputfile:
    json.dump(incremented_json_object, outputfile)

# Read and print the contents of the JSON file again to see the changes
with open(json_path, 'r') as outputfile:
    incremented_json_object = json.load(outputfile)
print(incremented_json_object)