Volumes

A volume is an OuterProduct resource that maps a name to a location in object storage. The volume stores metadata about that location; the data and objects remain in the backing object store. Volumes have two storage types:

OuterProduct-managed storage: OuterProduct allocates and manages the backing object-store location. This is the default when creating a volume.
External storage: The volume references an existing S3 bucket or Google Cloud Storage bucket. OuterProduct does not copy the existing objects.

Both types use the same Python API and can be mounted as files in a function’s runtime environment.

Creating a managed volume

client.create_volume() creates a volume and returns a VolumeHandle. OuterProduct manages the underlying object storage by default when the provider argument is omitted. Each volume has a name consisting of volumes/ followed by the string provided at creation time.

import outerproduct as op

client = op.init()
volume_handle: op.VolumeHandle = client.create_volume("datasets")
print(volume_handle.name)  # volumes/datasets

Volume names are case-insensitive and must be unique within a workspace. Creating a volume with an existing name raises an already_exists error.

Creating a volume based on an external object store

OuterProduct Volumes can also be created from existing S3 buckets and Google Cloud Storage buckets without copying their contents. To connect to an external store, cloud credentials should first be stored as OuterProduct Secrets.

Amazon S3
Google Cloud Storage

An S3 volume specifies a bucket and a stored AWS credential:

credential: op.Secret = client.get_secret("aws-storage")

volume_handle: op.VolumeHandle = client.create_volume(
    "external-datasets",
    provider=op.VolumeProvider.S3(
        bucket="my-bucket",
        prefix="datasets/",
        region="us-east-1",
        credential=credential.name,
    ),
)

With this prefix, training.csv in the volume corresponds to s3://my-bucket/datasets/training.csv. If no prefix is specified, paths will be relative to the bucket’s root.

A Google Cloud Storage volume specifies a bucket and a stored GCP credential:

credential: op.Secret = client.get_secret("gcp-storage")

volume_handle: op.VolumeHandle = client.create_volume(
    "external-datasets",
    provider=op.VolumeProvider.Gcs(
        bucket="my-bucket",
        prefix="datasets/",
        credential=credential.name,
    ),
)

With this prefix, training.csv in the volume corresponds to gs://my-bucket/datasets/training.csv. If no prefix is specified, paths will be relative to the bucket’s root.

Managing volume resources

client.list_volumes() returns accessible Volume metadata, while client.get_volume() takes a volume name (with or without the volumes/ prefix) and returns a handle to an existing volume. Use update_volume() to rename a volume or change its description without changing its identity. client.delete_volume() removes the volume resource and its metadata. Deleting an OuterProduct-managed volume also deletes its stored files. For a volume backed by your own object store, the files are retained.

for volume in client.list_volumes():
    print(volume.name)

volume_handle: op.VolumeHandle = client.get_volume("datasets")

client.update_volume(
    "datasets",
    new_name="training-data",
    description="Training inputs",
)

client.delete_volume("old-datasets")

Reading and writing data

VolumeHandle provides methods for reading and writing objects through obstore.

from pathlib import Path

from obstore import GetResult

volume_handle.put("training.csv", Path("training.csv"))

result: GetResult = volume_handle.get("training.csv")
Path("downloaded.csv").write_bytes(bytes(result.bytes()))

put() accepts bytes, local files, and streams. get() returns a result whose contents can be read into memory with .bytes() or consumed as a stream. When using these methods, object paths should be specified relative to the volume’s root. The handle also supports other standard methods for interacting with an object store, including listing objects, inspecting metadata, copying, renaming, and deleting data.

from obstore import ListResult, ObjectMeta

listing: ListResult = volume_handle.list_with_delimiter()
metadata: ObjectMeta = volume_handle.head("training.csv")

volume_handle.copy("training.csv", "archive/training.csv")

volume_handle.list(prefix) is an async iterator that yields batches of object metadata under the prefix. It is used with async for and has no synchronous form. See Async usage.

Mounting a volume

The volumes argument to client.run() is a dictionary mapping filesystem paths to volume handles, volume names, or UUIDs. At runtime, the volume’s contents are mounted at the specified path:

def count_rows(client) -> int:
    import csv

    with open("/data/training.csv") as file:
        return sum(1 for _ in csv.DictReader(file))


if __name__ == "__main__":
    result: op.Run[int] | op.Call[int] = client.run(
        fn=count_rows,
        image=image,
        volumes={"/data": volume_handle},  # count_rows can access the volume at /data
	   timeout=3600,  # mounts stop working an hour after the Attempt starts
    )

    print(result.wait())

Volume mounts are not inherited by child calls, so each child call must declare its own mounts. Mounting requires both read and write access to the volume. See Policies for more information on permissions in OuterProduct. Mounted volumes support reading files and sequential writes to new files. They do not support in-place edits, appending, or atomic renames.

Volume mounts use temporary credentials and stop working an hour after an Attempt starts. Set timeout=3600 or less on client.run() for Calls that mount volumes; client.run() warns otherwise. Calls that need longer should read and write the volume through VolumeHandle methods such as get() and put(), which refresh their credentials as needed.

Getting started

Execution

Governance

Creating a managed volume

Creating a volume based on an external object store

Managing volume resources

Reading and writing data

Mounting a volume

​Creating a managed volume

​Creating a volume based on an external object store

​Managing volume resources

​Reading and writing data

​Mounting a volume

Creating a managed volume

Creating a volume based on an external object store

Managing volume resources

Reading and writing data

Mounting a volume