Sorry, get-keys is the function:

interface "wasi:kv/data/get-many" {
  // get_many returns a stream of key-value pairs.
  // Notice hat this interface is non-atomic, meaning that the key-value pairs
  // returned may not be consistent.

  use { Error, payload } from "wasi:kv/types"
  
  get-many: func(keys: keys) -> result<stream<payload>, Error>

  get-keys: func() -> result<keys, Error>
}

I realize this is late in the review process for the SIP, so apologies.

Do we think that we could also add a list or get-all (as it is defined in the upstream WASI KV proposal) operation to the initial API?

I could imagine this being extremely useful for various (even early) scenarios, and I would like to use it in Spin components soon.

As I have started using the OCI functionality a bit more, I realized that "OCI" is a fairly specialized three letter acronym, and expecting a user to know it and immediately understand "this has something to do with registries" might not be reasonable.

This commit changes the top level spin oci command to spin registry, while still keeping an alias for spin oci.

This should be more intuitive for a user. Happy to close this PR if people think spin oci is more specific.

Signed-off-by: Radu Matei radu.matei@fermyon.com

I'll try to make my arguments from the previous comment slightly clearer:

1. Today, we are using the image manifest to distribute layers whose media types OCI 1.0 compliant registries MAY (and depending on how you interpret the spec, MUST) ignore. The fact that an OCI 1.0 compliant registry MUST accept the image manifest says nothing about whether the layer uploads will be successful. Saying "any OCI 1.0 registry will accept image manifests, so why do you need an artifact" only addresses the manifest, and not necessarily the content. If I am misreading the current image specification, and any OCI 1.0 compliant registry somehow accepted any layer (and config) media type and content, happy to concede this point.

2. As a consequence of 1, there is no way to know whether a given registry that claims OCI 1.0 compatibility would actually accept both the manifest and content. This leaves both artifact authors and their users in a state where all they can reason about is "at this point, registry X accepts our artifact, and we don't know about registry Y because no one tested it yet". This uncertainty is not instilling confidence in the specification, registries, or the tooling that uses them.

Finally, I am not trying to "prove anyone wrong" or "drop compatibility" because of any association. I am trying to express my opinion as an artifact author. If that opinion is not constructive or welcome in this thread, happy to not engage further.

(Background: I advocated for a referrers API, signatures, and new artifact types around 2019 with the CNAB project, and I am now working on WebAssembly tooling, where I am once again trying to use OCI to distribute things. Opinions here are my own.)

One of the points used in this argument is, paraphrasing, "the fact that Helm, Homebrew, or Cosign are successfully using the image manifest is proof that there is no reason to change".

I want to point out that, as an artifact author, without an explicit artifact media type being part of OCI 1.1, I have absolutely no guarantee that my users are able to distribute a signature, Wasm module, or other artifacts with a given registry; moreover, they also have no guarantee that, if at some point a registry did support those artifacts, that support is not going away.

Reading the image specification, the guidance given in this thread for artifact authors (just continue to use image manifests and set config.mediaType) is telling me that new artifacts are not guaranteed to be accepted by a registry, nor that tools (that are currently not being broken by them) would continue to support them:

• config.mediaType : "manifests concerned with portability SHOULD use [application/vnd.oci.image.config.v1+json](https://github.com/opencontainers/image-spec/blob/main/config.md)"
• layers: "Manifests concerned with portability SHOULD use one of the above media types (all OCI image media types). An encountered mediaType that is unknown to the implementation MUST be ignored."

Drawing the line at some tools not being broken by hiding random artifacts inside images sounds rather arbitrary. There are potentially equally as many that could be broken by not getting an image. (See @AaronFriel 's argument above.)

As an artifact author, the current state: 1) some registries supporting arbitrary artifacts in images, some not, 2) the image spec not being explicit about accepting them, and 3) the artifact manifest going away is concerning, and as someone who is/was considering building more tooling to distribute (Wasm) apps with OCI registries, I have to consider the possibility of a future PR restricting what even the image manifest could be used for.

ref #455

Before this commit, for a spin.toml file that had a URL module source for a component (such as the file server and redirect templates), subsequent runs of spin up would always download the module from the URL before starting the application.

https://github.com/fermyon/spin/pull/1014 added a content-addressed registry cache that can be used to store remote module sources.

This commit updates the Spin local loader to use that cache to store the component sources referenced through URL in spin.toml.

Before fetching a component, the loader will check the local cache first. If present, it will read the file and return its content; otherwise, it will fetch it, store it in the cache, then return its content.

Currently, the default cache root is always used. Once spin up will have a way to configure it, that will be propagated here as well.

Signed-off-by: Radu Matei radu.matei@fermyon.com

Add prototype for distributing Spin applications using OCI

This commit adds experimental support for distributing Spin applications using OCI registries.

Specifically, it uses the OCI Artifacts specification (https://github.com/opencontainers/artifacts) to package and distribute Spin applications.

This PR implements spin oci push, spin oci pull, and spin oci run to interact with a supporting container registry - for example:

$ spin oci push ghcr.io/<username>/my-spin-application:v1
INFO spin_publish::oci::client: Pushed "https://ghcr.io/v2/<username>/my-spin-application/manifests/sha256:9f4e7eebb27c0174fe6654ef5e0f908f1edc8f625324a9f49967ccde44a6516b"
$ spin oci pull ghcr.io/<username>/my-spin-application:v1
INFO spin_publish::oci::client: Pulled ghcr.io/<username>/my-spin-application:v1@sha256:9f4e7eebb27c0174fe6654ef5e0f908f1edc8f625324a9f49967ccde44a6516b
$ spin oci run ghcr.io/<username>/my-spin-application:v1
INFO spin_publish::oci::client: Pulled ghcr.io/<username>/my-spin-application:v1@sha256:9f4e7eebb27c0174fe6654ef5e0f908f1edc8f625324a9f49967ccde44a6516b

Following the SIP (https://github.com/fermyon/spin/pull/1033), this PR defines a new config.mediaType for a Spin application, application/vnd.fermyon.spin.application.v1+config, and two media types for the potential content that can be found in such an artifact: application/vnd.wasm.content.layer.v1+wasm for a Wasm module, and application/vnd.wasm.content.layer.v1+data for a static file. (Note that the media types can change in a future iteration of this experimental work if a community consensus on the media type used to represent Wasm is reached.) Following the SIP, this PR distributes the Spin lockfile for a given application as the OCI configuration object.

This PR also introduces a global cache for layers and manifests pulled from the registry. This is a content addressable cache, and its use will be extended in the future for Wasm modules pulled from HTTP sources.

Currently, spin oci pull (or spin oci run) will always make at least an initial request to the registry to fetch the manifest from the registry. After the manifest is fetched, already pulled layers will not be pulled again. In a future commit, the behavior of the initial manifest fetch regardless of its existence in the cache will be controllable by a flag.

Signed-off-by: Radu Matei radu.matei@fermyon.com

Merge pull request #1014 from radu-matei/oci

See SIP — #1033

Signed-off-by: Radu Matei radu.matei@fermyon.com

As of #1014, the spin oci set of commands can interact with an OCI registry. Currently, these commands will reuse the credentials in ~/.docker/config.json to authenticate to registries, as most registries have instructions on how to authenticate the Docker CLI.

While Spin should continue to use the credentials in ~/.docker/config.json, and it should continue to allow users to use registries they already authenticated for using their existing toolchains, it should not depend on the Docker CLI being installed on the target machine.

As a result, Spin should implement:

• spin oci login --username <> --password <> and
• <echo-password-or-get-it-using-cloud-specific-tool> | spin oci login --username <> --pasword-stdin

This would mirror the default ways to authenticate to container registries today.

#1014 implements running a Spin application from an OCI registry through the spin oci run command while we validate the implementation.

Ideally, we want to integrate this functionality with spin up. This issue tracks updating spin up to accept a reference from the registry and start the application.

Add prototype for distributing Spin applications using OCI

This commit adds experimental support for distributing Spin applications using OCI registries.

Specifically, it uses the OCI Artifacts specification (https://github.com/opencontainers/artifacts) to package and distribute Spin applications.

This PR implements spin oci push, spin oci pull, and spin oci run to interact with a supporting container registry - for example:

$ spin oci push ghcr.io/<username>/my-spin-application:v1
INFO spin_publish::oci::client: Pushed "https://ghcr.io/v2/<username>/my-spin-application/manifests/sha256:9f4e7eebb27c0174fe6654ef5e0f908f1edc8f625324a9f49967ccde44a6516b"
$ spin oci pull ghcr.io/<username>/my-spin-application:v1
INFO spin_publish::oci::client: Pulled ghcr.io/<username>/my-spin-application:v1@sha256:9f4e7eebb27c0174fe6654ef5e0f908f1edc8f625324a9f49967ccde44a6516b
$ spin oci run ghcr.io/<username>/my-spin-application:v1
INFO spin_publish::oci::client: Pulled ghcr.io/<username>/my-spin-application:v1@sha256:9f4e7eebb27c0174fe6654ef5e0f908f1edc8f625324a9f49967ccde44a6516b

Following the SIP (https://github.com/fermyon/spin/pull/1033), this PR defines a new config.mediaType for a Spin application, application/vnd.fermyon.spin.application.v1+config, and two media types for the potential content that can be found in such an artifact: application/vnd.wasm.content.layer.v1+wasm for a Wasm module, and application/vnd.wasm.content.layer.v1+data for a static file. (Note that the media types can change in a future iteration of this experimental work if a community consensus on the media type used to represent Wasm is reached.) Following the SIP, this PR distributes the Spin lockfile for a given application as the OCI configuration object.

This PR also introduces a global cache for layers and manifests pulled from the registry. This is a content addressable cache, and its use will be extended in the future for Wasm modules pulled from HTTP sources.

Currently, spin oci pull (or spin oci run) will always make at least an initial request to the registry to fetch the manifest from the registry. After the manifest is fetched, already pulled layers will not be pulled again. In a future commit, the behavior of the initial manifest fetch regardless of its existence in the cache will be controllable by a flag.

Signed-off-by: Radu Matei radu.matei@fermyon.com

Add note in spin oci doc about credentials

Signed-off-by: Radu Matei radu.matei@fermyon.com

Rename cache function oci_config_path to lockfile_path

Signed-off-by: Radu Matei radu.matei@fermyon.com

Fix Clippy and refactor OCI cache functions

Signed-off-by: Radu Matei radu.matei@fermyon.com