object

package
v0.0.0-...-3b62dd1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 4, 2025 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Index

Constants

View Source 
const SHA256V1FlatReferenceSizeBytes = 35

Variables

View Source 
var SHA256V1ReferenceFormat = ReferenceFormat{}

SHA256V1ReferenceFormat is a predeclared instance of ReferenceFormat that uses algorithm "SHA256_V1".

View Source 
var Unlimited = Limit{
	// contains filtered or unexported fields
}

Unlimited concurrency when attempting to traverse a graph of objects.

Functions

func NewDownloaderServer 

func NewDownloaderServer(downloader Downloader[GlobalReference]) object.DownloaderServer

NewDownloaderServer creates a gRPC server object for exposing a Downloader. This can be used to make objects contained in an object store accessible via the network.

func NewUploaderServer 

func NewUploaderServer(uploader Uploader[GlobalReference, []byte]) object.UploaderServer

NewUploaderServer creates a gRPC server object for exposing an Uploader. This can be used to make objects contained in an object store accessible via the network.

Types

type BasicReference 

type BasicReference interface {
	// Accessors of individual fields contained in the reference.
	GetHash() []byte
	GetSizeBytes() int
	GetHeight() int
	GetDegree() int
	GetMaximumTotalParentsSizeBytes(includeSelf bool) int

	// Binary representation.
	GetReferenceFormat() ReferenceFormat
	GetRawReference() []byte

	// Conversion to other reference types with the same value.
	GetLocalReference() LocalReference
	Flatten() FlatReference
}

BasicReference is a constraint to be used on generic functions and types that work with references, and need to gain access to at least some of the basic properties of a reference.

type Contents 

type Contents struct {
	LocalReference
	// contains filtered or unexported fields
}

Contents of an object read from storage, or to be written to storage.

func MustNewContents 

func MustNewContents(referenceFormatValue object.ReferenceFormat_Value, references []LocalReference, payload []byte) *Contents

MustNewContents constructs object contents given a list of outgoing references and a payload. This function may be used as part of tests.

func NewContentsFromFullData 

func NewContentsFromFullData(reference LocalReference, data []byte) (*Contents, error)

NewContentsFromFullData constructs object contents from raw data that is read from disk or an incoming RPC message. Construction fails if the provided data does not match the expected reference.

func (*Contents) DetachOutgoingReferences 

func (c *Contents) DetachOutgoingReferences() OutgoingReferences[LocalReference]

DetachOutgoingReferences copies all of the outgoing references into an OutgoingReferencesList, allowing the object contents to be garbage collected while keeping the outgoing references available.

func (*Contents) FlattenContents 

func (c *Contents) FlattenContents() *Contents

FlattenContents flattens the reference of the object, so that its height and degree are both zero.

This is used by the read caching backend, where we want to cache individual objects as opposed to graphs. When writing objects to the local cache, we set the height and degree to zero, so that there is no need to track any leases.

func (*Contents) GetFullData 

func (c *Contents) GetFullData() []byte

GetFullData returns the full object contents, including the binary representation of outgoing references that are stored at the beginning. This method can be used to encode contents for transmission across the network, or writing them to disk.

func (*Contents) GetOutgoingReference 

func (c *Contents) GetOutgoingReference(i int) LocalReference

GetOutgoingReference returns one of the outgoing references that is part of this object.

func (*Contents) GetPayload 

func (c *Contents) GetPayload() []byte

GetPayload returns the payload of the object, not including the outgoing references that are stored at the beginning.

func (*Contents) Unflatten 

func (c *Contents) Unflatten(newReference LocalReference) (*Contents, error)

Unflatten the reference of the object, so that its height and degree are set to the desired value.

This is used by the read caching backend, where we want to cache individual objects as opposed to graphs. When reading objects from the local cache, their heights and degrees will be set to zero. This method can be used to undo this transformation.

type Downloader 

type Downloader[TReference any] interface {
	DownloadObject(ctx context.Context, reference TReference) (*Contents, error)
}

Downloader of objects from storage.

type DownloaderForTesting 

type DownloaderForTesting Downloader[GlobalReference]

DownloaderForTesting is an instantiation of Downloader for generating mocks to be used as part of tests.

type FlatReference 

type FlatReference struct {
	// contains filtered or unexported fields
}

FlatReference is a simplified version of LocalReference that assumes that the height and degree of an object are zero. This means that only the hash and size need to be tracked, similar to REv2's Digest message.

Because the hash and size are stored at the beginning of a LocalReference, FlatReference uses the same representation as LocalReference with the last couple of bytes removed.

FlatReference is sufficient for the read caching and local storage backends, which are oblivious of references between objects.

func MustNewSHA256V1FlatReference 

func MustNewSHA256V1FlatReference(hash string, sizeBytes uint32) (flatReference FlatReference)

MustNewSHA256V1LocalReference creates a flat reference that uses reference format SHA256_V1. This function can be used as part of tests.

func (FlatReference) GetHash 

func (r FlatReference) GetHash() []byte

GetHash returns the hash of the contents of the object associated with the flat reference.

func (FlatReference) GetLocalReference 

func (r FlatReference) GetLocalReference() (localReference LocalReference)

GetLocalReference converts a flat reference to a corresponding local reference having height zero.

func (FlatReference) GetRawFlatReference 

func (r FlatReference) GetRawFlatReference() []byte

GetRawFlatReference returns the flat reference in binary form, so that it may be embedded into gRPC request bodies.

func (FlatReference) GetReferenceFormat 

func (FlatReference) GetReferenceFormat() ReferenceFormat

GetReferenceFormat returns the reference format that was used to generate the reference.

func (FlatReference) GetSizeBytes 

func (r FlatReference) GetSizeBytes() int

GetSizeBytes returns the size of the object associated with the flat reference.

type FlatStoreForTesting 

type FlatStoreForTesting Store[FlatReference, struct{}]

FlatStoreForTesting is an instantiation of Store for generating mocks to be used by tests that use flat references.

type GlobalReference 

type GlobalReference struct {
	InstanceName
	LocalReference
}

GlobalReference uniquely identifies an object across all namespaces.

func MustNewSHA256V1GlobalReference 

func MustNewSHA256V1GlobalReference(instanceName, hash string, sizeBytes uint32, height uint8, degree uint16, maximumTotalParentsSizeBytes uint64) GlobalReference

MustNewSHA256V1GlobalReference creates a global reference that uses reference format SHA256_V1. This function can be used as part of tests.

func (GlobalReference) CompareByHeight 

func (r GlobalReference) CompareByHeight(other GlobalReference) int

CompareByHeight returns -1 if the reference comes before another reference along a total order based on the references' height and maximum total parents size. Conversely, it returns 1 if the reference comes after another reference. 0 is returned if both references are equal.

This order can be used to perform bounded parallel traversal of DAGs in such a way that forward progress is guaranteed.

func (GlobalReference) GetNamespace 

func (r GlobalReference) GetNamespace() Namespace

GetNamespace returns the namespace that contains the object.

func (GlobalReference) String 

func (r GlobalReference) String() string

func (GlobalReference) WithLocalReference 

func (r GlobalReference) WithLocalReference(localReference LocalReference) GlobalReference

WithLocalReference returns a new GlobalReference that has its value replaced with another.

This method can be used to upgrade an outgoing reference of an object from a local reference to a global reference within the same namespace as the object.

type InstanceName 

type InstanceName struct {
	// contains filtered or unexported fields
}

InstanceName denotes the name of a namespace in storage.

In this implementation instance names can have arbitrary string values. This differs from REv2, where the instance name is path-like and cannot contain certain keywords.

func NewInstanceName 

func NewInstanceName(value string) InstanceName

NewInstanceName creates a new InstanceName that corresponds to the provided value.

func (InstanceName) String 

func (in InstanceName) String() string

func (InstanceName) WithLocalReference 

func (in InstanceName) WithLocalReference(localReference LocalReference) GlobalReference

WithLocalReference upgrades a LocalReference to a GlobalReference that is associated with the current instance name, allowing it to be used as part of an RPC.

type Limit 

type Limit struct {
	// contains filtered or unexported fields
}

Limit on the maximum number of objects to process in parallel.

func NewLimit 

func NewLimit(m *object.Limit) Limit

NewLimit converts limits on the maximum number of objects to process in parallel from a Protobuf message format to a native type.

func (*Limit) AcquireObjectAndChildren 

func (l *Limit) AcquireObjectAndChildren(reference LocalReference) bool

AcquireObjectAndChildren reduces the limits, so that resources are allocated for processing the subgraph rooted at the referenced object.

func (*Limit) CanAcquireObjectAndChildren 

func (l *Limit) CanAcquireObjectAndChildren(reference LocalReference) bool

CanAcquireObjectAndChildren returns whether the current limits allow processing the subgraph rooted at the referenced object.

func (Limit) Min 

func (l Limit) Min(other Limit) Limit

Min computes the lower bound on the maximum number of objects to process in parallel. This method can be used during client/server negotiation to pick a resource limit that can be respected by both sides.

func (*Limit) ReleaseChildren 

func (l *Limit) ReleaseChildren(reference LocalReference)

ReleaseChildren increases the limits, so that resources are released that were acquired for processing the children of the referenced object.

func (*Limit) ReleaseObject 

func (l *Limit) ReleaseObject(reference LocalReference)

ReleaseObject increases the limits, so that resources are released that were acquired for processing the referenced object (but not its children).

func (Limit) ToProto 

func (l Limit) ToProto() *object.Limit

ToProto converts a limit on the maximum number of objects to process in parallel to a Protobuf message, so that it can be embedded into a gRPC request message.

type LocalReference 

type LocalReference struct {
	// contains filtered or unexported fields
}

LocalReference uniquely identifies an object stored within a single storage namespace.

func MustNewSHA256V1LocalReference 

func MustNewSHA256V1LocalReference(hash string, sizeBytes uint32, height uint8, degree uint16, maximumTotalParentsSizeBytes uint64) LocalReference

MustNewSHA256V1LocalReference creates a local reference that uses reference format SHA256_V1. This function can be used as part of tests.

func (LocalReference) CompareByHeight 

func (r LocalReference) CompareByHeight(other LocalReference) int

CompareByHeight returns -1 if the reference comes before another reference along a total order based on the references' height and maximum total parents size. Conversely, it returns 1 if the reference comes after another reference. 0 is returned if both references are equal.

This order can be used to perform bounded parallel traversal of DAGs in such a way that forward progress is guaranteed.

func (LocalReference) Flatten 

func (r LocalReference) Flatten() (flatReference FlatReference)

Flatten a reference, so that its height and degree are both zero.

This is used by the read caching backend, where we want to cache individual objects as opposed to graphs. When writing objects to the local cache, we set the height and degree to zero, so that there is no need to track any leases.

func (LocalReference) GetDegree 

func (r LocalReference) GetDegree() int

GetDegree returns the number of children the object associated with the reference has.

func (LocalReference) GetHash 

func (r LocalReference) GetHash() []byte

GetHash returns the hash of the contents of the object associated with the reference.

func (LocalReference) GetHeight 

func (r LocalReference) GetHeight() int

GetHeight returns the maximum length of all paths to leaves underneath the current object. Objects without any children have height zero.

func (LocalReference) GetLocalReference 

func (r LocalReference) GetLocalReference() LocalReference

GetLocalReference trims all properties of the reference, so that only a LocalReference remains.

This method is provided to ensure that all types that are derived from LocalReference are easy to convert back to a LocalReference.

func (LocalReference) GetMaximumTotalParentsSizeBytes 

func (r LocalReference) GetMaximumTotalParentsSizeBytes(includeSelf bool) int

GetMaximumTotalParentsSizeBytes returns the maximum total size of objects along all paths to leaves underneath the current object, excluding the leaves themselves. Whether the size of the current object is taken into consideration is controlled through the includeSelf parameter.

This method can be used to determine the amount of memory that's needed to traverse the full graph. It is used by the DAG uploading and lease renewing code to traverse graphs in parallel, in such a way that memory usage is bounded.

func (LocalReference) GetRawReference 

func (r LocalReference) GetRawReference() []byte

GetRawReference returns the reference in binary form, so that it may be embedded into gRPC request bodies.

func (LocalReference) GetReferenceFormat 

func (LocalReference) GetReferenceFormat() ReferenceFormat

GetReferenceFormat returns the reference format that was used to generate the reference.

func (LocalReference) GetSizeBytes 

func (r LocalReference) GetSizeBytes() int

GetSizeBytes returns the size of the object associated with the reference. The size comprises both the outgoing references and the data payload.

func (LocalReference) String 

func (r LocalReference) String() string

func (LocalReference) WithLocalReference 

func (r LocalReference) WithLocalReference(localReference LocalReference) LocalReference

WithLocalReference returns a new LocalReference that has its value replaced with another.

This method is merely provided to satisfy some of the constraints of some generic functions and types that can operate both on local and global references.

type Namespace 

type Namespace struct {
	InstanceName
	ReferenceFormat
}

Namespace in which an object is stored.

A storage server may partition objects by project or tenant. Each of these partitions are called namespaces. It is not possible for objects to directly refer to objects in another namespace.

func NewNamespace 

func NewNamespace(namespaceMessage *object.Namespace) (Namespace, error)

NewNamespace creates a Namespace from a Protobuf message that was contained in a gRPC request.

func (Namespace) NewGlobalReference 

func (ns Namespace) NewGlobalReference(rawReference []byte) (GlobalReference, error)

NewGlobalReference creates a reference that uniquely refers to an object across all namespaces.

func (Namespace) ToProto 

func (ns Namespace) ToProto() *object.Namespace

ToProto converts a Namespace to a Protobuf message, so that it may be sent as part of a gRPC request.

type OutgoingReferences 

type OutgoingReferences[TReference any] interface {
	GetDegree() int
	GetOutgoingReference(index int) TReference

	// If the OutgoingReferences object is part of a larger object
	// (e.g., part of object.Contents), copy it, so that the
	// original instance may be garbage ollected.
	DetachOutgoingReferences() OutgoingReferences[TReference]
}

OutgoingReferences is a list of outgoing references of an object. It may either be implemented by a simple slice, or it's possible to provide an implementation that reads references directly out of the object's contents.

type OutgoingReferencesList 

type OutgoingReferencesList[T any] []T

OutgoingReferencesList is a list of outgoing references of an object that is backed by a simple slice.

func (OutgoingReferencesList[T]) DetachOutgoingReferences 

func (l OutgoingReferencesList[T]) DetachOutgoingReferences() OutgoingReferences[T]

DetachOutgoingReferences does nothing, as it is assumed that an OutgoingReferencesList is already stored in a separate memory allocation, and is not part of a larger object.

func (OutgoingReferencesList[T]) GetDegree 

func (l OutgoingReferencesList[T]) GetDegree() int

GetDegree returns the number of outgoing references in the list.

func (OutgoingReferencesList[T]) GetOutgoingReference 

func (l OutgoingReferencesList[T]) GetOutgoingReference(index int) T

GetOutgoingReference returns the reference at a given index in the outgoing refrences list. It is the caller's responsibility to ensure the index is within bounds.

type ReferenceFormat 

type ReferenceFormat struct{}

ReferenceFormat describes the algorithm to use to compute the LocalReference of an object. Right now only a single algorithm is supported, called "SHA256_V1".

func NewReferenceFormat 

func NewReferenceFormat(value object.ReferenceFormat_Value) (ReferenceFormat, error)

NewReferenceFormat converts a ReferenceFormat enumeration value that is used as part of gRPC requests to a native type. Right now it merely validates that the provided enumeration value corresponds to SHA256_V1.

func (ReferenceFormat) GetMaximumObjectSizeBytes 

func (ReferenceFormat) GetMaximumObjectSizeBytes() int

GetMaximumObjectSizeBytes returns the maximum size of objects encoded in this format. This size is guaranteed to be small enough to permit safe transmission of such objects as part of gRPC requests and responses, and to instantiate the object in memory of a reasonably modern computer.

func (ReferenceFormat) GetReferenceSizeBytes 

func (ReferenceFormat) GetReferenceSizeBytes() int

GetReferenceSizeBytes returns the size in bytes of references encoded in this format.

func (ReferenceFormat) NewContents 

func (rf ReferenceFormat) NewContents(outgoingReferences []LocalReference, payload []byte) (*Contents, error)

NewContents creates a new object having the provided set of outgoing references and data payload. It is assumed that outgoing references are provided in sorted order.

func (ReferenceFormat) NewFlatReference 

func (ReferenceFormat) NewFlatReference(rawReference []byte) (r FlatReference, err error)

func (ReferenceFormat) NewLocalReference 

func (ReferenceFormat) NewLocalReference(rawReference []byte) (r LocalReference, err error)

NewLocalReference converts a reference that is stored in binary format to an in-memory representation. It also validates that all fields contained in the reference are within bounds.

func (ReferenceFormat) ToProto 

func (ReferenceFormat) ToProto() object.ReferenceFormat_Value

ToProto converts a ReferenceFormat to an enumeration value that can be embedded into gRPC request messages.

type Store 

type Store[TReference any, TLease any] interface {
	Downloader[TReference]
	Uploader[TReference, TLease]
}

Store for objects, which is accessible for reading (downloading) and writing (uploading).

func NewStore 

func NewStore[TReference, TLease any](downloader Downloader[TReference], uploader Uploader[TReference, TLease]) Store[TReference, TLease]

NewStore is a helper function for creating a Store that is backed by separate instances of Downloader and Uploader.

type StoreForTesting 

type StoreForTesting Store[GlobalReference, any]

StoreForTesting is an instantiation of Store for generating mocks to be used by tests.

type UploadObjectComplete 

type UploadObjectComplete[TLease any] struct {
	Lease TLease
}

UploadObjectComplete indicates that an object is complete (i.e., present and all of its leases are valid).

type UploadObjectIncomplete 

type UploadObjectIncomplete[TLease any] struct {
	Contents                     *Contents
	WantOutgoingReferencesLeases []int
}

UploadObjectIncomplete indicates that an object is incomplete (i.e., present, but one or more leases have expired). It is expected that the caller attempts to obtain updated leases for the object's children and calls UploadObject() again to provide the updated leases.

type UploadObjectMissing 

type UploadObjectMissing[TLease any] struct{}

UploadObjectMissing indicates that an object is missing (i.e., not present in storage).

type UploadObjectResult 

type UploadObjectResult[TLease any] interface {
	// contains filtered or unexported methods
}

UploadObjectResult is the result returned by Uploader.UploadObject(). A backend may report that an object is complete (i.e., present and all of its leases are valid), incomplete (i.e., present, but one or more leases have expired) or missing (i.e., not present in storage).

type Uploader 

type Uploader[TReference any, TLease any] interface {
	UploadObject(ctx context.Context, reference TReference, contents *Contents, childrenLeases []TLease, wantContentsIfIncomplete bool) (UploadObjectResult[TLease], error)
}

Uploader of objects into storage.

When UploadObject() is called without contents and leases set to nil, the backend reports whether the object exists and whether leases to its children are still valid. If the object does not exist, or leases of its children have expired, another call can be performed to provide the object's contents or update its leases.

type UploaderForTesting 

type UploaderForTesting Uploader[GlobalReference, any]

UploaderForTesting is an instantiation of Uploader for generating mocks used by tests.

type UploaderForTestingBytes 

type UploaderForTestingBytes Uploader[GlobalReference, []byte]

UploaderForTestingBytes is an instantiation of Uploader for generating mocks used by tests.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.