core

package
v0.0.0-...-1b293c9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 3, 2025 License: Apache-2.0 Imports: 24 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ExistingFileBackedObjectLocation = FileBackedObjectLocation{
	// contains filtered or unexported fields
}

ExistingFileBackedObjectLocation is similar to dag.ExistingObjectContentsWalker. It is a placeholder value that results in a dag.ObjectContentsWalker that fails to complete.

Functions

func DecodableLocalReferenceToString 

func DecodableLocalReferenceToString[TReference object.BasicReference](reference Decodable[TReference]) string

DecodableLocalReferenceToString converts a reference containing decoding parameters to a string of the format "${reference}.${decodingParameters}", where both parts are base64. Strings of this format may be printed as part of logs and other diagnostic output.

func DecodableLocalReferenceToWeakProto 

func DecodableLocalReferenceToWeakProto[TReference object.BasicReference](reference Decodable[TReference]) *model_core_pb.WeakDecodableReference

DecodableLocalReferenceToWeakProto converts a reference containing decoding parameters to a Protobuf message, so that it may be used as a weak reference as part of RPC request/response bodies or objects in storage.

func FlattenReference 

func FlattenReference[TReference any](m Message[*model_core_pb.Reference, TReference]) (TReference, error)

FlattenReference returns the actual reference that is associated with a given Reference Protobuf message.

func FlattenReferenceSet 

func FlattenReferenceSet[TReference any](m Message[*model_core_pb.ReferenceSet, TReference]) (object.OutgoingReferences[TReference], error)

FlattenReferenceSet returns the actual references that are contained in a given ReferenceSet Protobuf message.

func GetIndexFromReferenceMessage 

func GetIndexFromReferenceMessage(referenceMessage *core.Reference, degree int) (int, error)

GetIndexFromReferenceMessage extracts the one-based index contained in a Reference message and converts it to a zero-based index that can, for example, be provided to OutgoingReferences.GetOutgoingReference().

func MarshalTopLevelMessage 

func MarshalTopLevelMessage[TMessage proto.Message, TReference object.BasicReference](m TopLevelMessage[TMessage, TReference]) ([]byte, error)

MarshalTopLevelMessage converts the contents of a top-level Protobuf message to a byte sequence. This byte sequence can be used to sort messages along a total order, or to act as a map key.

func MessagesEqual 

func MessagesEqual[
	TMessage proto.Message,
	TReference1, TReference2 object.BasicReference,
](m1 Message[TMessage, TReference1], m2 Message[TMessage, TReference2]) bool

MessagesEqual returns true if two messages contain the same data.

Nested messages belonging to different objects may contain the same data, but use different reference numbers. This function therefore needs to convert the provided messages to top-level messages, which is expensive. We therefore compare the message size first.

func PatchedMessagesEqual 

func PatchedMessagesEqual[
	TMessage proto.Message,
	TMetadata1, TMetadata2 ReferenceMetadata,
](m1 PatchedMessage[TMessage, TMetadata1], m2 PatchedMessage[TMessage, TMetadata2]) bool

PatchedMessagesEqual returns true if two patched messages contain the same data.

Nested messages belonging to different objects may contain the same data, but use different reference numbers. This function therefore needs to convert the provided messages to top-level messages, which is expensive. We therefore compare the message size first.

func TopLevelMessagesEqual 

func TopLevelMessagesEqual[
	TMessage proto.Message,
	TReference1, TReference2 object.BasicReference,
](m1 TopLevelMessage[TMessage, TReference1], m2 TopLevelMessage[TMessage, TReference2]) bool

TopLevelMessagesEqual returns true if two top-level messages contain the same data. This function can only be called against top-level messages, as only those consistently number any outgoing references.

Types

type CloneableReference 

type CloneableReference[TMetadata any] struct {
	object.LocalReference
	// contains filtered or unexported fields
}

CloneableReference is a basic implementation of object.BasicReference that holds on to any metadata that was extracted out of the ReferenceMessagePatcher by PatchedMessageToCloneable(). The metadata is later reinserted into one or more ReferenceMessagePatchers by calling Patch() using ClonedObjectManager.

type CloneableReferenceMetadata 

type CloneableReferenceMetadata interface {
	ReferenceMetadata

	IsCloneable()
}

CloneableReferenceMetadata is a type of ReferenceMetadata that is safe to place in multiple ReferenceMessagePatchers. For example, ReferenceMetadata that stores the contents of an object in a file may not be cloneable. A simple in-memory implementation like CreatedObject[T] is cloneable.

type CloningObjectManager 

type CloningObjectManager[TMetadata any] struct{}

func (CloningObjectManager[TMetadata]) CaptureExistingObject 

func (CloningObjectManager[TMetadata]) CaptureExistingObject(reference CloneableReference[TMetadata]) TMetadata

func (CloningObjectManager[TMetadata]) ReferenceObject 

func (CloningObjectManager[TMetadata]) ReferenceObject(reference object.LocalReference, metadata TMetadata) CloneableReference[TMetadata]

type CreatedObject 

type CreatedObject[TMetadata any] struct {
	*object.Contents
	Metadata []TMetadata
}

CreatedObject holds the contents of an object that was created using ReferenceMessagePatcher. It also holds the metadata that was provided to ReferenceMessagePatcher.AddReference(), provided in the same order as the outgoing references of the created object.

type CreatedObjectCapturer 

type CreatedObjectCapturer[TMetadata any] interface {
	CaptureCreatedObject(CreatedObject[TMetadata]) TMetadata
}

CreatedObjectCapturer can be used as a factory type for reference metadata. Given the contents of an object and the metadata of all of its children, it may yield new metadata. 

var DiscardingCreatedObjectCapturer CreatedObjectCapturer[NoopReferenceMetadata] = discardingCreatedObjectCapturer{}

DiscardingCreatedObjectCapturer is an implementation of CreatedObjectCapturer that discards any created objects. This can be used if the goal is to simply compute the root node of the Merkle tree, discarding any children that were created in the process. 

var WalkableCreatedObjectCapturer CreatedObjectCapturer[dag.ObjectContentsWalker] = walkableCreatedObjectCapturer{}

WalkableCreatedObjectCapturer is an implementation of ObjectCapturer that creates a dag.ObjectContentsWalker for each created object. This ends up keeping objects in memory and only allows them to be traversed as part of the upload process.

This implementation is sufficient when given existing Merkle trees in the form of a dag.ObjectContentsWalkers that need to be combined into single Merkle tree.

type CreatedObjectCapturerFunc 

type CreatedObjectCapturerFunc[TMetadata any] func(CreatedObject[TMetadata]) TMetadata

func (CreatedObjectCapturerFunc[TMetadata]) CaptureCreatedObject 

func (f CreatedObjectCapturerFunc[TMetadata]) CaptureCreatedObject(createdObject CreatedObject[TMetadata]) TMetadata

type CreatedObjectTree 

type CreatedObjectTree CreatedObject[CreatedObjectTree]

CreatedObjectTree is CreatedObject applied recursively. Namely, it can hold the contents of a tree of objects that were created using ReferenceMessagePatcher in memory.

func (CreatedObjectTree) Discard 

func (CreatedObjectTree) Discard()

Discard any resources owned by the CreatedObjectTree.

func (CreatedObjectTree) IsCloneable 

func (CreatedObjectTree) IsCloneable()

IsCloneable indicates that instances of CreatedObjectTree may safely be placed in multiple ReferenceMessagePatchers.

func (CreatedObjectTree) ToObjectContentsWalker 

func (t CreatedObjectTree) ToObjectContentsWalker() dag.ObjectContentsWalker

ToObjectContentsWalker returns a ObjectContentsWalker that allows traversing all objects contained in the CreatedObjectTree.

type Decodable 

type Decodable[T any] struct {
	Value T
	// contains filtered or unexported fields
}

Decodable can be used to annotate an object with parameters that are used to decode an object. For example, Decodable[CreatedObject[T]] can hold a created object that can subsequently be accessed. Similarly, Decodable[object.LocalReference] can be used to refer to an object that can subsequently be accessed.

Decodable is comparable if its value is comparable as well.

func CopyDecodable 

func CopyDecodable[T1, T2 any](from Decodable[T1], to T2) Decodable[T2]

CopyDecodable extracts the decoding parameters of a given Decodable[T] and attaches it to another object.

func FlattenDecodableReference 

func FlattenDecodableReference[TReference any](m Message[*model_core_pb.DecodableReference, TReference]) (Decodable[TReference], error)

FlattenDecodableReference returns the actual reference that is associated with a given DecodableReference Protobuf message, and attaches decoding parameters to it.

func MarshalAndEncode 

func MarshalAndEncode[TMetadata ReferenceMetadata](
	m PatchedMessage[Marshalable, TMetadata],
	referenceFormat object.ReferenceFormat,
	encoder model_encoding.BinaryEncoder,
) (Decodable[CreatedObject[TMetadata]], error)

MarshalAndEncode marshals a patched message, encodes it, and converts it to an object that can be written to storage.

func NewDecodable 

func NewDecodable[T any](value T, decodingParameters []byte) (Decodable[T], error)

NewDecodable is a helper function for creating instances of Decodable[T].

func NewDecodableLocalReferenceFromString 

func NewDecodableLocalReferenceFromString(referenceFormat object.ReferenceFormat, s string) (Decodable[object.LocalReference], error)

NewDecodableLocalReferenceFromString converts a string of the format "${reference}.${decodingParameters}", where both parts are base64 encoded, to a decodable local reference. Strings of this format may be printed as part of logs and other diagnostic output.

func NewDecodableLocalReferenceFromWeakProto 

func NewDecodableLocalReferenceFromWeakProto(referenceFormat object.ReferenceFormat, m *model_core_pb.WeakDecodableReference) (Decodable[object.LocalReference], error)

NewDecodableLocalReferenceFromWeakProto converts a WeakDecodableReference Protobuf messages to its native counterpart.

func (*Decodable[T]) GetDecodingParameters 

func (d *Decodable[T]) GetDecodingParameters() []byte

GetDecodingParameters returns the parameters needed to decode the object associated with the value.

type ExistingObjectCapturer 

type ExistingObjectCapturer[TReference, TMetadata any] interface {
	CaptureExistingObject(TReference) TMetadata
}

ExistingObjectCapturer can be used as a factory type for reference metadata. Given a reference of an object that already exists ins torage, it may yield metadata.

type FileBackedObjectLocation 

type FileBackedObjectLocation struct {
	// contains filtered or unexported fields
}

FileBackedObjectLocation contains the location at which an object was written to disk. It can be used to reobtain the contents of the object later.

func (FileBackedObjectLocation) Discard 

func (FileBackedObjectLocation) Discard()

Discard any resources associated with the FileBackedObjectLocation. This is merely provided to satisfy the ReferenceMetadata interface.

type FileReadingObjectContentsWalkerFactory 

type FileReadingObjectContentsWalkerFactory struct {
	// contains filtered or unexported fields
}

FileReadingObjectContentsWalkerFactory is a factory type for dag.ObjectContentsWalker that is capable of reading the contents of objects from disk, which were previously written by FileWritingObjectCapturer.

func NewFileReadingObjectContentsWalkerFactory 

func NewFileReadingObjectContentsWalkerFactory(reader ReaderAtCloser) *FileReadingObjectContentsWalkerFactory

NewFileReadingObjectContentsWalkerFactory creates a FileReadingObjectContentsWalkerFactory that is backed by a given file.

func (*FileReadingObjectContentsWalkerFactory) CreateObjectContentsWalker 

func (wf *FileReadingObjectContentsWalkerFactory) CreateObjectContentsWalker(reference object.LocalReference, location FileBackedObjectLocation) dag.ObjectContentsWalker

CreateObjectContentsWalker creates a dag.ObjectContentsWalker that reads the contents of an object from disk. The location of where the object is stored must be provided in the form of aFileBackedObjectLocation.

func (*FileReadingObjectContentsWalkerFactory) Release 

func (wf *FileReadingObjectContentsWalkerFactory) Release()

Release the underlying file by closing it. This function must be invoked after all calls to CreateObjectContentsWalker have been made.

type FileWritingObjectCapturer 

type FileWritingObjectCapturer struct {
	// contains filtered or unexported fields
}

FileWritingObjectCapturer is capable of sequentially writing the contents of objects to a file on disk. This can be used to temporarily store large Merkle trees on disk prior to being uploaded to a remote storage server. It may not be feasible to store the Merkle tree in memory.

The format in which data is stored on disk is simple. For each object, the contents are stored literally. After every object, the offset of any previously written children is stored. This makes it possible to traverse the full graph of objects afterwards.

func NewFileWritingObjectCapturer 

func NewFileWritingObjectCapturer(w io.Writer) *FileWritingObjectCapturer

NewFileWritingObjectCapturer creates a FileWritingObjectCapturer that is in the initial state (i.e., assuming the output file is empty).

func (*FileWritingObjectCapturer) CaptureCreatedObject 

func (c *FileWritingObjectCapturer) CaptureCreatedObject(createdObject CreatedObject[FileBackedObjectLocation]) FileBackedObjectLocation

CaptureObject writes the contents of a single object to disk.

func (*FileWritingObjectCapturer) Flush 

func (c *FileWritingObjectCapturer) Flush() error

Flush the contents of any lingering objects to disk. This function needs to be called to ensure that all objects can be retrieved afterwards.

type Marshalable 

type Marshalable interface {
	Marshal() ([]byte, error)
}

Marshalable represents a type of value that can be stored in an object.

This type is typically used in contexts where a PatchedMessage[TMessage, TMetadata] needs to be passed along, but no assumptions may be made about the message type. In some of those cases it's possible to use PatchedMessage[proto.Message, TMetadata], but that wouldn't work if TMessage is a list. This can be solved by using PatchedMessage[Marshalable, TMetadata] instead.

func NewProtoListMarshalable 

func NewProtoListMarshalable[TMessage proto.Message](messages []TMessage) Marshalable

NewProtoListMarshalable converts a list of Protobuf message to a Marshalable. This prevents further access to the message's contents, but does allow it to be passed along in contexts where the data is only expected to be marshaled and stored.

When marshaled, each message is prefixed with its size.

func NewProtoMarshalable 

func NewProtoMarshalable(message proto.Message) Marshalable

NewProtoMarshalable converts a Protobuf message to a Marshalable. This prevents further access to the message's contents, but does allow it to be passed along in contexts where the data is only expected to be marshaled and stored.

func NewRawMarshalable 

func NewRawMarshalable(data []byte) Marshalable

type Message 

type Message[TMessage any, TReference any] struct {
	Message            TMessage
	OutgoingReferences object.OutgoingReferences[TReference]
}

Message is a piece of data, typically a Protobuf message, that has zero or more outgoing references associated with it.

func Nested 

func Nested[TMessage1, TMessage2, TReference any](parent Message[TMessage1, TReference], child TMessage2) Message[TMessage2, TReference]

Nested is a helper function for creating instances of Message that refer to a message that was mebedded into another one.

func NewMessage 

func NewMessage[TMessage, TReference any](
	m TMessage,
	outgoingReferences object.OutgoingReferences[TReference],
) Message[TMessage, TReference]

NewMessage is a helper function for creating instances of Message.

func NewSimpleMessage 

func NewSimpleMessage[TReference, TMessage any](m TMessage) Message[TMessage, TReference]

NewSimpleMessage is a helper function for creating instances of Message for messages that don't contain any references.

func (*Message[TMessage, TReference]) Clear 

func (m *Message[TMessage, TReference]) Clear()

Clear the Message, releasing the data and outgoing references that are associated with it.

func (Message[TMessage, TReference]) IsSet 

func (m Message[TMessage, TReference]) IsSet() bool

IsSet returns true if the instance of Message is initialized.

type NoopReferenceMetadata 

type NoopReferenceMetadata struct{}

NoopReferenceMetadata is a trivial implementation of ReferenceMetadata that does not capture anything. This can be used if the goal is to simply compute the root node of the Merkle tree, discarding any children that were created in the process.

func (NoopReferenceMetadata) Discard 

func (NoopReferenceMetadata) Discard()

Discard all resources owned by this instance of NoopReferenceMetadata. This method is merely provided to satisfy the ReferenceMetadata interface.

type ObjectCapturer 

type ObjectCapturer[TReference, TMetadata any] interface {
	CreatedObjectCapturer[TMetadata]
	ExistingObjectCapturer[TReference, TMetadata]
}

ObjectCapturer is a combination of CreatedObjectCapturer and ExistingObjectCapturer, allowing the construction of metadata both for newly created objects and ones that exist in storage.

func NewDiscardingObjectCapturer 

func NewDiscardingObjectCapturer[TReference any]() ObjectCapturer[TReference, NoopReferenceMetadata]

NewDiscardingObjectCapturer creates is an implementation of ObjectCapturer that discards any created objects, and stores no metadata for any existing objects. This can be used if the goal is to simply compute the root node of the Merkle tree, discarding any children that were created in the process.

type ObjectManager 

type ObjectManager[TReference, TMetadata any] interface {
	ObjectCapturer[TReference, TMetadata]
	ObjectReferencer[TReference, TMetadata]
}

ObjectManager is an extension to ObjectCapturer, allowing metadata to be converted back to references. This can be of use in environments where objects also need to be accessible for reading right after they have been constructed, without explicitly waiting for them to be written to storage.

type ObjectReferencer 

type ObjectReferencer[TReference, TMetadata any] interface {
	ReferenceObject(object.LocalReference, TMetadata) TReference
}

type PatchedMessage 

type PatchedMessage[TMessage any, TMetadata ReferenceMetadata] struct {
	Message TMessage
	Patcher *ReferenceMessagePatcher[TMetadata]
}

PatchedMessage is a tuple for storing a Protobuf message that contains model_core_pb.Reference messages, and the associated ReferenceMessagePatcher that can be used to assign indices to these references.

func BuildPatchedMessage 

func BuildPatchedMessage[TMessage any, TMetadata ReferenceMetadata](
	builder func(*ReferenceMessagePatcher[TMetadata]) TMessage,
) PatchedMessage[TMessage, TMetadata]

BuildPatchedMessage is a convenience function for constructing PatchedMessage that has a newly created ReferenceMessagePatcher attached to it. A callback is invoked for creating a message that is managed by the ReferenceMessagePatcher.

func MarshalAny 

func MarshalAny[TMessage proto.Message, TMetadata ReferenceMetadata](m PatchedMessage[TMessage, TMetadata]) (PatchedMessage[*model_core_pb.Any, TMetadata], error)

MarshalAny wraps a patched message into a model_core_pb.Any message. References stored in the original message are kept intact.

func NewPatchedMessage 

func NewPatchedMessage[TMessage any, TMetadata ReferenceMetadata](
	message TMessage,
	patcher *ReferenceMessagePatcher[TMetadata],
) PatchedMessage[TMessage, TMetadata]

NewPatchedMessage creates a PatchedMessage, given an existing Protobuf message and reference message patcher.

func NewPatchedMessageFromExisting 

func NewPatchedMessageFromExisting[
	TMessage proto.Message,
	TMetadata ReferenceMetadata,
	TReference object.BasicReference,
](
	existing Message[TMessage, TReference],
	createMetadata ReferenceMetadataCreator[TMetadata],
) PatchedMessage[TMessage, TMetadata]

NewPatchedMessageFromExisting creates a PatchedMessage, given an existing Protobuf message that may contain one or more references to other objects. For each reference that is found, a callback is invoked to create metadata to associate with the reference.

func NewSimplePatchedMessage 

func NewSimplePatchedMessage[TMetadata ReferenceMetadata, TMessage any](v TMessage) PatchedMessage[TMessage, TMetadata]

NewSimplePatchedMessage is a helper function for creating instances of PatchedMessage for messages that don't contain any references.

func Patch 

func Patch[
	TMessage proto.Message,
	TMetadata ReferenceMetadata,
	TReference object.BasicReference,
](
	capturer ExistingObjectCapturer[TReference, TMetadata],
	m Message[TMessage, TReference],
) PatchedMessage[TMessage, TMetadata]

Patch an existing message, copying the message and causing all containing references to be managed by a ReferenceMessagePatcher. For each references contained within, metadata is created by calling into an ExistingObjectCapturer.

func PatchList 

func PatchList[
	TMessage any,
	TMessagePtr interface {
		*TMessage
		proto.Message
	},
	TMetadata ReferenceMetadata,
	TReference object.BasicReference,
](
	capturer ExistingObjectCapturer[TReference, TMetadata],
	existingList Message[[]*TMessage, TReference],
) PatchedMessage[[]*TMessage, TMetadata]

PatchList is identical to Patch(), except that it patches a list of messages.

func ProtoListToMarshalable 

func ProtoListToMarshalable[TMessage proto.Message, TMetadata ReferenceMetadata](m PatchedMessage[[]TMessage, TMetadata]) PatchedMessage[Marshalable, TMetadata]

func ProtoToMarshalable 

func ProtoToMarshalable[TMessage proto.Message, TMetadata ReferenceMetadata](m PatchedMessage[TMessage, TMetadata]) PatchedMessage[Marshalable, TMetadata]

func (*PatchedMessage[T, TMetadata]) Clear 

func (m *PatchedMessage[T, TMetadata]) Clear()

Clear the instance of PatchedMessage, disassociating it from its message and reference message patcher. The reference message patcher is not discarded, meaning that any resources owned by reference metadata is not released.

func (*PatchedMessage[T, TMetadata]) Discard 

func (m *PatchedMessage[T, TMetadata]) Discard()

Discard the reference message patcher, releasing any resources owned by reference metadata.

func (PatchedMessage[T, TMetadata]) IsSet 

func (m PatchedMessage[T, TMetadata]) IsSet() bool

IsSet returns true if the PatchedMessage is assigned to a message and its associated reference message patcher.

func (PatchedMessage[T, TMetadata]) SortAndSetReferences 

func (m PatchedMessage[T, TMetadata]) SortAndSetReferences() (TopLevelMessage[T, object.LocalReference], []TMetadata)

SortAndSetReferences assigns indices to outgoing references.

type ReaderAtCloser 

type ReaderAtCloser interface {
	io.ReaderAt
	io.Closer
}

ReaderAtCloser is used by FileReadingObjectContentsWalkerFactory to access objects that were written to disk.

type ReferenceMessagePatcher 

type ReferenceMessagePatcher[TMetadata ReferenceMetadata] struct {
	// contains filtered or unexported fields
}

ReferenceMessagePatcher keeps track of all Reference messages that are contained within a given Protobuf message. For each of these Reference messages, it stores the actual object.LocalReference that is associated with them.

Once the Protobuf message has been fully constructed, SortAndSetReferences() can be used to return a sorted, deduplicated list of all outgoing references of the message, and to update the indices in the Reference messages to refer to the correct outgoing reference.

func MapReferenceMessagePatcherMetadata 

func MapReferenceMessagePatcherMetadata[TOld, TNew ReferenceMetadata](pOld *ReferenceMessagePatcher[TOld], mapMetadata func(object.LocalReference, TOld) TNew) *ReferenceMessagePatcher[TNew]

MapReferenceMessagePatcherMetadata replaces a ReferenceMessagePatcher with a new instance that contains the same references, but has metadata mapped to other values, potentially of another type.

func MapReferenceMetadataToWalkers 

func MapReferenceMetadataToWalkers[TMetadata WalkableReferenceMetadata](p *ReferenceMessagePatcher[TMetadata]) *ReferenceMessagePatcher[dag.ObjectContentsWalker]

MapReferenceMetadataToWalkers replaces all the reference metadata in a patched message with instances of dag.ObjectContentsWalker.

This function can be used in contexts where we only care about transmitting the contents of a DAG to a server, and no longer need to access any of the additional metadata that was gathered.

func NewReferenceMessagePatcher 

func NewReferenceMessagePatcher[TMetadata ReferenceMetadata]() *ReferenceMessagePatcher[TMetadata]

NewReferenceMessagePatcher creates a new ReferenceMessagePatcher that does not contain any Reference messages.

func (*ReferenceMessagePatcher[TMetadata]) AddReference 

func (p *ReferenceMessagePatcher[TMetadata]) AddReference(reference object.LocalReference, metadata TMetadata) *core.Reference

AddReference allocates a new Reference message that is associated with a given object.LocalReference and caller provided metadata.

func (*ReferenceMessagePatcher[TMetadata]) CaptureAndAddDecodableReference 

func (p *ReferenceMessagePatcher[TMetadata]) CaptureAndAddDecodableReference(
	createdObject Decodable[CreatedObject[TMetadata]],
	capturer CreatedObjectCapturer[TMetadata],
) *core.DecodableReference

CaptureAndAddDecodableReference is a helper function for AddReference that can be called in the common case where a DecodableReference needs to be emitted, referring to a newly created object.

func (*ReferenceMessagePatcher[TMetadata]) Discard 

func (p *ReferenceMessagePatcher[TMetadata]) Discard()

Discard all resources owned by all metadata managed by this reference message patcher.

func (*ReferenceMessagePatcher[TMetadata]) GetHeight 

func (p *ReferenceMessagePatcher[TMetadata]) GetHeight() int

GetHeight returns the height that the object of the Protobuf message would have if it were created with the current set of outgoing references.

func (*ReferenceMessagePatcher[TMetadata]) GetReferencesSizeBytes 

func (p *ReferenceMessagePatcher[TMetadata]) GetReferencesSizeBytes() int

GetReferencesSizeBytes returns the size that all of the outgoing references would have if an object were created with the current set of outgoing references.

func (*ReferenceMessagePatcher[TMetadata]) Merge 

func (p *ReferenceMessagePatcher[TMetadata]) Merge(other *ReferenceMessagePatcher[TMetadata])

Merge multiple instances of ReferenceMessagePatcher together. This method can be used when multiple Protobuf messages are combined into a larger message, and are eventually stored as a single object.

func (*ReferenceMessagePatcher[TMetadata]) SortAndSetReferences 

func (p *ReferenceMessagePatcher[TMetadata]) SortAndSetReferences() (object.OutgoingReferencesList[object.LocalReference], []TMetadata)

SortAndSetReferences returns a sorted list of all outgoing references of the Protobuf message. This list can be provided to object.NewContents() to construct an actual object for storage. In addition to that, a list of user provided metadata is returned that sorted along the same order.

type ReferenceMetadata 

type ReferenceMetadata interface {
	Discard()
}

ReferenceMetadata is freeform metadata that can be associated with any reference managed by a ReferenceMessagePatcher.

Simple implementations of ReferenceMessagePatcher may capture the contents of the object referenced, causing the full Merkle tree to reside in memory. Alternatively, implementations may also store information on how the data that is referenced was obtained, so that it may be recomputed if needed.

As ReferenceMatadata may contain things like file descriptors, a Discard() method is provided to release such resources in case ReferenceMessagePatcher encounters duplicate references.

type ReferenceMetadataCreator 

type ReferenceMetadataCreator[T any] func(index int) T

ReferenceMetadataCreator is invoked by NewPatchedMessageFromExisting for every reference that is encountered, allowing metadata for the outgoing reference to be created.

type ReferenceMetadataCreatorForTesting 

type ReferenceMetadataCreatorForTesting ReferenceMetadataCreator[ReferenceMetadata]

ReferenceMetadataCreatorForTesting is an instantiation of ReferenceMetadataCreator for generating mocks to be used by tests.

type TopLevelMessage 

type TopLevelMessage[TMessage any, TReference any] struct {
	Message            TMessage
	OutgoingReferences object.OutgoingReferences[TReference]
}

TopLevelMessage is identical to Message, except that it may only refer to a message for which all embedded references are consecutively numbered, and the outgoing references list contains no references in addition to the ones embedded in the message.

As top-level messages are considered being canonical, this type provides some additional methods for generating keys and performing comparisons.

func FlattenAny 

func FlattenAny[TReference any](m Message[*model_core_pb.Any, TReference]) (TopLevelMessage[*anypb.Any, TReference], error)

FlattenAny extracts the anypb.Any message that's embedded in a model_core_pb.Any message. The resulting message only has access to the outgoing references of the inner message.

func MarshalTopLevelAny 

func MarshalTopLevelAny[TMessage proto.Message, TReference any](m TopLevelMessage[TMessage, TReference]) (TopLevelMessage[*anypb.Any, TReference], error)

MarshalTopLevelAny wraps a message in an anypb.Any.

This function assumes that the provided message is a top-level message. Only top-level messages are permitted to be wrapped in an anypb.Any, as it's impossible to create a model_core_pb.Any for messages having outgoing references with sparse indices.

func NewSimpleTopLevelMessage 

func NewSimpleTopLevelMessage[TReference, TMessage any](m TMessage) TopLevelMessage[TMessage, TReference]

NewSimpleTopLevelMessage creates a TopLevelMessage that has no outgoing references.

func NewTopLevelMessage 

func NewTopLevelMessage[TMessage, TReference any](
	m TMessage,
	outgoingReferences object.OutgoingReferences[TReference],
) TopLevelMessage[TMessage, TReference]

NewTopLevelMessage creates a TopLevelMessage that corresponds to an already existing Protobuf message.

func UnmarshalAnyNew 

func UnmarshalAnyNew[TReference any](m Message[*model_core_pb.Any, TReference]) (TopLevelMessage[proto.Message, TReference], error)

UnmarshalAnyNew extracts the message contained in a model_core.Any, and ensures that any references contained within are accessible.

func UnmarshalTopLevelAnyNew 

func UnmarshalTopLevelAnyNew[TReference any](m TopLevelMessage[*anypb.Any, TReference]) (TopLevelMessage[proto.Message, TReference], error)

UnmarshalTopLevelAnyNew extracts the message contained in an anypb.Any, and ensures that any references contained within are accessible.

This function assumes that the anypb.Any is a top-level message, meaning that the inner message has access to the same set of references as the outer message.

func UnmarshalTopLevelMessage 

func UnmarshalTopLevelMessage[
	TMessage any,
	TMessagePtr interface {
		*TMessage
		proto.Message
	},
](referenceFormat object.ReferenceFormat, data []byte) (TopLevelMessage[TMessagePtr, object.LocalReference], error)

UnmarshalTopLevelMessage performs the inverse of MarshalTopLevelMessage. It can be used to convert a byte sequence to a top-level Protobuf message.

func Unpatch 

func Unpatch[TMessage, TReference any, TMetadata ReferenceMetadata](
	referencer ObjectReferencer[TReference, TMetadata],
	m PatchedMessage[TMessage, TMetadata],
) TopLevelMessage[TMessage, TReference]

Unpatch performs the opposite of Patch(). Namely, it assigns indices to all references contained in the current message. Metadata that is tracked by the ReferenceMessagePatcher is converted to references by calling into an ObjectReferencer.

func (TopLevelMessage[TMessage, TReference]) Decay 

func (m TopLevelMessage[TMessage, TReference]) Decay() Message[TMessage, TReference]

Decay a top-level message to a plain message.

type WalkableReferenceMetadata 

type WalkableReferenceMetadata interface {
	ReferenceMetadata

	ToObjectContentsWalker() dag.ObjectContentsWalker
}

WalkableReferenceMetadata can be implemented by reference metadata to indicate that they can be converted to a dag.ObjectContentsWalker, allowing the contents of any captured objects to be obtained up to once. This allows them to be written to a storage server.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.