stemma

package module
v0.0.0-...-3756133 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 12, 2015 License: Apache-2.0 Imports: 26 Imported by: 0

README

stemma

Documentation

Index

Constants

View Source 
const EncodedObjectTypeSize = 1

EncodedObjectTypeSize is the size in bytes for an encoded object type which is placed at the begining of all object files.

Variables

View Source 
var (
	ErrInvalidDigestFormat = errors.New("invalid digest format")
	ErrInvalidDigestAlg    = errors.New("invalid digest algorithm")
)

Common Digest Errors.

View Source 
var (
	ErrNoSuchTag  = errors.New("no such tag")
	ErrinvalidTag = errors.New("invalid tag")
)

Common errors.

Functions

func EnsureObjectType 

func EnsureObjectType(r io.Reader, expected ObjectType) error

EnsureObjectType reads from the given reader to unmarshal an object type. The decoded object type is checked to ensure it is equal to the expected.

func MarshalDescriptor 

func MarshalDescriptor(w io.Writer, d Descriptor) error

marshal marshals this descriptor to its binary encoding to the given writer.

func MarshalTagDescriptors 

func MarshalTagDescriptors(w io.Writer, tagDescriptors map[string]Descriptor) error

func UnmarshalTagDescriptors 

func UnmarshalTagDescriptors(r io.Reader) (tagDescriptors map[string]Descriptor, err error)

Types

type Application 

type Application struct {
	Rootfs Rootfs
}

Application describes the rootfs and configuration for an application container.

func UnmarshalApplication 

func UnmarshalApplication(r io.Reader) (a Application, err error)

UnmarshalApplication unmarshals an Application from the binary encoding (little-endian) read from the given reader.

func (Application) Dependencies 

func (a Application) Dependencies() []Descriptor

Dependencies returs a list of Descriptors for the dependencies of this app.

func (Application) Marshal 

func (a Application) Marshal(w io.Writer) error

Marshal marshals this application to a binary encoding (little-endian) to the given writer.

type Descriptor 

type Descriptor interface {
	Digest() Digest
	Size() uint64
	Type() ObjectType

	// If this object is a directory, NumSubObjects is the total number
	// of objects references by the directory and all subdirectories and
	// SubObjectsSize is the total size of all of the referenced objects.
	NumSubObjects() uint32
	SubObjectsSize() uint64
}

Descriptor describes an object.

func UnmarshalDescriptor 

func UnmarshalDescriptor(r io.Reader) (Descriptor, error)

unmarshalDescriptor unmarshals a descriptor object from its binary encoding from the given reader.

type DescriptorQueue 

type DescriptorQueue interface {
	Len() int
	Empty() bool
	Full() bool
	PushBack(desc Descriptor)
	Peek() Descriptor
	Pop() Descriptor
}

DescriptorQueue is a FIFO queue of object descriptors.

func NewDescriptorQueue 

func NewDescriptorQueue(maxSize int) DescriptorQueue

NewDescriptorQueue creates a new FIFO queue with the given (advisory) maximum size. If maxSize <= 0, Full() will always return false.

type DescriptorStack 

type DescriptorStack interface {
	Len() int
	Empty() bool
	Full() bool
	PushFront(desc Descriptor)
	Peek() Descriptor
	Pop() Descriptor
}

DescriptorStack is a LIFO queue of object descriptors.

func NewDescriptorStack 

func NewDescriptorStack(maxSize int) DescriptorStack

NewDescriptorStack creates a new LIFO queue with the given (advisory) maximum size. If maxSize <= 0, Full() will always return false.

type Digest 

type Digest []byte

Digest is a byte sum from a cryptographic hash function tagged with an algorithm identifier prefix byte corresponding to one of the above Digest Algorithms.

func ParseDigest 

func ParseDigest(dgst string) (Digest, error)

ParseDigest parses a new digest from the given hexadecimal digest string.

func UnmarshalDigest 

func UnmarshalDigest(r io.Reader) (Digest, error)

UnmarshalDigest unmarshals a digest from the given reader.

func (Digest) Algorithm 

func (d Digest) Algorithm() DigestAlg

Algorithm returns the algorithm identifier for this digest.

func (Digest) Bytes 

func (d Digest) Bytes() []byte

Bytes returns a copy of the raw bytes of this digest (the first byte is the 1-byte algorithm identifier).

func (Digest) Equals 

func (d Digest) Equals(other Digest) bool

Equals returns whether this digest is equal to the given other digest.

func (Digest) Hex 

func (d Digest) Hex() string

Hex returns the hexadecimal encoding of this digest.

func (Digest) Marshal 

func (d Digest) Marshal(w io.Writer) error

Marshal marshals this digest into the given writer.

func (Digest) String 

func (d Digest) String() string

type DigestAlg 

type DigestAlg byte

DigestAlg is used to specify a digest algorithm. 255 possibilities should be enough to last us until the robots take over. The digest will be encoded to hex and prepended to hex digest strings (e.g., all SHA256 digests will begin with "01...", all SHA512_256 with "05..."). The remainder of the digest string will be the hex encoded digest of corresponding length. 

const (
	DigestAlgSHA224 DigestAlg = iota
	DigestAlgSHA256
	DigestAlgSHA384
	DigestAlgSHA512
	DigestAlgSHA512_224
	DigestAlgSHA512_256
	DigestAlgSHA3_224
	DigestAlgSHA3_256
	DigestAlgSHA3_384
	DigestAlgSHA3_512

	DigestAlgUnknown = 255
)

Digest algorithms.

func (DigestAlg) String 

func (a DigestAlg) String() string

type Digester 

type Digester interface {
	hash.Hash
	Algorithm() DigestAlg
	Digest() Digest
}

Digester is used to generate a digest from hashing raw object data.

func NewDigester 

func NewDigester(alg DigestAlg) (Digester, error)

NewDigester returns a new Digester using the given algorithm. If the given algorithm is not supported the returned error will be ErrInvalidDigestAlg.

type Directory 

type Directory []DirectoryEntry

Directory is a list of directory entries. Implements sort.Interface.

func UnmarshalDirectory 

func UnmarshalDirectory(r io.Reader) (d Directory, err error)

UnmarshalDirectory unmarshals the binary encoding (little-endian) of a directory from the given reader.

func (Directory) Dependencies 

func (d Directory) Dependencies() []Descriptor

Dependencies returns a list of Descriptors for the objects inside this directory.

func (Directory) Len 

func (d Directory) Len() int

func (Directory) Less 

func (d Directory) Less(i, j int) bool

func (Directory) Marshal 

func (d Directory) Marshal(w io.Writer) error

Marshal marshals the binary encoding (little-endian) of this directory into the given writer.

func (Directory) Swap 

func (d Directory) Swap(i, j int)

func (Directory) TotalNumSubOjbects 

func (d Directory) TotalNumSubOjbects() uint32

TotalNumSubOjbects returns the total number of objects referenced by this directory and any subdirectories.

func (Directory) TotalSubOjbectSize 

func (d Directory) TotalSubOjbectSize() uint64

TotalSubOjbectSize returns the total size of objects referenced by this directory and any subdirectories.

type DirectoryEntry 

type DirectoryEntry struct {
	Name string
	Type DirentType

	LinkTarget string // If type is DirentTypeLink.

	HeaderDigest Digest // Digest of the object's corresponding header.
	HeaderSize   uint64 // Size of Header object in Bytes

	ObjectDigest Digest // Digest of the object if regular file or dir.
	ObjectSize   uint64 // Size of Object in Bytes.

	// If this entry is a subdirectory, NumSubObjects is the number of
	// additional objects in the subdirectory (headers + regular files +
	// subdirectories(...)). SubObjectSize is the total size of each
	// subdirectory entry header + object size + recursive subobject size.
	NumSubObjects  uint32
	SubObjectsSize uint64
}

DirectoryEntry provides fields of a directory entry. ObjectSize is stored on the directory entry rather than inside the header object for 2 reasons: 1) Having the size in the directory entry allows for knowing the expected size of the object to fetch which can prevent an endless-data attack. 2) Not having the size in the header object (along with not storing timestamps in the header object) allows for reuse of the header object when only the value of the object has changed and all header fields have remained the same.

func UnmarshalDirectoryEntry 

func UnmarshalDirectoryEntry(r io.Reader) (de DirectoryEntry, err error)

UnmarshalDirectoryEntry unmarshals the binary encoding (little-endian) of a directory entry from the given reader.

func (DirectoryEntry) HeaderDescriptor 

func (de DirectoryEntry) HeaderDescriptor() Descriptor

HeaderDescriptor returns a descriptor for the header object associated with this directory entry.

func (DirectoryEntry) IsDir 

func (de DirectoryEntry) IsDir() bool

IsDir returns whether this directory entry is of type DirentTypeDirectory.

func (DirectoryEntry) Marshal 

func (de DirectoryEntry) Marshal(w io.Writer) error

Marshal marshals the binary encoding (little-endian) of this directory entry into the given writer.

func (DirectoryEntry) ObjectDescriptor 

func (de DirectoryEntry) ObjectDescriptor() Descriptor

ObjectDescriptor returns a descriptor for the file or directory object associated with this directory entry. If this directory entry is not a regular file or directory object, a nil descriptor is returned.

type DirectoryWriter 

type DirectoryWriter interface {
	Add(entry DirectoryEntry)
	Commit() (Descriptor, error)
}

DirectoryWriter provides a handle for writing a new directory object into a DirectoryStore.

type DirentType 

type DirentType byte

DirentType specifies the type of an entry in a directory listing. 

const (
	DirentTypeUnknown DirentType = iota
	DirentTypeBlockDevice
	DirentTypeCharDevice
	DirentTypeDirectory
	DirentTypeFifo
	DirentTypeLink
	DirentTypeRegular
	DirentTypeSocket
)

Available directory entry types.

type FileWriter 

type FileWriter interface {
	io.Writer
	// Digest returns the digest of the data which has been written so far.
	Digest() Digest
	// Commit completes the object writing process, cleaning up any
	// temporary resources. The new object is stored with the given object
	// type and computed digest.
	Commit() (Descriptor, error)
	// Cancel ends the writing process, cleaning up any temporary
	// resources.
	Cancel() error
}

FileWriter provides a handle for writing a new file object into a FileStore. 

type Header struct {
	Mode   os.FileMode       // File mode
	Rdev   uint32            // Device numbers
	UID    uint32            // Owner uid
	GID    uint32            // Group gid
	Xattrs map[string][]byte // Extended attributes
}

Header provides all filesystem header fields. Note that file size is not stored in this header as it is stored in the directory entry object instead. Datetimes for creation, last-modified, last-accessed, and last-change are not stored at all. This allows for many objects with the same mode and ownership to use the same header object whether they would not be able to if they had differing sizes or timestamps.

func NewHeader 

func NewHeader(path string) (Header, error)

NewHeader returns a new Header for the object at the given path.

func UnmarshalHeader 

func UnmarshalHeader(r io.Reader) (header Header, err error)

UnmarshalHeader unmarshals the binary encoding (little-endian) of a header from the given reader.

func (Header) DirentType 

func (h Header) DirentType() DirentType

DirentType returns the directory entry type for this header.

func (Header) Marshal 

func (h Header) Marshal(w io.Writer) error

Marshal marshals the binary encoding (little-endian) of this header into the given writer.

type MountSet 

type MountSet interface {
	List() (digests []Digest, err error)
	Add(digest Digest) error
	Remove(digest Digest) error
}

MountSet is the interface for managing mounts of application container rootfs directories.

type ObjectStore 

type ObjectStore interface {
	// Whether an object with the given digest exists in this store.
	Contains(digest Digest) bool
	// Get the header object with the given digest from this store.
	GetHeader(digest Digest) (Header, error)
	// Put the given header into this object store.
	PutHeader(header Header) (Descriptor, error)
	// Get the file object with the given digest from this store.
	GetFile(digest Digest) (ReadSeekCloser, error)
	// NewWriter begins the process of writing a new file using this store.
	NewFileWriter() (FileWriter, error)
	// Get the directory with the given digest from this store.
	GetDirectory(digest Digest) (Directory, error)
	// NewWriter begins the process of writing a new directory using this
	// store.
	NewDirectoryWriter(estimatedSize uint) (DirectoryWriter, error)
	// Get the application object with the given digest from this store.
	GetApplication(digest Digest) (Application, error)
	// Put the given application into this object store.
	PutApplication(a Application) (Descriptor, error)
}

ObjectStore is the interface for managing content-addressable objects.

type ObjectType 

type ObjectType byte

ObjectType is used to indicate the type of object which is stored. 

const (
	ObjectTypeFile ObjectType = iota
	ObjectTypeDirectory
	ObjectTypeHeader
	ObjectTypeApplication
)

Various object types.

func UnmarshalObjectType 

func UnmarshalObjectType(r io.Reader) (ot ObjectType, err error)

UnmarshalObjectType reads an encoded ObjectType from the given reader.

func (ObjectType) Marshal 

func (ot ObjectType) Marshal(w io.Writer) error

Marshal writes this object type as a single byte header for objects using the given writer.

func (ObjectType) String 

func (ot ObjectType) String() string

type ProgressMeter 

type ProgressMeter struct {
	TransferredObjects uint32
	SkippedObjects     uint32
	TotalObjects       uint32
	TransferredSize    uint64
	SkippedSize        uint64
	TotalSize          uint64
}

type ReadSeekCloser 

type ReadSeekCloser interface {
	io.Reader
	io.Seeker
	io.Closer
}

ReadSeekCloser is a combination reader, seeker, and closer.

type ReadWriteFlusher 

type ReadWriteFlusher interface {
	Read(p []byte) (n int, err error)
	Write(p []byte) (nn int, err error)
	Flush() error
}

type RemoteObjectFetcher 

type RemoteObjectFetcher interface {
	RequestObject(desc Descriptor) error
	SkipObject(desc Descriptor) error
	NextObject(size uint64) io.Reader
	SignalDone() error
}

RemoteObjectFetcher represents a session with a remote object store which we are fetching objects from.

type RemoteObjectStore 

type RemoteObjectStore interface {
	GetTag(name string) (Descriptor, error)
	ListTags() (map[string]Descriptor, error)
	Fetch(desc Descriptor, progress *ProgressMeter) error
	Push(desc Descriptor, progress *ProgressMeter) error
}

RemoteObjectStore represents a connection to a remote object store.

type Repository 

type Repository struct {

	// Advisory lock for the repository. Acquire an exclusive lock
	// if your opperation may modify refs or delete objects. Acquire a
	// shared lock if your opperation will only be reading refs or reading
	// or writing objects. Note: a shared lock for writing objects is okay
	// due to the content-addressibility of the object store.
	*sysutil.Lock
	// contains filtered or unexported fields
}

Repository represents a content-addressable repository of filesystem objects and application container metadata.

func NewRepository 

func NewRepository(root string) (*Repository, error)

NewRepository returns a repository storing objects at the given root directory.

func (*Repository) Contains 

func (r *Repository) Contains(digest Digest) bool

Contains returns whether an object with the given digest exists in this repository.

func (*Repository) GetApplication 

func (r *Repository) GetApplication(digest Digest) (a Application, err error)

GetApplication gets the application object with the given digest from this repository.

func (*Repository) GetDirectory 

func (r *Repository) GetDirectory(digest Digest) (Directory, error)

GetDirectory gets the directory with the given digest from this repository.

func (*Repository) GetFile 

func (r *Repository) GetFile(digest Digest) (ReadSeekCloser, error)

GetFile opens the file object with the given digest from this repository.

func (*Repository) GetHeader 

func (r *Repository) GetHeader(digest Digest) (header Header, err error)

GetHeader gets the header object with the given digest from this repository.

func (*Repository) HandleGetTag 

func (r *Repository) HandleGetTag(rw http.ResponseWriter, req *http.Request)

func (*Repository) HandleListTags 

func (r *Repository) HandleListTags(rw http.ResponseWriter, req *http.Request)

func (*Repository) HandleReceiveObjects 

func (r *Repository) HandleReceiveObjects(rw http.ResponseWriter, req *http.Request)

func (*Repository) HandleServeObjects 

func (r *Repository) HandleServeObjects(rw http.ResponseWriter, req *http.Request)

func (*Repository) NewDirectoryWriter 

func (r *Repository) NewDirectoryWriter(estimatedSize uint) (DirectoryWriter, error)

NewDirectoryWriter begins the process of writing a new directory into this repository.

func (*Repository) NewFileWriter 

func (r *Repository) NewFileWriter() (FileWriter, error)

NewFileWriter begins the process of writing a new file in this repository.

func (*Repository) PutApplication 

func (r *Repository) PutApplication(a Application) (Descriptor, error)

PutApplication puts the given application into this object repository.

func (*Repository) PutHeader 

func (r *Repository) PutHeader(header Header) (Descriptor, error)

PutHeader puts the given header into this repository.

func (*Repository) RemoteObjectStore 

func (r *Repository) RemoteObjectStore(remoteURL string) (RemoteObjectStore, error)

func (*Repository) ResolveRef 

func (r *Repository) ResolveRef(ref string) (Digest, error)

ResolveRef resolves the given reference string (either a tag or a hex- encoded digest) to a valid digest.

func (*Repository) StoreDirectory 

func (r *Repository) StoreDirectory(path string) (Descriptor, error)

StoreDirectory recursively stores the directory at the given path in this repository.

func (*Repository) StoreFile 

func (r *Repository) StoreFile(path string) (Descriptor, error)

StoreFile stores the file at the given path in this repository.

func (*Repository) TagStore 

func (r *Repository) TagStore() TagStore

TagStore returns the Tag Store for this repository.

type Rootfs 

type Rootfs struct {
	Header    RootfsHeader
	Directory RootfsDirectory
}

Rootfs describes the rootfs directory+header for an application containier.

func UnmarshalRootfs 

func UnmarshalRootfs(r io.Reader) (rfs Rootfs, err error)

UnmarshalRootfs unbarshals a Rootfs from the binary encoding (littel-endian) read from the given reader.

func (Rootfs) Marshal 

func (rfs Rootfs) Marshal(w io.Writer) error

Marshal marshals this Rootfs to a binary encoding (little-endian) to the given writer.

type RootfsDirectory 

type RootfsDirectory struct {
	Digest         Digest
	Size           uint64
	NumSubObjects  uint32
	SubObjectsSize uint64
}

RootfsDirectory describes the rootfs directory object for an application container.

func (RootfsDirectory) Descriptor 

func (d RootfsDirectory) Descriptor() Descriptor

Descriptor returns a descriptor for the rootfs directory object.

type RootfsHeader 

type RootfsHeader struct {
	Digest Digest
	Size   uint64
}

RootfsHeader describes the rootfs directory header object for an application container.

func (RootfsHeader) Descriptor 

func (h RootfsHeader) Descriptor() Descriptor

Descriptor returns a descriptor for the rootfs header object.

type TagStore 

type TagStore interface {
	Get(tag string) (Descriptor, error)
	Set(tag string, desc Descriptor) error
	List() (tags []string, err error)
	Remove(tag string) error
}

TagStore is the interface for managing mappings of simple strings to a content-addressable filesystem object digest.

func NewTagStore 

func NewTagStore(root string) (TagStore, error)

NewTagStore creates a new tag store using the given root directory.

type TempRef 

type TempRef interface {
	Commit() (Descriptor, error)
	Descriptor() Descriptor
}

TempRef refers to an object which has been downloaded but not yet commited to an object store. This is usually done when an object has been downloaded but we are waiting to fetch its dependencies.

type WriteFlusher 

type WriteFlusher interface {
	Write(p []byte) (nn int, err error)
	Flush() error
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
stemma-fetch
stemma-httpserver
stemma-mount
stemma-push
stemma-show-app
stemma-storedir

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.