repository

package
v0.17.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 26, 2024 License: BSD-2-Clause Imports: 37 Imported by: 0

Documentation

Overview

Package repository implements a restic repository on top of a backend. In the following the abstractions used for this package are listed. More information can be found in the restic design document.

File

A file is a named handle for some data saved in the backend. For the local backend, this corresponds to actual files saved to disk. Usually, the SHA256 hash of the content is used for a file's name (hexadecimal, in lower-case ASCII characters). An exception is the file `config`. Most files are encrypted before being saved in a backend. This means that the name is the hash of the ciphertext.

Blob

A blob is a number of bytes that has a type (data or tree). Blobs are identified by an ID, which is the SHA256 hash of the blobs' contents. One or more blobs are bundled together in a Pack and then saved to the backend. Blobs are always encrypted before being bundled in a Pack.

Pack

A Pack is a File in the backend that contains one or more (encrypted) blobs, followed by a header at the end of the Pack. The header is encrypted and contains the ID, type, length and offset for each blob contained in the Pack.

Index

Constants

View Source 
const (
	// KDFTimeout specifies the maximum runtime for the KDF.
	KDFTimeout = 500 * time.Millisecond

	// KDFMemory limits the memory the KDF is allowed to use.
	KDFMemory = 60
)
View Source 
const DefaultPackSize = 16 * 1024 * 1024
View Source 
const MaxPackSize = 128 * 1024 * 1024
View Source 
const MinPackSize = 4 * 1024 * 1024

Variables

View Source 
var (
	// ErrNoKeyFound is returned when no key for the repository could be decrypted.
	ErrNoKeyFound = errors.New("wrong password or no key found")

	// ErrMaxKeysReached is returned when the maximum number of keys was checked and no key could be found.
	ErrMaxKeysReached = errors.New("maximum number of keys reached")
)
View Source 
var ErrIndexIncomplete = errors.Fatal("index is not complete")
View Source 
var ErrPacksMissing = errors.Fatal("packs from index missing in repo")
View Source 
var ErrSizeNotMatching = errors.Fatal("pack size does not match calculated size from index")

Functions

func AsS3Backend added in v0.17.0 

func AsS3Backend(repo *Repository) *s3.Backend

AsS3Backend extracts the S3 backend from a repository TODO remove me once restic 0.17 was released

func BenchmarkAllVersions 

func BenchmarkAllVersions(b *testing.B, bench VersionedBenchmark)

func CheckPack added in v0.17.0 

func CheckPack(ctx context.Context, r *Repository, id restic.ID, blobs []restic.Blob, size int64, bufRd *bufio.Reader, dec *zstd.Decoder) error

CheckPack reads a pack and checks the integrity of all blobs.

func RemoveKey added in v0.17.0 

func RemoveKey(ctx context.Context, repo *Repository, id restic.ID) error

func Repack 

func Repack(ctx context.Context, repo restic.Repository, dstRepo restic.Repository, packs restic.IDSet, keepBlobs repackBlobSet, p *progress.Counter) (obsoletePacks restic.IDSet, err error)

Repack takes a list of packs together with a list of blobs contained in these packs. Each pack is loaded and the blobs listed in keepBlobs is saved into a new pack. Returned is the list of obsolete packs which can then be removed.

The map keepBlobs is modified by Repack, it is used to keep track of which blobs have been processed.

func RepairIndex added in v0.17.0 

func RepairIndex(ctx context.Context, repo *Repository, opts RepairIndexOptions, printer progress.Printer) error

func RepairPacks added in v0.17.0 

func RepairPacks(ctx context.Context, repo *Repository, ids restic.IDSet, printer progress.Printer) error

func TestAllVersions 

func TestAllVersions(t *testing.T, test VersionedTest)

func TestBackend 

func TestBackend(_ testing.TB) backend.Backend

TestBackend returns a fully configured in-memory backend.

func TestUseLowSecurityKDFParameters 

func TestUseLowSecurityKDFParameters(t logger)

TestUseLowSecurityKDFParameters configures low-security KDF parameters for testing.

func UpgradeRepo added in v0.17.0 

func UpgradeRepo(ctx context.Context, repo *Repository) error

func ZeroChunk 

func ZeroChunk() restic.ID

ZeroChunk computes and returns (cached) the ID of an all-zero chunk with size chunker.MinSize

Types

type CompressionMode 

type CompressionMode uint

CompressionMode configures if data should be compressed. 

const (
	CompressionAuto    CompressionMode = 0
	CompressionOff     CompressionMode = 1
	CompressionMax     CompressionMode = 2
	CompressionInvalid CompressionMode = 3
)

Constants for the different compression levels.

func (*CompressionMode) Set 

func (c *CompressionMode) Set(s string) error

Set implements the method needed for pflag command flag parsing.

func (*CompressionMode) String 

func (c *CompressionMode) String() string

func (*CompressionMode) Type 

func (c *CompressionMode) Type() string

type ErrPackData added in v0.17.0 

type ErrPackData struct {
	PackID restic.ID
	// contains filtered or unexported fields
}

ErrPackData is returned if errors are discovered while verifying a packfile

func (*ErrPackData) Error added in v0.17.0 

func (e *ErrPackData) Error() string

type Key 

type Key struct {
	Created  time.Time `json:"created"`
	Username string    `json:"username"`
	Hostname string    `json:"hostname"`

	KDF  string `json:"kdf"`
	N    int    `json:"N"`
	R    int    `json:"r"`
	P    int    `json:"p"`
	Salt []byte `json:"salt"`
	Data []byte `json:"data"`
	// contains filtered or unexported fields
}

Key represents an encrypted master key for a repository.

func AddKey 

func AddKey(ctx context.Context, s *Repository, password, username, hostname string, template *crypto.Key) (*Key, error)

AddKey adds a new key to an already existing repository.

func LoadKey 

func LoadKey(ctx context.Context, s *Repository, id restic.ID) (k *Key, err error)

LoadKey loads a key from the backend.

func OpenKey 

func OpenKey(ctx context.Context, s *Repository, id restic.ID, password string) (*Key, error)

OpenKey tries do decrypt the key specified by name with the given password.

func SearchKey 

func SearchKey(ctx context.Context, s *Repository, password string, maxKeys int, keyHint string) (k *Key, err error)

SearchKey tries to decrypt at most maxKeys keys in the backend with the given password. If none could be found, ErrNoKeyFound is returned. When maxKeys is reached, ErrMaxKeysReached is returned. When setting maxKeys to zero, all keys in the repo are checked.

func (Key) ID 

func (k Key) ID() restic.ID

ID returns an identifier for the key.

func (*Key) String 

func (k *Key) String() string

func (*Key) Valid 

func (k *Key) Valid() bool

Valid tests whether the mac and encryption keys are valid (i.e. not zero)

type Options 

type Options struct {
	Compression   CompressionMode
	PackSize      uint
	NoExtraVerify bool
}

type PruneOptions added in v0.17.0 

type PruneOptions struct {
	DryRun         bool
	UnsafeRecovery bool

	MaxUnusedBytes func(used uint64) (unused uint64) // calculates the number of unused bytes after repacking, according to MaxUnused
	MaxRepackBytes uint64

	RepackCacheableOnly bool
	RepackSmall         bool
	RepackUncompressed  bool
}

PruneOptions collects all options for the cleanup command.

type PrunePlan added in v0.17.0 

type PrunePlan struct {
	// contains filtered or unexported fields
}

func PlanPrune added in v0.17.0 

func PlanPrune(ctx context.Context, opts PruneOptions, repo *Repository, getUsedBlobs func(ctx context.Context, repo restic.Repository, usedBlobs restic.FindBlobSet) error, printer progress.Printer) (*PrunePlan, error)

PlanPrune selects which files to rewrite and which to delete and which blobs to keep. Also some summary statistics are returned.

func (*PrunePlan) Execute added in v0.17.0 

func (plan *PrunePlan) Execute(ctx context.Context, printer progress.Printer) error

Execute does the actual pruning: - remove unreferenced packs first - repack given pack files while keeping the given blobs - rebuild the index while ignoring all files that will be deleted - delete the files plan.removePacks and plan.ignorePacks are modified in this function.

func (*PrunePlan) Stats added in v0.17.0 

func (plan *PrunePlan) Stats() PruneStats

type PruneStats added in v0.17.0 

type PruneStats struct {
	Blobs struct {
		Used      uint
		Duplicate uint
		Unused    uint
		Remove    uint
		Repack    uint
		Repackrm  uint
	}
	Size struct {
		Used         uint64
		Duplicate    uint64
		Unused       uint64
		Remove       uint64
		Repack       uint64
		Repackrm     uint64
		Unref        uint64
		Uncompressed uint64
	}
	Packs struct {
		Used       uint
		Unused     uint
		PartlyUsed uint
		Unref      uint
		Keep       uint
		Repack     uint
		Remove     uint
	}
}

type RepairIndexOptions added in v0.17.0 

type RepairIndexOptions struct {
	ReadAllPacks bool
}

type Repository 

type Repository struct {
	Cache *cache.Cache
	// contains filtered or unexported fields
}

Repository is used to access a repository in a backend.

func New 

func New(be backend.Backend, opts Options) (*Repository, error)

New returns a new repository with backend be.

func TestFromFixture added in v0.17.0 

func TestFromFixture(t testing.TB, repoFixture string) (*Repository, backend.Backend, func())

func TestOpenBackend added in v0.17.0 

func TestOpenBackend(t testing.TB, be backend.Backend) *Repository

func TestOpenLocal 

func TestOpenLocal(t testing.TB, dir string) (*Repository, backend.Backend)

TestOpenLocal opens a local repository.

func TestRepository 

func TestRepository(t testing.TB) *Repository

TestRepository returns a repository initialized with a test password on an in-memory backend. When the environment variable RESTIC_TEST_REPO is set to a non-existing directory, a local backend is created there and this is used instead. The directory is not removed, but left there for inspection.

func TestRepositoryWithBackend 

func TestRepositoryWithBackend(t testing.TB, be backend.Backend, version uint, opts Options) (*Repository, backend.Backend)

TestRepositoryWithBackend returns a repository initialized with a test password. If be is nil, an in-memory backend is used. A constant polynomial is used for the chunker and low-security test parameters.

func TestRepositoryWithVersion 

func TestRepositoryWithVersion(t testing.TB, version uint) (*Repository, backend.Backend)

func (*Repository) Close 

func (r *Repository) Close() error

Close closes the repository by closing the backend.

func (*Repository) Config 

func (r *Repository) Config() restic.Config

Config returns the repository configuration.

func (*Repository) Connections 

func (r *Repository) Connections() uint

func (*Repository) Delete 

func (r *Repository) Delete(ctx context.Context) error

Delete calls backend.Delete() if implemented, and returns an error otherwise.

func (*Repository) Flush 

func (r *Repository) Flush(ctx context.Context) error

Flush saves all remaining packs and the index

func (*Repository) Init 

func (r *Repository) Init(ctx context.Context, version uint, password string, chunkerPolynomial *chunker.Pol) error

Init creates a new master key with the supplied password, initializes and saves the repository config.

func (*Repository) Key 

func (r *Repository) Key() *crypto.Key

Key returns the current master key.

func (*Repository) KeyID 

func (r *Repository) KeyID() restic.ID

KeyID returns the id of the current key in the backend.

func (*Repository) List 

func (r *Repository) List(ctx context.Context, t restic.FileType, fn func(restic.ID, int64) error) error

List runs fn for all files of type t in the repo.

func (*Repository) ListBlobs added in v0.17.0 

func (r *Repository) ListBlobs(ctx context.Context, fn func(restic.PackedBlob)) error

ListBlobs runs fn on all blobs known to the index. When the context is cancelled, the index iteration returns immediately with ctx.Err(). This blocks any modification of the index.

func (*Repository) ListPack 

func (r *Repository) ListPack(ctx context.Context, id restic.ID, size int64) ([]restic.Blob, uint32, error)

ListPack returns the list of blobs saved in the pack id and the length of the pack header.

func (*Repository) ListPacksFromIndex added in v0.17.0 

func (r *Repository) ListPacksFromIndex(ctx context.Context, packs restic.IDSet) <-chan restic.PackBlobs

func (*Repository) LoadBlob 

func (r *Repository) LoadBlob(ctx context.Context, t restic.BlobType, id restic.ID, buf []byte) ([]byte, error)

LoadBlob loads a blob of type t from the repository. It may use all of buf[:cap(buf)] as scratch space.

func (*Repository) LoadBlobsFromPack added in v0.17.0 

func (r *Repository) LoadBlobsFromPack(ctx context.Context, packID restic.ID, blobs []restic.Blob, handleBlobFn func(blob restic.BlobHandle, buf []byte, err error) error) error

LoadBlobsFromPack loads the listed blobs from the specified pack file. The plaintext blob is passed to the handleBlobFn callback or an error if decryption failed or the blob hash does not match. handleBlobFn is called at most once for each blob. If the callback returns an error, then LoadBlobsFromPack will abort and not retry it. The buf passed to the callback is only valid within this specific call. The callback must not keep a reference to buf.

func (*Repository) LoadIndex 

func (r *Repository) LoadIndex(ctx context.Context, p *progress.Counter) error

LoadIndex loads all index files from the backend in parallel and stores them

func (*Repository) LoadRaw added in v0.17.0 

func (r *Repository) LoadRaw(ctx context.Context, t restic.FileType, id restic.ID) (buf []byte, err error)

LoadRaw reads all data stored in the backend for the file with id and filetype t. If the backend returns data that does not match the id, then the buffer is returned along with an error that is a restic.ErrInvalidData error.

func (*Repository) LoadUnpacked 

func (r *Repository) LoadUnpacked(ctx context.Context, t restic.FileType, id restic.ID) ([]byte, error)

LoadUnpacked loads and decrypts the file with the given type and ID.

func (*Repository) LookupBlob added in v0.17.0 

func (r *Repository) LookupBlob(tpe restic.BlobType, id restic.ID) []restic.PackedBlob

func (*Repository) LookupBlobSize 

func (r *Repository) LookupBlobSize(tpe restic.BlobType, id restic.ID) (uint, bool)

LookupBlobSize returns the size of blob id.

func (*Repository) RemoveUnpacked added in v0.17.0 

func (r *Repository) RemoveUnpacked(ctx context.Context, t restic.FileType, id restic.ID) error

func (*Repository) SaveBlob 

func (r *Repository) SaveBlob(ctx context.Context, t restic.BlobType, buf []byte, id restic.ID, storeDuplicate bool) (newID restic.ID, known bool, size int, err error)

SaveBlob saves a blob of type t into the repository. It takes care that no duplicates are saved; this can be overwritten by setting storeDuplicate to true. If id is the null id, it will be computed and returned. Also returns if the blob was already known before. If the blob was not known before, it returns the number of bytes the blob occupies in the repo (compressed or not, including encryption overhead).

func (*Repository) SaveUnpacked 

func (r *Repository) SaveUnpacked(ctx context.Context, t restic.FileType, buf []byte) (id restic.ID, err error)

SaveUnpacked encrypts data and stores it in the backend. Returned is the storage hash.

func (*Repository) SearchKey 

func (r *Repository) SearchKey(ctx context.Context, password string, maxKeys int, keyHint string) error

SearchKey finds a key with the supplied password, afterwards the config is read and parsed. It tries at most maxKeys key files in the repo.

func (*Repository) SetDryRun 

func (r *Repository) SetDryRun()

SetDryRun sets the repo backend into dry-run mode.

func (*Repository) SetIndex 

func (r *Repository) SetIndex(i restic.MasterIndex) error

SetIndex instructs the repository to use the given index.

func (*Repository) StartPackUploader 

func (r *Repository) StartPackUploader(ctx context.Context, wg *errgroup.Group)

func (*Repository) UseCache 

func (r *Repository) UseCache(c *cache.Cache)

UseCache replaces the backend with the wrapped cache.

type Unlocker added in v0.17.0 

type Unlocker struct {
	// contains filtered or unexported fields
}

func Lock added in v0.17.0 

func Lock(ctx context.Context, repo *Repository, exclusive bool, retryLock time.Duration, printRetry func(msg string), logger func(format string, args ...interface{})) (*Unlocker, context.Context, error)

func (*Unlocker) Unlock added in v0.17.0 

func (l *Unlocker) Unlock()

type VersionedBenchmark 

type VersionedBenchmark func(b *testing.B, version uint)

type VersionedTest 

type VersionedTest func(t *testing.T, version uint)

Source Files

View all Source files

Directories

Path Synopsis
Package pack provides functions for combining and parsing pack files.
 Package pack provides functions for combining and parsing pack files.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.