archiver

package
v0.18.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 27, 2025 License: BSD-2-Clause Imports: 21 Imported by: 0

Documentation

Overview

Package archiver contains the code which reads files, splits them into chunks and saves the data to the repository.

Index

Constants

View Source
const (
	ChangeIgnoreCtime = 1 << iota
	ChangeIgnoreInode
)

Flags for the ChangeIgnoreFlags bitfield.

Variables

This section is empty.

Functions

func TestCreateFiles

func TestCreateFiles(t testing.TB, target string, dir TestDir)

TestCreateFiles creates a directory structure described by dir at target, which must already exist. On Windows, symlinks aren't created.

func TestEnsureFileContent

func TestEnsureFileContent(ctx context.Context, t testing.TB, repo restic.BlobLoader, filename string, node *restic.Node, file TestFile)

TestEnsureFileContent checks if the file in the repo is the same as file.

func TestEnsureFiles

func TestEnsureFiles(t testing.TB, target string, dir TestDir)

TestEnsureFiles tests if the directory structure at target is the same as described in dir.

func TestEnsureSnapshot

func TestEnsureSnapshot(t testing.TB, repo restic.Repository, snapshotID restic.ID, dir TestDir)

TestEnsureSnapshot tests if the snapshot in the repo has exactly the same structure as dir. On Windows, Symlinks are ignored.

func TestEnsureTree

func TestEnsureTree(ctx context.Context, t testing.TB, prefix string, repo restic.BlobLoader, treeID restic.ID, dir TestDir)

TestEnsureTree checks that the tree ID in the repo matches dir. On Windows, Symlinks are ignored.

func TestSnapshot

func TestSnapshot(t testing.TB, repo restic.Repository, path string, parent *restic.ID) *restic.Snapshot

TestSnapshot creates a new snapshot of path.

func TestWalkFiles

func TestWalkFiles(t testing.TB, target string, dir TestDir, fn TestWalkFunc)

TestWalkFiles runs fn for each file/directory in dir, the filename will be constructed with target as the prefix. Symlinks on Windows are ignored.

Types

type Archiver

type Archiver struct {
	Repo         archiverRepo
	SelectByName SelectByNameFunc
	Select       SelectFunc
	FS           fs.FS
	Options      Options

	// Error is called for all errors that occur during backup.
	Error ErrorFunc

	// CompleteItem is called for all files and dirs once they have been
	// processed successfully. The parameter item contains the path as it will
	// be in the snapshot after saving. s contains some statistics about this
	// particular file/dir.
	//
	// Once reading a file has completed successfully (but not saving it yet),
	// CompleteItem will be called with current == nil.
	//
	// CompleteItem may be called asynchronously from several different
	// goroutines!
	CompleteItem func(item string, previous, current *restic.Node, s ItemStats, d time.Duration)

	// StartFile is called when a file is being processed by a worker.
	StartFile func(filename string)

	// CompleteBlob is called for all saved blobs for files.
	CompleteBlob func(bytes uint64)

	// WithAtime configures if the access time for files and directories should
	// be saved. Enabling it may result in much metadata, so it's off by
	// default.
	WithAtime bool

	// Flags controlling change detection. See doc/040_backup.rst for details.
	ChangeIgnoreFlags uint
	// contains filtered or unexported fields
}

Archiver saves a directory structure to the repo.

An Archiver has a number of worker goroutines handling saving the different data structures to the repository, the details are implemented by the fileSaver, blobSaver, and treeSaver types.

The main goroutine (the one calling Snapshot()) traverses the directory tree and delegates all work to these worker pools. They return a futureNode which can be resolved later, by calling Wait() on it.

func New

func New(repo archiverRepo, filesystem fs.FS, opts Options) *Archiver

New initializes a new archiver.

func (*Archiver) Snapshot

func (arch *Archiver) Snapshot(ctx context.Context, targets []string, opts SnapshotOptions) (*restic.Snapshot, restic.ID, *Summary, error)

Snapshot saves several targets and returns a snapshot.

type ChangeStats added in v0.17.0

type ChangeStats struct {
	New       uint
	Changed   uint
	Unchanged uint
}

type ErrorFunc

type ErrorFunc func(file string, err error) error

ErrorFunc is called when an error during archiving occurs. When nil is returned, the archiver continues, otherwise it aborts and passes the error up the call stack.

type ItemStats

type ItemStats struct {
	DataBlobs      int    // number of new data blobs added for this item
	DataSize       uint64 // sum of the sizes of all new data blobs
	DataSizeInRepo uint64 // sum of the bytes added to the repo (including compression and crypto overhead)
	TreeBlobs      int    // number of new tree blobs added for this item
	TreeSize       uint64 // sum of the sizes of all new tree blobs
	TreeSizeInRepo uint64 // sum of the bytes added to the repo (including compression and crypto overhead)
}

ItemStats collects some statistics about a particular file or directory.

func (*ItemStats) Add

func (s *ItemStats) Add(other ItemStats)

Add adds other to the current ItemStats.

type Options

type Options struct {
	// ReadConcurrency sets how many files are read in concurrently. If
	// it's set to zero, at most two files are read in concurrently (which
	// turned out to be a good default for most situations).
	ReadConcurrency uint

	// SaveBlobConcurrency sets how many blobs are hashed and saved
	// concurrently. If it's set to zero, the default is the number of CPUs
	// available in the system.
	SaveBlobConcurrency uint

	// SaveTreeConcurrency sets how many trees are marshalled and saved to the
	// repo concurrently.
	SaveTreeConcurrency uint
}

Options is used to configure the archiver.

func (Options) ApplyDefaults

func (o Options) ApplyDefaults() Options

ApplyDefaults returns a copy of o with the default options set for all unset fields.

type RejectByNameFunc added in v0.18.0

type RejectByNameFunc func(path string) bool

RejectByNameFunc is a function that takes a filename of a file that would be included in the backup. The function returns true if it should be excluded (rejected) from the backup.

type RejectFunc added in v0.18.0

type RejectFunc func(path string, fi *fs.ExtendedFileInfo, fs fs.FS) bool

RejectFunc is a function that takes a filename and os.FileInfo of a file that would be included in the backup. The function returns true if it should be excluded (rejected) from the backup.

func RejectByDevice added in v0.18.0

func RejectByDevice(samples []string, filesystem fs.FS) (RejectFunc, error)

RejectByDevice returns a RejectFunc that rejects files which are on a different file systems than the files/dirs in samples.

func RejectBySize added in v0.18.0

func RejectBySize(maxSize int64) (RejectFunc, error)

func RejectCloudFiles added in v0.18.0

func RejectCloudFiles(warnf func(msg string, args ...interface{})) (RejectFunc, error)

RejectCloudFiles returns a func which rejects files which are online-only cloud files

func RejectIfPresent added in v0.18.0

func RejectIfPresent(excludeFileSpec string, warnf func(msg string, args ...interface{})) (RejectFunc, error)

RejectIfPresent returns a RejectByNameFunc which itself returns whether a path should be excluded. The RejectByNameFunc considers a file to be excluded when it resides in a directory with an exclusion file, that is specified by excludeFileSpec in the form "filename[:content]". The returned error is non-nil if the filename component of excludeFileSpec is empty. If rc is non-nil, it is going to be used in the RejectByNameFunc to expedite the evaluation of a directory based on previous visits.

type ScanStats

type ScanStats struct {
	Files, Dirs, Others uint
	Bytes               uint64
}

ScanStats collect statistics.

type Scanner

type Scanner struct {
	FS           fs.FS
	SelectByName SelectByNameFunc
	Select       SelectFunc
	Error        ErrorFunc
	Result       func(item string, s ScanStats)
}

Scanner traverses the targets and calls the function Result with cumulated stats concerning the files and folders found. Select is used to decide which items should be included. Error is called when an error occurs.

func NewScanner

func NewScanner(filesystem fs.FS) *Scanner

NewScanner initializes a new Scanner.

func (*Scanner) Scan

func (s *Scanner) Scan(ctx context.Context, targets []string) error

Scan traverses the targets. The function Result is called for each new item found, the complete result is also returned by Scan.

type SelectByNameFunc

type SelectByNameFunc func(item string) bool

SelectByNameFunc returns true for all items that should be included (files and dirs). If false is returned, files are ignored and dirs are not even walked.

func CombineRejectByNames added in v0.18.0

func CombineRejectByNames(funcs []RejectByNameFunc) SelectByNameFunc

type SelectFunc

type SelectFunc func(item string, fi *fs.ExtendedFileInfo, fs fs.FS) bool

SelectFunc returns true for all items that should be included (files and dirs). If false is returned, files are ignored and dirs are not even walked.

func CombineRejects added in v0.18.0

func CombineRejects(funcs []RejectFunc) SelectFunc

type SnapshotOptions

type SnapshotOptions struct {
	Tags           restic.TagList
	Hostname       string
	Excludes       []string
	BackupStart    time.Time
	Time           time.Time
	ParentSnapshot *restic.Snapshot
	ProgramVersion string
	// SkipIfUnchanged omits the snapshot creation if it is identical to the parent snapshot.
	SkipIfUnchanged bool
}

SnapshotOptions collect attributes for a new snapshot.

type Summary added in v0.17.0

type Summary struct {
	BackupStart    time.Time
	BackupEnd      time.Time
	Files, Dirs    ChangeStats
	ProcessedBytes uint64
	ItemStats
}

type TestDir

type TestDir map[string]interface{}

TestDir describes a directory structure to create for a test.

func (TestDir) String

func (d TestDir) String() string

type TestFile

type TestFile struct {
	Content string
}

TestFile describes a file created for a test.

func (TestFile) String

func (f TestFile) String() string
type TestHardlink struct {
	Target string
}

TestHardlink describes a hardlink created for a test.

func (TestHardlink) String added in v0.17.0

func (s TestHardlink) String() string
type TestSymlink struct {
	Target string
}

TestSymlink describes a symlink created for a test.

func (TestSymlink) String

func (s TestSymlink) String() string

type TestWalkFunc

type TestWalkFunc func(path string, item interface{}) error

TestWalkFunc is used by TestWalkFiles to traverse the dir. When an error is returned, traversal stops and the surrounding test is marked as failed.

type ToNoder added in v0.18.0

type ToNoder interface {
	ToNode(ignoreXattrListError bool) (*restic.Node, error)
}

ToNoder returns a restic.Node for a File.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL