storage

package
v0.5.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 8, 2025 License: Apache-2.0 Imports: 7 Imported by: 3

Documentation

Index

Constants

View Source
const DefaultPollWarmDuration = time.Minute

DefaultPollWarmDuration is the interval between requests to cloud providers, to get file status during files restore.

Variables

View Source
var (
	ErrEmptyStorage   = errors.New("empty storage")
	ErrArchivedObject = errors.New("archived object")
)

ErrEmptyStorage describes the empty storage error for the restore operation.

Functions

func CleanPath

func CleanPath(path string, isS3 bool) string

CleanPath sanitizes the input path string based on the storage type (S3 or non-S3). For S3 storage, it removes the root path "/" as S3 uses empty string for root. For all storage types, it ensures proper trailing slash format except for empty or root paths. Returns the cleaned path string.

func ErrToChan added in v0.5.0

func ErrToChan(ctx context.Context, ch chan<- error, err error)

ErrToChan checks context before sending an error to errors chan. If context is already canceled and the reader must be stopped, no need to send error to errors chan.

func IsDirectory

func IsDirectory(prefix, fileName string) bool

IsDirectory determines if a given file name represents a directory within the specified prefix. It considers three cases:

  1. File name ends with "/" (definite directory)
  2. File name is within a prefix and contains "/" after the prefix
  3. File name contains "/" (general case)

Returns true if the file name represents a directory, false otherwise.

func PreSort

func PreSort(ctx context.Context, r reader, path string) error

PreSort performs pre-processing of backup files by sorting them before reading. It retrieves a list of objects from the specified path, sorts them according to the backup file naming conventions, and configures the reader to stream the sorted list. Returns an error if listing objects fails or if the sorting operation fails.

Types

type Opt

type Opt func(*Options)

func WithAccessTier

func WithAccessTier(tier string) Opt

WithAccessTier sets the access tier for archived files. For readers, archived files are restored to the specified tier.

func WithChunkSize

func WithChunkSize(bytes int) Opt

WithChunkSize sets the chunk size for uploading files. Is used only for Writer.

func WithDir

func WithDir(path string) Opt

WithDir adds the directory to reading/writing files from/to.

func WithDirList

func WithDirList(pathList []string) Opt

WithDirList adds the directory list to read files from. Is used only for Reader.

func WithFile

func WithFile(path string) Opt

WithFile adds the file path to reading/writing from/to.

func WithFileList

func WithFileList(pathList []string) Opt

WithFileList adds the file list to read from. Is used only for Reader.

func WithLogger

func WithLogger(logger *slog.Logger) Opt

WithLogger sets the logger.

func WithNestedDir

func WithNestedDir() Opt

WithNestedDir adds WithNestedDir = true parameter. That means that we won't skip nested folders.

func WithRemoveFiles

func WithRemoveFiles() Opt

WithRemoveFiles adds remove files flag, so all files will be removed from backup folder before backup. Is used only for Writer.

func WithSkipDirCheck

func WithSkipDirCheck() Opt

WithSkipDirCheck adds skip dir check flags. Which means that backup directory won't be checked for emptiness.

func WithSorting

func WithSorting() Opt

WithSorting adds a sorting flag. Which means that files will be read from directory in the sorted order. Is used only for Reader.

func WithStartAfter

func WithStartAfter(v string) Opt

WithStartAfter adds start after parameter to list request. Is used only for Reader.

func WithStorageClass

func WithStorageClass(class string) Opt

WithStorageClass sets the storage class. For writers, files are saved to the specified class (in case of Azure it will be tier).

func WithUploadConcurrency

func WithUploadConcurrency(v int) Opt

WithUploadConcurrency defines max number of concurrent uploads to be performed to upload the file. Is used only for Azure Writer.

func WithValidator

func WithValidator(v validator) Opt

WithValidator adds the Validator to Reader, so files will be validated before reading. Is used only for Reader.

func WithWarmPollDuration

func WithWarmPollDuration(duration time.Duration) Opt

WithWarmPollDuration sets the restore status polling interval.

type Options

type Options struct {
	// PathList contains a list of files or directories.
	PathList []string
	// IsDir flag determines whether the path is a directory.
	IsDir bool
	// IsRemovingFiles flag specifies whether to clear the backup folder.
	IsRemovingFiles bool
	// Validator contains files Validator that is applied to files if IsDir is true.
	Validator validator
	// WithNestedDir determines whether the read/write operations should treat objects
	// identified as directories like regular files.
	// When we stream files or delete files in folder, we skip directories. Setting
	// WithNestedDir to true disables this directory skipping behavior.
	// Default: false
	WithNestedDir bool
	// StartAfter is an artificial parameter. Used to skip objects in the storage.
	// The Result will not include an object specified in startAfter.
	// If it is set, then we compare the names received from the storage lexicographically,
	// and if the name is less than the specified parameter, we skip this object.
	StartAfter string
	// SkipDirCheck, if true, the backup directory won't be checked.
	SkipDirCheck bool
	// SortFiles determines whether we need to sort files before reading.
	SortFiles bool

	// UploadConcurrency defines the max number of concurrent uploads to be performed to
	// upload the file. Each concurrent upload will create a buffer of size BlockSize.
	UploadConcurrency int

	// StorageClass specifies the type of storage class.
	// Supported values depend on the cloud provider being used.
	StorageClass string

	// The access tier that will be applied to back up files.
	AccessTier string

	// Logger contains logger.
	Logger *slog.Logger

	// How often restore status will be polled from cloud provider.
	PollWarmDuration time.Duration

	// Size of chunk to upload.
	ChunkSize int
}

Directories

Path Synopsis
aws
s3
azure
gcp

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL