s3

package module
v4.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 8, 2024 License: Apache-2.0 Imports: 16 Imported by: 0

README

s3-client

This client is a wrapper around minio-go.

Installation

go get github.com/Clarilab/s3-client/v4

Importing

import "github.com/Clarilab/s3-client/v4"

Features

// Client holds all callable methods.
type Client interface {
	// UploadFile uploads data under a given s3 path.
	UploadFile(ctx context.Context, upload Upload, options ...UploadOption) (*UploadInfo, error)

	// GetFile returns the file from given s3 path.
	GetFile(ctx context.Context, path string, options ...GetOption) (File, error)

	// GetObjectInfo returns an minio.ObjectInfo for the given s3 path.
	GetFileInfo(ctx context.Context, path string) (*FileInfo, error)

	// GetDirectory returns a list of files from given s3 folder.
	GetDirectory(ctx context.Context, path string, options ...GetOption) ([]File, error)

	// GetDirectoryInfos returns a list of file infos for all files from given s3 folder.
	GetDirectoryInfos(ctx context.Context, path string) ([]*FileInfo, error)

	// DownloadFile downloads the requested file to the file system under given localPath.
	DownloadFile(ctx context.Context, path, localPath string, options ...DownloadOption) error

	// DownloadDirectory downloads the requested folder to the file system.
	// The recursive option also downloads all sub folders.
	DownloadDirectory(ctx context.Context, path, localPath string, recursive bool, options ...DownloadOption) error

	// RemoveFile deletes the file under given s3 path.
	RemoveFile(ctx context.Context, path string, options ...RemoveOption) error

	// AddLifeCycleRule adds a lifecycle rule to the given folder.
	AddLifeCycleRule(ctx context.Context, ruleID, folderPath string, daysToExpiry int) error

	// CreateFileLink creates a link with expiration for a file under the given path.
	CreateFileLink(ctx context.Context, path string, expiration time.Duration) (*url.URL, error)

	// Close closes the s3 client.
	Close()

	// IsOnline reports true if the client is online. If the health-check has not been enabled this will always return true.
	IsOnline() bool
}

Integrity Support

Checksums

The Client features integrity support for CRC32C and MD5 checksums. When enabled:

  • the client will generate checksums accordingly when using the UploadFile method. The checksums will then be present in the UploadInfo.
  • using the GetFile method, the FileInfo will contain the checksums accordingly
  • using the GetFileInfo method, the FileInfo will contain the checksums accordingly if they were uploaded using this library version.
Integrity check
  • when using the GetFile method an integrity check can be performed when providing a comparison checksum to the according option.
  • when the checksums doesn't match an ErrChecksumMismatchwill be returned.
  • the integrity check can be performed even when the integrity support is disabled! When successful, the checksums will also be present in the FileInfo

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	// ErrEmptyHost occurs when the host is not specified.
	ErrEmptyHost = errors.New("host not specified")
	// ErrEmptyAccessKey occurs when the access key is not specified.
	ErrEmptyAccessKey = errors.New("access key not specified")
	// ErrEmptyAccessSecret occurs when the access secret is not specified.
	ErrEmptyAccessSecret = errors.New("access secret not specified")
	// ErrEmptyBucketName occurs when the bucket name is not specified.
	ErrEmptyBucketName = errors.New("bucket name not specified")
	// ErrNotFound indicates that the requested file does not exist.
	ErrNotFound = errors.New("file under specified filepath does not exist")
	// ErrChecksumMismatch occurs when the checksum of the downloaded file
	// does not match the expected checksum.
	ErrChecksumMismatch = errors.New("checksum mismatch")
)

Functions

func GenerateCheckSumCRC32C

func GenerateCheckSumCRC32C(data io.Reader) (string, error)

GenerateCheckSumCRC32C returns a CRC32C checksum of the given data.

func GenerateCheckSumMD5

func GenerateCheckSumMD5(data io.Reader) (string, error)

GenerateCheckSumMD5 returns a MD5 checksum of the given data.

Types

type BucketDoesNotExistError

type BucketDoesNotExistError struct {
	// contains filtered or unexported fields
}

BucketDoesNotExistError occurs when the given bucket does not exist.

func (*BucketDoesNotExistError) Error

func (e *BucketDoesNotExistError) Error() string

Error implements the error interface.

type Client

type Client interface {
	// UploadFile uploads data under a given s3 path.
	UploadFile(ctx context.Context, upload *Upload, options ...UploadOption) (*UploadInfo, error)

	// GetFile returns the file from given s3 path.
	GetFile(ctx context.Context, path string, options ...GetOption) (File, error)

	// GetObjectInfo returns an minio.ObjectInfo for the given s3 path.
	GetFileInfo(ctx context.Context, path string) (*FileInfo, error)

	// GetDirectory returns a list of files from given s3 folder.
	GetDirectory(ctx context.Context, path string, options ...GetDirectoryOption) ([]File, error)

	// GetDirectoryInfos returns a list of file infos for all files from given s3 folder.
	GetDirectoryInfos(ctx context.Context, path string) ([]*FileInfo, error)

	// DownloadFile downloads the requested file to the file system under given localPath.
	DownloadFile(ctx context.Context, path, localPath string, options ...DownloadOption) error

	// DownloadDirectory downloads the requested folder to the file system.
	// The recursive option also downloads all sub folders.
	DownloadDirectory(ctx context.Context, path, localPath string, recursive bool, options ...DownloadOption) error

	// RemoveFile deletes the file under given s3 path.
	RemoveFile(ctx context.Context, path string, options ...RemoveOption) error

	// AddLifeCycleRule adds a lifecycle rule to the given folder.
	AddLifeCycleRule(ctx context.Context, ruleID, folderPath string, daysToExpiry int) error

	// CreateFileLink creates a link with expiration for a file under the given path.
	CreateFileLink(ctx context.Context, path string, expiration time.Duration) (*url.URL, error)

	// Close closes the s3 client.
	Close()

	// IsOnline reports true if the client is online. If the health-check has not been enabled this will always return true.
	IsOnline() bool

	// IsHealthy reports true if the client is online. If the health-check has not been enabled this will always return true.
	IsHealthy() bool

	// GetName returns the name of the client.
	GetName() string
}

Client holds all callable methods.

func NewClient

func NewClient(details *ClientDetails, options ...ClientOption) (Client, error)

NewClient instantiates a s3.

type ClientDetails

type ClientDetails struct {
	Host         string
	AccessKey    string
	AccessSecret string
	BucketName   string
	Secure       bool
}

ClientDetails is a struct for all required connection details.

type ClientGetOptions

type ClientGetOptions minio.GetObjectOptions

ClientGetOptions is an alias for minio.GetObjectOptions.

type ClientOption

type ClientOption func(*client) error

ClientOption is an option for the s3 client.

func WithCRC32CIntegritySupport

func WithCRC32CIntegritySupport(enabled bool) ClientOption

WithCRC32CIntegritySupport enables or disables CRC32C integrity check support. By default it's enabled.

func WithHealthCheck

func WithHealthCheck(interval time.Duration) ClientOption

WithHealthCheck enables the health check for the s3 client.

func WithMD5IntegritySupport

func WithMD5IntegritySupport(enabled bool) ClientOption

WithMD5IntegritySupport enables or disables MD5 integrity check support. By default it's disabled.

type ClientRemoveOptions

type ClientRemoveOptions minio.RemoveObjectOptions

ClientRemoveOptions is an alias for minio.RemoveObjectOptions.

type ClientUploadOptions

type ClientUploadOptions minio.PutObjectOptions

ClientUploadOptions is an alias for minio.PutObjectOptions.

type DownloadOption

type DownloadOption func(*downloadOptions)

DownloadOption is an option for downloading a file.

func WithClientDownloadOptions

func WithClientDownloadOptions(options ClientGetOptions) DownloadOption

WithClientGetOptions sets client options for the get request.

type DownloadingFilesFailedError

type DownloadingFilesFailedError struct {
	// contains filtered or unexported fields
}

DownloadingFilesFailedError occurs when downloading files from s3 failed.

func (*DownloadingFilesFailedError) Error

DownloadingFilesFailedError implements the error interface.

type File

type File interface {
	io.ReadCloser
	// Info returns the file information.
	Info() *FileInfo
	// Bytes reads the entire file and returns its content as a byte slice.
	//
	// Note: The file is closed and must not be read after using this function!
	Bytes() ([]byte, error)
}

File is a file downloaded from s3.

type FileInfo

type FileInfo struct {
	Name         string
	Path         string
	Size         int64
	ContentType  string
	MetaData     map[string]string
	ModifiedDate time.Time
	Integrity
}

FileInfo contains information about a file.

type GetDirectoryOption

type GetDirectoryOption func(*getDirectoryOptions)

GetOption is an option for getting a file.

func WithGetDirectoryClientGetOptions

func WithGetDirectoryClientGetOptions(options ClientGetOptions) GetDirectoryOption

WithGetDirectoryClientGetOptions sets client options for the get directory request.

type GetOption

type GetOption func(*getOptions)

GetOption is an option for getting a file.

func WithClientGetOptions

func WithClientGetOptions(options ClientGetOptions) GetOption

WithClientGetOptions sets client options for the get request.

func WithIntegrityCheckCRC32C

func WithIntegrityCheckCRC32C(checksum string) GetOption

WithIntegrityCheckCRC32C checks if the CRC32C checksum of the downloaded file matches the given checksum.

func WithIntegrityCheckMD5

func WithIntegrityCheckMD5(checksum string) GetOption

WithIntegrityCheckMD5 checks if the MD5 checksum of the downloaded file matches the given checksum.

type Integrity

type Integrity struct {
	ChecksumCRC32C string // When CRC32C integrity support is disabled, ChecksumCRC32C will be empty if no explicit integrity check was requested via option
	ChecksumMD5    string // When MD5 integrity support is disabled, ChecksumMD5 will be empty if no explicit integrity check was requested via option
}

Integrity contains checksums for file integrity.

type RemoveOption

type RemoveOption func(*removeOptions)

UploadOption is an option for uploading a file.

func WithClientRemoveOptions

func WithClientRemoveOptions(options ClientRemoveOptions) RemoveOption

WithClientUploadOptions sets client options for the upload request.

type Upload

type Upload struct {
	io.ReadSeeker
	Path        string
	ContentType string
	MetaData    map[string]string
	Size        *int64
}

Upload represents a file that can be uploaded to the s3.

func NewUpload

func NewUpload(data io.ReadSeeker, size *int64, path, contentType string, metaData map[string]string) *Upload

NewUpload creates a new Upload instance.

type UploadInfo

type UploadInfo struct {
	Size int64
	Integrity
}

UploadInfo contains information about the uploaded file.

type UploadOption

type UploadOption func(*uploadOptions)

UploadOption is an option for uploading a file.

func WithClientUploadOptions

func WithClientUploadOptions(options ClientUploadOptions) UploadOption

WithClientUploadOptions sets client options for the upload request.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL