objstorageprovider

package
v2.0.7 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 22, 2025 License: BSD-3-Clause Imports: 23 Imported by: 1

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewFileWritable

func NewFileWritable(file vfs.File) objstorage.Writable

NewFileWritable returns a Writable that uses a file as underlying storage.

func NewRemoteReadable

func NewRemoteReadable(objReader remote.ObjectReader, size int64) objstorage.Readable

NewRemoteReadable creates an objstorage.Readable out of a remote.ObjectReader.

func NewRemoteWritable

func NewRemoteWritable(obj io.WriteCloser) objstorage.Writable

NewRemoteWritable creates an objstorage.Writable out of an io.WriteCloser.

func Open

func Open(settings Settings) (objstorage.Provider, error)

Open creates the provider.

func TestingCheckMaxReadahead

func TestingCheckMaxReadahead(rh objstorage.ReadHandle) bool

TestingCheckMaxReadahead returns true if the ReadHandle has switched to OS-level read-ahead.

func UsePreallocatedReadHandle

func UsePreallocatedReadHandle(
	readable objstorage.Readable,
	readBeforeSize objstorage.ReadBeforeSize,
	rh *PreallocatedReadHandle,
) objstorage.ReadHandle

UsePreallocatedReadHandle is equivalent to calling readable.NewReadHandle() but uses the existing storage of a PreallocatedReadHandle when possible (currently this happens if we are reading from a local file). The returned handle still needs to be closed.

Types

type PreallocatedReadHandle

type PreallocatedReadHandle struct {
	// contains filtered or unexported fields
}

PreallocatedReadHandle is used to avoid an allocation in NewReadHandle; see UsePreallocatedReadHandle.

func (*PreallocatedReadHandle) Close

func (rh *PreallocatedReadHandle) Close() error

Close is part of the objstorage.ReadHandle interface.

func (*PreallocatedReadHandle) ReadAt

func (rh *PreallocatedReadHandle) ReadAt(_ context.Context, p []byte, offset int64) error

ReadAt is part of the objstorage.ReadHandle interface.

func (*PreallocatedReadHandle) RecordCacheHit

func (rh *PreallocatedReadHandle) RecordCacheHit(_ context.Context, offset, size int64)

RecordCacheHit is part of the objstorage.ReadHandle interface.

func (*PreallocatedReadHandle) SetupForCompaction

func (rh *PreallocatedReadHandle) SetupForCompaction()

SetupForCompaction is part of the objstorage.ReadHandle interface.

type ReadaheadConfig

type ReadaheadConfig struct {
	// contains filtered or unexported fields
}

ReadaheadConfig is a container for the settings that control the use of read-ahead.

It stores two ReadaheadModes:

  • Informed is the type of read-ahead for operations that are known to read a large consecutive chunk of a file.
  • Speculative is the type of read-ahead used automatically, when consecutive reads are detected.

The settings can be changed and read atomically.

func NewReadaheadConfig

func NewReadaheadConfig() *ReadaheadConfig

NewReadaheadConfig returns a new readahead config container initialized with default values.

func (*ReadaheadConfig) Informed

func (rc *ReadaheadConfig) Informed() ReadaheadMode

Informed returns the type of read-ahead for operations that are known to read a large consecutive chunk of a file.

func (*ReadaheadConfig) Set

func (rc *ReadaheadConfig) Set(informed, speculative ReadaheadMode)

Set the informed and speculative readahead modes.

func (*ReadaheadConfig) Speculative

func (rc *ReadaheadConfig) Speculative() ReadaheadMode

Speculative returns the type of read-ahead used automatically, when consecutive reads are detected.

type ReadaheadMode

type ReadaheadMode uint8

ReadaheadMode indicates the type of read-ahead to use, either for informed read-ahead (e.g. compactions) or speculative read-ahead.

const (
	// NoReadahead disables readahead altogether.
	NoReadahead ReadaheadMode = iota

	// SysReadahead enables the use of SYS_READAHEAD call to prefetch data.
	// The prefetch window grows dynamically as consecutive writes are detected.
	SysReadahead

	// FadviseSequential enables the use of FADV_SEQUENTIAL. For informed
	// read-ahead, FADV_SEQUENTIAL is used from the beginning. For speculative
	// read-ahead, SYS_READAHEAD is first used until the window reaches the
	// maximum size, then we switch to FADV_SEQUENTIAL.
	FadviseSequential
)

type Settings

type Settings struct {
	Logger base.Logger

	// Local filesystem configuration.
	FS        vfs.FS
	FSDirName string

	// FSDirInitialListing is a listing of FSDirName at the time of calling Open.
	//
	// This is an optional optimization to avoid double listing on Open when the
	// higher layer already has a listing. When nil, we obtain the listing on
	// Open.
	FSDirInitialListing []string

	// Cleaner cleans obsolete files from the local filesystem.
	//
	// The default cleaner uses the DeleteCleaner.
	FSCleaner base.Cleaner

	// NoSyncOnClose decides whether the implementation will enforce a
	// close-time synchronization (e.g., fdatasync() or sync_file_range())
	// on files it writes to. Setting this to true removes the guarantee for a
	// sync on close. Some implementations can still issue a non-blocking sync.
	NoSyncOnClose bool

	// BytesPerSync enables periodic syncing of files in order to smooth out
	// writes to disk. This option does not provide any persistence guarantee, but
	// is used to avoid latency spikes if the OS automatically decides to write
	// out a large chunk of dirty filesystem buffers.
	BytesPerSync int

	// Local contains fields that are only relevant for files stored on the local
	// filesystem.
	Local struct {

		// ReadaheadConfig is used to retrieve the current readahead mode; it is
		// consulted whenever a read handle is initialized.
		ReadaheadConfig *ReadaheadConfig
	}

	// Fields here are set only if the provider is to support remote objects
	// (experimental).
	Remote struct {
		StorageFactory remote.StorageFactory

		// If CreateOnShared is non-zero, sstables are created on remote storage using
		// the CreateOnSharedLocator (when the PreferSharedStorage create option is
		// true).
		CreateOnShared        remote.CreateOnSharedStrategy
		CreateOnSharedLocator remote.Locator

		// CacheSizeBytes is the size of the on-disk block cache for objects
		// on remote storage. If it is 0, no cache is used.
		CacheSizeBytes int64

		// CacheBlockSize is the block size of the cache; if 0, the default of 32KB is used.
		CacheBlockSize int

		// ShardingBlockSize is the size of a shard block. The cache is split into contiguous
		// ShardingBlockSize units. The units are distributed across multiple independent shards
		// of the cache, via a hash(offset) modulo num shards operation. The cache replacement
		// policies operate at the level of shard, not whole cache. This is done to reduce lock
		// contention.
		//
		// If ShardingBlockSize is 0, the default of 1 MB is used.
		ShardingBlockSize int64

		// The number of independent shards the cache leverages. Each shard is the same size,
		// and a hash of filenum & offset map a read to a certain shard. If set to 0,
		// 2*runtime.GOMAXPROCS is used as the shard count.
		CacheShardCount int
	}
}

Settings that must be specified when creating the provider.

func DefaultSettings

func DefaultSettings(fs vfs.FS, dirName string) Settings

DefaultSettings initializes default settings (with no remote storage), suitable for tests and tools.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL