Documentation
¶
Index ¶
- Constants
- func Copy(srcLen uint32, srcTab Table, srcPack Pack, dstTab Table, dstPack Pack) (int, error)
- func CreatePackFile(root *os.Root, gen uint32, maxSize uint32) (*os.File, error)
- func CreateTableFile(shardRoot *os.Root, gen uint32, maxSize uint32) (*os.File, error)
- func IsErrShardFull(err error) bool
- func KeyCompare(a, b Key) int
- func LoadPackFile(root *os.Root, gen uint32) (*os.File, error)
- func LoadTableFile(shardRoot *os.Root, gen uint32) (*os.File, error)
- func PackFilename(gen uint32) string
- func SaveManifest(shardRoot *os.Root, manifest Manifest) error
- func TableFilename(gen uint32) string
- type Entry
- type ErrShardFull
- type FileKey
- type Key
- func (k Key) Bytes() []byte
- func (k Key) Data() (ret [16]byte)
- func (k Key) IsZero() bool
- func (k Key) RotateAway(i int) Key
- func (k Key) ShardID(numBits int) ShardID
- func (k Key) ShiftIn(i int) Key
- func (k Key) Uint16(i int) uint16
- func (k Key) Uint64(i int) uint64
- func (k Key) Uint8(i int) uint8
- func (k Key) Uint8Len() int
- type Manifest
- type Pack
- type Shard
- func (s *Shard) Close() error
- func (sh *Shard) Flush() error
- func (s *Shard) HasSpace(dataLen int) bool
- func (s *Shard) LocalAppend(key Key, data []byte) (bool, error)
- func (s *Shard) LocalDelete(key Key) (bool, error)
- func (sh *Shard) LocalExists(key Key) bool
- func (sh *Shard) LocalGet(key Key, fn func(data []byte)) (bool, error)
- type ShardID
- type Store
- type Table
Constants ¶
const ( TableFileExt = ".slot" PackFileExt = ".pack" )
const ( // TableEntrySize is the size of a row in the table. // It actually uses less space than this, but we want to align to 4096 bytes. TableEntrySize = 32 PageSize = 4096 EntriesPerPage = PageSize / TableEntrySize // DefaultMaxTableLen is the maximum length of a table in rows. DefaultMaxTableLen = (1 << 20) / TableEntrySize )
const DefaultMaxPackSize = 1 << 26
const ManifestFilename = "MF"
ManifestFilename is the filename for the manifest. This does not include the shard id, which will be in the directory path.
const ManifestSize = 32
ManifestSize is the size of the manifest in bytes.
Variables ¶
This section is empty.
Functions ¶
func Copy ¶
Copy copies entries from src{Tab,Pack} to dst{Tab,Pack}. If there is not enough space available in either the table or the pack, then an error is returned. Copy will not copy tombstones, and will sort all the entries by key before copying.
func CreatePackFile ¶
CreatePackFile creates a file configured for a pack in the filesystem, and returns it.
func CreateTableFile ¶
CreateTableFile creates a file configured for a table in the filesystem, and returns it. maxSize is the maximum size of the table in bytes, NOT the number of rows.
func IsErrShardFull ¶
func KeyCompare ¶
func PackFilename ¶
func TableFilename ¶
TableFilename returns the filename for a table. This is just the filename, the directory will contain the shard prefix.
Types ¶
type ErrShardFull ¶
type ErrShardFull struct{}
func (ErrShardFull) Error ¶
func (e ErrShardFull) Error() string
type FileKey ¶
type FileKey struct {
// Shard uniquely identifies the shard
ShardID ShardID
// Gen uniquely identifies the generation of the file within the shard
Gen uint32
}
FileKey uniquely identifies a {Table, Pack} file within the system.
type Key ¶
type Key [2]uint64
Key is a 128 bit key. The 0th bit is considered the first bit, and that is at k[0] & (1 << 0).
func KeyFromBytes ¶
func (Key) RotateAway ¶
RotateAway specifies the amount of bits to rotate the key away from the 0th bit.
func (Key) ShiftIn ¶
ShiftIn shifts the key into 0. The lowest bits are discarded, zeros are shifted in to the highest bits.
func (Key) Uint16 ¶
Uint16 returns the 16 bit integer at the given index. Indexes are interpretted modulo 8. Uint16(0) is bits [0, 15], Uint16(1) is bits [16, 31].
type Manifest ¶
type Manifest struct {
// Nonce is the number of times the manifest has been saved.
Nonce uint64
// Gen is the Shard's current generation.
Gen uint32
// Count is the number of entries in the table with the current generation.
TableLen uint32
}
Manifest is the source of truth for a Shard's state.
type Pack ¶
type Pack struct {
// contains filtered or unexported fields
}
Pack is an append-only file on disk.
type Shard ¶
type Shard struct {
// contains filtered or unexported fields
}
Shard is a directory on disk containing table and pack files, potentially across multiple generations. Each shard is independent of every other shard, and there are no consistency guarantees between shards. The Shard has no information about where it is in the trie. Shards only deal with files in their immediate directory, subdirectories (which correspond to children in the trie) are ignored by the Shard.
func CreateShard ¶
CreateShard creates a new shard in the filesystem. If the shard already exists, than os.ErrExist is returned. The root should be the global root.
func OpenShard ¶
OpenShard opens a shard that already exists in the filesystem. The root should be the global root.
func (*Shard) Flush ¶
Flush causes the Shard's current state to be written to disk. First the pack and table are flushed concurrently. Then once both of them have flushed successfully, the manifest is saved.
func (*Shard) LocalAppend ¶
LocalAppend appends the key and data to the table. It returns an error if the table is full or the pack is full. It returns (false, nil) if the data already exists. It returns (true, nil) if the data was appended successfully.
func (*Shard) LocalDelete ¶
LocalDelete deletes the key from the table. It returns (true, nil) if the key was deleted successfully.
func (*Shard) LocalExists ¶
LocalExists checks if the key exists in this shards local data. Local means that the children and grandchildren are not checked.
type ShardID ¶
type ShardID struct {
// contains filtered or unexported fields
}
ShardID is a prefix of at most 120 bits. ShardID takes up 128 bits. A prefix refers to a set of keys.
func NewShardID ¶
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store stores blobs on in the filesystem.
func (*Store) Get ¶
Get finds key if it exists and calls fn with the data. The data must not be used outside the callback.
type Table ¶
type Table struct {
// contains filtered or unexported fields
}
Table is an unordered append-only list of entries. Each entry points into a pack, and all entries are the same size.
func NewTable ¶
NewTable mmaps a file and returns a Table. count should be the number of rows in the table.
func (*Table) Close ¶
Close unmaps the table and closes the file. It DOES NOT flush the mmap to disk.
func (*Table) SlotOffset ¶
SlotOffset returns the offset of the i-th slot in the table. Slot 0 starts at 4 bytes, to make room for the row count at the beginning.
Source Files