vng

package
v1.18.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 20, 2024 License: BSD-3-Clause Imports: 18 Imported by: 0

Documentation

Overview

Package vng implements the reading and writing of VNG serialization objects. The VNG format is described at https://github.com/brimdata/zed/blob/main/docs/formats/vng.md.

A VNG object is created by allocating an Encoder for any top-level Zed type via NewEncoder, which recursively descends into the Zed type, allocating an Encoder for each node in the type tree. The top-level ZNG body is written via a call to Write. Each vector buffers its data in memory until the object is encoded.

After all of the Zed data is written, a metadata section is written consisting of a single Zed value describing the layout of all the vector data obtained by calling the Metadata method on the Encoder interface.

Nulls are encoded by a special Nulls object. Each type is wrapped by a NullsEncoder, which run-length encodes alternating sequences of nulls and values. If no nulls are encountered, then the Nulls object is omitted from the metadata.

Data is read from a VNG object by reading the metadata and creating vector Builders for each Zed type by calling NewBuilder with the metadata, which recusirvely creates Builders. An io.ReaderAt is passed to NewBuilder so each vector reader can access the underlying storage object and read its vector data effciently in large vector segments.

Once the metadata is assembled in memory, the recontructed Zed sequence data can be read from the vector segments by calling the Build method on the top-level Builder and passing in a zcode.Builder to reconstruct the Zed value.

Index

Constants

View Source 
const (
	Version     = 4
	HeaderSize  = 16
	MaxMetaSize = 100 * 1024 * 1024
	MaxDataSize = 2 * 1024 * 1024 * 1024
)
View Source 
const (
	CompressionFormatNone uint8 = 0 // No compression
	CompressionFormatLZ4  uint8 = 1 // LZ4 compression
)

Values for [Segment.CompressionFormat].

View Source 
const MaxDictSize = 256

Variables

View Source 
var Template = []interface{}{
	Record{},
	Array{},
	Set{},
	Map{},
	Union{},
	Primitive{},
	Named{},
	Error{},
	Nulls{},
	Const{},
	Variant{},
}

Functions

func NewZedReader added in v1.13.0 

func NewZedReader(zctx *zed.Context, meta Metadata, r io.ReaderAt) (zio.Reader, error)

func ReadIntVector 

func ReadIntVector(loc Segment, r io.ReaderAt) ([]int32, error)

XXX change this to single vector read

func ReadUint32Vector added in v1.13.0 

func ReadUint32Vector(loc Segment, r io.ReaderAt) ([]uint32, error)

Types

type Array added in v1.13.0 

type Array struct {
	Length  uint32
	Lengths Segment
	Values  Metadata
}

func (*Array) Len added in v1.13.0 

func (a *Array) Len() uint32

func (*Array) Type added in v1.13.0 

func (a *Array) Type(zctx *zed.Context) zed.Type

type ArrayBuilder added in v1.13.0 

type ArrayBuilder struct {
	Elems   Builder
	Lengths *Int64Decoder
}

func NewArrayBuilder added in v1.13.0 

func NewArrayBuilder(array *Array, r io.ReaderAt) (*ArrayBuilder, error)

func (*ArrayBuilder) Build added in v1.13.0 

func (a *ArrayBuilder) Build(b *zcode.Builder) error

type ArrayEncoder added in v1.13.0 

type ArrayEncoder struct {
	// contains filtered or unexported fields
}

func NewArrayEncoder added in v1.13.0 

func NewArrayEncoder(typ *zed.TypeArray) *ArrayEncoder

func (*ArrayEncoder) Emit added in v1.13.0 

func (a *ArrayEncoder) Emit(w io.Writer) error

func (*ArrayEncoder) Encode added in v1.13.0 

func (a *ArrayEncoder) Encode(group *errgroup.Group)

func (*ArrayEncoder) Metadata added in v1.13.0 

func (a *ArrayEncoder) Metadata(off uint64) (uint64, Metadata)

func (*ArrayEncoder) Write added in v1.13.0 

func (a *ArrayEncoder) Write(body zcode.Bytes)

type Builder added in v1.13.0 

type Builder interface {
	Build(*zcode.Builder) error
}

func NewBuilder added in v1.13.0 

func NewBuilder(meta Metadata, r io.ReaderAt) (Builder, error)

type Const added in v1.13.0 

type Const struct {
	Value zed.Value
	Count uint32
}

func (*Const) Len added in v1.13.0 

func (c *Const) Len() uint32

func (*Const) Type added in v1.13.0 

func (c *Const) Type(zctx *zed.Context) zed.Type

type ConstBuilder added in v1.13.0 

type ConstBuilder struct {
	Typ zed.Type
	// contains filtered or unexported fields
}

func NewConstBuilder added in v1.13.0 

func NewConstBuilder(c *Const) *ConstBuilder

func (*ConstBuilder) Build added in v1.13.0 

func (c *ConstBuilder) Build(b *zcode.Builder) error

type DictBuilder added in v1.13.0 

type DictBuilder struct {
	Typ zed.Type
	// contains filtered or unexported fields
}

func NewDictBuilder added in v1.13.0 

func NewDictBuilder(primitive *Primitive, reader io.ReaderAt) *DictBuilder

func (*DictBuilder) Build added in v1.13.0 

func (d *DictBuilder) Build(b *zcode.Builder) error

func (*DictBuilder) ReadBytes added in v1.13.0 

func (d *DictBuilder) ReadBytes() (zcode.Bytes, error)

type DictEntry added in v1.13.0 

type DictEntry struct {
	Value zed.Value
	Count uint32
}

type Encoder added in v1.13.0 

type Encoder interface {
	// Write collects up values to be encoded into memory.
	Write(zcode.Bytes)
	// Encode encodes all in-memory vector data into its storage-ready serialized format.
	// Vectors may be encoded concurrently and errgroup.Group is used to sync
	// and return errors.
	Encode(*errgroup.Group)
	// Metadata returns the data structure conforming to the VNG specification
	// describing the layout of vectors.  This is called after all data is
	// written and encoded by the Encode with the result marshaled to build
	// the header section of the VNG object.  An offset is passed down into
	// the traversal representing where in the data section the vector data
	// will land.  This is called in a sequential fashion (no parallelism) so
	// that the metadata can be computed and the VNG header written before the
	// vector data is written via Emit.
	Metadata(uint64) (uint64, Metadata)
	Emit(w io.Writer) error
}

func NewEncoder added in v1.13.0 

func NewEncoder(typ zed.Type) Encoder

type Error added in v1.18.0 

type Error struct {
	Values Metadata
}

func (*Error) Len added in v1.18.0 

func (e *Error) Len() uint32

func (*Error) Type added in v1.18.0 

func (e *Error) Type(zctx *zed.Context) zed.Type

type ErrorEncoder added in v1.18.0 

type ErrorEncoder struct {
	Encoder
}

func (*ErrorEncoder) Metadata added in v1.18.0 

func (e *ErrorEncoder) Metadata(off uint64) (uint64, Metadata)

type Field added in v1.13.0 

type Field struct {
	Name   string
	Values Metadata
}

type FieldBuilder added in v1.13.0 

type FieldBuilder struct {
	Values Builder
}

func NewFieldBuilder added in v1.13.0 

func NewFieldBuilder(field Field, r io.ReaderAt) (*FieldBuilder, error)

func (*FieldBuilder) Build added in v1.13.0 

func (f *FieldBuilder) Build(b *zcode.Builder) error

type FieldEncoder added in v1.13.0 

type FieldEncoder struct {
	// contains filtered or unexported fields
}

func (*FieldEncoder) Emit added in v1.13.0 

func (f *FieldEncoder) Emit(w io.Writer) error

func (*FieldEncoder) Encode added in v1.13.0 

func (f *FieldEncoder) Encode(group *errgroup.Group)

func (*FieldEncoder) Metadata added in v1.13.0 

func (f *FieldEncoder) Metadata(off uint64) (uint64, Field)
type Header struct {
	Version  uint32
	MetaSize uint32
	DataSize uint32
}

func ReadHeader added in v1.13.0 

func ReadHeader(r io.Reader) (Header, error)

func (*Header) Deserialize added in v1.13.0 

func (h *Header) Deserialize(bytes []byte) error

func (Header) Serialize added in v1.13.0 

func (h Header) Serialize() []byte

type Int64Decoder added in v1.13.0 

type Int64Decoder struct {
	PrimitiveBuilder
}

func NewInt64Decoder added in v1.13.0 

func NewInt64Decoder(loc Segment, r io.ReaderAt) *Int64Decoder

func (*Int64Decoder) Next added in v1.13.0 

func (p *Int64Decoder) Next() (int64, error)

type Int64Encoder added in v1.13.0 

type Int64Encoder struct {
	PrimitiveEncoder
}

func NewInt64Encoder added in v1.13.0 

func NewInt64Encoder() *Int64Encoder

func (*Int64Encoder) Write added in v1.13.0 

func (p *Int64Encoder) Write(v int64)

type Map added in v1.13.0 

type Map struct {
	Length  uint32
	Lengths Segment
	Keys    Metadata
	Values  Metadata
}

func (*Map) Len added in v1.13.0 

func (m *Map) Len() uint32

func (*Map) Type added in v1.13.0 

func (m *Map) Type(zctx *zed.Context) zed.Type

type MapBuilder added in v1.13.0 

type MapBuilder struct {
	Keys    Builder
	Values  Builder
	Lengths *Int64Decoder
}

func NewMapBuilder added in v1.13.0 

func NewMapBuilder(m *Map, r io.ReaderAt) (*MapBuilder, error)

func (*MapBuilder) Build added in v1.13.0 

func (m *MapBuilder) Build(b *zcode.Builder) error

type MapEncoder added in v1.13.0 

type MapEncoder struct {
	// contains filtered or unexported fields
}

func NewMapEncoder added in v1.13.0 

func NewMapEncoder(typ *zed.TypeMap) *MapEncoder

func (*MapEncoder) Emit added in v1.13.0 

func (m *MapEncoder) Emit(w io.Writer) error

func (*MapEncoder) Encode added in v1.13.0 

func (m *MapEncoder) Encode(group *errgroup.Group)

func (*MapEncoder) Metadata added in v1.13.0 

func (m *MapEncoder) Metadata(off uint64) (uint64, Metadata)

func (*MapEncoder) Write added in v1.13.0 

func (m *MapEncoder) Write(body zcode.Bytes)

type Metadata added in v1.13.0 

type Metadata interface {
	Type(*zed.Context) zed.Type
	Len() uint32
}

func Under added in v1.13.0 

func Under(meta Metadata) Metadata

type Named added in v1.13.0 

type Named struct {
	Name   string
	Values Metadata
}

func (*Named) Len added in v1.13.0 

func (n *Named) Len() uint32

func (*Named) Type added in v1.13.0 

func (n *Named) Type(zctx *zed.Context) zed.Type

type NamedEncoder added in v1.13.0 

type NamedEncoder struct {
	Encoder
	// contains filtered or unexported fields
}

func (*NamedEncoder) Metadata added in v1.13.0 

func (n *NamedEncoder) Metadata(off uint64) (uint64, Metadata)

type Nulls added in v1.13.0 

type Nulls struct {
	Runs   Segment
	Values Metadata
	Count  uint32 // Count of nulls
}

func (*Nulls) Len added in v1.13.0 

func (n *Nulls) Len() uint32

func (*Nulls) Type added in v1.13.0 

func (n *Nulls) Type(zctx *zed.Context) zed.Type

type NullsBuilder added in v1.13.0 

type NullsBuilder struct {
	Values Builder
	Runs   Int64Decoder
	// contains filtered or unexported fields
}

func NewNullsBuilder added in v1.13.0 

func NewNullsBuilder(values Builder, loc Segment, r io.ReaderAt) *NullsBuilder

func (*NullsBuilder) Build added in v1.13.0 

func (n *NullsBuilder) Build(b *zcode.Builder) error

type NullsEncoder added in v1.13.0 

type NullsEncoder struct {
	// contains filtered or unexported fields
}

NullsEncoder emits a sequence of runs of the length of alternating sequences of nulls and values, beginning with nulls. Every run is non-zero except for the first, which may be zero when the first value is non-null.

func NewNullsEncoder added in v1.13.0 

func NewNullsEncoder(values Encoder) *NullsEncoder

func (*NullsEncoder) Emit added in v1.13.0 

func (n *NullsEncoder) Emit(w io.Writer) error

func (*NullsEncoder) Encode added in v1.13.0 

func (n *NullsEncoder) Encode(group *errgroup.Group)

func (*NullsEncoder) Metadata added in v1.13.0 

func (n *NullsEncoder) Metadata(off uint64) (uint64, Metadata)

func (*NullsEncoder) Write added in v1.13.0 

func (n *NullsEncoder) Write(body zcode.Bytes)

type Object 

type Object struct {
	// contains filtered or unexported fields
}

func NewObject 

func NewObject(r io.ReaderAt) (*Object, error)

func (*Object) Close 

func (o *Object) Close() error

func (*Object) DataReader added in v1.13.0 

func (o *Object) DataReader() io.ReaderAt

func (*Object) Metadata added in v1.13.0 

func (o *Object) Metadata() Metadata

func (*Object) NewReader added in v1.13.0 

func (o *Object) NewReader(zctx *zed.Context) (zio.Reader, error)

type Primitive added in v1.13.0 

type Primitive struct {
	Typ      zed.Type `zed:"Type"`
	Location Segment
	Dict     []DictEntry
	Min      *zed.Value
	Max      *zed.Value
	Count    uint32
}

func (*Primitive) Len added in v1.13.0 

func (p *Primitive) Len() uint32

func (*Primitive) Type added in v1.13.0 

func (p *Primitive) Type(zctx *zed.Context) zed.Type

type PrimitiveBuilder added in v1.13.0 

type PrimitiveBuilder struct {
	Typ zed.Type
	// contains filtered or unexported fields
}

func NewPrimitiveBuilder added in v1.13.0 

func NewPrimitiveBuilder(primitive *Primitive, reader io.ReaderAt) *PrimitiveBuilder

func (*PrimitiveBuilder) Build added in v1.13.0 

func (p *PrimitiveBuilder) Build(b *zcode.Builder) error

func (*PrimitiveBuilder) ReadBytes added in v1.13.0 

func (p *PrimitiveBuilder) ReadBytes() (zcode.Bytes, error)

type PrimitiveEncoder added in v1.13.0 

type PrimitiveEncoder struct {
	// contains filtered or unexported fields
}

func NewPrimitiveEncoder added in v1.13.0 

func NewPrimitiveEncoder(typ zed.Type, useDict bool) *PrimitiveEncoder

func (*PrimitiveEncoder) Const added in v1.13.0 

func (p *PrimitiveEncoder) Const() *Const

func (*PrimitiveEncoder) Emit added in v1.13.0 

func (p *PrimitiveEncoder) Emit(w io.Writer) error

func (*PrimitiveEncoder) Encode added in v1.13.0 

func (p *PrimitiveEncoder) Encode(group *errgroup.Group)

func (*PrimitiveEncoder) Metadata added in v1.13.0 

func (p *PrimitiveEncoder) Metadata(off uint64) (uint64, Metadata)

func (*PrimitiveEncoder) Write added in v1.13.0 

func (p *PrimitiveEncoder) Write(body zcode.Bytes)

type Record added in v1.13.0 

type Record struct {
	Length uint32
	Fields []Field
}

func (*Record) Len added in v1.13.0 

func (r *Record) Len() uint32

func (*Record) Lookup added in v1.13.0 

func (r *Record) Lookup(path field.Path) *Field

func (*Record) LookupField added in v1.13.0 

func (r *Record) LookupField(name string) *Field

func (*Record) Type added in v1.13.0 

func (r *Record) Type(zctx *zed.Context) zed.Type

type RecordBuilder added in v1.13.0 

type RecordBuilder struct {
	Names  []string
	Values []FieldBuilder
}

func NewRecordBuilder added in v1.13.0 

func NewRecordBuilder(record *Record, reader io.ReaderAt) (*RecordBuilder, error)

func (*RecordBuilder) Build added in v1.13.0 

func (r *RecordBuilder) Build(b *zcode.Builder) error

type RecordEncoder added in v1.13.0 

type RecordEncoder struct {
	// contains filtered or unexported fields
}

func NewRecordEncoder added in v1.13.0 

func NewRecordEncoder(typ *zed.TypeRecord) *RecordEncoder

func (*RecordEncoder) Emit added in v1.13.0 

func (r *RecordEncoder) Emit(w io.Writer) error

func (*RecordEncoder) Encode added in v1.13.0 

func (r *RecordEncoder) Encode(group *errgroup.Group)

func (*RecordEncoder) Metadata added in v1.13.0 

func (r *RecordEncoder) Metadata(off uint64) (uint64, Metadata)

func (*RecordEncoder) Write added in v1.13.0 

func (r *RecordEncoder) Write(body zcode.Bytes)

type Segment added in v1.13.0 

type Segment struct {
	Offset            int64 // Offset relative to start of file
	Length            int32 // Length in file
	MemLength         int32 // Length in memory
	CompressionFormat uint8 // Compression format in file
}

func (*Segment) Read added in v1.13.0 

func (s *Segment) Read(r io.ReaderAt, b []byte) error

Read reads the segement r, uncompresses it if necessary, and stores it in the first s.MemLength bytes of b. If the length of b is less than s.MemLength, Read returns io.ErrShortBuffer.

type Set added in v1.13.0 

type Set Array

func (*Set) Len added in v1.13.0 

func (s *Set) Len() uint32

func (*Set) Type added in v1.13.0 

func (s *Set) Type(zctx *zed.Context) zed.Type

type SetEncoder added in v1.13.0 

type SetEncoder struct {
	ArrayEncoder
}

func NewSetEncoder added in v1.13.0 

func NewSetEncoder(typ *zed.TypeSet) *SetEncoder

func (*SetEncoder) Metadata added in v1.13.0 

func (s *SetEncoder) Metadata(off uint64) (uint64, Metadata)

type Union added in v1.13.0 

type Union struct {
	Length uint32
	Tags   Segment
	Values []Metadata
}

func (*Union) Len added in v1.13.0 

func (u *Union) Len() uint32

func (*Union) Type added in v1.13.0 

func (u *Union) Type(zctx *zed.Context) zed.Type

type UnionBuilder added in v1.13.0 

type UnionBuilder struct {
	// contains filtered or unexported fields
}

func NewUnionBuilder added in v1.13.0 

func NewUnionBuilder(union *Union, r io.ReaderAt) (*UnionBuilder, error)

func (*UnionBuilder) Build added in v1.13.0 

func (u *UnionBuilder) Build(b *zcode.Builder) error

type UnionEncoder added in v1.13.0 

type UnionEncoder struct {
	// contains filtered or unexported fields
}

func NewUnionEncoder added in v1.13.0 

func NewUnionEncoder(typ *zed.TypeUnion) *UnionEncoder

func (*UnionEncoder) Emit added in v1.13.0 

func (u *UnionEncoder) Emit(w io.Writer) error

func (*UnionEncoder) Encode added in v1.13.0 

func (u *UnionEncoder) Encode(group *errgroup.Group)

func (*UnionEncoder) Metadata added in v1.13.0 

func (u *UnionEncoder) Metadata(off uint64) (uint64, Metadata)

func (*UnionEncoder) Write added in v1.13.0 

func (u *UnionEncoder) Write(body zcode.Bytes)

type Variant added in v1.13.0 

type Variant struct {
	Tags   Segment
	Values []Metadata
	Length uint32
}

func (*Variant) Len added in v1.13.0 

func (v *Variant) Len() uint32

func (*Variant) Type added in v1.13.0 

func (*Variant) Type(zctx *zed.Context) zed.Type

type VariantEncoder added in v1.13.0 

type VariantEncoder struct {
	// contains filtered or unexported fields
}

func NewVariantEncoder added in v1.13.0 

func NewVariantEncoder() *VariantEncoder

func (*VariantEncoder) Emit added in v1.13.0 

func (v *VariantEncoder) Emit(w io.Writer) error

func (*VariantEncoder) Encode added in v1.13.0 

func (v *VariantEncoder) Encode() (Metadata, uint64, error)

func (*VariantEncoder) Write added in v1.13.0 

func (v *VariantEncoder) Write(val zed.Value) error

The variant encoder self-organizes around the types that are written to it. No need to define the schema up front! We track the types seen first-come, first-served and the VNG metadata structure follows accordingly.

type Writer 

type Writer struct {
	// contains filtered or unexported fields
}

Writer implements the zio.Writer interface. A Writer creates a vector VNG object from a stream of zed.Records.

func NewWriter 

func NewWriter(w io.WriteCloser) *Writer

func (*Writer) Close 

func (w *Writer) Close() error

func (*Writer) Write 

func (w *Writer) Write(val zed.Value) error

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.