Documentation
¶
Overview ¶
Package store implements a robust scheme for binary serialization and deserialization of plain data into and from byte buffers.
The purpose of this package is to serve as the serialization building block to the boltdb-backed storage engine, but it's general purpose enough that it can be used in any other context, such as serialization to/from files.
Serialization Buffer and Mode ¶
The basic building block is a `Buffer` struct that has a backing buffer (a byte slice) and a Mode, which can be either Serialize or Deserialize.
This allows the "serialization" function to fulfill the role of both reading and writing (serialization and deserialization) at the same time.
A serialization function takes a pointer to an object (to be (de)serialized) and a pointer to a Buffer.
If the buffer is in serialization mode, the function writes the binary representation of the object to the buffer. If the buffer is in deserialization mode, the function reads from the buffer into the object passed.
As a user of this package, you almost never have to worry about checking the mode in your own serialization code. At the high level, you just list the fields you want to serialize and what function to use to serialize them.
This allows the serialization and deserialization of complex objects to be robust: just serialize the relevant fields in order, and you're guaranteed that deserialization will happen in exactly the same order.
Versioning ¶
Robust serialization for long term storage requires supporting schema evolution through a version flag: before writing out the fields, write a version number.
In the future, when the struct changes, create a new serialization function, while keeping the old one and changing its content to account for the new version.
Example:
At time t0:
type XYZ struct {
Unit int
Energy int
Chapter string
President bool
}
func v1SerializeXYZ(xyz *XYZ, buf *store.Buffer) {
store.Int(&xyz.Unit, buf)
store.Int(&xyz.Energy, buf)
store.String(&xyz.Chapter, buf)
store.Bool(&xyz.President, buf)
}
func SerializeXYZ(xyz *XYZ, buf *store.Buffer) {
store.Versioned(xyz, buf,
// list out the version
v1SerializeXYZ,
)
}
Later at time t1, we remove the `Energy` field, and add a `Price` field.
type XYZ struct {
Unit int
Price int
Chapter string
President bool
}
// v1SerializeXYZ needs to be modified to accomodate the new structure,
// while still reading the fields in the same order
func v1SerializeXYZ(xyz *XYZ, buf *store.Buffer) {
var energy int // Field that used to exist in previous version
store.Int(&xyz.Unit, buf)
store.Int(&energy, buf) // reading into a local variable
store.String(&xyz.Chapter, buf)
store.Bool(&xyz.President, buf)
// version migration code:
// In the new version, price is energy * unit
xyz.Price = energy * xyz.Unit
}
func v2SerializeXYZ(xyz *XYZ, buf *store.Buffer) {
store.Int(&xyz.Unit, buf)
store.Int(&xyz.Price, buf)
store.String(&xyz.Chapter, buf)
store.Bool(&xyz.President, buf)
}
func SerializeXYZ(xyz *XYZ, buf *store.Buffer) {
store.Versioned(xyz, buf,
// list out the versions in order
v1SerializeXYZ,
v2SerializeXYZ,
)
}
Index ¶
- Constants
- Variables
- func BinaryMarshal(b Binary, buf *Buffer)
- func Bool(b *bool, buf *Buffer)
- func Byte(b *byte, buf *Buffer)
- func ByteSlice(s *[]byte, buf *Buffer)
- func FInt(n *int, buf *Buffer)
- func FInt64(n *int64, buf *Buffer)
- func FUInt16(n *uint16, buf *Buffer)
- func FUInt32(n *uint32, buf *Buffer)
- func FUInt64(n *uint64, buf *Buffer)
- func Float64(n *float64, buf *Buffer)
- func FromBytes[T any](data []byte, fn SerializeFn[T]) *T
- func FromBytesInto[T any](data []byte, obj *T, fn SerializeFn[T]) bool
- func Int(n *int, buf *Buffer)
- func IntEnum[T IntBased](n *T, buf *Buffer)
- func Map[K comparable, T any](m *map[K]T, keyFn SerializeFn[K], valFn SerializeFn[T], buf *Buffer)
- func Rune(r *rune, buf *Buffer)
- func SerializeUUID(id *UUID, buf *Buffer)
- func Slice[T any](list *[]T, fn SerializeFn[T], buf *Buffer)
- func String(s *string, buf *Buffer)
- func StringZ(s *string, buf *Buffer)
- func Time(t *time.Time, buf *Buffer)
- func ToBytes[T any](obj *T, fn SerializeFn[T]) []byte
- func UInt(n *uint, buf *Buffer)
- func UnixTime(t *time.Time, buf *Buffer)
- func UnixTimeKey(t *time.Time, buf *Buffer)
- func UnixTimeMilli(t *time.Time, buf *Buffer)
- func UnixTimeMilliKey(t *time.Time, buf *Buffer)
- func VInt64(n *int64, buf *Buffer)
- func VUInt64(n *uint64, buf *Buffer)
- func Versioned[T any](item *T, buf *Buffer, fns ...SerializeFn[T])
- type Binary
- type Buffer
- type IntBased
- type Mode
- type SerializeFn
- type UUID
Constants ¶
const UUID_SIZE = 16
Variables ¶
var BigEndian = binary.BigEndian
var GenericError = errors.New("Deserialization error")
var InvalidUUIDSize = errors.New("InvalidUUIDSize")
Functions ¶
func BinaryMarshal ¶
BinaryMarshal implements serialization for an object that implements the BinaryMarshaler and BinaryUnmarshaler interfaces from the standard library.
func ByteSlice ¶
ByteSlice implements serialization for a byte slice. It's more or less just like String.
func FInt ¶
FInt implements fixed size serialization of int (as 64 bits). It writes data in big endian, making it suitable for int keys to bolt.
func FInt64 ¶
FInt64 implements fixed size serialization of int64. It writes data in big endian, making it suitable for int keys to bolt.
func FUInt16 ¶
FUInt16 implements fixed size serialization of uint16. It writes data in big endian, making it suitable for int keys to bolt.
func FUInt32 ¶
FUInt32 implements fixed size serialization of uint32. It writes data in big endian, making it suitable for int keys to bolt.
func FUInt64 ¶
FUInt64 implements fixed size serialization of uint64. It writes data in big endian, making it suitable for int keys to bolt.
func FromBytes ¶
func FromBytes[T any](data []byte, fn SerializeFn[T]) *T
func FromBytesInto ¶
func FromBytesInto[T any](data []byte, obj *T, fn SerializeFn[T]) bool
func Int ¶
Int implements varint encoding for int (as int64). Varint users fewer bytes for small values.
func Map ¶
func Map[K comparable, T any](m *map[K]T, keyFn SerializeFn[K], valFn SerializeFn[T], buf *Buffer)
func SerializeUUID ¶
func Slice ¶
func Slice[T any](list *[]T, fn SerializeFn[T], buf *Buffer)
Slice is a helper for serialization a slice of some type, given its serialization function. It starts by reading/writing the length of the slice, then uses the provided serialization function to serialize each individual item in the slice.
func String ¶
String implements serialization for a string by first writing out the length in bytes (as a varint) then dumping the actual bytes into the buffer. When deserializing, it starts by reading the length (as a varint) then taking a slice of the input buffer and cloning it to a string
func StringZ ¶
StringZ implement serialization for a string using null-byte termination. This allows is to be used in the key of a boltdb key
func Time ¶
Time implement serialization for the std library's Time object using the Binary Marshalling interface
func ToBytes ¶
func ToBytes[T any](obj *T, fn SerializeFn[T]) []byte
func UInt ¶
UInt implements varint encoding for uint (as uint64). Varint users fewer bytes for small values.
func UnixTime ¶
UnixTime serializes Time as a unix timestamp, in other wrods, the resolution is truncated to the seconds level, and the location data is omitted. It also uses variable encoding to take as little space as possible.
It can store a reasonably accurate timestamp in 5 or 6 bytes.
If you require subsecond accuracy, don't use this function.
func UnixTimeKey ¶
UnixTimeKey is similar to UnixTime, but uses fixed encoding so the value is suitable for a bucket key so we can iterate by timestamp
If you require subsecond accuracy, don't use this function.
func UnixTimeMilli ¶
UnixTimeMilli is similar to UnixTime but truncates to the MilliSecond level making it more suitable for cases where sub-second accuracy is required
func UnixTimeMilliKey ¶
UnixTimeMilliKey is similar to UnixTimeMilli, but uses fixed encoding so the value is suitable for a bucket key so we can iterate by timestamp
func VInt64 ¶
VInt64 implements varint encoding for int64. Varint users fewer bytes for small values.
func VUInt64 ¶
VUInt64 implements varint encoding for uin64. Varint users fewer bytes for small values.
func Versioned ¶
func Versioned[T any](item *T, buf *Buffer, fns ...SerializeFn[T])
Versioned is a helper for creating versioned object serializers
Types ¶
type Binary ¶
type Binary interface {
encoding.BinaryMarshaler
encoding.BinaryUnmarshaler
}
type Buffer ¶
type Buffer struct {
Data []byte
Pos int // reading position; not used for writing
Error bool // TODO: something better than this to report errors with more info?
Mode Mode
}
Buffer is a byte buffer used to serialize data into, or deserialize data from, depending on the mode.
func NewReader ¶
NewReader prepares a Buffer for deserializing data from the backing byte buffer. The caller owns the data.
func NewWriter ¶
func NewWriter() *Buffer
NewWriter prepares a buffer for serializing data into. The backing buffer is owned by Buffer, but when serialization is done, the caller may use it.
func (*Buffer) EnsureSpace ¶
Ensure there's at least n bytes in the buffer starting from current position
func (*Buffer) ReadBytes ¶
ReadBytes does not expand the buffer to fit the required size. Instead, if there's no enough size, it sets the error flag.
func (*Buffer) ReadingDone ¶
func (*Buffer) WriteBytes ¶
type SerializeFn ¶
SerializeFn is a generic serialization function that can be used either to serialize or deserialize data, depending on the buffer's mode.
type UUID ¶
func GenerateUUID ¶
func GenerateUUID() UUID
func (*UUID) FromString ¶
func (UUID) MarshalJSON ¶
NOTE: this is important: the json marshaler should *not* be pointer based, but the unmarshaler *must* be pointer based
Source Files