canoto

Canoto

Canoto is a serialization format designed to be:

1. Fast
2. Compact
3. Canonical
4. Backwards compatible
5. Read compatible with Protocol Buffers.

Install

go install github.com/StephenButtolph/canoto/canoto@latest

Define Messages

Canoto messages are defined as normal golang structs:

type ExampleStruct0 struct {
	Int32              int32           `canoto:"int,1"`
	Int64              int64           `canoto:"int,2"`
	Uint32             uint32          `canoto:"uint,3"`
	Uint64             uint64          `canoto:"uint,4"`
	Sfixed32           int32           `canoto:"fint32,5"`
	Fixed32            uint32          `canoto:"fint32,6"`
	Sfixed64           int64           `canoto:"fint64,7"`
	Fixed64            uint64          `canoto:"fint64,8"`
	Bool               bool            `canoto:"bool,9"`
	String             string          `canoto:"string,10"`
	Bytes              []byte          `canoto:"bytes,11"`
	OtherStruct        ExampleStruct1  `canoto:"value,12"`
	OtherStructPointer *ExampleStruct1 `canoto:"pointer,13"`
	OtherStructField   *ExampleStruct1 `canoto:"field,14"`

	canotoData canotoData_ExampleStruct0
}

type ExampleStruct1 struct {
	Int32 int32 `canoto:"int,536870911"`

	canotoData canotoData_ExampleStruct1
}

All structs must include a field called canotoData that will cache the results of calculating the size of the struct.

The type canotoData_${structName} is automatically generated by Canoto.

For a given Struct, Canoto automatically implements the Message and FieldMaker[*Struct] interfaces:

// Message defines a type that can be a stand-alone Canoto message.
type Message interface {
	Field
	// MarshalCanoto returns the Canoto representation of this message.
	//
	// It is assumed that this message is ValidCanoto.
	MarshalCanoto() []byte
	// UnmarshalCanoto unmarshals a Canoto-encoded byte slice into the message.
	UnmarshalCanoto(bytes []byte) error
}

// Field defines a type that can be included inside of a Canoto message.
type Field interface {
	// CanotoSpec returns the specification of this canoto message.
	//
	// If there is not a valid specification of this type, it returns nil.
	CanotoSpec(types ...reflect.Type) *Spec
	// MarshalCanotoInto writes the field into a [Writer] and returns the
	// resulting [Writer].
	//
	// It is assumed that CalculateCanotoCache has been called since the last
	// modification to this field.
	//
	// It is assumed that this field is ValidCanoto.
	MarshalCanotoInto(w Writer) Writer
	// CalculateCanotoCache populates internal caches based on the current
	// values in the struct.
	CalculateCanotoCache()
	// CachedCanotoSize returns the previously calculated size of the Canoto
	// representation from CalculateCanotoCache.
	//
	// If CalculateCanotoCache has not yet been called, or the field has been
	// modified since the last call to CalculateCanotoCache, the returned size
	// may be incorrect.
	CachedCanotoSize() uint64
	// UnmarshalCanotoFrom populates the field from a [Reader].
	UnmarshalCanotoFrom(r Reader) error
	// ValidCanoto validates that the field can be correctly marshaled into the
	// Canoto format.
	ValidCanoto() bool
}

// FieldMaker is a Field that can create a new value of type T.
//
// The returned value must be able to be unmarshaled into.
//
// This type can be used when implementing a generic Field. However, if T is an
// interface, it is possible for generated code to compile and panic at runtime.
type FieldMaker[T any] interface {
	Field
	MakeCanoto() T
}

Generate

In order to generate canoto information for all of the structs in a file, simply run the canoto command with one or more files.

canoto example0.go example1.go

The above example will generate example0.canoto.go and example1.canoto.go.

The corresponding proto file for a canoto file can also be generated by adding the --proto.

canoto --proto example.go

The above example will generate example.canoto.go and example.proto.

go:generate

To automatically generate the .canoto.go version of a file, it is recommended to use go:generate

Placing

//go:generate canoto $GOFILE

at the top of a file will update the .canoto.go version of the file every time go generate ./... is run.

Best Practices

canoto only inspects a single golang file at a time, so it is recommended to define nested messages in the same file to be able to generate the most useful proto file.

Additionally, while fully supported in the canoto output, type aliases and generic types will result in proto files with default types. It is still guaranteed for the generated proto file to be able to parse canoto data, but the types may not be as specific as they could be.

If type aliases are needed, it may make sense to modify the generated proto file to specify the most specific proto type possible.

Generics

There are two ways to utilize generics with canoto.

Value and Pointer Types

To guarantee safe usage of a struct, type constraints can be used to implement a struct with a generic field T. Canoto inspects the generic types, so the struct must include a type parameter of canoto.FieldPointer[T]. Such as:

type GenericField[T any, _ canoto.FieldPointer[T]] struct {
	Value   T  `canoto:"value,1"`
	Pointer *T `canoto:"pointer,2"`

	canotoData canotoData_GenericField
}

If canoto.FieldPointer is aliased to a different type or is otherwise re-implemented, Canoto will not be able to correctly tie the type constraints together.

Field Types

Because using multiple types to constrain a single type is clunky, there is support for canoto.FieldMakers. canoto.FieldMakers can be used to allocate new messages during parsing. In order for canoto.FieldMakers to work safely, the implementing type must have a useful zero value.

[!WARNING] CanotoSpec, MakeCanoto, CalculateCanotoCache, CachedCanotoSize, ValidCanoto and MarshalCanotoInto must be able to be called with the zero value of the type implementing canoto.FieldMaker to avoid runtime panics. It is never safe to pass an interface as the canoto.FieldMaker type.

An example of correctly using canoto.FieldMaker:

type GenericField[T canoto.FieldMaker[T]] struct {
	Value T `canoto:"field,1"`

	canotoData canotoData_GenericField
}

var _ canoto.Message = (*GenericField[*ExampleStruct0])(nil)

An example of incorrectly using canoto.FieldMaker:

type GenericField[T canoto.FieldMaker[T]] struct {
	Value T `canoto:"field,1"`

	canotoData canotoData_GenericField
}

type BadUsage interface {
	canoto.Field
	MakeCanoto() BadUsage
}

var _ canoto.Message = (*GenericField[BadUsage])(nil)

Because BadUsage is an interface, it does not have a useful zero value and will panic when GenericField attempts to call its methods.

Pass-By-Value Messages

By default, the auto-generated canotoData struct includes atomic variables. This ensures that MarshalCanoto can be called at the same time on multiple threads, which is expected for what appears to be a read only method.

However, this results in being unable to pass messages by value due to the NoCopy included on atomic variables.

Because calling MarshalCanoto concurrently with copying a message is not safe, the canoto:"nocopy" tag can be added to the canotoData field to disallow message copying.

For example:

type NotPassableByValue struct {
	Int int64 `canoto:"int,1"`

	canotoData canotoData_NotPassableByValue `canoto:"nocopy"`
}

Standalone Implementations

In some instances, it may be desirable for the generated code to avoid introducing the dependency on this repo into the go.mod file. As an example, if the user must support having multiple versions of canoto utilized in the same application.

There are two CLI flags that enable using canoto without impacting the go.mod.

1. --library when specified generates the canoto library in the provided folder. For example --library="./internal" generates the canoto library in the ./internal/canoto package.
2. --import specifies the canoto library to depend on in any generated code.

For example:

canoto --library="./internal" --import="github.com/StephenButtolph/canoto/internal/canoto" ./canoto.go

Will generate the canoto library in ./internal/canoto and will import "github.com/StephenButtolph/canoto/internal/canoto" rather than the default "github.com/StephenButtolph/canoto" when generating ./canoto.canoto.go.

Supported Types

Non-standard encoding

It is valid to define a Field that implements a non-standard format. However, this format should still be canonical and the corresponding Proto file should report opaque bytes.

Why not Proto?

Proto is a fast, compact, encoding format with extensive language support. However, Proto is not canonical.

Proto is designed to be forwards-compatible. Almost by definition, a forwards-compatible serialization format can not be canonical. The format of a field can not validated to be canonical if the expected type of the field is not known during decoding.

Why is being canonical important?

In some cases, non-canonical serialization formats are subtle to work with.

For example, if the hash of the serialized data is important or if the serialized data is cryptographically signed.

In order to ensure that the hash of the serialized data does not change, it is important to carefully avoid re-serializing a message that was previously serialized.

For canonical serialization formats, the hash of the serialized data is guaranteed never to change. Every correct implementation of the format will produce the same hash.

Why be read compatible with Proto?

By being read compatible with Proto, users of the Canoto format inherit some Proto's cross language support.

If an application only needs to read Canoto messages, but not write them, it can simply treat the Canoto message as a Proto message.

Is Canoto Fast?

Canoto is typically more performant for both serialization and deserialization than Proto. However, Proto does not typically validate that fields are canonical. If a field is expensive to inspect, it's possible Canoto can be slightly slower.

Canoto is optimized to perform no unnecessary memory allocations, so careful management to ensure messages are stack allocated can significantly improve performance over Proto.

Is Canoto Forwards Compatible?

No. Canoto chooses to be a canonical serialization format rather than being forwards compatible.