Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Chain ¶
type Chain[T models.TokenConstraint] struct { // contains filtered or unexported fields }
The Chain contains a routine to process data. Chains will be running in parallel. Each routine is built from a Reader and/or Processor and/or Writer.
func NewReaderChain ¶
func NewReaderChain[T models.TokenConstraint](r Reader[T], p Processor[T]) (*Chain[T], chan T)
NewReaderChain returns a new Chain with a Reader and a Processor, and communication channel for backup operation.
func NewWriterChain ¶
func NewWriterChain[T models.TokenConstraint](w Writer[T], limiter *bandwidth.Limiter) (*Chain[T], chan T)
NewWriterChain returns a new Chain with a Writer, and communication channel for backup operation.
type Fanout ¶
type Fanout[T models.TokenConstraint] struct { Inputs []chan T Outputs []chan T // contains filtered or unexported fields }
Fanout routes messages between chain pools. FanoutStrategy controls the distribution of messages to output channels.
func NewFanout ¶
func NewFanout[T models.TokenConstraint]( inputs []chan T, outputs []chan T, strategy FanoutStrategy, ) (*Fanout[T], error)
NewFanout returns a new Fanout.
func (*Fanout[T]) GetMetrics ¶
GetMetrics returns the accumulated length for input and output channels.
type FanoutStrategy ¶
type FanoutStrategy int
FanoutStrategy represents a pipeline routing strategy.
const ( // Fixed strategy routes incoming tokens to output channels, establishing a // dedicated one-to-one mapping between input channels and output channels. // All tokens read from a specific input channel are routed to its pre-assigned // output channel. The number of output channels must equal the number of input // channels being processed. Fixed FanoutStrategy = iota // RoundRobin distributes incoming tokens between available output channels in a // fair, rotating manner. RoundRobin // Split routes incoming tokens to output channels using a custom routing function. // The routing function determines the destination channel for each token based on // its partition id. Split )
type Pipe ¶
type Pipe[T models.TokenConstraint] struct { // contains filtered or unexported fields }
Pipe is running and managing everything.
func NewPipe ¶
func NewPipe[T models.TokenConstraint]( pc ProcessorCreator[T], readers []Reader[T], writers []Writer[T], limiter *bandwidth.Limiter, strategy FanoutStrategy, ) (*Pipe[T], error)
NewPipe creates cew backup pipeline.
func (*Pipe[T]) GetMetrics ¶
GetMetrics returns the accumulated length for input and output channels.
type Pool ¶
type Pool[T models.TokenConstraint] struct { Chains []*Chain[T] // Outputs and Inputs are mutually exclusive. Inputs []chan T Outputs []chan T }
Pool is a pool of chains. All chains in a pool are running in parallel. Pools are communicating via fanout.
func NewReaderPool ¶
func NewReaderPool[T models.TokenConstraint](readers []Reader[T], pc ProcessorCreator[T]) *Pool[T]
NewReaderPool returns a new pool of Reader and Processor chains for backup operations, with the specified parallelism.
func NewWriterPool ¶
func NewWriterPool[T models.TokenConstraint](writers []Writer[T], limiter *bandwidth.Limiter) *Pool[T]
NewWriterPool creates a new pool of Writer chains for backup operations, with the specified parallelism and bandwidth.
type Processor ¶
type Processor[T models.TokenConstraint] interface { Process(T) (T, error) }
Processor describes data processors.
type ProcessorCreator ¶
type ProcessorCreator[T models.TokenConstraint] func() Processor[T]
ProcessorCreator is a function type that defines a creator for a Processor.
Source Files
Directories