Documentation
¶
Overview ¶
package iterators provide iterator implementations.
Summary ¶
An Iterator's goal is to decouple the origin of the data from the consumer who uses that data. Most commonly, iterators hide whether the data comes from a specific database, standard input, or elsewhere. This approach helps to design data consumers that are not dependent on the concrete implementation of the data source, while still allowing for the composition and various actions on the received data stream. An Iterator represents an iterable list of element, which length is not known until it is fully iterated, thus can range from zero to infinity. As a rule of thumb, if the consumer is not the final destination of the data stream, it should use the pipeline pattern to avoid bottlenecks with local resources such as memory.
Resources ¶
https://en.wikipedia.org/wiki/Iterator_pattern https://en.wikipedia.org/wiki/Pipeline_(software)
Index ¶
- Constants
- func Collect[T any](i Iterator[T]) (vs []T, err error)
- func Count[T any](i Iterator[T]) (total int, err error)
- func First[T any](i Iterator[T]) (value T, found bool, err error)
- func ForEach[T any](i Iterator[T], fn func(T) error) (rErr error)
- func Last[T any](i Iterator[T]) (value T, found bool, err error)
- func Must[T any](v T, err error) T
- func Pipe[T any]() (*PipeIn[T], *PipeOut[T])
- func Reduce[R, T any, FN ...](i Iterator[T], initial R, blk FN) (result R, rErr error)
- func Take[T any](iter Iterator[T], n int) ([]T, error)
- type CallbackOption
- type Iterator
- func Batch[T any](iter Iterator[T], size int) Iterator[[]T]
- func BatchWithTimeout[T any](i Iterator[T], size int, timeout time.Duration) Iterator[[]T]
- func BufioScanner[T string | []byte](s *bufio.Scanner, closer io.Closer) Iterator[T]
- func Empty[T any]() Iterator[T]
- func Error[T any](err error) Iterator[T]
- func Errorf[T any](format string, a ...interface{}) Iterator[T]
- func Filter[T any](i Iterator[T], filter func(T) bool) Iterator[T]
- func Func[T any](next func() (v T, ok bool, err error), callbackOptions ...CallbackOption) Iterator[T]
- func Head[T any](iter Iterator[T], n int) Iterator[T]
- func Limit[V any](iter Iterator[V], n int) Iterator[V]
- func Map[To any, From any](iter Iterator[From], transform func(From) (To, error)) Iterator[To]
- func Merge[T any](iters ...Iterator[T]) Iterator[T]
- func Offset[V any](iter Iterator[V], offset int) Iterator[V]
- func Paginate[T any](ctx context.Context, ...) Iterator[T]
- func SingleValue[T any](v T) Iterator[T]
- func Slice[T any](slice []T) Iterator[T]
- func WithCallback[T any](i Iterator[T], cs ...CallbackOption) Iterator[T]
- func WithConcurrentAccess[T any](i Iterator[T]) Iterator[T]
- func WithErr[T any](iter Iterator[T], err error) Iterator[T]
- type PipeIn
- type PipeOut
- type StubIter
Examples ¶
Constants ¶
const Break errorkit.Error = `iterators:break`
Variables ¶
This section is empty.
Functions ¶
func Count ¶
Count will iterate over and count the total iterations number
Good when all you want is count all the elements in an iterator but don't want to do anything else.
func Pipe ¶
Pipe return a receiver and a sender. This can be used with resources that
Example ¶
var ( i *iterators.PipeIn[int] o *iterators.PipeOut[int] ) i, o = iterators.Pipe[int]() _ = i // use it to send values _ = o // use it to consume values on each iteration (iter.Next())
func Reduce ¶
func Reduce[ R, T any, FN func(R, T) R | func(R, T) (R, error), ](i Iterator[T], initial R, blk FN) (result R, rErr error)
Example ¶
raw := iterators.Slice([]string{"1", "2", "42"})
_, _ = iterators.Reduce[[]int](raw, nil, func(vs []int, raw string) ([]int, error) {
v, err := strconv.Atoi(raw)
if err != nil {
return nil, err
}
return append(vs, v), nil
})
Types ¶
type CallbackOption ¶
type CallbackOption interface {
// contains filtered or unexported methods
}
func OnClose ¶
func OnClose(fn func() error) CallbackOption
type Iterator ¶
type Iterator[V any] interface { // Closer is required to make it able to cancel iterators where resources are being used behind the scene // for all other cases where the underling io is handled on a higher level, it should simply return nil io.Closer // Err return the error cause. Err() error // Next will ensure that Value returns the next item when executed. // If the next value is not retrievable, Next should return false and ensure Err() will return the error cause. Next() bool // Value returns the current value in the iterator. // The action should be repeatable without side effects. Value() V }
Iterator define a separate object that encapsulates accessing and traversing an aggregate object. Clients use an iterator to access and traverse an aggregate without knowing its representation (data structures). Interface design inspirited by https://golang.org/pkg/encoding/json/#Decoder https://en.wikipedia.org/wiki/Iterator_pattern
Example ¶
var iter iterators.Iterator[int]
defer iter.Close()
for iter.Next() {
v := iter.Value()
_ = v
}
if err := iter.Err(); err != nil {
// handle error
}
func BatchWithTimeout ¶
func BufioScanner ¶
func Empty ¶
Empty iterator is used to represent nil result with Null object pattern
Example ¶
iterators.Empty[any]()
func Error ¶
Error returns an Interface that only can do is returning an Err and never have next element
func Filter ¶
Example ¶
var iter iterators.Iterator[int]
iter = iterators.Slice([]int{0, 1, 2, 3, 4, 5, 6, 7, 8, 9})
iter = iterators.Filter[int](iter, func(n int) bool { return n > 2 })
defer iter.Close()
for iter.Next() {
n := iter.Value()
_ = n
}
if err := iter.Err(); err != nil {
log.Fatal(err)
}
func Func ¶
func Func[T any](next func() (v T, ok bool, err error), callbackOptions ...CallbackOption) Iterator[T]
Func enables you to create an iterator with a lambda expression. Func is very useful when you have to deal with non type safe iterators that you would like to map into a type safe variant. In case you need to close the currently mapped resource, use the OnClose callback option.
func Head ¶
Head takes the first n element, similarly how the coreutils "head" app works.
Example ¶
infStream := iterators.Func[int](func() (v int, ok bool, err error) {
return 42, true, nil
})
i := iterators.Head(infStream, 3)
vs, err := iterators.Collect(i)
_, _ = vs, err // []{42, 42, 42}, nil
func Map ¶
Map allows you to do additional transformation on the values. This is useful in cases, where you have to alter the input value, or change the type all together. Like when you read lines from an input stream, and then you map the line content to a certain data structure, in order to not expose what steps needed in order to deserialize the input stream, thus protect the business rules from this information.
Example ¶
rawNumbers := iterators.Slice([]string{"1", "2", "42"})
numbers := iterators.Map[int](rawNumbers, strconv.Atoi)
_ = numbers
func Paginate ¶
func Paginate[T any]( ctx context.Context, more func(ctx context.Context, offset int) (values []T, hasNext bool, _ error), ) Iterator[T]
Paginate will create an Iterator[T] which can be used like any other iterator, Under the hood the "more" function will be used to dynamically retrieve more values when the previously called values are already used up.
If the more function has a hard-coded true for the "has next page" return value, then the pagination will interpret an empty result as "no more pages left".
Example ¶
ctx := context.Background()
fetchMoreFoo := func(ctx context.Context, offset int) ([]Foo, bool, error) {
const limit = 10
query := url.Values{}
query.Set("limit", strconv.Itoa(limit))
query.Set("offset", strconv.Itoa(offset))
resp, err := http.Get("https://api.mydomain.com/v1/foos?" + query.Encode())
if err != nil {
return nil, false, err
}
var values []FooDTO
defer resp.Body.Close()
dec := json.NewDecoder(resp.Body)
dec.DisallowUnknownFields()
if err := dec.Decode(&values); err != nil {
return nil, false, err
}
vs, err := dtokit.Map[[]Foo](ctx, values)
if err != nil {
return nil, false, err
}
probablyHasNextPage := len(vs) == limit
return vs, probablyHasNextPage, nil
}
foos := iterators.Paginate(ctx, fetchMoreFoo)
_ = foos // foos can be called like any iterator,
// and under the hood, the fetchMoreFoo function will be used dynamically,
// to retrieve more values when the previously called values are already used up.
func SingleValue ¶
SingleValue creates an iterator that can return one single element and will ensure that Next can only be called once.
func WithCallback ¶
func WithCallback[T any](i Iterator[T], cs ...CallbackOption) Iterator[T]
func WithConcurrentAccess ¶
WithConcurrentAccess allows you to convert any iterator into one that is safe to use from concurrent access. The caveat with this is that this protection only allows 1 Decode call for each Next call.
type PipeIn ¶
type PipeIn[T any] struct { // contains filtered or unexported fields }
PipeIn provides access to feed a pipe receiver with entities
func (*PipeIn[T]) Close ¶
Close will close the feed and err channels, which eventually notify the receiver that no more value expected
type PipeOut ¶
type PipeOut[T any] struct { // contains filtered or unexported fields }
PipeOut implements iterator interface while it's still being able to receive values, used for streaming
func (*PipeOut[T]) Close ¶
Close sends a signal back that no more value should be sent because receiver stops listening
func (*PipeOut[T]) Err ¶
Err returns an error object that the pipe sender wants to present for the pipe receiver
Source Files
Directories