README
¶
Kronk
[!IMPORTANT] The Kronk provider for Fantasy is considered experimental! The API and behavior can change until we consider it stable.
Kronk is a Go package by ArdanLabs that uses yzma under the hood to provide hardware accelerated local inference with llama.cpp directly integrated into your applications. Kronk provides a high-level API that feels similar to using an OpenAI compatible API.
When using the Kronk provider in Fantasy, you only need to specify the model you want to use and it'll be automatically downloaded in your machine and used for inference.
To see which models are available for you, see the Kronk Catalog.
Examples on how to use it are available in examples/kronk.
Documentation
¶
Overview ¶
Package kronk provides an implementation of the fantasy AI SDK for local models using the Kronk SDK.
Index ¶
- Constants
- func DefaultMapFinishReasonFunc(finishReason string) fantasy.FinishReason
- func DefaultPrepareCallFunc(_ fantasy.LanguageModel, d model.D, call fantasy.Call) ([]fantasy.CallWarning, error)
- func DefaultToPrompt(prompt fantasy.Prompt, _ string, _ string) ([]model.D, []fantasy.CallWarning)
- func FmtLogger(_ context.Context, msg string, args ...any)
- func New(opts ...Option) (fantasy.Provider, error)
- func NewProviderOptions(opts *ProviderOptions) fantasy.ProviderOptions
- type LanguageModelMapFinishReasonFunc
- type LanguageModelOption
- func WithLanguageModelMapFinishReasonFunc(fn LanguageModelMapFinishReasonFunc) LanguageModelOption
- func WithLanguageModelObjectMode(om fantasy.ObjectMode) LanguageModelOption
- func WithLanguageModelPrepareCallFunc(fn LanguageModelPrepareCallFunc) LanguageModelOption
- func WithLanguageModelToPromptFunc(fn LanguageModelToPromptFunc) LanguageModelOption
- type LanguageModelPrepareCallFunc
- type LanguageModelToPromptFunc
- type Logger
- type Option
- type ProviderMetadata
- type ProviderOptions
Constants ¶
const ( TypeProviderOptions = Name + ".options" TypeProviderMetadata = Name + ".metadata" )
Global type identifiers for Kronk-specific provider data.
const (
// Name is the name of the Kronk provider.
Name = "kronk"
)
Variables ¶
This section is empty.
Functions ¶
func DefaultMapFinishReasonFunc ¶
func DefaultMapFinishReasonFunc(finishReason string) fantasy.FinishReason
DefaultMapFinishReasonFunc is the default implementation for mapping finish reasons.
func DefaultPrepareCallFunc ¶
func DefaultPrepareCallFunc(_ fantasy.LanguageModel, d model.D, call fantasy.Call) ([]fantasy.CallWarning, error)
DefaultPrepareCallFunc is the default implementation for preparing a call to the language model.
func DefaultToPrompt ¶
DefaultToPrompt is the default implementation for converting fantasy prompts to Kronk SDK messages.
func NewProviderOptions ¶
func NewProviderOptions(opts *ProviderOptions) fantasy.ProviderOptions
NewProviderOptions creates new provider options for Kronk.
Types ¶
type LanguageModelMapFinishReasonFunc ¶
type LanguageModelMapFinishReasonFunc func(finishReason string) fantasy.FinishReason
LanguageModelMapFinishReasonFunc is a function that maps the finish reason for the language model.
type LanguageModelOption ¶
type LanguageModelOption func(*languageModel)
LanguageModelOption is a function that configures a languageModel.
func WithLanguageModelMapFinishReasonFunc ¶
func WithLanguageModelMapFinishReasonFunc(fn LanguageModelMapFinishReasonFunc) LanguageModelOption
WithLanguageModelMapFinishReasonFunc sets the map finish reason function for the language model.
func WithLanguageModelObjectMode ¶
func WithLanguageModelObjectMode(om fantasy.ObjectMode) LanguageModelOption
WithLanguageModelObjectMode sets the object generation mode.
func WithLanguageModelPrepareCallFunc ¶
func WithLanguageModelPrepareCallFunc(fn LanguageModelPrepareCallFunc) LanguageModelOption
WithLanguageModelPrepareCallFunc sets the prepare call function for the language model.
func WithLanguageModelToPromptFunc ¶
func WithLanguageModelToPromptFunc(fn LanguageModelToPromptFunc) LanguageModelOption
WithLanguageModelToPromptFunc sets the to prompt function for the language model.
type LanguageModelPrepareCallFunc ¶
type LanguageModelPrepareCallFunc func(lm fantasy.LanguageModel, d model.D, call fantasy.Call) ([]fantasy.CallWarning, error)
LanguageModelPrepareCallFunc is a function that prepares the call for the language model.
type LanguageModelToPromptFunc ¶
type LanguageModelToPromptFunc func(prompt fantasy.Prompt, provider, modelID string) ([]model.D, []fantasy.CallWarning)
LanguageModelToPromptFunc is a function that handles converting fantasy prompts to Kronk SDK messages.
type Option ¶
type Option func(*options)
Option defines a function that configures Kronk provider options.
func WithLanguageModelOptions ¶
func WithLanguageModelOptions(opts ...LanguageModelOption) Option
WithLanguageModelOptions sets the language model options for the Kronk provider.
func WithLogger ¶
WithLogger sets the logger function for download progress.
func WithModelConfig ¶
WithModelConfig sets additional model configuration options.
func WithObjectMode ¶
func WithObjectMode(om fantasy.ObjectMode) Option
WithObjectMode sets the object generation mode.
type ProviderMetadata ¶
type ProviderMetadata struct {
TokensPerSecond float64 `json:"tokens_per_second"`
OutputTokens int64 `json:"output_tokens"`
}
ProviderMetadata represents additional metadata from Kronk provider.
func (ProviderMetadata) MarshalJSON ¶
func (m ProviderMetadata) MarshalJSON() ([]byte, error)
MarshalJSON implements custom JSON marshaling with type info for ProviderMetadata.
func (*ProviderMetadata) Options ¶
func (*ProviderMetadata) Options()
Options implements the ProviderOptionsData interface.
func (*ProviderMetadata) UnmarshalJSON ¶
func (m *ProviderMetadata) UnmarshalJSON(data []byte) error
UnmarshalJSON implements custom JSON unmarshaling with type info for ProviderMetadata.
type ProviderOptions ¶
type ProviderOptions struct {
TopK *int64 `json:"top_k"`
RepeatPenalty *float64 `json:"repeat_penalty"`
Seed *int64 `json:"seed"`
MinP *float64 `json:"min_p"`
NumPredict *int64 `json:"num_predict"`
Stop []string `json:"stop"`
}
ProviderOptions represents additional options for Kronk provider.
func ParseOptions ¶
func ParseOptions(data map[string]any) (*ProviderOptions, error)
ParseOptions parses provider options from a map.
func (ProviderOptions) MarshalJSON ¶
func (o ProviderOptions) MarshalJSON() ([]byte, error)
MarshalJSON implements custom JSON marshaling with type info for ProviderOptions.
func (*ProviderOptions) Options ¶
func (*ProviderOptions) Options()
Options implements the ProviderOptionsData interface.
func (*ProviderOptions) UnmarshalJSON ¶
func (o *ProviderOptions) UnmarshalJSON(data []byte) error
UnmarshalJSON implements custom JSON unmarshaling with type info for ProviderOptions.
Source Files