apoco

package
v0.0.29 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 16, 2021 License: MIT Imports: 20 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AgreeingOCRs 

func AgreeingOCRs(t T, i, n int) (float64, bool)

AgreeingOCRs returns the number of OCRs that aggree with the master OCR token.

func ApplyOCRToCorrection 

func ApplyOCRToCorrection(ocr, sug string) string

ApplyOCRToCorrection applies the casing of the master OCR string to the correction's candidate suggestion and prepends and appends any punctuation of the master OCR to the suggestion.

func CandidateAgreeingOCR 

func CandidateAgreeingOCR(t T, i, n int) (float64, bool)

CandidateAgreeingOCR returns the number of OCR tokens that agree with the specific profiler candidate of the token.

func CandidateHistPatternConf 

func CandidateHistPatternConf(t T, i, n int) (float64, bool)

CandidateHistPatternConf returns the product of the confidences of the primary OCR characters for the assumed historical rewrite pattern of the connected candidate.

func CandidateHistPatternConfLog added in v0.0.17 

func CandidateHistPatternConfLog(t T, i, n int) (float64, bool)

CandidateHistPatternConfLog returns the sum of the logrithm of the confidences of the primary OCR characters for the assumed historical rewrite pattern of the connected candidate.

func CandidateLen 

func CandidateLen(t T, i, n int) (float64, bool)

CandidateLen returns the length of the connected profiler candidate.

func CandidateLevenshteinDist 

func CandidateLevenshteinDist(t T, i, n int) (float64, bool)

CandidateLevenshteinDist returns the levenshtein distance between the OCR token and the token's connected profiler candidate. For the master OCR the according Distance from the profiler candidate is used, whereas for support OCRs the levenshtein distance is calculated.

func CandidateMatchesOCR 

func CandidateMatchesOCR(t T, i, n int) (float64, bool)

CandidateMatchesOCR returns true if the according ocr matches the connected candidate and false otherwise.

func CandidateMaxTrigramFreq 

func CandidateMaxTrigramFreq(t T, i, n int) (float64, bool)

CandidateMaxTrigramFreq returns the maximal trigram frequenzy for the connected candidate.

func CandidateMinTrigramFreq 

func CandidateMinTrigramFreq(t T, i, n int) (float64, bool)

CandidateMinTrigramFreq returns the minimal trigram frequezy for the connected candidate.

func CandidateOCRPatternConf 

func CandidateOCRPatternConf(t T, i, n int) (float64, bool)

CandidateOCRPatternConf returns the product of the confidences of the primary OCR characters for the assumed OCR error pattern of the connected candidate. TODO: rename to CandiateErrPatternConf

func CandidateOCRPatternConfLog added in v0.0.16 

func CandidateOCRPatternConfLog(t T, i, n int) (float64, bool)

CandidateOCRPatternConfLog returns the sum of the logarithm of the confidences of the primary OCR characters for the assumed OCR error pattern of the connected candidate.

func CandidateProfilerWeight 

func CandidateProfilerWeight(t T, i, n int) (float64, bool)

CandidateProfilerWeight returns the profiler confidence value for tokens candidate.

func CandidateTrigramFreq 

func CandidateTrigramFreq(t T, i, n int) (float64, bool)

CandidateTrigramFreq returns the product of the candidate's trigrams.

func CandidateTrigramFreqLog added in v0.0.17 

func CandidateTrigramFreqLog(t T, i, n int) (float64, bool)

CandidateTrigramFreqLog returns the product of the candidate's trigrams.

func CandidateUnigramFreq 

func CandidateUnigramFreq(t T, i, n int) (float64, bool)

CandidateUnigramFreq returns the relative frequency of the token's candidate.

func DocumentLexicality 

func DocumentLexicality(t T, i, n int) (float64, bool)

DocumentLexicality returns the (global) lexicality of the given token's document. Using this feature only makes sense if the training contains at least more than one training document.

func EachToken 

func EachToken(ctx context.Context, in <-chan T, f func(T) error) error

EachToken iterates over the tokens in the input channel and calls the callback function for each token.

func EachTokenGroup added in v0.0.6 

func EachTokenGroup(ctx context.Context, in <-chan T, f func(string, ...T) error) error

EachTokenGroup iterates over the tokens grouping them together based on their groups. The given callback function is called for each group of tokens.

func EachTokenLM added in v0.0.29 

func EachTokenLM(ctx context.Context, in <-chan T, f func(*LanguageModel, ...T) error) error

EachTokenLM iterates over the tokens grouping them together based on their language models. The given callback function is called for each group of tokens. This function assumes that the tokens are connected with a language model.

func EachTrigram added in v0.0.18 

func EachTrigram(str string, f func(string))

EachTrigram calls the given callback function for each trigram in the given string.

func Log added in v0.0.21 

func Log(f string, args ...interface{})

Log logs the given message if logging is enabled. This function uses log.Printf for logging, so it is save to be used concurrently.

func LogEnabled added in v0.0.29 

func LogEnabled() bool

LogEnabled returns true if logging is currently enabled.

func OCRLevenshteinDist added in v0.0.17 

func OCRLevenshteinDist(t T, i, n int) (float64, bool)

OCRLevenshteinDist returns the levenshtein distance between the secondary OCRs with the primary OCR.

func OCRMaxCharConf 

func OCRMaxCharConf(t T, i, n int) (float64, bool)

OCRMaxCharConf returns the maximal character confidence of the master OCR token.

func OCRMaxTrigramFreq 

func OCRMaxTrigramFreq(t T, i, n int) (float64, bool)

OCRMaxTrigramFreq returns the maximal trigram relative frequenzy confidence of the tokens.

func OCRMinCharConf 

func OCRMinCharConf(t T, i, n int) (float64, bool)

OCRMinCharConf returns the minimal character confidence of the master OCR token.

func OCRMinTrigramFreq 

func OCRMinTrigramFreq(t T, i, n int) (float64, bool)

OCRMinTrigramFreq returns the minimal trigram relative frequenzy confidence of the tokens.

func OCRTokenLen 

func OCRTokenLen(t T, i, n int) (float64, bool)

OCRTokenLen returns the length of the OCR token. It operates on any configuration.

func OCRTrigramFreq 

func OCRTrigramFreq(t T, i, n int) (float64, bool)

OCRTrigramFreq returns the product of the OCR token's trigrams.

func OCRUnigramFreq 

func OCRUnigramFreq(t T, i, n int) (float64, bool)

OCRUnigramFreq returns the relative frequency of the OCR token in the unigram language model.

func Pipe 

func Pipe(ctx context.Context, fns ...StreamFunc) error

Pipe pipes multiple stream funcs together, making shure to run all of them concurently. The first function in the list (the reader) is called with a nil input channel. The last function is always called with a nil output channel. To clarify: the first function must never read from its input channel and the last function must never write to its output channel.

StreamFunctions should transform the input tokens to output tokens. They must never close any channels. They should use the SendTokens, ReadToken and EachToken utility functions to ensure proper handling of context cancelation.

func RankingCandidateConfDiffToNext 

func RankingCandidateConfDiffToNext(t T, i, n int) (float64, bool)

RankingCandidateConfDiffToNext returns the top ranked candidate's weight minus the the weight of the next (or 0).

func RankingConf 

func RankingConf(t T, i, n int) (float64, bool)

RankingConf returns the confidence of the best ranked correction candidate for the given token.

func RankingConfDiffToNext 

func RankingConfDiffToNext(t T, i, n int) (float64, bool)

RankingConfDiffToNext returns the difference of the best ranked correction candidate's confidence to the next. If only one correction candidate is available, the next ranking's confidence is assumed to be 0.

func ReadProfile added in v0.0.17 

func ReadProfile(name string) (gofiler.Profile, error)

ReadProfile reads the profile from a gzipped json formatted file.

func RunProfiler added in v0.0.17 

func RunProfiler(ctx context.Context, exe, config string, tokens ...T) (gofiler.Profile, error)

RunProfiler runs the profiler over the given tokens (using the token entries at index 0) with the given executable and config file. The profiler's output is logged to stderr.

func SendTokens 

func SendTokens(ctx context.Context, out chan<- T, tokens ...T) error

SendTokens writes tokens into the given output channel. This function should always be used to write tokens into output channels.

func SetLog added in v0.0.19 

func SetLog(enable bool)

SetLog enables or disables logging. This function is not safe for concurrent usage and should be used once at application start.

func WriteProfile added in v0.0.17 

func WriteProfile(name string, profile gofiler.Profile) error

WriteProfile writes the profile as gzipped json formatted file.

Types

type Char 

type Char struct {
	Conf float64 // confidence of the rune
	Char rune    // rune
}

Char represents an OCR char with its confidence.

func (Char) String 

func (char Char) String() string

type Chars 

type Chars []Char

Chars represents the master OCR chars with the respective confidences.

func (Chars) Confs added in v0.0.9 

func (chars Chars) Confs() []float64

Confs returns the confidences as array.

func (Chars) String 

func (chars Chars) String() string

String converts a char array to a string.

type Correction 

type Correction struct {
	Candidate *gofiler.Candidate
	Conf      float64
}

Correction represents a correction decision for tokens.

type FeatureFunc 

type FeatureFunc func(t T, i, n int) (float64, bool)

FeatureFunc defines the function a feature needs to implement. A feature func gets a token and a configuration (the current OCR-index i and the total number of parallel OCRs n). The function then should return the feature value for the given token and wether this feature applies for the given configuration (i and n).

type FeatureSet 

type FeatureSet []FeatureFunc

FeatureSet is just a list of feature funcs.

func NewFeatureSet 

func NewFeatureSet(names ...string) (FeatureSet, error)

NewFeatureSet creates a new feature set from the list of feature function names.

func (FeatureSet) Calculate 

func (fs FeatureSet) Calculate(xs []float64, t T, n int) []float64

Calculate calculates the feature vector for the given feature functions for the given token and the given number of OCRs and appends it to the given vector. Any given feature function that does not apply to the given configuration (and returns false as it second return parameter for the configuration) is omitted and not appended to the resulting feature vector.

func (FeatureSet) Names added in v0.0.21 

func (fs FeatureSet) Names(names []string, nocr int, dm bool) []string

Names returns the names of the features including the features for different values of OCR's. This function panics if the length of the feature set differs from the length of the given feature names.

type FreqList 

type FreqList struct {
	FreqList map[string]int `json:"freqList"`
	Total    int            `json:"total"`
}

FreqList is a simple frequenzy map.

type LanguageModel 

type LanguageModel struct {
	Ngrams     FreqList
	Unigrams   FreqList
	Profile    gofiler.Profile
	Lexicality float64
}

LanguageModel consists of holds the language model for tokens.

func (*LanguageModel) AddUnigram 

func (lm *LanguageModel) AddUnigram(token string)

AddUnigram adds the token to the language model's unigram map.

func (*LanguageModel) EachTrigram 

func (lm *LanguageModel) EachTrigram(str string, f func(float64))

EachTrigram looks up the trigrams of the given token and returns the product of the token's trigrams.

func (*LanguageModel) LoadGzippedNGram 

func (lm *LanguageModel) LoadGzippedNGram(path string) error

LoadGzippedNGram loads the (gzipped) ngram model file. The expected format for each line is `%d,%s`.

func (*LanguageModel) ReadProfile added in v0.0.29 

func (lm *LanguageModel) ReadProfile(ctx context.Context, exe, config string, cache bool, tokens ...T) error

ReadProfile loads the profile for the master OCR tokens.

func (*LanguageModel) Trigram 

func (lm *LanguageModel) Trigram(str string) float64

Trigram looks up the trigrams of the given token and returns the product of the token's trigrams.

func (*LanguageModel) TrigramLog added in v0.0.17 

func (lm *LanguageModel) TrigramLog(str string) float64

TrigramLog looks up the trigrams of the given token and returns the sum of the logarithmic relative frequency of the token's trigrams.

func (*LanguageModel) Unigram 

func (lm *LanguageModel) Unigram(str string) float64

Unigram looks up the given token in the unigram list (or 0 if the unigram is not present).

type Model 

type Model struct {
	Models             map[string]map[int]ModelData
	GlobalHistPatterns map[string]float64
	GlobalOCRPatterns  map[string]float64
	Ngrams             FreqList
}

Model holds the different models for the different number of OCRs.

func ReadModel 

func ReadModel(model, ngrams string) (Model, error)

ReadModel reads a model from a gob compressed input file. If the given file does not exist, an empty model is returned. If the model does not contain a valid ngram frequency list, the list is loaded from the given path.

func (Model) Get 

func (m Model) Get(mod string, nocr int) (*ml.LR, FeatureSet, error)

Get loads the the model and the according feature set for the given configuration.

func (Model) Put 

func (m Model) Put(mod string, nocr int, lr *ml.LR, fs []string)

Put inserts the weights and the according feature set for the given configuration into this model.

func (Model) Write 

func (m Model) Write(name string) (err error)

Write writes the model as gob encoded, gziped file to the given path overwriting any previous existing models.

type ModelData added in v0.0.7 

type ModelData struct {
	Features []string
	Model    *ml.LR
}

ModelData holds a linear regression model.

type Ranking 

type Ranking struct {
	Candidate *gofiler.Candidate
	Prob      float64
}

Ranking maps correction candidates of tokens to their predicted probabilities.

type StreamFunc 

type StreamFunc func(context.Context, <-chan T, chan<- T) error

StreamFunc is a type def for stream functions. A stream function is used to transform tokens from the input channel to the output channel. They should be used with the Pipe function to chain multiple functions together.

func Combine added in v0.0.14 

func Combine(ctx context.Context, fns ...StreamFunc) StreamFunc

Combine lets you combine stream functions. All functions are run concurently in their own error group.

func ConnectCandidates 

func ConnectCandidates() StreamFunc

ConnectCandidates returns a stream function that connects tokens with their respective candidates to the stream. Tokens with no candidates or tokens with only a modern interpretation are filtered from the stream.

func ConnectCorrections 

func ConnectCorrections(lr *ml.LR, fs FeatureSet, n int) StreamFunc

ConnectCorrections connects the tokens with the decider's correction decisions.

func ConnectLM 

func ConnectLM(ngrams FreqList) StreamFunc

ConnectLM connects the language model to the tokens.

func ConnectProfile added in v0.0.29 

func ConnectProfile(exe, config string, cache bool) StreamFunc

ConnectProfiler connects the profile with the tokens. This function must be called after ConnectLM.

func ConnectRankings 

func ConnectRankings(lr *ml.LR, fs FeatureSet, n int) StreamFunc

ConnectRankings connects the tokens of the input stream with their respective rankings.

func ConnectUnigrams added in v0.0.29 

func ConnectUnigrams() StreamFunc

ConnectUnigrams adds the unigrams to the tokens's language model.

func FilterBad 

func FilterBad(min int) StreamFunc

FilterBad returns a astream function that filters tokens with not enough ocr and/or gt tokens.

func FilterLexiconEntries 

func FilterLexiconEntries() StreamFunc

FilterLexiconEntries returns a stream function that filters all tokens that are lexicon entries from the stream.

func FilterShort 

func FilterShort(min int) StreamFunc

FilterShort returns a stream function that filters short master OCR tokens from the input stream. Short tokens are tokens, with less than min unicode characters.

func Normalize 

func Normalize() StreamFunc

Normalize returns a stream function that trims all leading and subsequent punctionation from the tokens, converts them to lowercase and replaces any whitespace (in the case of merges due to alignment) with a '_'.

func Tee added in v0.0.14 

func Tee(fns ...func(T) error) StreamFunc

Tee calls all the given callback function for each token. After all functions have been called, if the output channel is not nil, the token is send to the output channel.

type T added in v0.0.14 

type T struct {
	LM      *LanguageModel // language model for this token
	Payload interface{}    // token payload; *gofiler.Candidate, []Ranking or Correction
	Cor     string         // Correction for the token
	File    string         // the file of the token
	Group   string         // file group of the token
	ID      string         // id of the token in this file
	Chars   Chars          // master OCR chars with their confidences
	Tokens  []string       // master and support OCRs and gt
}

T represents aligned OCR-tokens.

func ReadToken 

func ReadToken(ctx context.Context, in <-chan T) (T, bool, error)

ReadToken reads one token from the given channel. This function should alsways be used to read single tokens from input channels.

func (T) IsLexiconEntry added in v0.0.14 

func (t T) IsLexiconEntry() bool

IsLexiconEntry returns true if this token is a normal lexicon entry for its connected language model.

func (T) String added in v0.0.14 

func (t T) String() string

Source Files

View all Source files

Directories

Path Synopsis
Package node provides helper functions to work with queryxml.Node pointers.
 Package node provides helper functions to work with queryxml.Node pointers.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.