Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Extensions ¶
type Extensions []string
Extensions is used to tokenize snippets in directories using the list of file extensions.
func (Extensions) Tokenize ¶
func (e Extensions) Tokenize(dirs ...string) apoco.StreamFunc
Tokenize tokenizes tokens from line snippets TSV files (identyfied by the given file extensions) and alignes them accordingly. If a extension ends with `.txt`, one line is read from the text file (no confidences). Otherwise the file is read as a TSV file expecting on char and its confidence on each line.
Source Files
Click to show internal directories.
Click to hide internal directories.