span

Span

The span tools convert to and from an intermediate schema and support license tagging and quality assurance.

The intermediate schema is a normalization vehicle, spec: https://github.com/ubleipzig/intermediateschema

Install with

$ go get github.com/miku/span/cmd/...

or via deb or rpm packages.

Formats

• CrossRef API, works and members
• JATS Journal Archiving and Interchange Tag Set, with various flavours for JSTOR and others
• DOAJ exports
• FINC Intermediate Format
• Various FINC SOLR Schema
• GENIOS Profile XML
• Elsevier Transport
• Thieme TM Style
• Formeta
• IEEE IDAMS Exchange V2.0.0

Also:

• KBART

Addings data sources

The following kinds of data shapes are supported at the moment:

• A stream of XML, containing zero, one or more records, identified by an XML tag. Moderately fast.
• Newline delimited JSON, containing zero, one or more records, one record per line. Fast.
• Single records of arbitrary shape. Slow.

Use span, if metafacture or jq or a Python snippet are not sufficient.

Steps:

• Add a new subpackage for your format, e.g. dummy.
• Add a struct representing the original record (XML, JSON, bytes).
• Implement the conversion functions required, e.g. ToIntermediateSchema
• Add an entry into the format map for span-import
• Decide, which kind of source this is (XML stream, newline delimited JSON, single records, something else)
• Recompile and ship.

Ideas for span 0.2.0

TODO:

• Do not require recompilation for mapping updates (allow various sources)
• Decouple format from source. Things like SourceID and MegaCollection are per source, not format.
• Allow loadable assets from ~/.config/span/maps, some specified location or a single JSON file.

More taggable formats:

Let formats implement a single function interface, that takes a filter value. How to make this filter generic? Or use Tag as an adapeter?

// Tag alters the document by applying a filter to it.
func (doc *Document) Tag(f Filter) error {
    // Adapt.
    record = FilterRecord{
	ISSN: doc.ISSN,
	Title: doc.CombinedTitle(),
	// ...
    }
    // Apply.
    return f.Apply(record)
}

DONE:

• Reuse more generic code, e.g. parallel
• Make conversions a simpler with xmlstream, zek

Licence

• GPLv3
• This project uses the Compact Language Detector 2 - CLD2, Apache License Version 2.0