metalcore

metalcore

Metalcore is the many-task computing framework for the cloud age!

"... but what does that mean?". Glad you asked, metalcore's goals and reasons for existing are:

• to provide a simple way to quickly submit tasks and receive results from any pool size of compute resources
• to reduce your TCO (license costs, cloud resources, stack dependencies)
• to be hardware, virtualization, and containerization wise as agnostic as possible (runs everywhere [Linux, Windows, MacOS, amd64, arm, arm64, ...], can use everything [CPUs, GPUs, FPGAs, ...])
• to scale billable cloud resources aligned to actual demand and/or upfront plan, and to do so quickly
• to be highly available and fault tolerant, so that you can use the cheapest metal to run your workloads on (think: spot instances)
• to be easy to understand, deploy, operate, build, and maintain
• to be lean in protocols used and fast in execution = being a viable solution for low-latency workloads

"Alright, sounds good, what can I use it for?". Broadly speaking, everything that benefits from massive scale, horizontally or vertically. At its core (pun intended), metalcore implements a work/task queue and distributes the tasks from the client machines to worker nodes, and submits the results back to the clients.

Some usage examples are:

• financial risk calculations, like Monte Carlo and others
• video or photo transformation for large scale sites and applications

Feature highlights:

• Support for many programming languages to implement your clients and service instances
• shared data support (tasks can share the same input, and you can update it at any time)
• batching of tasks and results, to save on network round-trips
• ... + prefetching of entire batches of tasks and results to save even more
• configuration via simple yaml file per application
• single binaries compiled for many common OS/arch combinations, no dependencies

Documentation

You can find the latest documentation built and ready to read on readthedocs. The documentation source files can be found under docs here in the repo.