filed

File d'attente

File d'attente (queue in French) is a concurrent, file-based job queue, written in Go.

File d'attente uses files and directories for queue manipulation. Create a job with "printf cmd > /pending/$id", view running jobs with "ls /active", and restart a failed job with "mv /failed/$id /pending".

The tool is intended for single-server workloads, as a companion queue to another application. File d'attente comes with sandboxing, automatic retries, timeout, and backoff built-in.

Installation

File d'attente is built in Go and depends on sqlite and fuse (make sure fusermount is available in path).

$ git clone https://sr.ht/~marcc/filed/
$ cd filed
$ go install
$ go install cmd/filed-launch.go

To build the docs you need scdoc

$ for f in filed*.scd; do
    scdoc < "$f" > "${f%.scd}"
done
# mv filed.5 /usr/local/man/man5
# mv filed.config.5 /usr/local/man/man5
# mv filed-launch.1 /usr/local/man/man1

Getting started

It is recommended to read the man pages for more complete documentation and security considerations, but below is a small example to get you started.

filed requires a job directory and a state file location (defaulting to XDG_DATA_HOME). Afterward, you can start the daemon:

$ mkdir /tmp/filed-jobs
$ filed -rof "/usr/bin/echo" -ro "/lib" /tmp/filed-jobs

filed mounts the directory filed-jobs and exposes a few files and directories. With the above script, each job will launch in a sandboxed-mode and only have access to echo and lib.

A job can then be added by creating a file in the newly available pending directory:

$ printf "echo 'hello world'" > /tmp/filed-jobs/pending/1

If all went well, you can see the job output in /complete:

$ cat /tmp/filed-jobs/complete/1
>>> echo 'hello world'
hello world

By default, a job retries 3 times, and if unsuccessful, gets moved to /failed. You can inspect the logs to see what went wrong:

$ printf "ech this-will-fail" > /tmp/filed-jobs/pending/2
# Wait for a bit until it finishes retrying
$ cat /tmp/filed-jobs/failed/2
>>> ech this-will-fail
sh: 1: ech: not found


[System Error]: exit status 127

You can restart a job by moving the job back to pending:

$ mv /tmp/filed-jobs/failed/2 /tmp/filed-jobs/pending

Finally, if you want to remove completed or failed jobs:

$ rm /tmp/filed-jobs/failed/2

Documentation, security considerations, maintenance etc.

Available in the manpages:

• filed.5
• filed.config.5
• filed-launch.1

Design & Motivation

I wanted to create a queue that would be easy to use for self-hosted web applications, that could be used by any programming language. I also wanted to make it easy for admins to understand why a job fails, and to rerun jobs if there is an error.

I was inspired by 9p, and files proved to be a great abstraction since directories model state transitions quite well. File d'attente makes it very easy to inspect the state, without needing to build an admin portal with separate sign in. Instead, all admin operations can be done by just SSHing into the server, and the operations for manipulating, securing and automating the system become very intuitive. The source code can then be very slimmed down, while still packing a lot of features.

TODO

• Support chmod and chown
• State is configured via environment variable
• Customizable backoff and timeout before retries
• Last modified and created at are correctly rendered for jobs
• "Landlock"-mode for sandboxing
  • Add filed-launch - a script that can be used to restrict command access
  • Add command arguments to filed to lock it down, but still allow it access to state files, and remove that access in filed-launch
• Support landlock cli to only take -ro or -rw, use stat to determine if it's a file.
• Support network restrictions
• A reusable systemd unit file
• Notification on failure. Unfortunately inotify does not work with fuse, which would have been elegant otherwise.
• Notify forget and other updates.
• Package for Alpine Linux (with reusable openrc script)
• Add support for removing/moving active jobs
  • When moved to failed, the job should be killed immediately
  • When removed, the job should be killed immediately

Contributing

bugs/patches can be submitted by email to ~marcc/public-inbox@lists.sr.ht

Status

File d'attente is tested, but not battle-tested. There are probably quite a few warts and inefficiencies.

Alternatives

• nq - nq is simpler and not a persistent process, but does not feature retries. They serve different purposes: nq for ad-hoc queuing of command lines. filed serves well as a job manager for your server, where you want admins to see jobs and be able to rerun them.
• task-spooler - ts has better control over how you want the task executed (GPU or CPU), and a lot of other features. It does (AFAIK) not support retries, which are supported in filed.
• bull - bull is only for node and javascript. It features a graphical UI, and a few other features not found in filed. filed eschews a GUI in favor of simple files, allowing it to better interoperate with other systems, and allows it to use regular unix permissions for access management.
• sqs - requires you to setup most infrastructure around retries yourself. sqs is far more complex, more focused on message passing, harder to inspect, but far more flexible. Sqs scales better and fits more workloads.