slogparquet

slog: Parquet handler

A parquet Handler for slog Go library.

See also:

• slog-multi: slog.Handler chaining, fanout, routing, failover, load balancing...
• slog-formatter: slog attribute formatting
• slog-sampling: slog sampling policy
• slog-gin: Gin middleware for slog logger
• slog-echo: Echo middleware for slog logger
• slog-fiber: Fiber middleware for slog logger
• slog-datadog: A slog handler for Datadog
• slog-rollbar: A slog handler for Rollbar
• slog-sentry: A slog handler for Sentry
• slog-syslog: A slog handler for Syslog
• slog-logstash: A slog handler for Logstash
• slog-fluentd: A slog handler for Fluentd
• slog-graylog: A slog handler for Graylog
• slog-loki: A slog handler for Loki
• slog-slack: A slog handler for Slack
• slog-telegram: A slog handler for Telegram
• slog-mattermost: A slog handler for Mattermost
• slog-microsoft-teams: A slog handler for Microsoft Teams
• slog-webhook: A slog handler for Webhook
• slog-kafka: A slog handler for Kafka
• slog-parquet: A slog handler for Parquet + Object Storage
• slog-zap: A slog handler for Zap
• slog-zerolog: A slog handler for Zerolog
• slog-logrus: A slog handler for Logrus

🚀 Install

go get github.com/samber/slog-parquet

Compatibility: go >= 1.21

No breaking changes will be made to exported APIs before v2.0.0.

💡 Usage

GoDoc: https://pkg.go.dev/github.com/samber/slog-parquet

Handler options

type Option struct {
    // log level (default: debug)
	Level slog.Leveler

	// parquet rows buffer
	Buffer slogparquet.ParquetBuffer

	// optional: customize json payload builder
	Converter Converter
}

Parquet buffer

func NewParquetBuffer(bucket objstore.Bucket, prefix string, maxRecords int, maxInterval time.Duration) slogparquet.ParquetBuffer

Attributes will be injected in log payload.

Object storage

See github.com/thanos-io/objstore.

Example

import (
	"log/slog"

	slogparquet "github.com/samber/slog-parquet"
	"github.com/thanos-io/objstore/providers/s3"
)

func main() {
    bucket, _ := s3.NewBucketWithConfig(
        slogparquet.NewLogger(),
        s3.Config{
            Endpoint:  os.Getenv("AWS_S3_ENDPOINT"),
            Region:    os.Getenv("AWS_S3_REGION"),
            Bucket:    os.Getenv("AWS_S3_BUCKET"),
            AccessKey: os.Getenv("AWS_ACCESS_KEY"),
            SecretKey: os.Getenv("AWS_SECRET_KEY"),
            PartSize:  16 * 1024 * 1024, // 16MB
        },
        "logger",
    )

    buffer := slogparquet.NewParquetBuffer(bucket, "api/logs", 10*1024*1024)

    logger := slog.New(slogparquet.Option{Level: slog.LevelDebug, Buffer: buffer}.NewParquetHandler())
    logger = logger.
        With("environment", "dev").
        With("release", "v1.0.0")

    // log error
    logger.
        With("category", "sql").
        With("query.statement", "SELECT COUNT(*) FROM users;").
        With("query.duration", 1*time.Second).
        With("error", fmt.Errorf("could not count users")).
        Error("caramba!")

    // log user signup
    logger.
        With(
            slog.Group("user",
                slog.String("id", "user-123"),
                slog.Time("created_at", time.Now()),
            ),
        ).
        Info("user registration")

    buffer.Flush(true)
    bucket.Close()
}

Output:

$ parquet meta ~/Downloads/00_17_08.d4d9f.parquet

File path:  /Users/samber/Downloads/00_17_08.d4d9f.parquet
Created by: github.com/samber/slog-parquet version (devel)(build )
Properties: (none)
Schema:
message log {
  required int64 time (TIMESTAMP(NANOS,true));
  required binary log_level (STRING);
  required binary message (STRING);
  required binary attributes;
  required binary source (STRING);
}


Row group 0:  count: 2  279.00 B records  start: 51  total(compressed): 558 B total(uncompressed):644 B 
--------------------------------------------------------------------------------
            type      encodings count     avg size   nulls   min / max
time        INT64     F   _     2         22.50 B            "2023-08-19T00:17:08.14408..." / "2023-08-19T00:17:08.14420..."
log_level   BINARY    F         2         26.50 B            "ERROR" / "INFO"
message     BINARY    F         2         35.00 B            "caramba!" / "user registration"
attributes  BINARY    F         2         155.50 B           "0x7B2263617465676F7279223..." / "0x7B22656E7669726F6E6D656..."
source      BINARY    F         2         39.50 B            "samber/slog-parquet" / "samber/slog-parquet"

🤝 Contributing

• Ping me on twitter @samuelberthe (DMs, mentions, whatever :))
• Fork the project
• Fix open issues or request new features

Don't hesitate ;)

# Install some dev dependencies
make tools

# Run tests
make test
# or
make watch-test

👤 Contributors

💫 Show your support

Give a ⭐️ if this project helped you!

📝 License

Copyright © 2023 Samuel Berthe.

This project is MIT licensed.