llcm

package module
v0.0.16 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 7, 2025 License: MIT Imports: 27 Imported by: 0

README

llcm

CI Go Report Card GitHub GitHub

llcm is a CLI tool to manage the lifecycle of Amazon CloudWatch Logs log groups. List, update, and delete log groups to manage their lifecycle. It handles multiple regions at high speed while avoiding throttling errors. It can also return simulation results based on desired states.

Features

  • List: Fast listing of log groups for specified multiple regions.
  • Preview: By passing the desired state as an argument, the log group is listed with the results of the reduction simulation.
  • Apply: The desired state passed in the argument is actually applied to the listed log groups.

All of these subcommands can be passed the filter expressions to narrow down the target log groups.

Commands

NAME:
   llcm - AWS Log groups lifecycle manager

USAGE:
   llcm [global options] command [command options]

VERSION:
   0.0.0

DESCRIPTION:
   A listing, updating, and deleting tool to manage the lifecycle of Amazon CloudWatch Logs.
   It handles multiple regions fast while avoiding throttling. It can also return simulation
   results based on the desired state.

COMMANDS:
   list        List log group entries with specified format
   preview     Preview simulation results based on desired state
   apply       Apply desired state to log group entries

GLOBAL OPTIONS:
   --help, -h     show help
   --version, -v  print the version
List
NAME:
   llcm list - List log group entries with specified format

USAGE:
   llcm list [command options]

DESCRIPTION:
   List collects basic information about log groups from multiple specified regions and
   returns it in a specified format.

OPTIONS:
   --profile value, -p value                              set aws profile [$AWS_PROFILE]
   --log-level value, -l value                            set log level (default: "info") [$LLCM_LOG_LEVEL]
   --region value, -r value [ --region value, -r value ]  set target regions (default: all regions with no opt-in)
   --filter value, -f value [ --filter value, -f value ]  set expressions to filter log groups
   --output value, -o value                               set output type (default: "compressedtext") [$LLCM_OUTPUT_TYPE]
   --help, -h                                             show help
Preview
NAME:
   llcm preview - Preview simulation results based on desired state

USAGE:
   llcm preview [command options]

DESCRIPTION:
   Preview performs a simple calculation based on `DesiredState` specified in the argument
   and returns a simulated list including `ReducibleBytes`, `RemainingBytes`, etc.

OPTIONS:
   --profile value, -p value                              set aws profile [$AWS_PROFILE]
   --log-level value, -l value                            set log level (default: "info") [$LLCM_LOG_LEVEL]
   --region value, -r value [ --region value, -r value ]  set target regions (default: all regions with no opt-in)
   --filter value, -f value [ --filter value, -f value ]  set expressions to filter log groups
   --desired value, -d value                              set the desired state
   --output value, -o value                               set output type (default: "compressedtext") [$LLCM_OUTPUT_TYPE]
   --help, -h                                             show help
Apply
NAME:
   llcm apply - Apply desired state to log group entries

USAGE:
   llcm apply [command options]

DESCRIPTION:
   Apply deletes and updates target log groups in batches based on `DesiredState`.
   It is fast across multiple regions, but cleverly avoids throttling.

OPTIONS:
   --profile value, -p value                              set aws profile [$AWS_PROFILE]
   --log-level value, -l value                            set log level (default: "info") [$LLCM_LOG_LEVEL]
   --region value, -r value [ --region value, -r value ]  set target regions (default: all regions with no opt-in)
   --filter value, -f value [ --filter value, -f value ]  set expressions to filter log groups
   --desired value, -d value                              set the desired state
   --help, -h                                             show help

Options

The following values can be passed for each option.

Option Values Default value Environment Variable
--profile value -p value - - AWS_PROFILE
--log-level value -l value debug info warn error info LLCM_LOG_LEVEL
--region value1,value2... -r value1,value2... af-south-1 ap-east-1 ap-northeast-1 ap-northeast-2 ap-northeast-3 ap-south-1 ap-south-2 ap-southeast-1 ap-southeast-2 ap-southeast-3 ap-southeast-4 ap-southeast-5 ap-southeast-7 ca-central-1 ca-west-1 eu-central-1 eu-central-2 eu-north-1 eu-south-1 eu-south-2 eu-west-1 eu-west-2 eu-west-3 il-central-1 me-central-1 me-south-1 mx-central-1 sa-east-1 us-east-1 us-east-2 us-west-1 us-west-2 All regions with no opt-in -
--filter value1,value2... -f value1,value2... key, operator, and value separated by spaces, as in bytes != 0
key: name class elapsed retention bytes
operator: > >= < <= == ==* != !=* =~ =~* !~ !~*
- -
--desired value -d value delete 1day 3days 5days 1week 2weeks 1month 2months 3months 4months 5months 6months 1year 13months 18months 2years 3years 5years 6years 7years 8years 9years 10years infinite - -
--output value -o value json prettyjson text compressedtext markdown backlog tsv compressedtext LLCM_OUTPUT_TYPE
--help -h - - -
--version -v - - -

Examples

Case 1
  • List logs generated by Lambda from all regions that are not empty and have a retention period longer than one year, and list the reduction effect if the retention period is changed uniformly to one year. The output shall be in markdown table format.

[!NOTE] Note that this simulation is simply a pro-rata calculation of the log bytes.

llcm preview --desired 1year --filter 'name =~ ^/aws/lambda/.*','bytes != 0','retention > 1year' --output markdown

# The following outputs are obtained
| Name                 | Region         | Class    | CreatedAt                 | ElapsedDays | RetentionInDays | StoredBytes  | BytesPerDay | DesiredState | ReductionInDays | ReducibleBytes | RemainingBytes |
| -------------------- | -------------- | -------- | ------------------------- | ----------- | --------------- | ------------ | ----------- | ------------ | --------------- | -------------- | -------------- |
| /aws/lambda/tokyo-1  | ap-northeast-1 | STANDARD | 2019-04-15T21:50:12+09:00 | 2107        | 731             | 161094000389 | 220374829   | 365          | 366             | 80657187414    | 80436812975    |
| /aws/lambda/tokyo-2  | ap-northeast-1 | STANDARD | 2020-08-26T23:45:50+09:00 | 1608        | 731             | 30273686566  | 41414071    | 365          | 366             | 15157549986    | 15116136580    |
| /aws/lambda/oregon-1 | us-west-2      | STANDARD | 2020-08-27T14:34:54+09:00 | 1607        | 731             | 28578246408  | 39094728    | 365          | 366             | 14308670448    | 14269575960    |
| /aws/lambda/oregon-2 | us-west-2      | STANDARD | 2020-08-26T23:48:51+09:00 | 1608        | 731             | 22822519036  | 31220956    | 365          | 366             | 11426869896    | 11395649140    |
...
  • Apply the desired retention period to the log groups identified above.
llcm apply --desired 1year --filter 'name =~ ^/aws/lambda/.*','bytes != 0','retention > 1year'
Case 2
  • List log groups for Tokyo and Oregon that are empty and have been created more than one year ago in the backlog table format.
llcm list --filter 'bytes == 0','elapsed > 365' --region ap-northeast-1,us-west-2 --output backlog

# The following outputs are obtained
| Name   | Region         | Class    | CreatedAt                 | ElapsedDays | RetentionInDays | StoredBytes |h
| test-1 | ap-northeast-1 | STANDARD | 2017-12-07T13:16:02+09:00 |        2601 |             731 |           0 |
| test-2 | us-west-2      | STANDARD | 2017-12-07T12:44:45+09:00 |        2601 |             731 |           0 |
| test-3 | ap-northeast-1 | STANDARD | 2017-12-07T13:21:09+09:00 |        2601 |             731 |           0 |
| test-4 | us-west-2      | STANDARD | 2017-12-07T12:50:11+09:00 |        2601 |             731 |           0 |
...
  • Delete the log groups identified above in a batch.
llcm apply --desired delete --filter 'bytes == 0','elapsed > 365' --region ap-northeast-1,us-west-2
Case 3
  • For a more serious investigation, copy all log groups for that account in TSV format to the clipboard and paste them into a spreadsheet.
# macOS
llcm list --output tsv | pbcopy

# Windows
llcm list --output tsv | Set-Clipboard
Case 4
  • Visualize the reductions in order to submit the rate reduction plan in a document. A simple stacked bar chart will appear in your browser in one stroke!
llcm preview --desired 1year --output chart

chart

  • If the output type is chart in the list command, a pie chart is displayed.
llcm list --output chart

chart

Moreover

In addition to use from the command line, it is designed to be used as is as a package; see the example of running it with Lambda defined in the AWS CDK.

Warnings

  • Consider enclosing strings passed to the filter in single quotes. Unintended expansion may occur, e.g., history expansion by the shell (Try typing this command in your shell environment: echo "name !~ ^test.*")
  • The Preview command is the best used to simulate reductions, but note that it is only a simple calculation of the log volume pro-rated by day.
  • The fields such as ElapsedDays and ReductionInDays represent the number of days, but are rounded down to the nearest whole number when cast to int64. This means that the reduction simulation will not be inflated beyond what is expected.
  • The minimum value for BytesPerDay is 1. Note this specification if you have a large number of log groups that have just been created and are small in size.

Installation

Install with homebrew

brew install nekrassov01/tap/llcm

Install with go

go install github.com/nekrassov01/llcm@latest

Or download binary from releases

Shell completion

Supported Shells are as follows:

  • bash
  • zsh
  • pwsh
llcm completion bash|zsh|pwsh

Todo

  • Add to readme an example of implementing as a lambda function
  • Implement logical operators: && ||
  • Implement visualization of simulation results
  • Support streaming output (in github.com/nekrassov01/mintab)

Author

nekrassov01

License

MIT

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	// PageTitle is the title of the HTML page.
	PageTitle = "llcm"

	// BaseName is the base name of the HTML file.
	BaseName = "llcm"

	// MaxPieChartItems is the maximum number of items in a pie chart.
	MaxPieChartItems = 11

	// MaxBarChartItems is the maximum number of items in a bar chart.
	MaxBarChartItems = 31

	// PieChartTitle is the title of the pie chart.
	PieChartTitle = "Stored bytes of log groups"

	// barChartTitle is the title of the bar chart.
	BarChartTitle = "The simulation of reductions in log groups"
)
View Source
var (
	// DefaultRegion is the region speficied by default.
	DefaultRegion = "us-east-1"

	// DefaultRegions is the default target regions.
	DefaultRegions = []string{
		"ap-northeast-1",
		"ap-northeast-2",
		"ap-northeast-3",
		"ap-south-1",
		"ap-southeast-1",
		"ap-southeast-2",
		"ca-central-1",
		"eu-central-1",
		"eu-west-1",
		"eu-west-2",
		"eu-west-3",
		"eu-north-1",
		"sa-east-1",
		"us-east-1",
		"us-east-2",
		"us-west-1",
		"us-west-2",
	}
)
View Source
var (
	// TotalStoredBytesLabel is the label of the total stored bytes.
	TotalStoredBytesLabel = "storedBytes"

	// TotalReducibleBytesLabel is the label of the total reducible bytes.
	TotalReducibleBytesLabel = "reducibleBytes"

	// TotalRemainingBytesLabel is the label of the total remaining bytes.
	TotalRemainingBytesLabel = "remainingBytes"
)
View Source
var (
	// MaxRetryAttempts is the maximum number of retry attempts.
	MaxRetryAttempts = 10

	// DelayTimeSec is the sleep time in seconds for retry.
	DelayTimeSec = 3
)
View Source
var NumWorker = int64(runtime.NumCPU()*2 + 1)

NumWorker is the number of workers for concurrent processing.

Functions

func LoadConfig

func LoadConfig(ctx context.Context, profile string) (aws.Config, error)

LoadConfig loads the aws config.

func SortEntries

func SortEntries[E Entry, D EntryData[E]](data D)

SortEntries sorts the entries by bytes and name.

Types

type API

API represents an interface for CloudWatch Logs.

type Client

type Client struct {
	API
}

Client represents a client for CloudWatch Logs.

func NewClient

func NewClient(cfg aws.Config) *Client

NewClient creates a new client.

type DesiredState

type DesiredState int32

DesiredState represents the desired state of the log group.

const (
	DesiredStateNone           DesiredState = -9999 // A value meaning none.
	DesiredStateZero           DesiredState = 0     // A value meaning delete the log group.
	DesiredStateOneDay         DesiredState = 1     // A value meaning retain the log group for one day.
	DesiredStateThreeDays      DesiredState = 3     // A value meaning retain the log group for three days.
	DesiredStateFiveDays       DesiredState = 5     // A value meaning retain the log group for five days.
	DesiredStateOneWeek        DesiredState = 7     // A value meaning retain the log group for one week.
	DesiredStateTwoWeeks       DesiredState = 14    // A value meaning retain the log group for two weeks.
	DesiredStateOneMonth       DesiredState = 30    // A value meaning retain the log group for one month.
	DesiredStateTwoMonths      DesiredState = 60    // A value meaning retain the log group for two months.
	DesiredStateThreeMonths    DesiredState = 90    // A value meaning retain the log group for three months.
	DesiredStateFourMonths     DesiredState = 120   // A value meaning retain the log group for four months.
	DesiredStateFiveMonths     DesiredState = 150   // A value meaning retain the log group for five months.
	DesiredStateSixMonths      DesiredState = 180   // A value meaning retain the log group for six months.
	DesiredStateOneYear        DesiredState = 365   // A value meaning retain the log group for one year.
	DesiredStateThirteenMonths DesiredState = 400   // A value meaning retain the log group for thirteen months.
	DesiredStateEighteenMonths DesiredState = 545   // A value meaning retain the log group for eighteen months.
	DesiredStateTwoYears       DesiredState = 731   // A value meaning retain the log group for two years.
	DesiredStateThreeYears     DesiredState = 1096  // A value meaning retain the log group for three years.
	DesiredStateFiveYears      DesiredState = 1827  // A value meaning retain the log group for five years.
	DesiredStateSixYears       DesiredState = 2192  // A value meaning retain the log group for six years.
	DesiredStateSevenYears     DesiredState = 255   // A value meaning retain the log group for seven years.
	DesiredStateEightYears     DesiredState = 2922  // A value meaning retain the log group for eight years.
	DesiredStateNineYears      DesiredState = 3288  // A value meaning retain the log group for nine years.
	DesiredStateTenYears       DesiredState = 3653  // A value meaning retain the log group for ten years.
	DesiredStateInfinite       DesiredState = 9999  // A value meaning retain the log group infinity.
)

func ParseDesiredState

func ParseDesiredState(s string) (DesiredState, error)

ParseDesiredState parses a string into a DesiredState.

func (DesiredState) MarshalJSON

func (t DesiredState) MarshalJSON() ([]byte, error)

MarshalJSON returns the JSON representation of the DesiredState.

func (DesiredState) String

func (t DesiredState) String() string

String returns the string representation of the DesiredState.

type Entry

type Entry interface {
	Name() string              // Name returns the name of the entry.
	Bytes() int64              // Bytes returns the stored bytes of the entry.
	DataSet() map[string]int64 // DataSet returns map for plotting the chart.
	// contains filtered or unexported methods
}

Entry is an interface for log group entry.

type EntryData

type EntryData[T Entry] interface {
	Header() []string
	Entries() []T
	Total() map[string]int64
	Chart() error
}

EntryData represents the collection of entries.

type Filter

type Filter struct {
	Key      FilterKey      // The key of the filter expression
	Operator FilterOperator // The operator of the filter expression
	Value    string         // The value of the filter expression
}

Filter represents a filter expression for filtering log group entries.

func EvaluateFilter

func EvaluateFilter(expressions []string) ([]Filter, error)

EvaluateFilter evaluates the filter expressions.

type FilterKey

type FilterKey int

FilterKey represents the key to filter by.

const (
	FilterKeyNone      FilterKey = iota // The key meaning none.
	FilterKeyName                       // The key meaning "name" corresponding LogGroupName.
	FilterKeyClass                      // The key meaning "class" corresponding Class.
	FilterKeyElapsed                    // The key meaning "elapsed" corresponding ElapsedDays.
	FilterKeyRetention                  // The key meaning "retention" corresponding RetentionInDays.
	FilterKeyBytes                      // The key meaning "bytes" corresponding StoredBytes.
)

func (FilterKey) MarshalJSON

func (t FilterKey) MarshalJSON() ([]byte, error)

MarshalJSON returns the JSON representation of the FilterKey.

func (FilterKey) String

func (t FilterKey) String() string

String returns the string representation of the FilterKey.

type FilterOperator

type FilterOperator int

FilterOperator represents the operator to filter by.

const (
	FilterOperatorNone  FilterOperator = iota // The operator meaning none.
	FilterOperatorGT                          // The operator meaning "greater than".
	FilterOperatorGTE                         // The operator meaning "greater than or equal to".
	FilterOperatorLT                          // The operator meaning "less than".
	FilterOperatorLTE                         // The operator meaning "less than or equal to".
	FilterOperatorEQ                          // The operator meaning "equal".
	FilterOperatorEQI                         // The operator meaning "equal" (case-insensitive).
	FilterOperatorNEQ                         // The operator meaning "not equal".
	FilterOperatorNEQI                        // The operator meaning "not equal" (case-insensitive).
	FilterOperatorREQ                         // The operator meaning regular expression match.
	FilterOperatorREQI                        // The operator meaning regular expression match (case-insensitive).
	FilterOperatorNREQ                        // The operator meaning regular expression unmatch.
	FilterOperatorNREQI                       // The operator meaning regular expression unmatch (case-insensitive).

)

func (FilterOperator) MarshalJSON

func (t FilterOperator) MarshalJSON() ([]byte, error)

MarshalJSON returns the JSON representation of the FilterOperator.

func (FilterOperator) String

func (t FilterOperator) String() string

String returns the string representation of the FilterOperator.

type ListEntry

type ListEntry struct {
	// contains filtered or unexported fields
}

ListEntry represents an entry to list log group.

func (ListEntry) Bytes

func (e ListEntry) Bytes() int64

Bytes returns the stored bytes of the entry.

func (*ListEntry) DataSet added in v0.0.13

func (e *ListEntry) DataSet() map[string]int64

DataSet returns map for plotting the chart.

func (ListEntry) Name

func (e ListEntry) Name() string

Name returns the name of the entry.

type ListEntryData

type ListEntryData struct {
	TotalStoredBytes int64 // The total stored bytes of the log groups.
	// contains filtered or unexported fields
}

ListEntryData represents the collection of ListEntry.

func (*ListEntryData) Chart added in v0.0.14

func (d *ListEntryData) Chart() error

Chart generates a pie chart for the ListEntryData.

func (*ListEntryData) Entries

func (d *ListEntryData) Entries() []*ListEntry

Entries returns the entries of the ListEntryData.

func (*ListEntryData) Header

func (d *ListEntryData) Header() []string

Header returns the header of the ListEntryData.

func (*ListEntryData) Total

func (d *ListEntryData) Total() map[string]int64

Total returns the total of the ListEntryData.

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

Manager represents a log group lifecycle manager.

func NewManager

func NewManager(ctx context.Context, client *Client) *Manager

NewManager creates a new manager for log group lifecycle management.

func (*Manager) Apply

func (man *Manager) Apply(w io.Writer) (int32, error)

Apply applies the desired state to the log groups.

func (*Manager) List

func (man *Manager) List() (*ListEntryData, error)

List lists the log group entries.

func (*Manager) Preview

func (man *Manager) Preview() (*PreviewEntryData, error)

Preview returns the log group entries with the desired state and its simulated results.

func (*Manager) SetDesiredState

func (man *Manager) SetDesiredState(desired DesiredState) error

SetDesiredState sets the desired state.

func (*Manager) SetFilter

func (man *Manager) SetFilter(filters []Filter) error

SetFilter sets the filter expressions.

func (*Manager) SetRegion added in v0.0.9

func (man *Manager) SetRegion(regions []string) error

SetRegion sets the specified regions.

func (*Manager) String

func (man *Manager) String() string

String returns the string representation of the manager.

type OutputType

type OutputType int

OutputType represents the type of output to render.

const (
	OutputTypeNone           OutputType = iota // The output type that means none.
	OutputTypeJSON                             // The output type of JSON format.
	OutputTypePrettyJSON                       // The output type of pretty JSON format.
	OutputTypeText                             // The output type of text table format.
	OutputTypeCompressedText                   // The output type of compressed text table format.
	OutputTypeMarkdown                         // The output type of markdown table format.
	OutputTypeBacklog                          // The output type of backlog table format.
	OutputTypeTSV                              // The output type of tab-separated values.
	OutputTypeChart                            // The output type that means pie chart.
)

func ParseOutputType

func ParseOutputType(s string) (OutputType, error)

ParseOutputType parses a string into an OutputType.

func (OutputType) MarshalJSON

func (t OutputType) MarshalJSON() ([]byte, error)

MarshalJSON returns the JSON representation of the OutputType.

func (OutputType) String

func (t OutputType) String() string

String returns the string representation of the OutputType.

type PreviewEntry

type PreviewEntry struct {
	BytesPerDay     int64 // The bytes per day of the log group.
	DesiredState    int64 // The desired state of the log group.
	ReductionInDays int64 // The number of days to be reduced after the action.
	ReducibleBytes  int64 // The number of bytes that can be reduced after the action.
	RemainingBytes  int64 // The number of bytes that remain after the action.
	// contains filtered or unexported fields
}

PreviewEntry is an extended representation of entry with the desired state and its simulated results.

func (PreviewEntry) Bytes

func (e PreviewEntry) Bytes() int64

Bytes returns the stored bytes of the entry.

func (*PreviewEntry) DataSet added in v0.0.13

func (e *PreviewEntry) DataSet() map[string]int64

DataSet returns map for plotting the chart.

func (PreviewEntry) Name

func (e PreviewEntry) Name() string

Name returns the name of the entry.

type PreviewEntryData

type PreviewEntryData struct {
	TotalStoredBytes    int64 // The total stored bytes of the log groups.
	TotalReducibleBytes int64 // The total reducible bytes of the log groups.
	TotalRemainingBytes int64 // The total remaining bytes of the log groups.
	// contains filtered or unexported fields
}

PreviewEntryData represents the collection of PreviewEntry.

func (*PreviewEntryData) Chart added in v0.0.14

func (d *PreviewEntryData) Chart() error

Chart generates a bar chart for the PreviewEntryData.

func (*PreviewEntryData) Entries

func (d *PreviewEntryData) Entries() []*PreviewEntry

Entries returns the entries of the PreviewEntryData.

func (*PreviewEntryData) Header

func (d *PreviewEntryData) Header() []string

Header returns the header of the PreviewEntryData.

func (*PreviewEntryData) Total

func (d *PreviewEntryData) Total() map[string]int64

Total returns the total of the PreviewEntryData.

type Renderer

type Renderer[E Entry, D EntryData[E]] struct {
	Data       D
	OutputType OutputType
	// contains filtered or unexported fields
}

OutputType represents the type of the output.

func NewRenderer

func NewRenderer[E Entry, D EntryData[E]](w io.Writer, data D, outputType OutputType) *Renderer[E, D]

NewRenderer creates a new renderer with the specified parameters.

func (*Renderer[E, D]) Render

func (ren *Renderer[E, D]) Render() error

Render renders the output.

func (*Renderer[E, D]) String

func (ren *Renderer[E, D]) String() string

String returns the string representation of the renderer.

type Retryer

type Retryer struct {
	// contains filtered or unexported fields
}

Retryer represents a retryer for avoiding api rate limit exceeded.

func NewRetryer

func NewRetryer(isErrorRetryableFunc func(error) bool, delayTimeSec int) *Retryer

NewRetryer creates a new retryer.

func (*Retryer) GetAttemptToken

func (r *Retryer) GetAttemptToken(context.Context) (func(error) error, error)

GetAttemptToken returns the attempt token.

func (*Retryer) GetInitialToken

func (r *Retryer) GetInitialToken() func(error) error

GetInitialToken returns the initial token.

func (*Retryer) GetRetryToken

func (r *Retryer) GetRetryToken(context.Context, error) (func(error) error, error)

GetRetryToken returns the retry token.

func (*Retryer) IsErrorRetryable

func (r *Retryer) IsErrorRetryable(err error) bool

IsErrorRetryable checks if the error is retryable.

func (*Retryer) MaxAttempts

func (r *Retryer) MaxAttempts() int

MaxAttempts returns the maximum number of retry attempts.

func (*Retryer) RetryDelay

func (r *Retryer) RetryDelay(int, error) (time.Duration, error)

RetryDelay returns the delay time for retry.

Directories

Path Synopsis
cmd

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL