Filtering Log Output

When providers write logs, there may be sensitive data which should not be present in log messages or structured log fields. The tflog package supports masking data or omitting log entries entirely before they are output via the provider root logger or provider-defined subsystem loggers.

NOTE: While log filtering can help hide sensitive data, it is important to ensure the provider implementation works as expected before creating production provider releases.

Masking Log Output

Masking data in log messages or structured log fields is the process of replacing a sensitive piece of data with a placeholder piece of data. The tflog package uses *** as the replacement.

Masking Messages Via Exact Strings

Use the tflog.MaskMessageStrings() function before writing logs:

tflog.MaskMessageStrings(ctx, "my-sensitive-data")

// Will output a message of: example message with *** masked
tflog.Trace(ctx, "example message with my-sensitive-data masked")

For subsystems, use the tflog.SubsystemMaskMessageStrings() function before writing logs:

tflog.MaskMessageStrings(ctx, "my-subsystem", "my-sensitive-data")

Both functions can accept multiple string values at once to simplify filtering implementations.

This example shows how to provide multiple values either through a slice, which may be defined somewhere else, or inline:

sensitiveStrings := []string{"my-sensitive-data", "more-sensitive-data"}
tflog.MaskMessageStrings(ctx, sensitiveStrings...)
tflog.MaskMessageStrings(ctx, "yet-more-sensitive-data", "final-sensitive-data")

Masking Messages Via Regular Expressions

Use the tflog.MaskMessageRegexes() function before writing logs:

tflog.MaskMessageRegexes(ctx, regexp.MustCompile(`[0-9]{4}-[0-9]{4}-[0-9]{4}-[0-9]{4}`))

// Will output a message of: example message with *** masked
tflog.Trace(ctx, "example message with 1234-1234-1234-1234 masked")

For subsystems, use the tflog.SubsystemMaskMessageRegexes() function before writing logs:

tflog.MaskMessageRegexes(ctx, "my-subsystem", regexp.MustCompile(`[0-9]{4}-[0-9]{4}-[0-9]{4}-[0-9]{4}`))

Both functions can accept multiple regular expressions at once to simplify filtering implementations.

This example shows how to provide multiple regular expresions either through a slice, which may be defined somewhere else, or inline:

sensitiveRegexes := []regexp.Regexp{
    regexp.MustCompile(`[0-9]{4}-[0-9]{4}-[0-9]{4}-[0-9]{4}`),
    regexp.MustCompile(`[0-9]{3}-[0-9]{2}-[0-9]{4}`),
}
tflog.MaskMessageRegexes(ctx, sensitiveRegexes...)
tflog.MaskMessageRegexes(ctx, regexp.MustCompile(`(top-)?secret`), regexp.MustCompile(`me@example\.(com|org)`))

Masking Entire Field Values Via Field Keys

Use the tflog.MaskFieldValuesWithFieldKeys() function before writing logs:

tflog.MaskFieldValuesWithFieldKeys(ctx, "my-sensitive-field")

// Will output: example message: my-sensitive-field=***
tflog.Trace(ctx, "example message", map[string]interface{}{"my-sensitive-field": "some-sensitive-data"})

For subsystems, use the tflog.SubsystemMaskFieldValuesWithFieldKeys() function before writing logs:

tflog.MaskFieldValuesWithFieldKeys(ctx, "my-subsystem", "my-sensitive-field")

Both functions can accept multiple string values at once to simplify filtering implementations.

This example shows how to provide multiple values either through a slice, which may be defined somewhere else, or inline:

sensitiveFields := []string{"my-sensitive-field", "another-sensitive-field"}
tflog.MaskFieldValuesWithFieldKeys(ctx, sensitiveFields...)
tflog.MaskFieldValuesWithFieldKeys(ctx, "yet-another-sensitive-field", "final-sensitive-field")

Masking Field Values Via Regular Expressions

Use the tflog.MaskAllFieldValuesRegexes() function before writing logs:

tflog.MaskAllFieldValuesRegexes(ctx, regexp.MustCompile(`(\w{3}_SECRET)`))

// Will output: example message: contains-secret=my-super-***
tflog.Trace(ctx, "example message", map[string]interface{}{"contains-secret": "my-super-TOP_SECRET"})

For subsystems, use the tflog.SubsystemMaskAllFieldValuesRegexes() function before writing logs:

tflog.SubsystemMaskAllFieldValuesRegexes(ctx, "my-subsystem", regexp.MustCompile(`(\w{3}_SECRET)`))

Both functions can accept multiple string values at once to simplify filtering implementations.

Masking Field Values Via Exact Strings

Use the tflog.MaskAllFieldValuesStrings() function before writing logs:

tflog.MaskAllFieldValuesStrings(ctx, "TOP_SECRET")

// Will output: example message: contains-secret=my-super-***
tflog.Trace(ctx, "example message", map[string]interface{}{"contains-secret": "my-super-TOP_SECRET"})

For subsystems, use the tflog.SubsystemMaskAllFieldValuesStrings() function before writing logs:

tflog.SubsystemMaskAllFieldValuesStrings(ctx, "my-subsystem", "TOP_SECRET")

Both functions can accept multiple string values at once to simplify filtering implementations.

Masking Messages and Field Values via Regular Expressions

Use the tflog.MaskLogRegexes() function to obtain the same configuration and behaviour as if you had used the same input on tflog.MaskMessageRegexes() and tflog.MaskAllFieldValuesRegexes().

The same applies, respectively, for tflog.SubsystemMaskLogRegexes() function, and the functions tflog.SubsystemMaskMessageRegexes() and tflog.SubsystemMaskAllFieldValuesRegexes().

Masking Messages and Field Values via Exact Strings

Use the tflog.MaskLogStrings() function to obtain the same configuration and behaviour as if you had used the same input on tflog.MaskMessageStrings() and tflog.MaskAllFieldValuesStrings().

The same applies, respectively, for tflog.SubsystemMaskLogStrings() function, and the functions tflog.SubsystemMaskMessageStrings() and tflog.SubsystemMaskAllFieldValuesStrings().

Omitting Log Output

Omitting logs is the process of skipping the entire output of a log message and its fields when encountering sensitive data.

Omitting Logs Via Message Exact Strings

Use the tflog.OmitLogWithMessageStrings() function before writing logs:

tflog.OmitLogWithMessageStrings(ctx, "my-sensitive-data")

// Will not be output
tflog.Trace(ctx, "example message with my-sensitive-data masked")

For subsystems, use the tflog.SubsystemOmitLogWithMessageStrings() function before writing logs:

tflog.OmitLogWithMessageStrings(ctx, "my-subsystem", "my-sensitive-data")

Both functions can accept multiple string values at once to simplify filtering implementations.

This example shows how to provide multiple values either through a slice, which may be defined somewhere else, or inline:

sensitiveStrings := []string{"my-sensitive-data", "more-sensitive-data"}
tflog.OmitLogWithMessageStrings(ctx, sensitiveStrings...)
tflog.OmitLogWithMessageStrings(ctx, "yet-more-sensitive-data", "final-sensitive-data")

Omitting Logs Via Message Regular Expressions

Use the tflog.OmitLogWithMessageRegexes() function before writing logs:

tflog.OmitLogWithMessageRegexes(ctx, regexp.MustCompile(`[0-9]{4}-[0-9]{4}-[0-9]{4}-[0-9]{4}`))

// Will not be output
tflog.Trace(ctx, "example message with 1234-1234-1234-1234 masked")

For subsystems, use the tflog.SubsystemOmitLogWithMessageRegexes() function before writing logs:

tflog.OmitLogWithMessageRegexes(ctx, "my-subsystem", regexp.MustCompile(`[0-9]{4}-[0-9]{4}-[0-9]{4}-[0-9]{4}`))

Both functions can accept multiple regular expressions at once to simplify filtering implementations.

This example shows how to provide multiple regular expresions either through a slice, which may be defined somewhere else, or inline:

sensitiveRegexes := []regexp.Regexp{
    regexp.MustCompile(`[0-9]{4}-[0-9]{4}-[0-9]{4}-[0-9]{4}`),
    regexp.MustCompile(`[0-9]{3}-[0-9]{2}-[0-9]{4}`),
}
tflog.MaskMessageRegexes(ctx, sensitiveRegexes...)
tflog.MaskMessageRegexes(ctx, regexp.MustCompile(`(top-)?secret`), regexp.MustCompile(`me@example\.(com|org)`))

Omitting Logs Via Field Keys

Use the tflog.OmitLogWithFieldKeys() function before writing logs:

tflog.OmitLogWithFieldKeys(ctx, "my-sensitive-field")

// Will not be output
tflog.Trace(ctx, "example message", map[string]interface{}{"my-sensitive-field": "some-sensitive-data"})

For subsystems, use the tflog.SubsystemOmitLogWithFieldKeys() function before writing logs:

tflog.OmitLogWithFieldKeys(ctx, "my-subsystem", "my-sensitive-field")

Both functions can accept multiple string values at once to simplify filtering implementations.

This example shows how to provide multiple values either through a slice, which may be defined somewhere else, or inline:

sensitiveFields := []string{"my-sensitive-field", "another-sensitive-field"}
tflog.OmitLogWithFieldKeys(ctx, sensitiveFields...)
tflog.OmitLogWithFieldKeys(ctx, "yet-another-sensitive-field", "final-sensitive-field")