Consul
Fault Injection
This topic describes the process to inject different types of faults into the services in Consul's service mesh to help you test your network's resilience.
Enterprise
This feature is available in Consul Enterprise.
Introduction
Consul allows you to configure fault injection filters to alter the responses from an upstream service. By injecting HTTP and gRPC statuses, bandwidth limits, or delays, users can test the resilience of their system to different unexpected issues.
Consul applies fault injection filters to an upstream service as part of its service defaults. For example, to simulate that an upstream returned a 500 HTTP status code 50 percent of the time, add an Abort configuration to the Envoy extensions block in the service default configuration for the upstream service.
The fault injection filters may be used individually, or in a combination of the three types by adding multiple blocks to the Arguments.Config section of the configuration.
Requirements
Consul Enterprise v1.18.0 or later
Inject delays
Specify the fault injection behavior in the service defaults configuration entry.
To inject a delay fault when Consul runs on virtual machines, configure the following parameters:
Arguments.Config.Delay.Duration: The delay in milliseconds to use when injecting a delay type fault.Arguments.Config.Delay.Percentage: The percentage of responses to inject the delay behavior into.
The following example configures the default behavior for a service named billing. This configuration injects a 100 ms delay into 50 percent of responses from the billing service.
Kind = "service-defaults"
Name = "billing"
EnvoyExtensions = [
{
Name = "builtin/fault-injection"
Arguments = {
Config = {
Delay = {
Duration = 100
Percentage = 50
}
}
}
}
]
Refer to the service defaults configuration entry reference for additional specifications and example configurations.
Inject statuses
To inject a status code when aborting a response, configure the following parameters:
Arguments.Config.Abort.HttpStatus: The HTTP status code to inject into responses when injecting an abort type fault. You may specify either a HTTP status or a gRPC status but not both.Arguments.Config.Abort.GrpcStatus: The gRPC status to inject into responses when injecting an abort type fault. You may specify either a HTTP status or a gRPC status but not both.Arguments.Config.Abort.Percentage: The percentage of responses to inject the abort behavior into.
The following example configures the default behavior for a service named billing. This configuration injects an HTTP status code of 503 into 5 percent of responses from the billing service.
Kind = "service-defaults"
Name = "billing"
EnvoyExtensions = [
{
Name = "builtin/fault-injection"
Arguments = {
Config = {
Abort = {
HttpStatus = 503
Percentage = 5
}
}
}
}
]
Refer to the service defaults configuration entry reference for additional specifications and example configurations.
Inject bandwidth limit
To inject a rate limit into a response, configure the following parameters:
Arguments.Config.Bandwidth.Limit: The amount of data allowed through the filter when applied. This value is specified in KiB/s. When the limit is exceeded, requests, operations, or connections will be subjected to rate limiting.Arguments.Config.Bandwidth.Percentage: The percent of responses to inject the bandwidth limit behavior into.
The following example configures the default behavior for a service named billing. This configuration limits the bandwidth of outgoing requests to 100 KiB/s for 50 percent of responses from the billing service.
Kind = "service-defaults"
Name = "billing"
EnvoyExtensions = [
{
Name = "builtin/fault-injection"
Arguments = {
Config = {
Bandwidth = {
Limit = 100
Percentage = 50
}
}
}
}
]
Refer to the service defaults configuration entry reference for additional specifications and example configurations.