How to improve resilience with Apache Camel and Hystrix

Today we are going to look at one of the patterns to improve the resilience of microservices. In this particular case, it will be the Circuit Breaker pattern. And we will be able to see it applied to a microservice with Apache Camel through Netflix Hystrix. As always, we will start with a little bit of theory. We will explain what is the Circuit Breaker pattern and what is its functionality.

This pattern is based on the idea that some intercommunications within the microservice may fail. So in a normal flow of the application where there are external systems down, there is therefore wasted time and resources in making invocations that are possibly unsuccessful. The idea of the pattern is to monitor the external service that we are going to invoke so that in case of any error, redirect the flow to another service or business logic that makes us keep our microservice operational and with low latency.

This pattern bases its operation on the state control of the circuit itself. It has three different states:

• Closed: Operational circuit, everything works correctly.
• Open: Faulty circuit. The flow is redirected to do another logic, allowing to reduce the loss of resources and continue offering service.
• Semi-open: Intermediate state that will check if the external service is available or not and if it keeps the circuit open or closes it.

Image from Apache Camel Circuit Breaker Documentation

To sum up, the circuit breaker pattern will act as a proxy that will allow us to check if the external service is running before making the call to that system, in order to save resources and time.

Within Apache Camel, there are three implementations to carry out this pattern. These are:

• Hystrix: Netflix library on which we will base the example. Currently (05/2021) it is no longer under development and is in maintenance status.
• Resilience4j: Library based on Hystrix and oriented to the implementation of several patterns for the implementation of resilience.
• MicroProfile Fault Tolerance: Eclipse foundation library for the implementation of resilience patterns.

Therefore, the use of the Circuit Breaker pattern at the code level will be the same regardless of the library selected for its implementation. The only thing that will change will be the library we indicate as a dependency and its configuration. In this case:

<dependency>
	<groupId>org.apache.camel.springboot</groupId>
	<artifactId>camel-hystrix-starter</artifactId>
</dependency>

In the case of Hystrix, there are up to 31 configuration options that we can enter in the Apache Camel application configuration file: application.properties.

For our example, we will modify some options. This way we will not have to make so many invocations to appreciate the Circuit Breaker pattern. These options are:

• circuitBreakerErrorThresholdPercentage: Percentage of error from which it considers the circuit open and does not accept invocations.
• circuitBreakerSleepWindowInMilliseconds: Time to pass before accepting requests again.
• metricsRollingStatisticalWindowInMilliseconds: Duration time of the associated metrics.
• circuitBreakerRequestVolumeThreshold: Minimum number of requests that exist during the duration of the metrics. If this minimum number of invocations is not reached, the circuit will not be considered open, regardless of the error percentage.

As mentioned in the previous list, there is another point to highlight. These are the metrics associated with the execution of our microservice. Depending on these metrics and the configuration values, we will decide the state of the circuit and how to act.

If we use Spring in our Apache Camel application, we can access these metrics easily through the URL /hystrix.stream.

Now we will move on to the first example. To carry it out we will rely on examples that we have done before and that you can see here. In this example, we will invoke a database (which we will raise with docker-compose). Depending on if the DB is operational or not, we will return one or another result. In the case that there is no DB, we will simply return an empty list.

Through the circuitBreaker method, we will indicate that we are going to implement this pattern. Then we can indicate if we are going to use some configuration, in case we do not indicate any, the default configuration will be used. We must indicate a route to redirect the call, which normally will be an external service. In this case, the DB. And we must indicate what we are going to do in case of error, for this case we will always send the same result, which we can indicate through the onFallback method. And finally indicate that the logic associated with the circuit breaker is finished.

If we raise the docker-compose, the application, and make invocations to the following URL, we will be able to see how we obtain correctly all the books of the DB.

curl --location --request GET 'http://localhost:9090/hystrix/sql'

Once the application is started, we can access the Portainer, through the URL http://localhost:9000, and stop the MySQL container. At that moment we can invoke the application again and see how the calls return an empty list of elements.

The onFallback method allows us to answer a message immediately. But if what we want to do is to resend the call to another external service, we will have to use the onFallbackViaNetwork method.

In the following example that invokes a Wiremock we will be able to check this second invocation. In addition, it will also allow us to visualize through the Wiremock console if when the circuit is open, the first service is still invoked or not. An aspect that was more difficult to verify with the MySQL container.

Therefore we will create a mapping for Wiremock that responds to the path /book with the same response that the DB returns. And our example will try to invoke the path /fail, which not existing will always fail and then invoke /book.

If we invoke the service and at the same time we consult the Wiremock console, we will be able to see how it calls first to a service and then invoke the second one. Returning our microservice a list of books, as the previous method did.

curl --location --request GET 'http://localhost:9090/hystrix/fail'

And based on the above configuration, if we make a large number of calls. We can see in the Wiremock console, as when the circuit is open, no calls are made to the first service.

Finally, in previous versions of Spring, there was a dependency called spring-cloud-starter-hystrix-dashboard that generated a dashboard for the visualization of Hystrix metrics. But it is not compatible with the current Spring versions used in this example. Instead of using that library, we will use a docker image called mlabouardy/hystrix-dashboard that will allow us to access the dashboard.

To do this we must start the image on port 8080 and access the URL: http://localhost:8080/hystrix. The next step will be to indicate the URL where Hystrix is showing the metrics. Keep in mind that if the Apache Camel application is not being launched through an image on the same network as the dashboard we must indicate the URL as follows on Windows or Mac systems: http://host.docker.internal:9090/hystrix.stream. Below is an example of how these metrics can be visualized with the help of this image:

That’s it, folks. We’ve already learned a bit more about microservice resilience and how to make it work easily in our Apache Camel application. If you want access to the code, you have it here.