Enterprise Integrations with Apache Camel

24.11.2021 Stefan Welsch
Tech event-driven distributed-systems java

Many companies want to make their services available in the cloud. The connection to different systems plays an extremely important role. For example, if you want to make a certain product proposal to customers in your app because this product is currently being offered at a reduced price, you have to read data from a recommendation engine, enrich it with data from a SAP system and ultimately make it available to the app in a certain form. Before the user can access this data at all, he must of course be authenticated and authorized.

It is precisely with these tasks that Apache Camel, as an integration platform, should help us, and therefore I want to show you the advantages and disadvantages of Apache Camel today. But since I want to take up the topic of the cloud, I will also introduce you to Apache Camel K at the end. This makes it possible to run Camel applications directly native in the Kubernetes cluster.

But first, as always, a few facts about Apache Camel. The initial release of Camel came out in June 2007. So Camel is probably the oldest project that we have presented in our TechUp series so far. The framework was developed by the Apache Software Foundation and is written in Java and XML. The tool was built on the book Enterprise Integration Pattern written by the two authors Gregor Hohpe and Bobby Woolf, which were specially created for the design of enterprise application integration and message-oriented middleware-based systems.

What is Apache Camel

The EIPs (Enterprise Integration Pattern) show us that we should only use messages both as input and as output when communicating between components. There is a full set of patterns that we can choose from and implement into our various components which then together make up the entire system.

Apache Camel offers interfaces for the EIPs, the basic objects, frequently required implementations, debugging tools, a configuration system and many other helpers that save us a lot of time when we want to implement a solution according to the EIPs. So it is a fully production-ready framework for developers who want to implement their solutions according to the EIPs.

Apache Camel architecture

Camel Context

Apache Camel consists of a core called CamelContext. This is basically the runtime that holds everything else together. The Camel Context gives you access to the most important services, components, type converters, registry, endpoints, routes, data formats and languages.

Routes

The integration flow is defined in a camel route. For example, in order to integrate two systems, a Camel route can be coded, which indicates how these systems are integrated. For example, if you want to load data from a url and write it to a file, you would define the following route. Please note that routes are defined with one of Camel's DSLs. We only use Java DSL in our examples. But here, too, a distinction is made between the Standard Route Builder Java DSL and the Endpoint Route Builder Java DSL. The latter of the two offers us type security, as we can see in the following example. I therefore clearly prefer the Enpoint Route Builder variant.

Standard Route Builder Java DSL

from("http:myserver/myurl")
  .to("http:mydataprovider/myrequesteddata?bridgeEndpoint=true)
  .to("file:///path/to/your/file?fileName=yourFileName);

Endpoint Route Builder Java DSL

from(http("myserver/myurl"))
  .to(http("mydataprovider/myrequesteddata").bridgeEndpoint(true))
  .to(file("/path/to/your/file").fileName("yourFileName")));

Processors

Processors are used to transform and manipulate messages during routing. For example, if we have to modify data that we load from an url before we pass it on, we can use a processor for this. I'll explain why I write "can" in a later step when we look at beans.

Components

Components describe the extension points in Camel. This allows us to add connectivity to other systems. These components provide other Camel resources with an interface so that they can use the component.

Apache Camel in practice

In order to understand what exactly Apache Camel does, let's create a sample project that receives data from a residual endpoint, processes and enriches it. The data is then available again at a remaining endpoint.

There are several options for creating a Camel project. Today we want to create a Spring Boot application. I will not go into further details on the setup of Camel Spring Boot. This is described in the official documentation.

To need to have local test data, I use a Json Mock Server to archive this. To do this, a db.json file is simply created in the project and the Json server is then started with this. This server creates two endpoints for us ( todos und todo-details) which return the data to us as Json.

{
  "todos": [
    {
      "id": 1,
      "title": "delectus aut autem"
    }
  ], 
  "todo-details": [
    {
      "userId": 1,
      "id": 1,
      "completed": false
    }
  ]
}

RestRoute

First, let's take care of the endpoint that is called by the user. So there is an interface /todos which returns all ToDos with all information. So first we create the endpoint.

package com.bnova.camel;

import org.apache.camel.builder.endpoint.EndpointRouteBuilder;
import org.springframework.stereotype.Component;

@Component
public class RestRoute extends EndpointRouteBuilder {

	@Override
	public void configure() throws Exception {
		restConfiguration().host("localhost").port(8080);

		rest().get("/todos")
				.consumes("application/json")
				.produces("application/json")
				.to("log:info");
	}
}

Line 6: Registration of our route as a component. The route is added automatically when the application is started.

Line 7: We create our route based on the EndpointRestBuilder

Line 11: Set the host and port for our rest API

Lines 13-16: Create a rest endpoint on the path /todos, which expects JSON and also outputs JSON again. We first write the call as information in the log.

We can now start the application first and should then receive the following log.

mvn spring-boot:run

....
2021-11-08 15:09:25.403  INFO 67677 --- [           main] com.bnova.camel.CamelApplication         : Starting CamelApplication using Java 16.0.1 on Stefan-Welsch-MacBook-Pro.fritz.box with PID 67677 (/Users/swelsch/Development/b-nova-techub/techup-camel/target/classes started by swelsch in /Users/swelsch/Development/b-nova-techub/techup-camel)
2021-11-08 15:09:25.404  INFO 67677 --- [           main] com.bnova.camel.CamelApplication         : No active profile set, falling back to default profiles: default
2021-11-08 15:09:27.075  INFO 67677 --- [           main] c.s.b.CamelSpringBootApplicationListener : Starting CamelMainRunController to ensure the main thread keeps running
2021-11-08 15:09:27.091  INFO 67677 --- [           main] org.eclipse.jetty.util.log               : Logging initialized @2275ms to org.eclipse.jetty.util.log.Slf4jLog
2021-11-08 15:09:27.134  WARN 67677 --- [           main] o.a.c.c.jetty.JettyHttpComponent         : You use localhost interface! It means that no external connections will be available. Don't you want to use 0.0.0.0 instead (all network interfaces)? jetty:http://localhost:8080/todos?httpMethodRestrict=GET
2021-11-08 15:09:27.178  INFO 67677 --- [           main] org.eclipse.jetty.server.Server          : jetty-9.4.44.v20210927; built: 2021-09-27T23:02:44.612Z; git: 8da83308eeca865e495e53ef315a249d63ba9332; jvm 16.0.1+0
2021-11-08 15:09:27.206  INFO 67677 --- [           main] o.e.jetty.server.handler.ContextHandler  : Started o.e.j.s.ServletContextHandler@1c5d3a37{/,null,AVAILABLE}
2021-11-08 15:09:27.234  INFO 67677 --- [           main] o.e.jetty.server.AbstractConnector       : Started ServerConnector@41def031{HTTP/1.1, (http/1.1)}{localhost:8080}
2021-11-08 15:09:27.234  INFO 67677 --- [           main] org.eclipse.jetty.server.Server          : Started @2420ms
2021-11-08 15:09:27.236  INFO 67677 --- [           main] o.a.c.impl.engine.AbstractCamelContext   : Routes startup summary (total:1 started:1)
2021-11-08 15:09:27.236  INFO 67677 --- [           main] o.a.c.impl.engine.AbstractCamelContext   :     Started route1 (rest://get:/todos)
2021-11-08 15:09:27.236  INFO 67677 --- [           main] o.a.c.impl.engine.AbstractCamelContext   : Apache Camel 3.12.0 (camel-1) started in 401ms (build:46ms init:190ms start:165ms)
2021-11-08 15:09:27.240  INFO 67677 --- [           main] com.bnova.camel.CamelApplication         : Started CamelApplication in 2.119 seconds (JVM running for 2.426)

In line 14 we can see that the route has been registered. We can now call up the route once via curl and then see the info message in the log, as expected.

curl localhost:8080/todos
2021-11-08 15:27:00.147  INFO 67677 --- [qtp177267393-29] info                                     : Exchange[ExchangePattern: InOut, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: [Body is instance of org.apache.camel.StreamCache]]

ToDo route

Now we want to create the route that retrieves the data from the backend service and prepares it accordingly. For this we create a new class ToDoRoute. In this we define our second route.

package com.bnova.camel;

import org.apache.camel.Exchange;
import org.apache.camel.builder.endpoint.EndpointRouteBuilder;
import org.eclipse.jetty.http.HttpMethod;
import org.springframework.stereotype.Component;

@Component
public class ToDoRoute extends EndpointRouteBuilder {

	@Override
	public void configure() throws Exception {
		getContext().setMessageHistory(true);

		from(direct("todo-route"))
				.routeId("todor-route")
				.removeHeaders("*")
				.setHeader(Exchange.HTTP_METHOD, constant(HttpMethod.GET))
				.setHeader(Exchange.CONTENT_TYPE, constant("application/json"))
				.to(http( "localhost:3000/todos").bridgeEndpoint(true));
	}
}

Line 15:

We start our route with a direct entry point, which we will specify later in our RestRoute.

Line 16:

We delete all headers. There is an error every second call to the upstream, because some E-tag is evaluated.

Line 17 + 18:

We define the headers for the call to the upstream.

Line 19:

This is where the definitive call to the upstream is made.

Line 20:

We log the body.

Now that we have created the lower ToDoRoute, we want to call it up in our RestRoute. To do this, we change the following line in the rest of the route as follows.

rest().get("/todos")
				.consumes("application/json")
				.produces("application/json")
				.to("log:info") --> .to(direct("todo-route"));

Now we have to restart the application and can do a curl again. As we can see, the output is the Json of the todos endpoint of our backend rest service.

$ curl localhost:8080/todos
[
  {
    "id": 1,
    "title": "delectus aut autem"
  }
]

Now we want to have the detailed information of the todo output. So we want to enrich the information from one backend service with the information from the second backend service. Camel offers us a enricher for this purpose, which was made precisely for this purpose, namely to enrich content. To do this, we are expanding our ToDo route as follows:

package com.bnova.camel;

import org.apache.camel.AggregationStrategy;
import org.apache.camel.Exchange;
import org.apache.camel.builder.endpoint.EndpointRouteBuilder;
import org.apache.camel.component.jackson.JacksonDataFormat;
import org.apache.camel.component.jackson.ListJacksonDataFormat;
import org.apache.camel.model.dataformat.JsonLibrary;
import org.eclipse.jetty.http.HttpMethod;
import org.springframework.stereotype.Component;

import javax.annotation.PostConstruct;
import java.io.InputStream;
import java.util.Collection;
import java.util.List;
import java.util.function.Function;
import java.util.stream.Collectors;

@Component
public class ToDoRoute extends EndpointRouteBuilder {

	private JacksonDataFormat jacksonDataFormat;

	@PostConstruct
	public void init() {
		this.jacksonDataFormat = new ListJacksonDataFormat(ToDo.class);
	}

	AggregationStrategy aggregationStrategy = new AggregationStrategy() {

		@Override
		public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
			List<ToDo> toDo = oldExchange.getIn().getBody(List.class);
			try {
				InputStream body = newExchange.getIn().getBody(InputStream.class);
				List<ToDo> toDoDetails = (List<ToDo>) jacksonDataFormat.unmarshal(newExchange, body);
				toDo.addAll(toDoDetails);
				Collection<ToDo> todos = toDo.stream().collect(Collectors.toMap(ToDo::getId, Function.identity(), (left, right) -> {
					left.setCompleted(right.isCompleted());
					left.setUserId(right.getUserId());
					return left;
				})).values();
				oldExchange.getIn().setBody(todos);
			} catch (Exception e) {
				e.printStackTrace();
			}

			return oldExchange;
		}
	};


	@Override
	public void configure() throws Exception {
		getContext().setMessageHistory(true);

		from(direct("todo-route"))
				.removeHeaders("*")
				.setHeader(Exchange.HTTP_METHOD, constant(HttpMethod.GET))
				.setHeader(Exchange.CONTENT_TYPE, constant("application/json"))
				.to(
						http( "localhost:3000/todos")
								.bridgeEndpoint(true))
				.unmarshal(jacksonDataFormat)
				.enrich(http("localhost:3000/todo-details"), aggregationStrategy)
				.marshal().json(JsonLibrary.Jackson, ToDo.class);
	}
}

Row 29-50: We define our Enricher and do some Java Stream Magic so that we can enrich one list with the data from the other list.

Line 64: Here we convert the response from the first call to the upstream into a list of ToDo items

Line 65: We use the enrich-component to enrich the first list with a call to the second backend endpoint.

Line 66: Now we make a string out of the list again, which we output to the consumer.

Let's look at the result after restarting the server.

$ curl localhost:8080/todos1                                                                                                                                                                                                                      2.6.3 16:02:19
[ {
  "userId" : 1,
  "id" : 1,
  "title" : "delectus aut autem",
  "completed" : false
}, {
  "userId" : 1,
  "id" : 2,
  "title" : "quis ut nam facilis et officia qui",
  "completed" : false
}, {
  "userId" : 2,
  "id" : 3,
  "title" : "fugiat veniam minus",
  "completed" : false
}, {
  "userId" : 2,
  "id" : 4,
  "title" : "et porro tempora",
  "completed" : true
}, {
  "userId" : 3,
  "id" : 5,
  "title" : "laboriosam mollitia et enim quasi adipisci quia provident illum",
  "completed" : false
} ]% 

As we can see, our data from both systems is now aggregated.

In this techup, we only scratched the surface of Apache Camel to show how effectively you can build endpoints for consumers that have to aggregate and process data from multiple backends. So if you want to conjure up a fast integration with an established framework and little code, then Apache Camel is the right choice!

(blue star)


This text was automatically translated with our golang markdown translator.

Stefan Welsch - pioneer, stuntman, mentor. As the founder of b-nova, Stefan is always looking for new and promising fields of development. He is a pragmatist through and through and therefore prefers to write articles that are as close as possible to real-world scenarios.