Routing Slip

The first method to alter a route’s process is the Routing Slip pattern. The Routing Slip allows you to define a list of endpoints that should receive the message. In the Camel DSL, this looks like:

from("direct:start").routingSlip("decisionSlip");

The “decisionSlip” is header placed on the exchange containing a delimited list of the destination endpoints. The header can either be set on the ProducerTemplate:

producerTemplate.sendBodyAndHeader("direct:start", 
  order, "decisionSlip", 
  "direct:verify-account,direct:check-inventory,direct:place-order");

or using a bean method returning a list of endpoints:

from("direct:start")
  .setHeader("decisionSlip").method(OrderProcessorBean.class, "calculateRoute")
  .routingSlip("decisionSlip");

Additionally, you can use the @RoutingSlip annotation on a bean method, then call the bean in your route. First, the bean:

public class OrderProcessorBean {
  @RoutingSlip
  public String calculateRoute(@Body Order order) {...}
}

Calling the the OrderProcessorBean and invoking the annotated method looks like any other Camel route:

from("direct:start").bean(OrderProcessorBean.class);

While the @RoutingSlip annotation has its uses, I prefer to use other methods as the annotation isn’t obvious to the person reading or maintaining the route.

There are two important things to note about the Routing Slip. First and most obvious, the header containing the delimited list must be set in the route before the routingSlip(…) call is reached. The second point is the Routing Slip is only evaluated once, not after each step in the route. To determine the destination endpoint after each point in the route, we need to use a Dynamic Router.

Dynamic Router

Building on the Routing Slip is the Dynamic Router. Unlike the Routing Slip pattern, the Dynamic Router calculates the next endpoint at each step in the route. For example, if the route “direct:verify-account” returned an unverified account, the Dynamic Router can reroute to “direct:create-account”. The two steps to using a Dynamic Router are 1) define the logic and 2) define the route. The routing logic can be defined in a bean:

public class DynamicOrderProcessorBean {
  public String calculateRoute(@Body Order order, 
                               @Header(Exchange.SLIP_PREVIOUS) String prev) {
    if (prev == null) {
      return "direct:verify-account";
    }
    else if (order.getStatus() == OrderStatus.NO_ACCOUNT) {
      return "direct:create-account";
    }
    else if (order.getStatus() == OrderStatus.CHECK_INVENTORY) {
      return "direct:check-inventory";
    }
    else if (order.getStatus() == OrderStatus.IN_STOCK) {
      return "direct:place-order";
    }
    else {
      return null;
    }
  }
}

The above code is obviously contrived, but you get the idea. With the bean defined, the next step is to define the route:

from("direct:start")
  .dynamicRouter(DynamicOrderProcessorBean.class, "calculateRoute");

That’s really all there is to the Dynamic Router. Camel also supports a @DynamicRouter annotation which has similar usage to the @RoutingSlip annotation.

To find out more information on dynamic routing with Apache Camel, be sure to read the documentation on the Routing Slip and Dynamic Router patterns.

MongoDB, my choice for NoSQL data stores, addressed this shortcoming with its built-in map/reduce functionality. Suppose you want to count number of blog posts Bill has made. In a basic relational model, the SQL statement is straightforward:

select count(*) from blogposts where username='Bill';

Things are not so simple in MongoDB. To accomplish the same thing as the SQL statement shown above, you need to break out a map/reduce function. One representation of this function may look like:

db.blogPosts.group({
    cond: {username: 'Bill'},
    reduce: function(obj,prev) { prev.count++ },
    initial: {count : 0},
});

We can see how this new function maps to the SQL statement. The ‘cond’ parameter defines the conditions we want to match. In this case we’re selecting documents where the username attribute equals “Bill”. Next we define the reduce function. Put simply, this function does something with the objects the condition returns. The reduce function is passed the current object (obj) and the counter object (prev). Finally, the initial parameter initializes the attributes on the counter object. Executing the above function returns the count:

[ { "count" : 27 } ]

If you’re trying to accomplish this aggregation with the Spring Framework’s Data project, it’s quite a bit more cumbersome:

DBCollection collection = mongoTemplate.getCollection("blogPosts");
GroupCommand cmd = new GroupCommand(collection,
    null,
    new BasicDBObject("username", "Bill"),
    new BasicDBObject("count", 0),
    "function(obj,prev) {prev.count++;}",
    null);
return collection.group(cmd);

The ‘collection.group(cmd)’ statement returns a DBObject containing the count results.

MongoDB 2.2 introduces a simplified aggregation framework, making these operations much easier to understand and implement. This new aggregation framework will be the subject of a future post. In the meantime, I hope this has shed some light on how to perform operations in MongoDB you would normally take for granted in relational databases. This post has only scratched the surface of the current state of aggregation in MongoDB. For the full scoop, refer to the MongoDB Aggregation Framework documentation.

Unfortunately, ZooKeeper isn’t the easiest collection of services to just drop into your application. The learning curve can be fairly steep. This is where another Apache project comes in: Camel. Camel provides a ZooKeeper component and some supporting classes to take advantage of ZooKeeper’s more powerful features. The feature I’m most interested in is master election.

Camel’s ZooKeeper component supports master election using a standard route policy. The ZooKeeperRoutePolicy monitors a common znode on the ZooKeeper ensemble. For example:

ZooKeeperRoutePolicy routePolicy = 
    new ZooKeeperRoutePolicy("zookeeper://10.211.55.4:2181/status/reportgen", 1);

This route policy allows one client to bind to the “/status/reportgen” znode. If I wanted three clients to be able to bind to the znode, I would simply change the number in the route policy to 3. Once the route policy is defined, it is plugged into the route:

from("quartz://cluster/heartbeat?cron=0/30+*+*+*+*+?")
    .routePolicy(routePolicy)
    .to("mock:generate-reports");

The above Camel route executes every 30 seconds. The “mock:generate-reports” is only executed on the client able to bind to the relevant znode.

Before I discovered Camel’s ZooKeeper component, I tried several options for distributed master election, including ActiveMQ’s exclusive consumer and rolling my own master election process. From my experience, ZooKeeper offers the easiest out of the box solution for a distributed environment.

In the future, I’ll explore some of the additional challenges I faced implementing Camel’s ZooKeeper component. If you’re using ZooKeeper, let me know in the comments.

After firing up YourKit, probably my second-favorite Java development utility to IDEA, the problem became readily apparent. The Camel route passed a payload from CXF as a MessageContentsList to a custom Processor implementation which called the following:

DecisionRequest req = exchange.getIn().getBody(DecisionRequest.class);

Since a conversion from MessageContentsList to DecisionRequest was required, Camel looks up the necessary converter in the BaseTypeConverterRegistry’s getOrFindTypeConverter(…) method. Here’s the version of that method in Apache Camel 2.9.0:

    protected <T> TypeConverter getOrFindTypeConverter(Class<?> toType, Object value) {
        Class<?> fromType = null;
        if (value != null) {
            fromType = value.getClass();
        }
        TypeMapping key = new TypeMapping(toType, fromType);
        TypeConverter converter;
        synchronized (typeMappings) {
            converter = typeMappings.get(key);
            if (converter == null) {
                converter = lookup(toType, fromType);
                if (converter != null) {
                    typeMappings.put(key, converter);
                }
            }
        }
        return converter;
    }

By synchronizing access to the collection of typeMappings, threads were forced to block until they could access the collection. The easiest workaround was to avoid using Camel’s conversion mechanism and implement a custom converter in the processor.

Luckily, the Apache Camel team was quick to recognize and resolve the problem. The version of this method in Apache Camel 2.9.1 and 2.10 removes this block.

In 2010, Invidi closed a $23m investment round. The round was led by Google, arguably the keeper of more personal data than any other company, except for possibly Facebook. The Google investment isn’t earth-shattering. After all, Google’s quarterly golden egg is driven by advertising. Another company taking part in Invidi’s Series D round is far more disturbing.

Experian, the global purveyor of consumer information, not only invested in Invidi, but formed a partnership and obtained an observer seat on Invidi’s board of directors. Experian was able to buy a place at the digital advertising table for a few million dollars. The close relationship between a source of consumer data and a company able to translate that into pervasive advertising decisions signals the further erosion of consumer privacy.