jCavallotti - Java Development: May 2013

I'm working on a particular project which uses Mule ESB as a central piece, for adding a new feature to my project I need to consume Google's custom search API so for making it beautiful I'm going to create a connector using Mule's DevKit and showing step by step the process of building a connector that consumes a REST API.

First, I create a DevKit module from the archetype:

mvn archetype:generate -DarchetypeGroupId=org.mule.tools.devkit -DarchetypeArtifactId=mule-devkit-archetype-generic -DarchetypeVersion=3.4.0 -DarchetypeRepository=http://repository.mulesoft.org/releases/ -DgroupId=com.mycompany -DartifactId=mule-module-google-custom-search -Dversion=1.0.0-SNAPSHOT -DmuleVersion=3.4.0 -DmuleModuleName=GoogleSearch -Dpackage=com.mulesoft.module.googlesearch -DarchetypeRepository=http://repository.mulesoft.org/releases

Since I'm building a cloud connector for google's custom search API it is handy to have it's reference at hand:

https://developers.google.com/custom-search/v1/cse/list

You may also want to create a custom search engine:

http://www.google.com/cse/create/new

Lastly: here is the link for the source code of the project:

https://github.com/juancavallotti/mule-module-google-custom-search

Now let's the coding begin! First of all I want to clean up the sample module, pretty much removing everything except the class and it's annotations. I have picked to be a module (instead of a proper connector) because it's a fair simple functionality which does not require work for setting up a connection. The class looks like this:

/**
 * Google custom search Module.
 *
 * @author Juan Alberto López Cavallotti.
 */
@Module(name="google-search", schemaVersion="1.0.0-SNAPSHOT")
public class GoogleSearchModule {

}

So first we will add the required configuration parameters as stated at the documentation, these parameters are the API Key, and the custom search engine ID or it's url (one of both are required). The search query is also required but we'll save it for passing it as a parameter to the only operation this connector will have. I will give nice names to these parameters instead of the really short ones they currently have, this way is more self-documented.

@Module(name="google-search", schemaVersion="1.0.0-SNAPSHOT")
public class GoogleSearchModule {

    @Configurable
    private String apiKey;

    @Configurable
    @Optional @Default("")
    private String searchEngineId;

    @Configurable
    @Optional @Default("")
    private String searchEngineUrl;

    //getters - setters
}

DevKit demands us to write javadoc for each element on the class so it is able to generate the connector's documentation properly. I won't show this documentation on this post unless I have something to say about it, having said that, you might want to take a look at the final version of this module in order to get more insight on its development.

Next, I want to create the one and only operation as a method of this module definition, this method has a lot of parameters, most of all with default values. For me this is pretty ugly and DevKit allows me to do better so I will do, I will pick the most important parameters for direct setting and the others would be gathered through a POJO. The return value I will keep it as the JSON String the service returns (for the sake of simplicity) but kind of more beautiful would be to create an Object Structure and take advantage that Mule has bundled the Jackson Library.

/**
 * Perform a Google custom search.
 *
 * {@sample.xml ../../../doc/GoogleSearchModule-connector.xml.sample google-search:search}
 *
 * @param query The search query to send to google search.
 * @param siteSearch The site where to search.
 * @param searchType The type of the search to be performed.
 * @param searchConfiguration Configuration for this google search.
 *
 * @return The JSON result as returned by the google custom search API.
 */
@Processor
public String search(String query, @Optional @Default("") String siteSearch, 
                     @Optional SearchType searchType, @Optional SearchConfiguration searchConfiguration) {
    return null;
}

You need to consider the following:

The method must be annotated with one of the annotations provided by devkit to generate different types of message processors. (Each type has its own signature requirements).
Every operation should be documented properly.
You need to create a sample usage of the operation on the sample file, (which luckily it gets created by the archetype).
You need to document EVERY parameter if you wish the build to succeed.

<!-- BEGIN_INCLUDE(google-search:search) -->
<google-search:search query="#[header:inbound:query]" />
<!-- END_INCLUDE(google-search:search) -->

Now, the sample XML looks like the following:

I want to perform an initial configuration and validation when the module starts just to make sure we're ready to go when making a search so I create a method to take advantage of the configuration element lifecycle:

private HashMap<String, String> connectorConfigs;
 
/**
 * Perform module initialization.
 */
@Start
public void initializeConfiguration() {
    connectorConfigs = new HashMap<String, String>();
 
    connectorConfigs.put("key", apiKey);

    if (StringUtils.isBlank(searchEngineId) && StringUtils.isBlank(searchEngineUrl)) {
        throw new IllegalArgumentException("You must configure either searchEngineId or searchEngineUrl");
    }
 
    if (StringUtils.isNotBlank(searchEngineId) && StringUtils.isNotBlank(searchEngineUrl)) {
        throw new IllegalArgumentException("You must configure a reference to the custom search engine.");
    }
 
    addIfNotBlank(connectorConfigs, "cx" , searchEngineId);
    addIfNotBlank(connectorConfigs, "cref", searchEngineUrl);
}

This basically creates a map that will be the base parameters for every request. Every request should have at least a reference to the API key (which is the way google charges us for our searches) and the reference to our custom search engine (which we need to create manually in order to use the API).

Next, let's dig into the actual implementation. I will want to access the MuleEvent for this message processor so, I will modify slightly the signature of the method. What this implementation does is: create a map with the parameters to send to the API, then convert these maps into the API url and finally dispatch it through Mule's HTTPS connector. This is very convenient since it allows the configuration of the HTTPS connector parameters even though there is still room for improvement!! I could have made the http connector to use configurable but I won't just to keep it simple. The search method now looks like this:

@Processor
@Inject
@Mime("application/json")
public String search(MuleEvent event, String query, @Optional @Default("") String siteSearch,
                     @Optional @Default("WEB_SEARCH") SearchType searchType, @Optional SearchConfiguration searchConfiguration) {
 
    MuleContext context = event.getMuleContext();
 
    MuleMessage message = event.getMessage();
 
    HashMap<String, String> searchParams = buildSearchParams(query, siteSearch, searchType, searchConfiguration);
 
    String apiUrl = buildSearchUrl(searchParams);
 
    try {
        OutboundEndpoint endpoint = context.getEndpointFactory().getOutboundEndpoint(apiUrl);
 
        //configure the message.
        message.setOutboundProperty("http.method", "GET");
 
        MuleEvent responseEvent = endpoint.process(event);
        //return the payload.
        return responseEvent.getMessage().getPayload(String.class);
    } catch (MuleException e) {
        logger.error("Error while querying the google custom search API", e);
    }
    return null;
}

This is pretty straightforward. I won't get into the implementation of the auxiliary methods because it is just boilerplate. I want to highlight the @Inject annotation I used, this is used to get the actual MuleEvent (and distinguish this parameter from what the user needs to provide) and also the @Mime annotation which will generate the appropriate message header.

Now we just want to create the MuleStudio update site so we can install and try this module:

mvn clean package -Ddevkit.studio.package.skip=false

We can import this through update site and use it! There are furhter configurations I can (and probably will) make to this connector, DevKit has annotations for customizing the studio dialogs and much more!!

Here is a sample project which uses the recently created connector:

<mule xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns:google-search="http://www.mulesoft.org/schema/mule/google-search" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:spring="http://www.springframework.org/schema/beans" version="EE-3.4.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/google-search http://www.mulesoft.org/schema/mule/google-search/1.0.0-SNAPSHOT/mule-google-search.xsd">
    
    <google-search:config name="Google_Search" apiKey="<api key>" searchEngineId="<engine id>" doc:name="Google Search"/>
    
    <flow name="mule-configFlow1" doc:name="mule-configFlow1">
        <http:inbound-endpoint exchange-pattern="request-response" host="localhost" port="8081" doc:name="HTTP"/>
        <google-search:search config-ref="Google_Search" query="http connector" doc:name="Google Search" />
        <logger level="ERROR" message="Payload is: #[payload]" doc:name="Logger"/>
        <logger level="ERROR" doc:name="Logger"/>
    </flow>
</mule>

Please continue reading the DevKit cookbook for much more info:

http://www.mulesoft.org/documentation/display/current/Cloud+Connector+Devkit+Cookbook

The Issue

I have been developing lately an annotation processor and found that unit-testing it is not as trivial as it might sound.

Some would say "You just need to mock all of the java.lang.model.* packages and that's it!" Unfortunately mocking these elements is not as easy as it might appear and I don't really find a reason why to do this other than speeding up the test suite, this alternative has to be taken real seriously and prioritize the test time / development effort.

The solution

The path I went through is invoking the Java compiler within my unit tests. Doing this is kind of straightforward but some considerations must be taken.

In order for the tests to be repeatable, please make sure your working directory is invariant, maven does a good job at this.
It's easier if your annotation processor can be auto-discovered. This is place the appropriate "javax.annotation.processing.Processor" file on META-INF/services/
Make sure you delete the junk files after you have tested. Since we're going to compile classes some .class files may appear, it is not mandatory to delete them but they would get mixed with the source code which is not nice at all.

Now that we've taken care of the errands, I present you a generic annotation processing test case which you can use for your unit tests.

Please note that this is not the only (or strictly the best) way of doing this but is a way and I like it so, here is the code:

First, we have a simple interface which we will implement to build our test cases, implementations return the list of files to compile and then might verify if the compilation has failed of there were some compilation messages.

Next, we have a really simple sample implementation of a smoke test which verifies that no "Critical Warnings" or "Compiler Errors" have been issued from a valid configuration.

Finally we have the test class. For this I make use of the parameterized test runner that JUnit 4 provides and the parameters are exactly the different implementations of our simple interface.

Please note that in order to be completely generic, this unit test must obtain its parameters from configuration nevertheless I made it like this for the sake of simplicity and just keeping note of the room for improvement.

After this it is just boilerplate code, get the compiler, configure the file manager, compiling the relevant files, invoking the test and finally cleaning up after ourselves.

Please feel free to modify or upgrade this test case and I hope you find it as useful as it is to me.

jCavallotti - Java Development

Saturday, May 18, 2013

How to build a cloud connector for Mule ESB

Tuesday, May 14, 2013

How To Unit-Test an Annotation Processor

The Issue

The solution

Monday, May 6, 2013

jDTO Binder gains experimental support for compile-time config validation

Annotation Config Validation