Sunday, December 7, 2014

Basic integration with Spring Boot

Introduction

The less I hack into a third-party piece of software, the better. Even if it’s open source software I know that any modification to the core system is likely to make support and upgrades more difficult. Integration through Application Programming Interfaces (APIs) is a really handy approach to take that can reduce the impact of coercing systems to work together. Essentially, you have data coming in and out of the system from/to different sources. Unfortunately, these sources don’t often work in a manner that just lets you “wire up” the communications.

Enterprise Integration Patterns provides a highly useful set of patterns that can help you solve a variety of integration problems. Spring Integration provides a number of mechanisms for integrating systems. In this article I’ll outline and build a very basic integration example.

For incoming data (from our source system) an “inbound channel” is used. The inbound data may then be handled in some way (maybe enhanced or transformed). The result is then sent to an “outbound channel”(to our client/receiving system). Spring Integration provides us with a very useful set of ready-to-go inbound/outbound channels:

  • Message Queues
  • Filesystem
  • FTP
  • Databases (JDBC, JPA)
  • Mail
  • Syslog
  • Twitter
  • Web sockets
  • Web services

Sitting between the inbound and outbound channel is some logic that might convert, extract, enhance etc the data.

The example I’ve prepared is based on a situation I’ve seen over numerous projects. Essentially, a source system will create a data export (usually a comma-separated file) on a periodic basis and put the file into a directory. The client system grabs the file from the directory and then processes it in some way. Often, the client system can’t accept the data as-is and needs to manipulate it in some way to meet its needs. A SysAdmin will turn to bash and cron to create a file poller and the developer will maybe reach for perl. I’d recommend that both could easily start to reach for Groovy and Spring Boot + Integration.

Implementation

Design

The example system is a very basic implementation of the File Transfer pattern that will:

  • Monitor a directory for new files that match a name pattern (Inbound file channel)
  • Copy new files to another directory (Outbound file channel)
  • Log that this is being done (Basic service activator)

Drawn using the enterprise integration icons the process appears thus:

Integration diagram for inbound and outbound file channels

It’s a very basic model and, of itself, isn’t something with a great utility. However, it provides an basic implementation for building on. The key features in the example are:

  • Spring Boot takes almost all of the heavy lifting around preparing the application
    • Including allowance for application properties to use defaults, be passed in via the command line, or be provided in a configuration file
  • Spring Integration’s built-in file channel adapter handles the inbound and outbound file
  • The file is passed (by the service activator) through to a POGO (Plain Old Groovy Object) handler
    • At present this handler just logs the fact it’s been called and returns the file without any processing.

Components

The following components are utilised in the solution:

Code

The code for this article is located in the Workbench Bitbucket repository.

Discussion

The build.gradle file defines the build and dependencies. You’ll see it makes use of the spring-boot plugin and the spring-boot-starter-integration dependency. As I’ve used the Gradle wrapper you should be able to run the code without installing Gradle - just run ./gradlew run.

When the system is running you can look in /tmp and see the input and output directories. Add a file into input and you’ll see that the running Gradle project log will show you a message from demo.Handler that the file was copied. Check out the input directory and the file is gone - it’s now in output.

Let’s take a look at what’s happening:

  • The src/main/resources/application.properties file sets the default inputDir, inputFilePattern and outputDir values
  • The src/main/resources/integration.xml file defines the Spring Integration:
    • An inboud-channel-adapter monitors files in inputDir that match the inputFilePattern (which is an Apache Ant-style pattern)
      • It also performs auto-create-directory - to make sure the inputDir is there
      • You’ll see that a poller is defined to run periodically
    • An outbound-file-channel uses the outputDir configuration for the output directory
      • Again, auto-create-directory is used
      • The mode uses REPLACE so that old copies of files in outputDir are written over
      • You’ll also see delete-source-files is used - this removes the inputDir copy
    • A service-channel is configured for the outbound channel.
    • The service-activator really brings it all together by putting the demo.Handler class’s handleFile method between the inbound and the outbound channels.
      • This class is defined in src/main/groovy/demo/Handler.groovy and it’s pretty basic
  • To get the whole thing running, src/main/groovy/demo/Application.groovy loads the configuration (including integration.xml) and starts up the system.

Whilst you can run this all in Gradle, take a look in the build/libs directory and you’ll find boot-integration-demo-0.1.0.jar - it’s quite large because it’s self-contained. You can run it using java -jar build/libs/boot-integration-demo-0.1.0.jar and this makes it easily distributable.

Spring Boot also gives us the ability to override the settings in application.properties via a few options. Each property can be changed using double dash (--) parameters on the command line - allowing the command line below to reconfigure the essentials:

java -jar build/libs/boot-integration-demo-0.1.0.jar --inputDir=/tmp/input2 --outputDir=/tmp/output2 --inputFilePattern=*.txt

This is really a very basic example to build off. There’s a wide range of directions you could take, including:

  • Push the file at a database, web service or message queue
  • Create a more sophisticated Handler.java that manipulates the file or extracts specific information

Anyway, Spring Integration combined with Spring Boot and Groovy provides a really strong toolset for tying together your software ecosystem.

Further reading

If you’re new to Spring Boot and Spring Integration then it’s a good idea to head to spring.io and look at their documentation and tutorials - they’re really useful.

Check out Enterprise Integration Patterns and the seminal book on the topic:

G. Hohpe and B. Woolf, Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions. Addison-Wesley Professional, 2004. 

The Apache Camel website also provides a great index of the patterns as well as links to drawing tool stencils.

Tuesday, December 2, 2014

An HTTP server for testing

Introduction

At various times I've needed to check what an HTTP server is receiving from a client system. This can be useful in many scenarios such as if an upstream system is posting data and you don't think it's sending what's expected. The code I present here is a diagnostic tool that lets you see what an HTTP client is sending through.

Implementation

Design

As this will be a tool I want to use in a variety of environments I've gone for an individual Groovy script. This will let me copy it to a server and get it running quickly. Other than that I'm really just needing a bare-bones HTTP service and the ability to log what it receives.

This approach assumes that Groovy is installed on the server but, if needed, I could package up the compiled Groovy into a jar file and run it under a plain old Java VM.

Components

The following components are utilised in the solution:

Code

The code for this article is located in the Workbench Bitbucket repository.

The codebase consists of a single script (http.groovy) to instantiate an HTTP server and log incoming requests.

To start the HTTP server I just run groovy http.groovy but the following is also supported:

  • groovy http.groovy -h - for usage info
  • groovy http.groovy -p <port> - to change the server port

To send a request to the server I could use a number of tools (even a browser) but the following curl examples are useful:

  • curl --data "param1=value1&param2=value2" http://127.0.0.1:9999 
    • Performs a basic POST request with some query items
  • curl -G --data "param1=value1&param2=value2" http://127.0.0.1:9999
    • Performs a basic GET request with some query items
  • curl -d @test.xml --header "Content-Type:application/xml" http://127.0.0.1:9999
    • POSTs an XML file

Discussion

I won't break down the script here as I've added enough information within the code. As a diagnostic tool it's important not to see the solution as universal. In its current state the script will provide basic information and, provided you're only POSTing a text-based file, should give you a starting point. I would generally copy this for each specific use and adapt the code (primarily the HttpHandler class) to meet the incoming request.

Further reading

Monday, December 1, 2014

Kick-off

This Workbench blog is an effort to publish and describe some of the mini coding projects I undertake. The goal isn't production code - it's mainly problem solving and, at the moment, it's mainly in Groovy. Watch this space.