Enterprise Integration Patterns - Split

Overview

The purpose of the split pattern is to take a single input message and output one or more output messages based on a boundary condition. The boundary condition can be specified in a wide variety of languages and is generally tied to the type of data being split (e.g. XML -> XPath). Some split expressions are more powerful than others, with the option to mangle or enrich the data as a side effect of being split.

Use Cases

The split pattern is very common in the integration space, particularly in batch-oriented environments. Some of the more common use cases follow (feel free to add your own!).

Large Message Processing

Take a large input message and split it into more digestible pieces, also known as chunking. This allows for more efficient memory utilization, as chunked pieces can be reclaimed by the garbage collector individually. Chunking also allows for a divide and conquer approach, where parallel processing results in faster overall throughput for a given message size.

Dispatch

A message needs to be unwrapped into individual pieces so that each piece can be processed by the appropriate message stream. Commonly paired with content-based routing.

Enveloped Data

Refinement of dispatch centered around an envelope structure in the input message. A classic example of this use case is receipt of an ANSI X12 EDI message which contains multiple transaction types inside a single input message. Split can be used to break out each transaction type and forward on to the appropriate back-end processing system.

Keep the Baby, Lose the Bath Water

Avoids situations where a large message is completed rejected due to one or two bad records.

IFL

Syntax

split [type] [name | expression]
type := splitter implementation (e.g. xpath, regex, xsl)
name := named configuration which contains the split expression(s)
expression := inline split configuration

Examples

Split with inline configuration

route do
   from "batch-inbound"
   split xpath ("/PurchaseOrder")
   to "po-process"
end

Note: I'm wondering if it would be more appropriate for the in-line configuration to be contained within 'do/end' blocks, similar to how broadcast is handled today. This may be a good general-purpose construct for scoping all IFL language features. Thoughts?

Split with external configuration

route do
   from "batch-inbound"
   split xpath "po-split"
   to "po-process"

> cat po-split.xml
/PurchaseOrder

Misc.

• The IFL parser should not place any constraints on the type identifier in a split definition. This means that the value for type can only be verified at the time of service artifact generation.
• Known types to the parser/editor should provide contextual help in the form of syntax highlighting, code completion, etc. For the time being, this will be limited to xpath, regex, and xsl configuration.
• Inline configuration for unknown split types present a problem for the IFL parser. There is really no other choice than allowing for late-binding validation in the plugin for the particular service type (see Configuration).

Configuration

Each type implementation will have its own specific configuration. There however is a configuration which can be defined for all types in general:

Attribute	Description	Default
batchSize	Limits the of output messages that are processed in parallel.	Default is 1000. If the inbound message exchange is transactional (I.E. MessageExchange.isTransacted() returns true) the batchSize is forced to 1

The service artifact layer in Fuji will have a reasonable default behavior for all split types. Individual split types can override the default behavior by implementing the following contracts:

• Generation - the configuration generated from IFL for a given split type. A facility must exist for discovering template service artifacts for a given split type. This could be an SPI or a convention for resource location inside a bundle/jar file. For external configuration, a default implementation should be provided that results in an empty file with the split configuration name used as the file name. Generation is not relevant for inline configuration.
• Validation - during the packaging phase of the application build, split types will have an opportunity to validate configuration. This should be exposed as an SPI that a split type provider can implement.

The directory layout for generated configuration should be:

app-root/
   split/
      {name}/
           [configuration]

Where '{name}' is the named configuration provided in IFL.

XPath

The XPath filter type provides a simple facility for breaking up a single XML input message into multiple XML output messages. This filter is useful for messages which have a recurring element that can be processed as is (i.e. without enrichment or other processing).

Input Message

<orders>
   <order>
      <item>bang001</item>
      <desc>guns</desc>
      <amount>50</amount>
   </order>
   <order>
      <item>parkayXYZ</item>
      <desc>butter</desc>
      <amount>100</amount>
   </order>
</orders>

XPath Configuration

/orders/order

Output Messages

<order>
   <item>bang001</item>
   <desc>guns</desc>
   <amount>50</amount>
</order>

<order>
   <item>parkayXYZ</item>
   <desc>butter</desc>
   <amount>100</amount>
</order>

XSL

Like the XPath filter, the XSL filter allows for data from an input message to be extracted and forwarded as one or more output messages. The additional capability that XSL provides is the ability to mangle the data as part of the split. A common mangling use case is the inclusion of correlation data from a parent element in a child element that is split into a separate message.

Input Message

<orders>
   <orderNum>PO123</orderNum>
   <order>
      <item>bang001</item>
      <desc>guns</desc>
      <amount>50</amount>
   </order>
   <order>
      <item>parkayXYZ</item>
      <desc>butter</desc>
      <amount>100</amount>
   </order>
</orders>

XSL Configuration

Output Messages

<orders>
   <orderNum>PO123</orderNum>
   <order>
      <item>bang001</item>
      <desc>guns</desc>
      <amount>50</amount>
   </order>
</orders>

<orders>
   <orderNum>PO123</orderNum>
   <order>
      <item>parkayXYZ</item>
      <desc>butter</desc>
      <amount>100</amount>
   </order>
</orders>

Content enrichment is another common use case, where data is inserted into a split message that is not defined in the original message content model.

Regex

The regex (regular expression) split type is useful for non-XML data. The user provides a regular expression as the configuration for the split and each match of the expression marks a message boundary for the split.

Input Message

ORDER|PO123
ITEM|bang001|guns|50
ITEM|parkayXYZ|butter|200
ORDER|PO456
ITEM|oopsABC|huggies|300

Regex Configuration

"ORDER"

Output Messages

ORDER|PO123
ITEM|bang001|guns|50
ITEM|parkayXYZ|butter|200

ORDER|PO456
ITEM|oopsABC|huggies|300

Other

• X12 - ANSI X12 EDI data is encapsulated within interchange, group, and document envelopes. This split can be configured to extract messages at any of these levels. In addition, information from parent envelopes can be included in the split data if required.
• EDIFACT - Another EDI data format. Pretty much the same story (with different syntax) as X12.
• HL7

Packaging

At application build time, all split configurations should be packaged as part of the application. Since the ultimate consumer of this configuration information is the runtime split implementation, and not a component, it is not appropriate to package the individual configuration instances into a service unit. Rather, the following convention should be followed:

app-root /
      META-INF /
            flow/
                 split/
                    [configuration]

Where '{name}' is the named configuration provided in IFL. This layout mimics the configuration layout, but includes META-INF/ and flow/ as parent directories.

All split configuration should be included in this directory, including inline configuration. Since inline configuration is not named, a name must be generated. Using a pattern of 'split$n' should work, where 'n > 0' and increments for each inline split definition in the IFL.

Individual split types may have specific requirements around how the service artifacts are packaged, so an SPI should exist to allow these types to process the split configuration output before it is bundled inside an application.

Runtime

This page (revision-14) was last changed on 21-Jul-09 09:42 AM, -0700 by MarkWhite.
This page was created on 14-Aug-08 19:29 PM, -0700 by KeithBabo.

More info...