Overall Functionality - Delimited Data Parsing (Parser) and Encoding (Encoder), Fixed Length Data Parsing and Encoding

• Custom Encoder, also known as User Defined OTD in Java CAPS 5, facilitates user to handle custom coded (more specifically speaking, it is delimiter or fixed-length coded) structural data.
• The implementation of custom encoder provides tools for user to define and test custom encoders, as well as provides runtime support (e.g. decoding, encoding) for custom encoders.
• Custom encoder definition, which is represented in form of XSD, is used in WSDL just like normal XSDs with the convention that usually an element declaration is used in WSDL, not any of the XML types.

In composite applications, custom encoder is used the same way as all other encoder types are used, which is, in a WSDL's binding section specifying the encoding style in the encoding style property of a message type. Currently the encoding style value for using custom encoder is "customencoder-1.0".

To learn more about encoder in general, please visit An Introduction to Encoder.

Keywords: Parsing Delimited Data, Parsing Fixed Length Data, Parsing, Parser, Encoding, Encoder, XML, Conversion, Converter, XML Schema

High Level Requirements

The requirements of custom encoder fall into the following two categories:

1. The design time tools: The design time tools shall facilitate user to create, mutate and validate custom encoder definition, as well as discover its runtime behavior at design time.
2. The runtime support: The runtime support shall provide efficient and fast algorithms to decode custom coded message (more specifically, it is delimiter or fixed-length coded) into XML message and encode XML message into custom coded message.

Use Cases

• Create abstract message definition. This use case shall be covered by NetBeans XSD editor.
• Apply the "custom encoding" style to the abstract message definition.
• Define detail custom encoding rules, such as
  • Define node type, delimited, fixed or transient?
  • Define the top node.
  • Specifying different types of character set for the top node, such as the input character set, the parsing character set etc. Specifying these character sets may needed because the decoding and encoding processes are byte-based.
  • Define the order of the children of a complex content node.
  • Define field length if the field is a fixed-length field.
  • Define field match, alignment if special matching or alignment is needed during parsing.
  • Define a delimiter list on a node, which defines a set of delimiters that will be applied to all descendant nodes of that node if they do not have their own delimiter list defined (a.k.a., delimiter inheritance). There are smaller use cases about defining the delimiter list, such as
    • Create delimiter level.
    • Create delimiter. Specify delimiter type, precedence, optional mode, terminator mode, offset and length etc.
• Validate custom encoder definition. This function shall detect any inconformity in a custom encoder definition and give out verbal description of all inconformity.
• Test custom encoder at design time. This function shall allow user to test the encoder's runtime encoding/decoding behavior at design time, which shortens the overall encoder design/build process. This use case shall be covered by the generic encoder testing tool.
• Specify the “custom encoding” style for different binding styles (such as FTP, FILE, JMS etc.)
• Provide custom decoding/encoding support at runtime

Please also refer to the Java CAPS 6 eGate User Guide/User-Defined OTD section for some detail use cases of this encoder.

Technical Details

• Custom encoder definition is stored in an XSD
• The abstract message structure is defined by normal XSD constructs
• The detail custom encoding rules is stored in XSD's Annotation/AppInfo constructs
• Custom encoder editor is NetBeans XSD editor plus an overlay/plug-in (which is called AppInfo provider) on top of it. The plug-in shows a custom node underneath the Annotation node.

XSD Governing the AppInfo

The following link pointing to an XSD, which describes the XML structure that is used to in AppInfo defining custom encoding rules:

XSD Governing the AppInfo

XSD Editor Plug-in for Custom Encoder

As stated in the Encoder Introduction section, each encoder type implementation may provide its own plug-in to XSD editor to support editing of its specific encoding rules. The implementation of custom encoder type does the same thing. Once an XSD gets custom encoding style applied, a node called "custom encoding" shows up underneath every element declaration's Annotation node, as below: Please note in the screenshot that a set of encoding related custom properties are shown for user to edit. When user changes the property values in the property sheet, the AppInfo in the underlying XSD model will be mutated. Complex property editing is also supported. For example, if user clicks on the Delimiter List property, the delimiter list editor screen will show up (as below). Once the value of this complex property changes, the value of the delimiter property of all the descendant element declarations of the current element declaration will also change (This is called delimiter inheritance).

A Simple Example

This example illustrates how to use the custom encoder to decode/encode the following simple custom coded message:
a,b|AABB
This message is first delimited by a vertical bar '|' and then delimited by ',' and the other part includes two fixed length fields (2 characters and 2 characters respectively), we can define following abstract message structure using XSD editor:

• element: root
  • element: groupNode
    • element: element1
      • element: delimField1
      • element: delimField2
    • element: element2
      • element: fixedField1
      • element: fixedField2

Then we can apply custom encoding style to this XSD and define detail encoding rules through the "custom encoding" node:

• element: root (NodeType=group, DelimiterList=2 levels ')
  • element: groupNode (NodeType=group)
    • element: element1 (NodeType=delimited, delimiter inherited as '|')
      • element: delimField1 (NodeType=delimited, delimiter inherited as ',')
      • element: delimField2 (NodeType=delimited, delimiter inherited as ',')
    • element: element2 (NodeType=delimited, delimiter inherited as '|')
      • element: fixedField1 (NodeType=fixed length, length=2)
      • element: fixedField2 (NodeType=fixed length, length=2)

Please note:

• Delimited node (element) does not have to define delimiter locally. Delimiter can be inherited from its ascendant nodes (elements). For example, element1, delimField1 and delimField2 all inherit delimiter from root.
• Only delimited node (element) will consume delimiter (actually the delimiter level) from the delimiter list. For example, root and groupNode do not consume delimiters, so element1 gets the delimiter '|' from delimiter level 1. After that, delimField1 gets delimiter ',' from delimiter level 2.

See a Demo

Custom Encoder Demo - an End-to-end scenario of Custom Encoder invoked in File BC.
TestCustomEncoderProjects2_and_Data.zip - NetBeans Projects and input data for the demo.

Want to Try It Out?

Please follow the steps outlined at the Encoder Feature Installation Guide page to proceed with the encoder feature installation.

This page (revision-56) was last changed on 30-Apr-09 13:29 PM, -0700 by LixinTang.
This page was created on 22-Jan-07 16:42 PM, -0800 by JunXu.

More info...