Index Changes

Difference between version and version     

Back to Large Message Support, or Large Message Support Info


At line 1 added 1 line.
[{TableOfContents }]
At line 3 added 1 line.
At line 4 changed 1 line.
When Open-ESB is put into serious use to handle real messages, the ability and efficiency to handle messages by encoders and JBI components, especially the ability to process large messages, may become an important requirement in the related business scenarios. This can also become a big differentiator in JBI implementations.
When Open-ESB is put into serious use to handle real business messages, the ability and efficiency to handle messages by encoders and JBI components, especially the ability to process data of large size, may become an important requirement in the related business scenarios. This can also become a big differentiator in JBI implementations.
At line 11 changed 1 line.
Support of large message handling may be achieved through at least the following supports:
Handling of large message may be achieved through at least the following supports:
At line 15 removed 2 lines.
\\
Different use cases may require different approach with or without advanced parsing technology.
At line 18 added 2 lines.
Different use cases may require different approaches with or without advanced parsing technology.
At line 20 changed 1 line.
* Service data layer: API and implementation for an easy and unified way of handling data.
* Service data layer refers to the API and implementation for an easy and unified way of handling data.
At line 22 changed 3 lines.
* In JBI, currently DOM is used as the service data layer.
** Pros: standard, easy to use (reads whole document into memory), fast access with XPath
** Cons: performance, large memory footprint, cannot handle large messages, lack of validation (too loose), lack of meta-data introspection
* In JBI, DOM is currently used as the service data layer.
** Pros:
*** Standard (XML tree representation of whole document loaded in the memory)
*** Easy to use, very flexible API supporting CRUD operations
*** Fast access with XPath
** Cons:
*** Performance problem
*** Large memory footprint
*** Can not handle large messages
*** Lack of validation (too loose)
*** Lack of metadata introspection
At line 26 changed 2 lines.
* We need a SDL better than DOM. The proposed SDL overcomes drawbacks of DOM:
** better performance, small footprint, large message handling, validation, meta-data introspection
* So we need a better SDL than the current DOM.
* The proposed SDL overcomes drawbacks of DOM as follows:
** Better performance
** Small footprint
** Large message handling
** Validation
** Metadata introspection
At line 29 changed 1 line.
* Characteristics of the proposed SDL (many of these may be borrowed from SDO spec)
* Characteristics of the proposed SDL (many may overlap with SDO spec?)
At line 31 changed 2 lines.
** XSD: meta-model language
** non-XML data: converted to XML for processing in NMR or stored as an attachment
** XSD: metadata modeling language
** Non-XML data: either converted to XML for processing in NMR, or simply stored as an attachment
At line 37 changed 4 lines.
** Support dynamic metadata introspection – getDataType
** Support single base type inheritance and dynamic type substitution and enforcement (xsi:type) - have it with DOM – don’t want to lose it
** Native XPath support, convenient data conversion (helper methods) and copy support - SDL object has Xpath support built in - more efficient than applying Xpath externally on DOM
** Support partial scan of original non-XMLized data for XPath evaluation (handy for content based routing) - parsing the header without converting the whole document to XML
** Support dynamic metadata introspection – getDataType()
** Support single base type inheritance and dynamic type substitution and enforcement (xsi:type)
*** have it with DOM – don’t want to lose it
** Native XPath support with convenient data conversion (helper methods) and copy support
*** SDL object has Xpath support built in - more efficient than applying XPath externally on DOM
** Support partial scan of original non-XML data for XPath evaluation (handy for content based routing scenario)
*** parses the header without converting the whole document to XML
At line 44 removed 1 line.
At line 46 changed 1 line.
** The benchmark should cover parsing, XPath evaluation (for both reading and updating), building large dataset and marshaling.
** Benchmark should cover parsing, XPath evaluation (for both reading and updating), building large dataset and marshaling.
At line 50 changed 1 line.
** May use existing XML APIs as a “view” or ’façade”
** May use existing XML APIs as a “view” or “façade”
At line 54 changed 1 line.
** vtd-xml: http://vtd-xml.sourceforge.net/.
** [vtd-xml|http://vtd-xml.sourceforge.net].
At line 61 changed 1 line.
* URIs can be used to reference attachments as illustrated in XOP (XML-binary Optimized Packaging, see http://www.w3.org/TR/xop10/)
* URIs can be used to reference attachments as illustrated in XML-binary Optimized Packaging (W3C XOP)
At line 70 changed 1 line.
!!! Streaming (and partial evaluation)
!!! Streaming (and partial scan/evaluation)
At line 72 changed 1 line.
* No easy way to stream messages in JBI.
* No easy way to stream messages in JBI architecture.
At line 74 changed 1 line.
* One-pass streaming process is not feasible for Open-ESB.
* One-pass streaming process is not feasible for Open-ESB, for reasons below:
At line 81 changed 1 line.
** Currently all existing encoder providers (Custom Encoder, HL7 encoder, Cobol Copybook Encoder) use SAXSouce as input or output in order to avoid object creation. But the drawbacks of SAXSource are:
** Currently all existing encoder providers (i.e. Custom Encoder, HL7 encoder, Cobol Copybook Encoder) use SAXSouce as input or output in order to avoid object creation. But the drawbacks of SAX Source are:
At line 83 changed 6 lines.
*** lacking write capabilities.
** Proposed solution:
*** Enhance the encoder providers to also implement StAX Source - a “Pull” model XML source that support both read and write.
*** Develop features like partial XPath evaluation, where XPath expression can be evaluated while decoding and the process stops as soon as the value is retrieved.
*** Should be very useful for content based routing or heterogeneous message process.
*** only reading, no writing capabilities.
*** requires entire document to be read.
** Proposed solutions:
*** (1) Enhance the encoder providers to also implement StAX Source
**** StAX is a “Pull” model XML source that support both reading and writing.
**** StAX does not require to read the whole document. Portions can be skipped
*** (2) Develop features like partial XPath evaluation, where XPath expression can be evaluated while decoding and the process stops as soon as the value is retrieved.
**** Should be very useful for content based routing or heterogeneous message process.
At line 92 removed 1 line.
At line 95 changed 1 line.
Different use case may require different approach to handle data. For example, handling data in content based routing may require only peeking the header data via one-pass partial scan, whereas handling data in BPEL process may require random access to any piece of the data inside a fully parsed message. We have identified some uses cases of data handling (see following), which may be addressed using layered approach.
Different use case may require different approach to handle data. For example, handling data in content based routing may require only peeking the header data via one-pass partial scan, whereas handling data in BPEL process may require random access to any piece of the data inside a fully parsed message.
At line 97 changed 4 lines.
* Fully decoded small or mid-size message with random R/W access
** This is the most common case (found in BPEL, XSLT, XQuery etc.) and we must support it very well.
** Includes both building a message from scratch and updating an existing one.
** We cannot compromise on performance for this when supporting large message handling.
We have identified some uses cases of data handling (see following), which may be addressed using layered approach.
At line 102 changed 1 line.
* Content based-routing:
# Fully decoded small to mid-size message with random R/W access
** This is the most common case (found in BPEL, XSLT, XQuery etc.) and we must support it very well.
** Includes both creating a message from scratch and updating an existing one.
** We can not compromise on performance for this use case when supporting large message handling.
# Content based-routing:
At line 104 changed 1 line.
** Message size might be small or large.
** Message can be of small, mid or large sizes.
At line 106 changed 8 lines.
* Deferred payload processing:
** Message is composed of a header/envelope and payload.
** The payload size might be small or large.
** For decoding, read the header, and extract raw payload out as intact. Then based on header information, perform action with payload (e.g. persist payload to a DB).
** For encoding, construct a header and insert a raw payload.
* Fully decoded large message with sequential R/W access.
# Deferred payload processing:
** Message is composed of a header (or envelope) and payload.
** The payload size might be small, mid or large.
** For decoding, it reads the header, and extracts raw payload out as intact. Then based on header information, performs action with payload (e.g. persist payload to a DB).
** For encoding, it constructs a header and inserts a raw payload.
# Fully decoded large message with sequential R/W access.
At line 115 changed 3 lines.
** Encodeing example: Build from scratch an HL7 batch message with a batch header, a number of message payloads of same or different types, and a batch trailer.
* Fully decoded large message with random R/W access.
** Encodeing example: Create from scratch an HL7 batch message with a batch header, a number of message payloads of same or different types, and a batch trailer.
# Fully decoded large message with random R/W access.

JSPWiki v2.4.100
[RSS]
« Home Index Changes Prefs
This page (revision-6) was last changed on 02-Apr-09 14:00 PM, -0700 by LixinTang