FTP BC Recovery

Wiki Main

FTP BC realizes recovery using "inbound message processing journaling" - a local file system or NFS based journal which records (persists) the inbound messaging context for each every inbound message consuming, it ensures that an inbound message consuming can survive the followings:

• Uncorrectable Failure During Inbound Message Consuming (connection interruption, IO error, etc.)
• Runtime Termination (Crash of Application, Components - BC or SE etc., Container, or Application Server)

And, when the application starts next time, the concerned message consuming can be resumed and the corresponding message routing completes based on the context information persisted in the recovery journal.

For details, please refer to FTP BC Transactional Messaging & Clustering Support & Message Sequence Numbering .

Recovery After Uncorrectable Error

Wiki Main

The followings demonstrate how an uncorrectable error interrupts inbound message consuming and gets recovered when the application gets fixed and restarted again.

Application Diagram:

This application uses FTPBC to poll the remote target directory <message_exchange_base_dir>/inbox for incoming message (files that match specified name pattern), when found, move to inbound staging area <message_exchange_base_dir>/inselect, then start reading the content of the file, when complete, parse the content and wrap it up in a JBI message (normalization), when done, send it into NMR, NMR routes the message to the downstream component - in this case - a BPEL SE executing some bpel script (a pass thru), the bpel script invokes local file write service which uses FILE BC outbound to: denormalize the message and write the payload to a local file - output%d.txt.

For details on how to create SOA BPEL module and adding new WSDL documents and BPEL processes, as well as adding BPEL modules to a SOA composite application, please refer to : Create Composite Application Using FTPBC

The following screen shots refelects the steps showing recovery in action:

Step 1: Create SOA BPEL Module, Add FTP Inbound One Way Messaging Service, Add FILE Outbound Service, Add BPEL process: GETFROM_FTP_2_LOCAL.bpel

• Create a SOA BPEL module: DOAN_LOAD_FTP
• Define a local file service to write a message to a local file, save the definition to FILE_WRITE.wsdl
• Intentionally specify a local path with a non existing drive letter, so that the operation will fail - create a uncorrectable error

• Defined a ftp service to poll a remote message, save the definition to FTP_GET.wsdl

• Create a bpel process doing a copy through, save the bpel script to GETFROM_FTP_2_LOCAL.bpel
• Configure the BP as atomic so that the file outbound error will bounce back to the ftp inbound

Step 2: Create SOA Composite Application

• Create a composite application service assembly: DOWN_LOAD_FTP_COMP_APP, and adding the BPEL module: DOWN_LOAD_FTP.jar

• Build the composite application and edit it in CASA editer

Step 3: Build and Deploy the composite application

• Build and deploy

• Exam the deployed project through JBI manager

Step 4: Start Messaging by feeding a inbound message

• Manually feed a file at the ftp directory where the inbound processor is polling

• Exam the local directory to see if the file is written, since the path d:\temp is invalid, the attempt to write to d:\temp\output0.txt will fail

• The recovery log is stored at the persistence store, located at:

• Location of the message which the message routing is not completed

• Content of a recovery log entry : it records the context info for the message that failed delivery, which will be used by the recovery logic when next time the application starts

Step 5: Undeploy and Fix the WSDL

Step 6: Build and Deploy Again

Step 7: Exam recovery log and local directory

• Recovery log entry is removed - processed
• There is the output message at the local directory c:\temp\output0.txt

Recovery After System Crash

Wiki Main

In following shows how recovery works when an inbound consuming was interrupted by application server crash.

The same composite application used in previous section "Recovery After Uncorrectable Error" is used for the demo.

Step 1: Build and deploy the composite application

Step 2: Enabled debug for bpel se

Enabled debugging in BPEL SE

Step 3: Set break point in the bpel script

Set break point in netbeans IDE
Set break point in the bpel script source code

Step 4: Attach to the bpel se

Set the composite application as main project, get ready to attach


Set debug info on on the debugger side - netbeans IDE
Attach!!

Step 5: Feed inbound message

Feed inbound message, triggering the message consuming

Suspended after message polled but not ACKed or NACKed

Step 6: Kill the jvm for the application server to simulate a system crash

Locate and Kill the app server to simulate a crash



There is no output on the local destination



A recovery log entry persisted in the app server jbi persistence storage



Go to netbeans "Services" tab and refresh "Servers" node, the app server glassfish is not running

Step 7: Restart the application server which will also start the deployed composite application

When a service unit is started, the recovery will happen before the inbound processor goes into periodical polling:


Restarted the app server glassfish, and observing the server log

Server log shows that the particular recovery log entry has been processed successfully

Step 8: Exam recovery log and local destination directory

Check the local destination, the output has been written

Check the recovery log, it is cleaned up

Done!!

Malformed Message Handling

Wiki Main


Malformed messages are caught during normalization, when an external message is obtained from external system, it is normalized (parsed using a XML schema or custom encoder) and wrapped in a JBI Message object, then the normalized message is set in a message exchange and sent into NMR (Normalized Message Router).


If a message is malformed, i.e., does not comply with its schema, it is moved out of the way (in inbound staging area), and marked with a suffix ".err" indicating such, a error log is persisted in a dedicated location at the application server which records the details of the error and location of the offending message.

This process applies to all the scenarios that a message is retrieved from remote and going through JBI Message normalization before being pumped into NMR, in another words, this process applies to:

• Polling Side of: "Poll Request Message", "Put Request Message and Poll Response", "Poll Request Message and Put Response"
• Receiving Side of: "Receive Request", "Receive Request and Send Response", "Send Request and Receive Response"
• On Demand Get Message, On Demand Receive Transfer

For samples projects using complex XML message and custom encoded message please refer to:


Create Composite Application Using FTPBC - HelloDukeUsingComplexType

Create Composite Application Using FTPBC - HelloDukeUsingCustomEncoder

The following demo uses a composite application that poll a remote directory for inbound messages (e.g. Hello) and process the message and generate a response accordingly (e.g. Hello) and put the response to outbound directory.

The messages are delimited as is defined by Greeting.xsd with annotation specifying multi-level delimiters info:

A request message looks as follows:

Hello|Duke,this|is|Jay

A response message looks like follows:

Hello|Jay,this|is|Duke

Step 1: Create BPEL Module and Composite Application

Step 1.1: Create "POLL_AND_PUT_ENCODER" Module

Create Greeting.xsd annotated with custom delimited encoding info, for details of creating such a annotated xsd refer to: Create Composite Application Using FTPBC - HelloDukeUsingCustomEncoder
details omitted here for brevity:


The resulted annotated xsd containing custom encoder delimiter info.
Create the FTPBC WSDL POLL_AND_PUT_ENCODER.wsdl as follows:

Note, in the following screen shots, the WSDL is generated using FTPBC WSDL binding wizard, wsdl can also be generated using text editor, or generic WSDL editor from netbeans IDE, for examples, refer to: Create and Validate WSDLs

Set connection info

Set poll request message binding info : select "encoded data" as message type

Set poll request message binding info : select "root" element type for message part

Set poll request message binding info : specify the name of the encoder - 'customencoder-1.0' used by the runtime to lookup the corresponding encoder library

Set put response message binding info - through the same process as with request message binding info



The resulted WSDL for the definition of a service that poll a request (Hello) and process it and generate a personalized response bound to FTP transport


The resulted composite application rendered in CASA editor


Create BPEL process and tie it with the service defined above


BPEL script does the mapping between request message and the response - generating a personalized response

Step 1.2: Create Composite Application "POLL_AND_PUT_ENCODER_COMP_APP"

Create composite application "POLL_AND_PUT_ENCODER_COMP_APP" and add BPEL module "POLL_AND_PUT_ENCODER" to it.

Step 2: Build and Deploy

Details omitted for brevity, the following shows that the composite application is deployed successfully and it is up and running.
Exam the deployed composite application through JBI manager.

Step 3: Prepare sample messages

In order to show the behavior of malformed message handling, two messages are prepared:

one is a conforming message:



the other is a malformed message:

Step 4: Feed the two messages into the message processing

Drop the two messages in the inbox - they will be picked up in seconds

Step 5: Exam the various places to see malformed message handling in action

Good message has been through the message routing and the message has been archived after processed successfully.



Good message has been through the message routing and the corresponding response end up in outbox.




Bad message will be marked as malformed message and stay in the staging area.
Meanwhile, an error log will be created on the dedicated location at the application server(see below).


Location of the message error log at the application server.




The content of the message error log: it contains info about the location of the bad message and the detailed error cause.

Step 6: Resolve the malformed message

When a message is malformed, it is moved out of the way, it is the responsibility of the system administrator to exam the message error log, and fix the error, e.g. make the message conform to the schema, and put the message back into the processing again.

---

Wiki Main

This page (revision-51) was last changed on 27-Feb-09 18:37 PM, -0800 by jimfu.
This page was created on 24-Feb-09 17:22 PM, -0800 by jimfu.

More info...