1. Introduction 
    1.1. Project/Component Working Name:
      ETL (Extract-Transform-Load) Service Engine

    1.2. Name(s) and e-mail address of Document Author(s)/Supplier: 

      Ahimanikya Satapathy : Ahimanikya.Satapathy@sun.com
      Srinivasan Rengarajan: srengara@dev.java.net
      Nilesh Apte: nilesh_apte@dev.java.net


    1.3. Date of This Document: 
       April 01, 2007

2. Project Summary
    2.1. Project Description:
      ETL stands for Extraction, Transformation and Loading. ETL processes are 
      used with database for bulk data transfer, data warehouse, data cleansing
      etc. ETL Service Engine is a Java Business Integration (JSR-208) based 
      Service Engine which can expose the ETL operations as web services. Once 
      exposed, the ETL operation can be invoked by any external client. From ETL
      point of view it is always a service provider and someone in going to 
      consume that service.

      ETL has three parts: ETL Editor, ETL Project, ETL Service Engine
      ETL Editor is a design time NetBeans module which allow user to design ETL 
      process in graphical way. The end result of using editor is a etl definition
      xml file which describes the etl process.

      ETL Project is also a design time NetBeans module which allows user to 
      package ETL related artifacts in a jar file which could be deployed onto 
      ETL service engine. ETL project is a container for all etl definition xml 
      file. User can create an ETL project and then create as many etl definition 
      xml file (.etl files) using etl editor.

      ETL service engine is JBI compliant service engine implementation which can
      be deployed to a JBI container. The service unit jar which is produced by
      ETL project is consumed by ETL service engine. ETL Engine is the core kernel
      which implements the ETL functionality. ETL service engine exposes this  
      functionality in the JBI based ESB environment. 

    2.2.Risks and Assumptions:

     **TBD**

3. Business Summary
    3.1. Problem area
      ETL, Short for Extract, Transform, Load; three database functions that are
      combined into one tool that automates the process to pull data out of one 
      database and place it into another database.
    
    3.2. Market/Requester:
      No concrete one yet, will target customers with CAPS experience and used
      eTL in the past. Data Integration is an integral part of Business Integration
      problem and hence this Service Engine tries to address this aspect so that
      Customers applying OpenESB for their Business Problems can leverage this engine
      to solve Data Integration as well. 

    3.3. Business Justification:
      A common mistake is to write custom programs to perform the extraction, 
      transformation, and load functions. Writing an ETL program by hand may 
      seem to be a viable option because the program does not appear to be too
      complex and programmers are available. However, there are serious 
      problems with hand-coded ETL programs.

      1) Unlike OLTP applications, the functions to be supported by individual 
      data marts cannot be predicted in advance.

      2) Metadata is not generated automatically by hand-generated ETL programs.

      3) Hand-coded ETL programs are likely to have a slower speed of execution,
      compared with directly executable code generated by off-the-shelf ETL tools.


    3.4. Competitive Analysis:
      **TBD**

   3.5. Opportunity Window/Exposure:
      **TBD**

4. Technical Description
    4.1. Details
      ETLSE wsdl extension element is installed as part of ent-pack in NetBeans.
      etlse.jar is installed from Application Server JBI category.

      4.1.1 . Installation and Packaging.

            Composite Application SA contents
               provider- ETL SU
               consumer - Httpbc SU
               META-INF/jbi.xml
 
            ETL Service Unit contents
               etlEngine.xml
               etlEngine.wsdl
               etlMap.xml
               META-INF/jbi.xml


      4.1.2. Dependence: 
       The ETL SE has dependence on following third party libraries:
       1. axiondb.jar
       2. wsdl4.jar
       3. commons-logging.jar
       4. commons-codec.jar
       5. commons-primitive.jar

       The ETL SE also depends on following shared libraries provided by JBI:
       1. WSDL 1.1 wrapper
       2. Status Provider
       3. Management Message
       4. Runtime Configuration


      4.1.3. Startup.
       By default, the lifecycle module starts during appserver startup.

      4.1.4. Administration integration:
       Fully integrated with JBI administration GUI or CAM.

5. Reference Documents:

JBI documents. 
http://www.glassfishwiki.org/jbiwiki/Wiki.jsp?page=GlassfishJbiIntegration

ETL Inegrator @ glassfish wiki
http://www.glassfishwiki.org/jbiwiki/Wiki.jsp?page=ETLSE

This page (revision-4) was last changed on 06-Apr-07 05:02 AM, -0700 by AhiSatapathy.
This page was created on 06-Apr-07 04:01 AM, -0700 by rsrini.

More info...