Skip to main content

Design a mapping to load the first record from a flat file into one table A, the last record from a flat file into table B and the remaining records into table C?


                                                                                                                                                                      
Scenario:
Design a mapping to load the first record from a flat file into one table A, the last record from a flat file into table B and the remaining records into table C?


Solution:

Please follow the below steps
  1. From source qualifier pass data to the exp1 transformation, Add an variable port as
  2. V_row_number. We can assign value to variable V_row_number by two ways
    1. By using sequence generator
    2. By using below logic in the expression transformation
V_ row_number =V_ row_number +1
O_ row_number =V_ row_number

Input, O_ row_number
a,                            1,                           
b,                            2,                           
c,                             3,                           
d,                            4,                           
e,                            5,                            

  1. Table A - In one pipeline, send data from exp transformation to filter where you filter out first row as O_ row_number = 1 to table A.
  2. Table B - Now again there are two ways to identify last records,
    1. Pass all rows from exp1 transformation to agg transformation and don’t select any column in group by port,it will sent last record to table B.
    2. By using max in agg
  3. Table c - Now send out of step 4 to an exp2 transformation, where you will get O_ row_number=5 then add a dummy port into a same exp with value 1 now join this exp2 with the very first exp1 so that you will get output like below
Input, O_ row_number, O_ last_row_number
a,                            1,                            5
b,                            2,                            5
c,                             3,                            5
d,                            4,                            5
e,                            5,                            5

Now pass the data to filter and add condition add O_ row_number <> 1 and O_ row_number <> O_ last_row_number


Comments

Post a Comment

Popular posts from this blog

CMN_1650 A duplicate row was attempted to be inserted into a dynamic lookup cache Dynamic lookup error.

Scenario: I have 2 ports going through a dynamic lookup, and then to a router. In the router it is a simple case of inserting new target rows (NewRowLookup=1) or rejecting existing rows (NewRowLookup=0). However, when I run the session I'm getting the error: "CMN_1650 A duplicate row was attempted to be inserted into a dynamic lookup cache Dynamic lookup error. The dynamic lookup cache only supports unique condition keys." I thought that I was bringing through duplicate values so I put a distinct on the SQ. There is also a not null filter on both ports. However, whilst investigating the initial error that is logged for a specific pair of values from the source, there is only 1 set of them (no duplicates). The pair exists on the target so surely should just return from the dynamic lookup newrowlookup=0. Is this some kind of persistent data in the cache that is causing this to think that it is duplicate data? I haven't got the persistent cache or...
Post a Comment
Read more

SQL Transformation with examples

============================================================================================= SQL Transformation with examples   Use : SQL Transformation is a connected transformation used to process SQL queries in the midstream of a pipeline . We can insert, update, delete and retrieve rows from the database at run time using the SQL transformation. Use SQL transformation in script mode to run DDL (data definition language) statements like creating or dropping the tables. The following SQL statements can be used in the SQL transformation. Data Definition Statements (CREATE, ALTER, DROP, TRUNCATE, RENAME) DATA MANIPULATION statements (INSERT, UPDATE, DELETE, MERGE) DATA Retrieval Statement (SELECT) DATA Control Language Statements (GRANT, REVOKE) Transaction Control Statements (COMMIT, ROLLBACK) Scenario: Let’s say we want to create a temporary table in mapping while workflow is running for some intermediate calculation. We can use SQL transformat...
1 comment
Read more

Load the session statistics such as Session Start & End Time, Success Rows, Failed Rows and Rejected Rows etc. into a database table for audit/log purpose.

                                                                                                                                                                     ...
Post a Comment
Read more