how to do the incremental loading using mapping variable

Scenario:

Solution:

Step 1: create a mapping variable, $$MappingDateVariable. In the source qualifier, create a filter to read only rows whose transaction date equals $$MappingDateVariable, such as: transaction_date = $$MappingDateVariable

(I am assuming you have a column in your source table called, transaction date or any date column to help do incremental load).

Step 2: Create a mapping variable, $$MappingDateVariable and hard code the value from which date you need to extract.

Step 3: In the mapping, use the variable function to set the variable value to increment one day each time the session runs.

lets say you set the initial value of $$MappingDateVariable as 11/16/2010. The first time the integration service runs the session, it reads only rows dated 11/16/2010. And would set $$MappingDateVariable to 11/17/2010. It saves 11/17/2010 to the repository at the end of the session. The next time it runs the session, it reads only rows from 11/17/2010.

Comments

SQL Transformation with examples

============================================================================================= SQL Transformation with examples Use : SQL Transformation is a connected transformation used to process SQL queries in the midstream of a pipeline . We can insert, update, delete and retrieve rows from the database at run time using the SQL transformation. Use SQL transformation in script mode to run DDL (data definition language) statements like creating or dropping the tables. The following SQL statements can be used in the SQL transformation. Data Definition Statements (CREATE, ALTER, DROP, TRUNCATE, RENAME) DATA MANIPULATION statements (INSERT, UPDATE, DELETE, MERGE) DATA Retrieval Statement (SELECT) DATA Control Language Statements (GRANT, REVOKE) Transaction Control Statements (COMMIT, ROLLBACK) Scenario: Let’s say we want to create a temporary table in mapping while workflow is running for some intermediate calculation. We can use SQL transformat...

CMN_1650 A duplicate row was attempted to be inserted into a dynamic lookup cache Dynamic lookup error.

Scenario: I have 2 ports going through a dynamic lookup, and then to a router. In the router it is a simple case of inserting new target rows (NewRowLookup=1) or rejecting existing rows (NewRowLookup=0). However, when I run the session I'm getting the error: "CMN_1650 A duplicate row was attempted to be inserted into a dynamic lookup cache Dynamic lookup error. The dynamic lookup cache only supports unique condition keys." I thought that I was bringing through duplicate values so I put a distinct on the SQ. There is also a not null filter on both ports. However, whilst investigating the initial error that is logged for a specific pair of values from the source, there is only 1 set of them (no duplicates). The pair exists on the target so surely should just return from the dynamic lookup newrowlookup=0. Is this some kind of persistent data in the cache that is causing this to think that it is duplicate data? I haven't got the persistent cache or...

Load the session statistics such as Session Start & End Time, Success Rows, Failed Rows and Rejected Rows etc. into a database table for audit/log purpose.

...

Solution for Informatica problems

Search This Blog