Optimize SSIS package for fewer queries
3
votes
2
answers
213
views
I need to associate customers' order with their "level" (Silver, Gold, etc.) _when they placed the order_:
CRM server::CRM db::CRM table
----------
CustomerID PreviousLevel NewLevel NewLevelGrantedOn
Order server::Order db::Order table
----------
OrderID CustomerID OrderPlacedOn
In an SSIS package I did this:
1. Execute SQL against Order db to extract the orders and put them in an object variable;
1. Loop through each order using "foreach container", in which I put a data flow task
select top 1 * where CustomerID = ? and LevelGrantedOn < ? order by LevelGrantedOn desc
that extract data from CRM db (both parameters come from step one), derive some columns and write the output to another table.
There are more then twenty thousand records in the order db, which means the data flow task will be executed for more than twenty thousand times. The CRM db will be queried for more than twenty thousand times, too. It takes more than an hour to do these.
Can I utilize some built-in features to speed up these (or do it in a "smart" way)? And, is an hour a long time, in the context of ETL and / or SSIS?
Asked by Ryan
(313 rep)
Jun 21, 2017, 01:56 PM
Last activity: Jun 30, 2017, 10:01 PM
Last activity: Jun 30, 2017, 10:01 PM