I always get into a debate where people say that ODI doesn't do bulk integration or very high volume of data integration that smoothly as compared to its competitors. So is it really true ? I deal with this in sections
1. Native Essbase Data load vs. Oracle Data Integrator Load
Native Essbase load (using Load rules and EAS console) is faster than integration done using Oracle Data Integrator. But the amount of flexibilty that ODI provides in terms of mappings and transformations sometimes outweighs the lesser load time taken by native technique. Also, the use or selection of Staging area is crucial in deciding the load time. Using Sunopsis memory engine (Java heap in other words) as Staging drastically increases the load time, sometimes even failing the process. Staging area should always be a good relational database or if possible, the source itself.
2. Relational Databases
Having worked on integration projects involving pure relational tables and lots of data. I have come to know that there are few things that decide how good or bad the load times are going to be
1. Selection of Staging area
2. Deciding which transformation is executed where
3. Selection of the right Knowledge module
4. Table joins and lookups.
4 And ofcourse, how good the source, target and staging area relational databases are in terms of memory and processing speeds
The best part is that ODI gives a vast amount of flexibility when dealing the question of 'how to load data most efficiently'. It is through trial and errors that we arrive at the best possible solution. So, in my opinion, ODI is not that bad when it comes to loading high volumes of data.
1. Native Essbase Data load vs. Oracle Data Integrator Load
Native Essbase load (using Load rules and EAS console) is faster than integration done using Oracle Data Integrator. But the amount of flexibilty that ODI provides in terms of mappings and transformations sometimes outweighs the lesser load time taken by native technique. Also, the use or selection of Staging area is crucial in deciding the load time. Using Sunopsis memory engine (Java heap in other words) as Staging drastically increases the load time, sometimes even failing the process. Staging area should always be a good relational database or if possible, the source itself.
2. Relational Databases
Having worked on integration projects involving pure relational tables and lots of data. I have come to know that there are few things that decide how good or bad the load times are going to be
1. Selection of Staging area
2. Deciding which transformation is executed where
3. Selection of the right Knowledge module
4. Table joins and lookups.
4 And ofcourse, how good the source, target and staging area relational databases are in terms of memory and processing speeds
The best part is that ODI gives a vast amount of flexibility when dealing the question of 'how to load data most efficiently'. It is through trial and errors that we arrive at the best possible solution. So, in my opinion, ODI is not that bad when it comes to loading high volumes of data.
No comments:
Post a Comment