Sqoop: Incremental import

Using Sqoop incremental import, we can import the delta changes from previous import to current import.

 

For example, Suppose you imported employee table (with 50000 records) from RDBMS to HDFS yesterday. There are 1000 new records added to employee table today, do you really want to reimport all the 51000 records again to HDFS? No.

 

Using incremental import technique, we can onboard newly added records, updated records after previous sync to the HDFS.

 

Sqoop incremental modes

Sqoop supports two incremental modes.

a.   append mode: Use this mode, when you are only interested in adding newly created records to the existing dataset.

b.   lastmodified mode: Use this mode, when you want to import the data by updated time stamp.

Previous                                                    Next                                                    Home