Sqoop: Specify number of mappers while executing sqoop command

We can customize the number of mappers using either -m or --num-mappers options.

 

Example 1: Below snippet set number of mappers to 10 using -m option.

sqoop import --connect "jdbc:mysql://quickstart.cloudera:3306/retail_db" --username "root" --password "cloudera" --table "orders" --target-dir /no_of_mappers_demo -m 10

Once above command execute successfully, you can query the folder /no_of_mappers_demo.

 

[cloudera@quickstart Desktop]$ hadoop fs -ls /no_of_mappers_demo 
Found 11 items
-rw-r--r--   1 cloudera supergroup          0 2022-04-01 01:59 /no_of_mappers_demo/_SUCCESS
-rw-r--r--   1 cloudera supergroup     292973 2022-04-01 01:59 /no_of_mappers_demo/part-m-00000
-rw-r--r--   1 cloudera supergroup     298045 2022-04-01 01:59 /no_of_mappers_demo/part-m-00001
-rw-r--r--   1 cloudera supergroup     301249 2022-04-01 01:59 /no_of_mappers_demo/part-m-00002
-rw-r--r--   1 cloudera supergroup     301346 2022-04-01 01:59 /no_of_mappers_demo/part-m-00003
-rw-r--r--   1 cloudera supergroup     301023 2022-04-01 01:59 /no_of_mappers_demo/part-m-00004
-rw-r--r--   1 cloudera supergroup     300917 2022-04-01 01:59 /no_of_mappers_demo/part-m-00005
-rw-r--r--   1 cloudera supergroup     301036 2022-04-01 01:59 /no_of_mappers_demo/part-m-00006
-rw-r--r--   1 cloudera supergroup     300931 2022-04-01 01:59 /no_of_mappers_demo/part-m-00007
-rw-r--r--   1 cloudera supergroup     301150 2022-04-01 01:59 /no_of_mappers_demo/part-m-00008
-rw-r--r--   1 cloudera supergroup     301274 2022-04-01 01:59 /no_of_mappers_demo/part-m-00009

 

Since we set the number of mappers to 10, we can see 10 part files generated in the above output (one part file for one mapper).

 

Example 2: Below snippet set number of mappers to 5 using --num-mappers option. 

sqoop import --connect "jdbc:mysql://quickstart.cloudera:3306/retail_db" --username "root" --password "cloudera" --table "orders" --target-dir /no_of_mappers_demo_1 --num-mappers 5

  Once above command execute successfully, you can query the folder /no_of_mappers_demo_1.

 

[cloudera@quickstart Desktop]$ hadoop fs -ls /no_of_mappers_demo_1
Found 6 items
-rw-r--r--   1 cloudera supergroup          0 2022-04-01 02:04 /no_of_mappers_demo_1/_SUCCESS
-rw-r--r--   1 cloudera supergroup     590975 2022-04-01 02:04 /no_of_mappers_demo_1/part-m-00000
-rw-r--r--   1 cloudera supergroup     602638 2022-04-01 02:04 /no_of_mappers_demo_1/part-m-00001
-rw-r--r--   1 cloudera supergroup     601940 2022-04-01 02:04 /no_of_mappers_demo_1/part-m-00002
-rw-r--r--   1 cloudera supergroup     601967 2022-04-01 02:04 /no_of_mappers_demo_1/part-m-00003
-rw-r--r--   1 cloudera supergroup     602424 2022-04-01 02:04 /no_of_mappers_demo_1/part-m-00004
[cloudera@quickstart Desktop]$

 

 

Previous                                                    Next                                                    Home