问题描述:我试图对数据进行sqoop,但sqoop返回零记录而没有任何错误。但是当我尝试使用某个限制检索记录时,它会获取数据,但是一旦我进一步使用更大的限制,它就不会获取任何记录。
The QUERY that was passes using the Sqoop is as mentioned below:
select usr.id,usr.login,usr.auto_login,usr.password,usr.password_salt,usr.member,usr.first_name,usr.middle_name,usr.last_name,usr.user_type,usr.locale,usr.lastactivity_date,usr.lastpwdupdate,usr.generatedpwd,usr.registration_date,usr.email,usr.email_status,usr.receive_email,usr.last_emailed,usr.gender,usr.date_of_birth,usr.securitystatus,usr.description,usr.realm_id,usr.password_kdf,dcspp_order.last_modified_date, 20151223080640 FROM <TABLE_NAME> usr JOIN atgprdcore.dcspp_order ON (usr.id = dcspp_order.profile_id ) WHERE $CONDITIONS'
Generated SQOOP Command: sqoop job -Dmapred.child.java.opts="-Djava.security.egd=file:/dev/../dev/urandom" -libjars /<COMP>/stage/da_data/DataAqusition_ATG/dm-sqoop-1.0.0/lib/tdgssconfig.jar,/<COMP>/stage/da_data/DataAqusition_ATG/dm-sqoop-1.0.0/lib/ojdbc6.jar,/<COMP>/stage/da_data/DataAqusition_ATG/dm-sqoop-1.0.0/lib/nzjdbc3.jar,/<COMP>/stage/da_data/DataAqusition_ATG/dm-sqoop-1.0.0/lib/terajdbc4.jar -Dfile.encoding=UTF-8 -Dmapreduce.job.queuename=long_running -Dmapreduce.job.name=sample-job-name --create Sqoop_Utility1253423780 -- import --connect jdbc:oracle:thin:@10.202.201.15:9101:KOHLDBSA1 --username XXXXXX --password-file /tmp/sqoop-nzhdusr/27c6d6d50fccdc67342374a4f560d1d6-asdfg.txt --fetch-size 100 --query 'select usr.id,usr.login,usr.auto_login,usr.password,usr.password_salt,usr.member,usr.first_name,usr.middle_name,usr.last_name,usr.user_type,usr.locale,usr.lastactivity_date,usr.lastpwdupdate,usr.generatedpwd,usr.registration_date,usr.email,usr.email_status,usr.receive_email,usr.last_emailed,usr.gender,usr.date_of_birth,usr.securitystatus,usr.description,usr.realm_id,usr.password_kdf,dcspp_order.last_modified_date, 20151223080640 FROM <database>.<tablename> usr JOIN atgprdcore.dcspp_order ON (usr.id = dcspp_order.profile_id ) WHERE $CONDITIONS' --hive-drop-import-delims --null-string "" --target-dir /tmp/sqoop-nzhdusr/dps_user --num-mappers 1 --fields-terminated-by "|"
[INFO] running sqoop
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
15/12/23 08:07:10 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4.2.1.10.0-881
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
15/12/23 08:07:15 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4.2.1.10.0-881
15/12/23 08:07:18 INFO tool.CodeGenTool: Beginning code generation
15/12/23 08:07:19 INFO manager.OracleManager: Time zone has been set to GMT
15/12/23 08:07:19 INFO manager.SqlManager: Executing SQL statement: select usr.id,usr.login,usr.auto_login,usr.password,usr.password_salt,usr.member,usr.first_name,usr.middle_name,usr.last_name,usr.user_type,usr.locale,usr.lastactivity_date,usr.lastpwdupdate,usr.generatedpwd,usr.registration_date,usr.email,usr.email_status,usr.receive_email,usr.last_emailed,usr.gender,usr.date_of_birth,usr.securitystatus,usr.description,usr.realm_id,usr.password_kdf,dcspp_order.last_modified_date, 20151223080640 FROM <database>.<tablename> tab1 JOIN atgprdcore.dcspp_order ON (usr.id = dcspp_order.profile_id ) WHERE (1 = 0)
15/12/23 08:07:19 INFO manager.SqlManager: Executing SQL statement: select usr.id,usr.login,usr.auto_login,usr.password,usr.password_salt,usr.member,usr.first_name,usr.middle_name,usr.last_name,usr.user_type,usr.locale,usr.lastactivity_date,usr.lastpwdupdate,usr.generatedpwd,usr.registration_date,usr.email,usr.email_status,usr.receive_email,usr.last_emailed,usr.gender,usr.date_of_birth,usr.securitystatus,usr.description,usr.realm_id,usr.password_kdf,dcspp_order.last_modified_date, 20151223080640 FROM <database>.<tablename> tab2 JOIN atgprdcore.dcspp_order ON (usr.id = dcspp_order.profile_id ) WHERE (1 = 0)
15/12/23 08:07:19 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-nzhdusr/compile/ed8d5029fc473715d385a2c0b7e002c4/QueryResult.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
15/12/23 08:07:21 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-nzhdusr/compile/ed8d5029fc473715d385a2c0b7e002c4/QueryResult.jar
15/12/23 08:07:21 INFO mapreduce.ImportJobBase: Beginning query import.
15/12/23 08:07:21 INFO client.RMProxy: Connecting to ResourceManager at nhga0002.tst.<COMP>.com/10.200.0.3:8050
15/12/23 08:07:21 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 47174 for nzhdusr on ha-hdfs:<URL>
15/12/23 08:07:21 INFO security.TokenCache: Got dt for hdfs://<URL>; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:<URL>, Ident: (HDFS_DELEGATION_TOKEN token 47174 for nzhdusr)
15/12/23 08:07:23 INFO db.DBInputFormat: Using read commited transaction isolation
15/12/23 08:07:24 INFO mapreduce.JobSubmitter: number of splits:1
15/12/23 08:07:24 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1444949527622_18165
15/12/23 08:07:24 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:<URL>, Ident: (HDFS_DELEGATION_TOKEN token 47174 for nzhdusr)
15/12/23 08:07:24 INFO impl.YarnClientImpl: Submitted application application_1444949527622_18165
15/12/23 08:07:25 INFO mapreduce.Job: The url to track the job: https://nhga0002.tst.<COMP>.com:8090/proxy/application_1444949527622_18165/
15/12/23 08:07:25 INFO mapreduce.Job: Running job: job_1444949527622_18165
15/12/23 08:07:35 INFO mapreduce.Job: Job job_1444949527622_18165 running in uber mode : false
15/12/23 08:07:35 INFO mapreduce.Job: map 0% reduce 0%
15/12/23 08:24:57 INFO mapreduce.Job: map 100% reduce 0%
15/12/23 08:24:57 INFO mapreduce.Job: Job job_1444949527622_18165 completed successfully
15/12/23 08:24:57 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=117614
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=87
HDFS: Number of bytes written=0
HDFS: Number of read operations=4
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Other local map tasks=1
Total time spent by all maps in occupied slots (ms)=1039640
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=1039640
Total vcore-seconds taken by all map tasks=1039640
Total megabyte-seconds taken by all map tasks=6919843840
Map-Reduce Framework
Map input records=0
Map output records=0
Input split bytes=87
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=119
CPU time spent (ms)=7760
Physical memory (bytes) snapshot=315817984
Virtual memory (bytes) snapshot=6523957248
Total committed heap usage (bytes)=1114112000
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=0
15/12/23 08:24:57 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 1,055.9463 seconds (0 bytes/sec)
15/12/23 08:24:57 INFO mapreduce.ImportJobBase: Retrieved 0 records.