Импорт из SQL Server, типы данных не преобразуются должным образом

Импорт из SQL Server, типы данных не преобразуются правильно Stack: Установлен HDP-2.3.2.0-2950 использованием Ambari 2.1Импорт из SQL Server, типы данных не преобразуются должным образом

Цель:

Импорт таблицы из SQL Server на HDFS в формате Avro
Создать ВНЕШНЕЕ Hive Avro (SerDe) таблицы, которые имеют все данные
Создание таблиц ВНЕШНЕЕ Hive ORC и вставить в ORC выбрать * из Avro таблиц
уронить Avro таблицы и выполнить тестирование на столах ORC

Один из таблиц:

ECU_DTC_ID   int 
DTC_CDE    nchar(20) 
ECU_NAME   nvarchar(15) 
ECU_FAMILY_NAME  nvarchar(15) 
DTC_DESC   nvarchar(MAX) 
INSERTED_BY   nvarchar(64) 
INSERTION_DATE  datetime 
DTC_CDE_DECIMAL  int

Когда я исполняю нормальный импорт sqoop, то дата и время превращаются в длинную, NCHAR и NVARCHAR нанизывать. Файл полученный AVSC будет, как показано, когда я создаю таблицу Hive Avro, она не включает в себя сгенерированные Avro файлы, таким образом, оставляя за пустым столом:

{ 
    "type" : "record", 
    "name" : "DimECUDTCCode", 
    "doc" : "Sqoop import of DimECUDTCCode", 
    "fields" : [ { 
    "name" : "ECU_DTC_ID", 
    "type" : [ "null", "int" ], 
    "default" : null, 
    "columnName" : "ECU_DTC_ID", 
    "sqlType" : "4" 
    }, { 
    "name" : "DTC_CDE", 
    "type" : [ "null", "string" ], 
    "default" : null, 
    "columnName" : "DTC_CDE", 
    "sqlType" : "-15" 
    }, { 
    "name" : "ECU_NAME", 
    "type" : [ "null", "string" ], 
    "default" : null, 
    "columnName" : "ECU_NAME", 
    "sqlType" : "-9" 
    }, { 
    "name" : "ECU_FAMILY_NAME", 
    "type" : [ "null", "string" ], 
    "default" : null, 
    "columnName" : "ECU_FAMILY_NAME", 
    "sqlType" : "-9" 
    }, { 
    "name" : "DTC_DESC", 
    "type" : [ "null", "string" ], 
    "default" : null, 
    "columnName" : "DTC_DESC", 
    "sqlType" : "-9" 
    }, { 
    "name" : "INSERTED_BY", 
    "type" : [ "null", "string" ], 
    "default" : null, 
    "columnName" : "INSERTED_BY", 
    "sqlType" : "-9" 
    }, { 
    "name" : "INSERTION_DATE", 
    "type" : [ "null", "long" ], 
    "default" : null, 
    "columnName" : "INSERTION_DATE", 
    "sqlType" : "93" 
    }, { 
    "name" : "DTC_CDE_DECIMAL", 
    "type" : [ "null", "int" ], 
    "default" : null, 
    "columnName" : "DTC_CDE_DECIMAL", 
    "sqlType" : "4" 
    } ], 
    "tableName" : "DimECUDTCCode"

я решил включить --map-столбец -java:

sqoop import --connect 'jdbc:sqlserver://somedbserver;database=somedb' --username someusername--password somepassword --as-avrodatafile --num-mappers 8 --table DimECUDTCCode --map-column-java DTC_CDE=string,ECU_NAME=string,ECU_FAMILY_NAME=string,DTC_DESC=string,INSERTED_BY=string,INSERTION_DATE=timestamp --warehouse-dir /dataload/tohdfs/reio/odpdw/may2016 --verbose

, но я получаю следующее сообщение об ошибке:

16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 ERROR orm.ClassWriter: No ResultSet method for Java type string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string 
16/05/12 09:43:12 ERROR orm.ClassWriter: No ResultSet method for Java type string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string 
16/05/12 09:43:12 ERROR orm.ClassWriter: No ResultSet method for Java type string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string 
16/05/12 09:43:12 ERROR orm.ClassWriter: No ResultSet method for Java type string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string 
16/05/12 09:43:12 ERROR orm.ClassWriter: No ResultSet method for Java type string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp 
16/05/12 09:43:12 ERROR orm.ClassWriter: No ResultSet method for Java type timestamp 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 ERROR tool.ImportTool: Imported Failed: No ResultSet method for Java type string 
[[email protected] root]$

Что я упускаю?

источник

2016-05-12 Kaliyug Antagonist

вы можете попробовать '--map-column-hive' и напрямую сопоставить столбцы SQL Server с столбцами улья. –

Но почему улей, я хочу использовать java, которая не преуспевает –

Да, вы должны попытаться найти проблему, используя '--map-column-java'. Я просто дал альтернативу, если вы застряли, потому что я попробовал «-map-column-hive», и это сработало. –

Оказывается, что SQOOP обрабатывают по-разному STRING и String и string. Правильный путь: String.

источник

2018-01-09 06:10:40 peeyush

Импорт из SQL Server, типы данных не преобразуются должным образом

ответ

Смежные вопросы