Каким образом следует искрообразовать sql для доступа к метастоуму улья?

Question

Я пытаюсь использовать Spark SQL для чтения таблицы из метастата улья, но Spark дает ошибку о том, что таблица не найдена. Я боюсь, что Spark SQL создает совершенно новый пустой метастор.

Я представляю искровой задачу через эту команду:

spark-submit --class etl.EIServerSpark --driver-class-path '/opt/cloudera/parcels/CDH/lib/hive/lib/*' --driver-java-options '-Dspark.executor.extraClassPath=/opt/cloudera/parcels/CDH/lib/hive/lib/*' --jars $HIVE_CLASSPATH --files /etc/hive/conf/hive-site.xml,/etc/hadoop/conf/yarn-site.xml --master yarn-client /root/etl.jar 

Это ошибка:

2015-06-30 17:50:51,563 INFO [main] util.Utils (Logging.scala:logInfo(59)) - Copying /etc/hive/conf/hive-site.xml to /tmp/spark-568de027-8b66-40fa-97a4-2ec50614f486/hive-site.xml 
2015-06-30 17:50:51,568 INFO [main] spark.SparkContext (Logging.scala:logInfo(59)) - Added file file:/etc/hive/conf/hive-site.xml at http://10.136.149.126:43349/files/hive-site.xml with timestamp 1435683051561 
2015-06-30 17:50:51,568 INFO [main] util.Utils (Logging.scala:logInfo(59)) - Copying /etc/hadoop/conf/yarn-site.xml to /tmp/spark-568de027-8b66-40fa-97a4-2ec50614f486/yarn-site.xml 
2015-06-30 17:50:51,570 INFO [main] spark.SparkContext (Logging.scala:logInfo(59)) - Added file file:/etc/hadoop/conf/yarn-site.xml at http://10.136.149.126:43349/files/yarn-site.xml with timestamp 1435683051568 
2015-06-30 17:50:51,637 INFO [sparkDriver-akka.actor.default-dispatcher-5] util.AkkaUtils (Logging.scala:logInfo(59)) - Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@gateway.edp.hadoop:52818/user/HeartbeatReceiver 
2015-06-30 17:50:51,756 INFO [main] netty.NettyBlockTransferService (Logging.scala:logInfo(59)) - Server created on 40198 
2015-06-30 17:50:51,757 INFO [main] storage.BlockManagerMaster (Logging.scala:logInfo(59)) - Trying to register BlockManager 
2015-06-30 17:50:51,759 INFO [sparkDriver-akka.actor.default-dispatcher-2] storage.BlockManagerMasterActor (Logging.scala:logInfo(59)) - Registering block manager localhost:40198 with 265.4 MB RAM, BlockManagerId(<driver>, localhost, 40198) 
2015-06-30 17:50:51,761 INFO [main] storage.BlockManagerMaster (Logging.scala:logInfo(59)) - Registered BlockManager 
2015-06-30 17:50:52,840 INFO [main] parse.ParseDriver (ParseDriver.java:parse(185)) - Parsing command: SELECT id, name FROM eiserver.eismpt 
2015-06-30 17:50:53,141 INFO [main] parse.ParseDriver (ParseDriver.java:parse(206)) - Parse Completed 
2015-06-30 17:50:54,041 INFO [main] metastore.HiveMetaStore (HiveMetaStore.java:newRawStore(502)) - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 
2015-06-30 17:50:54,064 INFO [main] metastore.ObjectStore (ObjectStore.java:initialize(247)) - ObjectStore, initialize called 
2015-06-30 17:50:54,227 WARN [main] DataNucleus.General (Log4JLogger.java:warn(96)) - Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hive/lib/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/jars/datanucleus-rdbms-3.2.9.jar." 
2015-06-30 17:50:54,268 WARN [main] DataNucleus.General (Log4JLogger.java:warn(96)) - Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hive/lib/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/jars/datanucleus-api-jdo-3.2.6.jar." 
2015-06-30 17:50:54,274 WARN [main] DataNucleus.General (Log4JLogger.java:warn(96)) - Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/lib/hive/lib/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/opt/cloudera/parcels/CDH-5.3.2-1.cdh5.3.2.p0.10/jars/datanucleus-core-3.2.10.jar." 
2015-06-30 17:50:54,314 INFO [main] DataNucleus.Persistence (Log4JLogger.java:info(77)) - Property datanucleus.cache.level2 unknown - will be ignored 
2015-06-30 17:50:54,315 INFO [main] DataNucleus.Persistence (Log4JLogger.java:info(77)) - Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 
2015-06-30 17:50:56,109 INFO [main] metastore.ObjectStore (ObjectStore.java:getPMF(318)) - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 
2015-06-30 17:50:56,170 INFO [main] metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(110)) - MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: "@" (64), after : "". 
2015-06-30 17:50:57,315 INFO [main] DataNucleus.Datastore (Log4JLogger.java:info(77)) - The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table. 
2015-06-30 17:50:57,316 INFO [main] DataNucleus.Datastore (Log4JLogger.java:info(77)) - The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 
2015-06-30 17:50:57,688 INFO [main] DataNucleus.Datastore (Log4JLogger.java:info(77)) - The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table. 
2015-06-30 17:50:57,688 INFO [main] DataNucleus.Datastore (Log4JLogger.java:info(77)) - The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 
2015-06-30 17:50:57,842 INFO [main] DataNucleus.Query (Log4JLogger.java:info(77)) - Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing 
2015-06-30 17:50:57,844 INFO [main] metastore.ObjectStore (ObjectStore.java:setConf(230)) - Initialized ObjectStore 
2015-06-30 17:50:58,113 INFO [main] metastore.HiveMetaStore (HiveMetaStore.java:createDefaultRoles(560)) - Added admin role in metastore 
2015-06-30 17:50:58,115 INFO [main] metastore.HiveMetaStore (HiveMetaStore.java:createDefaultRoles(569)) - Added public role in metastore 
2015-06-30 17:50:58,198 INFO [main] metastore.HiveMetaStore (HiveMetaStore.java:addAdminUsers(597)) - No user is added in admin role, since config is empty 
2015-06-30 17:50:58,376 INFO [main] session.SessionState (SessionState.java:start(383)) - No Tez session required at this point. hive.execution.engine=mr. 
2015-06-30 17:50:58,525 INFO [main] metastore.HiveMetaStore (HiveMetaStore.java:logInfo(632)) - 0: get_table : db=eiserver tbl=eismpt 
2015-06-30 17:50:58,525 INFO [main] HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(314)) - ugi=root  ip=unknown-ip-addr  cmd=get_table : db=eiserver tbl=eismpt 
2015-06-30 17:50:58,567 ERROR [main] metadata.Hive (Hive.java:getTable(1003)) - NoSuchObjectException(message:eiserver.eismpt table not found) 
     at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1569) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 

Как настроить искровой SQL для доступа улья metastore развернут на Postgres? Я использую CDH 5.3.2.

Спасибо

Answer 1

The doc говорит положить spark.sql.hive.metastore.sharedPrefixes = org.postgresql в конфигурационном файле, вы попробуете это?

Answer 2

Настройка Спарк использовать улей metastore thriftserver:

Редактировать $SPARK_HOME/conf/hive-site.xml удалить информацию прямого подключения и добавить это свойство:

<configuration> 
    <property> 
    <name>hive.metastore.uris</name> 
    <value>thrift://localhost:9083</value> /*make sure to replace with your hive-metastore service's thrift url*/ 
    <description>URI for client to contact metastore server</description> 
    </property> 
</configuration> 

Если hive-site.xml не существует в $SPARK_HOME/conf то, чтобы подключитесь к метафору улья, вам нужно скопировать файл hive-site.xml в каталог spark/conf. Так выполните следующую команду после входа в систему с правами суперпользователя,

cp /usr/lib/hive/conf/hive-site.xml /usr/lib/spark/conf/ 

Создать улей контекст

На scala> REPL строке введите следующее:

import org.apache.spark.sql.hive.HiveContext 
val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc) 

Создать улей таблицу

hiveContext.sql("CREATE TABLE IF NOT EXISTS TestTable (key INT, value STRING)") 

Показать ульев Столы

scala> hiveContext.hql("SHOW TABLES").collect().foreach(println) 

Проверяют конфигурации (необязательно)

• Stop Спарк SQL thriftserver с cd $SPARK_HOME; sbin/stop-thriftserver.sh
• Запуск metastore Hive thriftserver с cd;./start-thriftserver.sh
• Check журналы на $HIVE_HOME/logs/metastore.out для любой ошибки s.
• Spread SQL thriftserver не запускается, пока он не сможет успешно подключиться к этому серверу, поэтому он должен работать.
• Запустите Spark SQL triftserver с cd $SPARK_HOME; sbin/start-thriftserver.sh Проверьте файл журнала, указанный в возвращенной строке.
• Вы должны увидеть линии, как это:

16/12/29 20:22:19 INFO metastore: Trying to connect to metastore with URI thrift://localhost:9083 
16/12/29 20:22:19 INFO metastore: Connected to metastore. 

Выполнить $SPARK_HOME/bin/beeline -u 'jdbc:hive2://localhost:10000/' и попробовать команду !tables, чтобы убедиться, что вы можете перечислить метаданные.