Hive 报 message:Metastore contains multiple versions 错误

当前位置：首页 > 分布式 > 正文

Nov222016

作者：Fisher 发布：2016-11-22 11:18 分类：分布式抢沙发

[摘要] 早上看到从4点起所有 Hive 任务都跑失败了，手动启动Hive也同样报错，说是Metastore的version存在多个：MetaException(message:Metastore contains multiple versions)，google了一下，问题解决，这里把解决办法记录一下。

执行Hive命令：

$ hive
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
	at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
	... 8 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
	... 14 more
Caused by: MetaException(message:Metastore contains multiple versions (3) [ version = 1.2.0, comment = Set by MetaStore hdfs@10.10.100.164 ] [ version = 1.2.0, comment = Set by MetaStore hdfs@10.10.100.164 ] [ version = 1.2.0, comment = Set by MetaStore hdfs@10.10.100.164 ])
	at org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6745)
	at org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:6703)
	at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6661)
	at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6645)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
	at com.sun.proxy.$Proxy5.verifySchema(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:572)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:620)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:66)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:199)
	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
	... 19 more

$ hive

Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)

at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)

at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)

at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)

at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)

at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)

at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)

at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)

at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)

... 8 more

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)

... 14 more

Caused by: MetaException(message:Metastore contains multiple versions (3) [ version = 1.2.0, comment = Set by MetaStore hdfs@10.10.100.164 ] [ version = 1.2.0, comment = Set by MetaStore hdfs@10.10.100.164 ] [ version = 1.2.0, comment = Set by MetaStore hdfs@10.10.100.164 ])

at org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6745)

at org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:6703)

at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6661)

at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6645)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)

at com.sun.proxy.$Proxy5.verifySchema(Unknown Source)

at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:572)

at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:620)

at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461)

at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:66)

at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)

at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762)

at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:199)

at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)

... 19 more

看了一下Hive的元数据表：

mysql> select * from version;
+--------+----------------+-------------------------------------+
| VER_ID | SCHEMA_VERSION | VERSION_COMMENT                     |
+--------+----------------+-------------------------------------+
|      1 | 1.2.0          | Set by MetaStore hdfs@10.10.100.164 |
|     16 | 1.2.0          | Set by MetaStore hdfs@10.10.100.164 |
|     21 | 1.2.0          | Set by MetaStore hdfs@10.10.100.164 |
+--------+----------------+-------------------------------------+
3 rows in set (0.00 sec)

mysql> select * from version;

+--------+----------------+-------------------------------------+

| VER_ID | SCHEMA_VERSION | VERSION_COMMENT |

+--------+----------------+-------------------------------------+

| 1 | 1.2.0 | Set by MetaStore hdfs@10.10.100.164 |

| 16 | 1.2.0 | Set by MetaStore hdfs@10.10.100.164 |

| 21 | 1.2.0 | Set by MetaStore hdfs@10.10.100.164 |

+--------+----------------+-------------------------------------+

3 rows in set (0.00 sec)

发现version表中有3条数据。

解决办法：

delete from version where VER_ID = '16';
delete from version where VER_ID = '21';

1 2	delete from version where VER_ID = '16'; delete from version where VER_ID = '21';

产生的原因？
初步认为是并发跑任务造成的，有一个大任务在连续循环执行，另外还有cronjob再跑，可能是由于并发起了hive造成了version表的多条记录。

接下来就是重跑脚本，补数据了:(

本文固定链接: https://www.sudops.com/hive-message-metastore-contains-multiple-versions-error.html | 运维·速度

该日志由 Fisher 于2016年11月22日发表在分布式分类下，你可以发表评论，并在保留原文地址及作者的情况下引用到你的网站或博客。
原创文章转载请注明: Hive 报 message:Metastore contains multiple versions 错误 | 运维·速度
关键字: hive

【上一篇】devops 的七宗罪与如何避免
【下一篇】wget整站下载被robots.txt阻挡时的处理方法

Hive 报 message:Metastore contains multiple versions 错误

您可能还会对这些文章感兴趣！

Hive 报 message:Metastore contains multiple versions 错误：等您坐沙发呢！

发表评论

最新日志热评日志随机日志

赞助链接

标签云

最新文章

最近评论

最活跃的读者

欢迎关注运维·速度微信公众号

最新评论

标签云集

博客统计

Hive 报 message:Metastore contains multiple versions 错误

您可能还会对这些文章感兴趣！

Hive 报 message:Metastore contains multiple versions 错误：等您坐沙发呢！

发表评论

最新日志热评日志随机日志

赞助链接

标签云

最新文章

最近评论

最活跃的读者

欢迎关注 运维·速度 微信公众号

最新评论

标签云集

博客统计

欢迎关注运维·速度微信公众号