您好,欢迎来到五一七教育网。
搜索
您的当前位置:首页一起学Hive——使用MSCK命令修复Hive分区

一起学Hive——使用MSCK命令修复Hive分区

来源:五一七教育网

幸运的是Hive提供了MSCK命令,用于修复表的分区,我们先看Hive官方是如果介绍MCSK命令的:

翻译成中文的大概意思就是:Hive将每个表的分区信息保存在metastore中,如果通过hadoop fs -put命令直接将分区信息添加到HDFS,metastore是不会感知到这些新增的分区,除非执行了ALTER TABLE table_name ADD PARTITION命令。但是用户可以运行metastore检查命令MSCK REPAIR TABLE table_name;该命令将关于分区的元信息添加到Hive metastore中,这是对于那些没有元信息的分区来说的。换句话说,就是将任何存在于HDFS上但不在metastore上的分区添加到metastore。

hive> show partitions bigdata17_partition;
OK
Time taken: 1.121 seconds

然后执行MSCK REPAIR TABLE bigdata17_partition;命令添加分区:

hive> MSCK REPAIR TABLE bigdata17_partition;
18/10/11 17:27:15 WARN log: Updating partition stats fast for: bigdata17_partition;
18/10/11 17:27:15 WARN log: Updated size to 10124
18/10/11 17:27:15 WARN log: Updating partition stats fast for: bigdata17_partition;
18/10/11 17:27:15 WARN log: Updated size to 21234
18/10/11 17:27:15 WARN log: Updating partition stats fast for: bigdata17_partition;
18/10/11 17:27:15 WARN log: Updated size to 346783
18/10/11 17:27:15 WARN log: Updating partition stats fast for: bigdata17_partition;
18/10/11 17:27:15 WARN log: Updated size to 2532162
18/10/11 17:27:15 WARN log: Updating partition stats fast for: bigdata17_partition;
18/10/11 17:27:15 WARN log: Updated size to 2901198
18/10/11 17:27:15 WARN log: Updating partition stats fast for: bigdata17_partition;
18/10/11 17:27:15 WARN log: Updated size to 3129087
18/10/11 17:27:15 WARN log: Updating partition stats fast for: bigdata17_partition;
18/10/11 17:27:15 WARN log: Updated size to 23190876
OK
Partitions not in metastore:    bigdata17_partition;:dt=2018-09-15    bigdata17_partition;:dt=2018-09-16    bigdata17_partition;:dt=2018-09-17    bigdata17_partition;:dt=2018-09-18    bigdata17_partition;:dt=2018-09-19     bigdata17_partition;:dt=2018-09-20    bigdata17_partition;:dt=2018-09-21
Repair: Added partition to metastore bigdata17_partition;:dt=2018-09-15
Repair: Added partition to metastore bigdata17_partition;:dt=2018-09-16
Repair: Added partition to metastore bigdata17_partition;:dt=2018-09-17
Repair: Added partition to metastore bigdata17_partition;:dt=2018-09-18
Repair: Added partition to metastore bigdata17_partition;:dt=2018-09-19
Repair: Added partition to metastore bigdata17_partition;:dt=2018-09-20
Repair: Added partition to metastore bigdata17_partition;:dt=2018-09-21
Time taken: 0.613 seconds, Fetched 8 row(s)

通过上述的结果可以看到已经将bigdata17_partition表的分区信息添加到Hive metastore中,和add partition命令比起来既方便又高效。

转载于:https://www.cnblogs.com/airnew/p/9776023.html

因篇幅问题不能全部显示,请点此查看更多更全内容

Copyright © 2019- 517ttc.cn 版权所有 赣ICP备2024042791号-8

违法及侵权请联系:TEL:199 18 7713 E-MAIL:2724546146@qq.com

本站由北京市万商天勤律师事务所王兴未律师提供法律服务