幸运的是Hive提供了MSCK命令,用于修复表的分区,我们先看Hive官方是如果介绍MCSK命令的:
翻译成中文的大概意思就是:Hive将每个表的分区信息保存在metastore中,如果通过hadoop fs -put命令直接将分区信息添加到HDFS,metastore是不会感知到这些新增的分区,除非执行了ALTER TABLE table_name ADD PARTITION命令。但是用户可以运行metastore检查命令MSCK REPAIR TABLE table_name;该命令将关于分区的元信息添加到Hive metastore中,这是对于那些没有元信息的分区来说的。换句话说,就是将任何存在于HDFS上但不在metastore上的分区添加到metastore。
hive> show partitions bigdata17_partition;
OK
Time taken: 1.121 seconds
然后执行MSCK REPAIR TABLE bigdata17_partition;命令添加分区:
hive> MSCK REPAIR TABLE bigdata17_partition;
18/10/11 17:27:15 WARN log: Updating partition stats fast for: bigdata17_partition;
18/10/11 17:27:15 WARN log: Updated size to 10124
18/10/11 17:27:15 WARN log: Updating partition stats fast for: bigdata17_partition;
18/10/11 17:27:15 WARN log: Updated size to 21234
18/10/11 17:27:15 WARN log: Updating partition stats fast for: bigdata17_partition;
18/10/11 17:27:15 WARN log: Updated size to 346783
18/10/11 17:27:15 WARN log: Updating partition stats fast for: bigdata17_partition;
18/10/11 17:27:15 WARN log: Updated size to 2532162
18/10/11 17:27:15 WARN log: Updating partition stats fast for: bigdata17_partition;
18/10/11 17:27:15 WARN log: Updated size to 2901198
18/10/11 17:27:15 WARN log: Updating partition stats fast for: bigdata17_partition;
18/10/11 17:27:15 WARN log: Updated size to 3129087
18/10/11 17:27:15 WARN log: Updating partition stats fast for: bigdata17_partition;
18/10/11 17:27:15 WARN log: Updated size to 23190876
OK
Partitions not in metastore: bigdata17_partition;:dt=2018-09-15 bigdata17_partition;:dt=2018-09-16 bigdata17_partition;:dt=2018-09-17 bigdata17_partition;:dt=2018-09-18 bigdata17_partition;:dt=2018-09-19 bigdata17_partition;:dt=2018-09-20 bigdata17_partition;:dt=2018-09-21
Repair: Added partition to metastore bigdata17_partition;:dt=2018-09-15
Repair: Added partition to metastore bigdata17_partition;:dt=2018-09-16
Repair: Added partition to metastore bigdata17_partition;:dt=2018-09-17
Repair: Added partition to metastore bigdata17_partition;:dt=2018-09-18
Repair: Added partition to metastore bigdata17_partition;:dt=2018-09-19
Repair: Added partition to metastore bigdata17_partition;:dt=2018-09-20
Repair: Added partition to metastore bigdata17_partition;:dt=2018-09-21
Time taken: 0.613 seconds, Fetched 8 row(s)
通过上述的结果可以看到已经将bigdata17_partition表的分区信息添加到Hive metastore中,和add partition命令比起来既方便又高效。