Github user gvramana commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1706#discussion_r158316333
--- Diff: core/src/main/java/org/apache/carbondata/core/metadata/PartitionMapFileStore.java ---
@@ -253,6 +282,96 @@ public void commitPartitions(String segmentPath, final String uniqueId, boolean
}
}
+ /**
+ * Clean up invalid data after drop partition in all segments of table
+ * @param table
+ * @param currentPartitions Current partitions of table
+ * @param forceDelete Whether it should be deleted force or check the time for an hour creation
+ * to delete data.
+ * @throws IOException
+ */
+ public void cleanSegments(
+ CarbonTable table,
+ List<String> currentPartitions,
+ boolean forceDelete) throws IOException {
+ SegmentStatusManager ssm = new SegmentStatusManager(table.getAbsoluteTableIdentifier());
+
+ CarbonTablePath carbonTablePath = CarbonStorePath
+ .getCarbonTablePath(table.getAbsoluteTableIdentifier().getTablePath(),
+ table.getAbsoluteTableIdentifier().getCarbonTableIdentifier());
+
+ LoadMetadataDetails[] details = ssm.readLoadMetadata(table.getMetaDataFilepath());
+ // scan through each segment.
+
+ for (LoadMetadataDetails segment : details) {
+
+ // if this segment is valid then only we will go for deletion of related
+ // dropped partition files. if the segment is mark for delete or compacted then any way
+ // it will get deleted.
+
+ if (segment.getSegmentStatus() == SegmentStatus.SUCCESS
+ || segment.getSegmentStatus() == SegmentStatus.LOAD_PARTIAL_SUCCESS) {
+ List<String> toBeDeletedIndexFiles = new ArrayList<>();
+ List<String> toBeDeletedDataFiles = new ArrayList<>();
+ // take the list of files from this segment.
+ String segmentPath = carbonTablePath.getCarbonDataDirectoryPath("0", segment.getLoadName());
+ String partitionFilePath = getPartitionFilePath(segmentPath);
+ if (partitionFilePath != null) {
+ PartitionMapper partitionMapper = readPartitionMap(partitionFilePath);
+ DataFileFooterConverter fileFooterConverter = new DataFileFooterConverter();
+ SegmentIndexFileStore indexFileStore = new SegmentIndexFileStore();
+ indexFileStore.readAllIIndexOfSegment(segmentPath);
+ Set<String> indexFilesFromSegment = indexFileStore.getCarbonIndexMap().keySet();
+ for (String indexFile : indexFilesFromSegment) {
+ // Check the partition information in the partiton mapper
+ List<String> indexPartitions = partitionMapper.partitionMap.get(indexFile);
+ if (indexPartitions == null || !currentPartitions.containsAll(indexPartitions)) {
+ Long fileTimestamp = CarbonUpdateUtil.getTimeStampAsLong(indexFile
+ .substring(indexFile.lastIndexOf(CarbonCommonConstants.HYPHEN) + 1,
+ indexFile.length() - CarbonTablePath.INDEX_FILE_EXT.length()));
+ if (CarbonUpdateUtil.isMaxQueryTimeoutExceeded(fileTimestamp) || forceDelete) {
--- End diff --
1. mergeindex also should be read based on transactiontimestamp, otherwise if droppartition is called and then clean files is called immediately , select can read previous index files which might get deleted during reading.
2. Alter drop partition can also recreate mergeindex map in same transactiontimestamp.
This can be handled along with partitionmap transactiontimestamp implementation.
---
Locked
