ravipesala commented on a change in pull request #3179: [CARBONDATA-3338] Support Incremental DataLoad for MV Datamap[with single parent table]
URL:
https://github.com/apache/carbondata/pull/3179#discussion_r278068979
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/status/DiskBasedDataMapStatusProvider.java
##########
@@ -196,4 +207,278 @@ private static ICarbonLock getDataMapStatusLock() {
.getSystemLevelCarbonLockObj(CarbonProperties.getInstance().getSystemFolderLocation(),
LockUsage.DATAMAP_STATUS_LOCK);
}
+
+ /**
+ * Reads and returns dataMapSegmentStatusDetail
+ *
+ * @param dataMapSchema
+ * @throws IOException
+ */
+ public DataMapSegmentStatusDetail getDataMapSegmentStatus(DataMapSchema dataMapSchema)
+ throws IOException {
+ String statusPath = getDatamapSegmentStatusFile(dataMapSchema.getDataMapName());
+ Gson gsonObjectToRead = new Gson();
+ DataInputStream dataInputStream = null;
+ BufferedReader buffReader = null;
+ InputStreamReader inStream = null;
+ DataMapSegmentStatusDetail dataMapSegmentStatusDetail;
+ try {
+ if (!FileFactory.isFileExist(statusPath)) {
+ return new DataMapSegmentStatusDetail();
+ }
+ dataInputStream =
+ FileFactory.getDataInputStream(statusPath, FileFactory.getFileType(statusPath));
+ inStream = new InputStreamReader(dataInputStream,
+ Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
+ buffReader = new BufferedReader(inStream);
+ dataMapSegmentStatusDetail =
+ gsonObjectToRead.fromJson(buffReader, DataMapSegmentStatusDetail.class);
+ } catch (IOException e) {
+ LOG.error("Failed to read datamap segment status", e);
+ throw e;
+ } finally {
+ CarbonUtil.closeStreams(buffReader, inStream, dataInputStream);
+ }
+
+ if (null == dataMapSegmentStatusDetail) {
+ return new DataMapSegmentStatusDetail();
+ }
+ return dataMapSegmentStatusDetail;
+ }
+
+ /**
+ * After each data load to mv datamap, update the segment status mapping. Get the new load name
+ * from datamap table loadMetaDetails and map newly loaded main table segments against the datamap
+ * table new load entry
+ *
+ * @param dataMapSchema
+ * @throws IOException
+ */
+ public void updateSegmentMapping(DataMapSchema dataMapSchema) throws IOException {
Review comment:
Segment mapping should be passed explicitly after the finish of datamap load. Don't read the main table status file to get it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[hidden email]
With regards,
Apache Git Services
Locked
