[GitHub] [carbondata] akashrn5 opened a new pull request #3686: [WIP]refactor segmentRefreshInfo and fix cache issue in multiple application scenario

[GitBox]

akashrn5 opened a new pull request #3686: [WIP]refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686
 
 
   
    ### Why is this PR needed?
   
   
    ### What changes were proposed in this PR?
   
       
    ### Does this PR introduce any user interface change?
    - No
    - Yes. (please explain the change and update document)
   
    ### Is any new testcase added?
    - No
    - Yes
   
       
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3686: [WIP]refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#issuecomment-605443370
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2584/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3686: [WIP]refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#issuecomment-605443769
 
 
   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/876/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3686: [WIP]refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#issuecomment-605447204
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2585/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3686: [WIP]refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#issuecomment-605447041
 
 
   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/877/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3686: [WIP]refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#issuecomment-605465256
 
 
   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/878/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3686: [WIP]refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#issuecomment-605465429
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2586/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3686: [WIP]refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#issuecomment-605595334
 
 
   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/880/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3686: [WIP]refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#issuecomment-605595492
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2588/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3686: [WIP]refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#issuecomment-605607382
 
 
   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/881/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3686: [WIP]refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#issuecomment-605607429
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2589/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3686: [WIP]refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#issuecomment-605625433
 
 
   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/882/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3686: [WIP]refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#issuecomment-605625510
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2590/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

kunal642 commented on a change in pull request #3686: [CARBONDATA-3759]Refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#discussion_r405259513
 
 

 ##########
 File path: core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java
 ##########
 @@ -783,26 +783,28 @@ private static void closeStreams(Closeable... streams) {
 
   /**
    * Returns the invalid timestamp range of a segment.
-   * @param segmentId
-   * @return
    */
-  public UpdateVO getInvalidTimestampRange(String segmentId) {
+  public static UpdateVO getInvalidTimestampRange(LoadMetadataDetails loadMetadataDetails) {
     UpdateVO range = new UpdateVO();
-    for (LoadMetadataDetails segment : segmentDetails) {
-      if (segment.getLoadName().equalsIgnoreCase(segmentId)) {
-        range.setSegmentId(segmentId);
-        range.setFactTimestamp(segment.getLoadStartTime());
-        if (!segment.getUpdateDeltaStartTimestamp().isEmpty() && !segment
-            .getUpdateDeltaEndTimestamp().isEmpty()) {
-          range.setUpdateDeltaStartTimestamp(
-              CarbonUpdateUtil.getTimeStampAsLong(segment.getUpdateDeltaStartTimestamp()));
-          range.setLatestUpdateTimestamp(
-              CarbonUpdateUtil.getTimeStampAsLong(segment.getUpdateDeltaEndTimestamp()));
-        }
-        return range;
+    if (getInvalidTimeStampRange(range, loadMetadataDetails)) return range;
+    return range;
 
 Review comment:
   if check can be removed

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3686: [CARBONDATA-3759]Refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#discussion_r405266072
 
 

 ##########
 File path: core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java
 ##########
 @@ -783,26 +783,28 @@ private static void closeStreams(Closeable... streams) {
 
   /**
    * Returns the invalid timestamp range of a segment.
-   * @param segmentId
-   * @return
    */
-  public UpdateVO getInvalidTimestampRange(String segmentId) {
+  public static UpdateVO getInvalidTimestampRange(LoadMetadataDetails loadMetadataDetails) {
     UpdateVO range = new UpdateVO();
-    for (LoadMetadataDetails segment : segmentDetails) {
-      if (segment.getLoadName().equalsIgnoreCase(segmentId)) {
-        range.setSegmentId(segmentId);
-        range.setFactTimestamp(segment.getLoadStartTime());
-        if (!segment.getUpdateDeltaStartTimestamp().isEmpty() && !segment
-            .getUpdateDeltaEndTimestamp().isEmpty()) {
-          range.setUpdateDeltaStartTimestamp(
-              CarbonUpdateUtil.getTimeStampAsLong(segment.getUpdateDeltaStartTimestamp()));
-          range.setLatestUpdateTimestamp(
-              CarbonUpdateUtil.getTimeStampAsLong(segment.getUpdateDeltaEndTimestamp()));
-        }
-        return range;
+    if (getInvalidTimeStampRange(range, loadMetadataDetails)) return range;
+    return range;
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

kunal642 commented on a change in pull request #3686: [CARBONDATA-3759]Refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#discussion_r405277532
 
 

 ##########
 File path: core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java
 ##########
 @@ -783,26 +783,26 @@ private static void closeStreams(Closeable... streams) {
 
   /**
    * Returns the invalid timestamp range of a segment.
-   * @param segmentId
-   * @return
    */
-  public UpdateVO getInvalidTimestampRange(String segmentId) {
+  public static UpdateVO getInvalidTimestampRange(LoadMetadataDetails loadMetadataDetails) {
     UpdateVO range = new UpdateVO();
-    for (LoadMetadataDetails segment : segmentDetails) {
-      if (segment.getLoadName().equalsIgnoreCase(segmentId)) {
-        range.setSegmentId(segmentId);
-        range.setFactTimestamp(segment.getLoadStartTime());
-        if (!segment.getUpdateDeltaStartTimestamp().isEmpty() && !segment
-            .getUpdateDeltaEndTimestamp().isEmpty()) {
-          range.setUpdateDeltaStartTimestamp(
-              CarbonUpdateUtil.getTimeStampAsLong(segment.getUpdateDeltaStartTimestamp()));
-          range.setLatestUpdateTimestamp(
-              CarbonUpdateUtil.getTimeStampAsLong(segment.getUpdateDeltaEndTimestamp()));
-        }
-        return range;
+    getInvalidTimeStampRange(range, loadMetadataDetails);
+    return range;
+  }
+
+  private static void getInvalidTimeStampRange(UpdateVO range,
 
 Review comment:
   Can we conbine this with the above method as it is not called from anywhere else

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

kunal642 commented on a change in pull request #3686: [CARBONDATA-3759]Refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#discussion_r405277532
 
 

 ##########
 File path: core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java
 ##########
 @@ -783,26 +783,26 @@ private static void closeStreams(Closeable... streams) {
 
   /**
    * Returns the invalid timestamp range of a segment.
-   * @param segmentId
-   * @return
    */
-  public UpdateVO getInvalidTimestampRange(String segmentId) {
+  public static UpdateVO getInvalidTimestampRange(LoadMetadataDetails loadMetadataDetails) {
     UpdateVO range = new UpdateVO();
-    for (LoadMetadataDetails segment : segmentDetails) {
-      if (segment.getLoadName().equalsIgnoreCase(segmentId)) {
-        range.setSegmentId(segmentId);
-        range.setFactTimestamp(segment.getLoadStartTime());
-        if (!segment.getUpdateDeltaStartTimestamp().isEmpty() && !segment
-            .getUpdateDeltaEndTimestamp().isEmpty()) {
-          range.setUpdateDeltaStartTimestamp(
-              CarbonUpdateUtil.getTimeStampAsLong(segment.getUpdateDeltaStartTimestamp()));
-          range.setLatestUpdateTimestamp(
-              CarbonUpdateUtil.getTimeStampAsLong(segment.getUpdateDeltaEndTimestamp()));
-        }
-        return range;
+    getInvalidTimeStampRange(range, loadMetadataDetails);
+    return range;
+  }
+
+  private static void getInvalidTimeStampRange(UpdateVO range,
 
 Review comment:
   Can we combine this with the above method as it is not called from anywhere else

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3686: [CARBONDATA-3759]Refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#discussion_r405286913
 
 

 ##########
 File path: core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java
 ##########
 @@ -783,26 +783,26 @@ private static void closeStreams(Closeable... streams) {
 
   /**
    * Returns the invalid timestamp range of a segment.
-   * @param segmentId
-   * @return
    */
-  public UpdateVO getInvalidTimestampRange(String segmentId) {
+  public static UpdateVO getInvalidTimestampRange(LoadMetadataDetails loadMetadataDetails) {
     UpdateVO range = new UpdateVO();
-    for (LoadMetadataDetails segment : segmentDetails) {
-      if (segment.getLoadName().equalsIgnoreCase(segmentId)) {
-        range.setSegmentId(segmentId);
-        range.setFactTimestamp(segment.getLoadStartTime());
-        if (!segment.getUpdateDeltaStartTimestamp().isEmpty() && !segment
-            .getUpdateDeltaEndTimestamp().isEmpty()) {
-          range.setUpdateDeltaStartTimestamp(
-              CarbonUpdateUtil.getTimeStampAsLong(segment.getUpdateDeltaStartTimestamp()));
-          range.setLatestUpdateTimestamp(
-              CarbonUpdateUtil.getTimeStampAsLong(segment.getUpdateDeltaEndTimestamp()));
-        }
-        return range;
+    getInvalidTimeStampRange(range, loadMetadataDetails);
+    return range;
+  }
+
+  private static void getInvalidTimeStampRange(UpdateVO range,
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3686: [CARBONDATA-3759]Refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#issuecomment-610837294
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2676/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3686: [CARBONDATA-3759]Refactor segmentRefreshInfo and fix cache issue in multiple application scenario
URL: https://github.com/apache/carbondata/pull/3686#issuecomment-610844552
 
 
   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/964/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services