[GitHub] [carbondata] vikramahuja1001 opened a new pull request #3679: [WIP]: Update_delete_issue

[GitBox]

vikramahuja1001 opened a new pull request #3679: [WIP]:  Update_delete_issue
URL: https://github.com/apache/carbondata/pull/3679
 
 
    ### Why is this PR needed?
   
   
    ### What changes were proposed in this PR?
   
       
    ### Does this PR introduce any user interface change?
    - No
    - Yes. (please explain the change and update document)
   
    ### Is any new testcase added?
    - No
    - Yes
   
       
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3679: [WIP]:  Update_delete_issue
URL: https://github.com/apache/carbondata/pull/3679#issuecomment-603865095
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2557/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3679: [WIP]:  Update_delete_issue
URL: https://github.com/apache/carbondata/pull/3679#issuecomment-603873233
 
 
   Build Failed  with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/850/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

ajantha-bhat commented on a change in pull request #3679: [WIP]:  Update_delete_issue
URL: https://github.com/apache/carbondata/pull/3679#discussion_r398312583
 
 

 ##########
 File path: integration/spark/src/main/scala/org/apache/carbondata/indexserver/DistributedPruneRDD.scala
 ##########
 @@ -143,6 +143,7 @@ private[indexserver] class DistributedPruneRDD(@transient private val ss: SparkS
         val reader = dataMapFormat.createRecordReader(inputSplit, attemptContextImpl)
         reader.initialize(inputSplit, attemptContextImpl)
         while (reader.nextKeyValue()) {
+          reader.getCurrentValue.setRowCount(reader.getCurrentValue.getDetailInfo.getRowCount)
 
 Review comment:
   getCurrentValue is called 3 times here now !
   
   better to call once and set value or handle it inside.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

kunal642 commented on a change in pull request #3679: [WIP]:  Update_delete_issue
URL: https://github.com/apache/carbondata/pull/3679#discussion_r398314590
 
 

 ##########
 File path: integration/spark/src/main/scala/org/apache/carbondata/indexserver/DistributedPruneRDD.scala
 ##########
 @@ -143,6 +143,7 @@ private[indexserver] class DistributedPruneRDD(@transient private val ss: SparkS
         val reader = dataMapFormat.createRecordReader(inputSplit, attemptContextImpl)
         reader.initialize(inputSplit, attemptContextImpl)
         while (reader.nextKeyValue()) {
+          reader.getCurrentValue.setRowCount(reader.getCurrentValue.getDetailInfo.getRowCount)
 
 Review comment:
   I dont think this is the root cause of this issue. We need to check why CarbonInputSplit in ExtendedBlocklet has the wrong count

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3679: [WIP]:  Update_delete_issue
URL: https://github.com/apache/carbondata/pull/3679#issuecomment-604292383
 
 
   Build Failed  with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/853/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3679: [WIP]:  Update_delete_issue
URL: https://github.com/apache/carbondata/pull/3679#issuecomment-604292780
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2560/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

kunal642 commented on a change in pull request #3679: [WIP]:  Update_delete_issue
URL: https://github.com/apache/carbondata/pull/3679#discussion_r398405170
 
 

 ##########
 File path: integration/spark/src/main/scala/org/apache/carbondata/indexserver/DistributedPruneRDD.scala
 ##########
 @@ -145,6 +145,7 @@ private[indexserver] class DistributedPruneRDD(@transient private val ss: SparkS
         while (reader.nextKeyValue()) {
           blocklets.add(reader.getCurrentValue)
         }
+        reader.close()
 
 Review comment:
   no need to call close. nextKeyValue is already handling

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

kunal642 commented on issue #3679: [WIP]:  Update_delete_issue
URL: https://github.com/apache/carbondata/pull/3679#issuecomment-604306385
 
 
   @kumarvishal09 Please review
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

vikramahuja1001 commented on a change in pull request #3679: [CARBONDATA-3751]:  Segments are not Marked for delete if everything is deleted in a segment with index server enabled
URL: https://github.com/apache/carbondata/pull/3679#discussion_r398464643
 
 

 ##########
 File path: integration/spark/src/main/scala/org/apache/carbondata/indexserver/DistributedPruneRDD.scala
 ##########
 @@ -145,6 +145,7 @@ private[indexserver] class DistributedPruneRDD(@transient private val ss: SparkS
         while (reader.nextKeyValue()) {
           blocklets.add(reader.getCurrentValue)
         }
+        reader.close()
 
 Review comment:
   removed

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3679: [CARBONDATA-3751]:  Segments are not Marked for delete if everything is deleted in a segment with index server enabled
URL: https://github.com/apache/carbondata/pull/3679#issuecomment-604360058
 
 
   Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/854/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3679: [CARBONDATA-3751]:  Segments are not Marked for delete if everything is deleted in a segment with index server enabled
URL: https://github.com/apache/carbondata/pull/3679#issuecomment-604397670
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2563/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3679: [CARBONDATA-3751]:  Segments are not Marked for delete if everything is deleted in a segment with index server enabled
URL: https://github.com/apache/carbondata/pull/3679#issuecomment-604415074
 
 
   Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/857/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

kunal642 commented on issue #3679: [CARBONDATA-3751]:  Segments are not Marked for delete if everything is deleted in a segment with index server enabled
URL: https://github.com/apache/carbondata/pull/3679#issuecomment-608230730
 
 
   @kumarvishal09 Please review

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3679: [CARBONDATA-3751]:  Segments are not Marked for delete if everything is deleted in a segment with index server enabled
URL: https://github.com/apache/carbondata/pull/3679#issuecomment-609551072
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2640/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3679: [CARBONDATA-3751]:  Segments are not Marked for delete if everything is deleted in a segment with index server enabled
URL: https://github.com/apache/carbondata/pull/3679#issuecomment-609551752
 
 
   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/931/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3679: [CARBONDATA-3751]:  Segments are not Marked for delete if everything is deleted in a segment with index server enabled
URL: https://github.com/apache/carbondata/pull/3679#issuecomment-609551890
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2641/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3679: [CARBONDATA-3751]:  Segments are not Marked for delete if everything is deleted in a segment with index server enabled
URL: https://github.com/apache/carbondata/pull/3679#issuecomment-609557656
 
 
   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/934/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3679: [CARBONDATA-3751]:  Segments are not Marked for delete if everything is deleted in a segment with index server enabled
URL: https://github.com/apache/carbondata/pull/3679#issuecomment-609557905
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2644/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3679: [CARBONDATA-3751]:  Segments are not Marked for delete if everything is deleted in a segment with index server enabled
URL: https://github.com/apache/carbondata/pull/3679#issuecomment-609602209
 
 
   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/935/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services