[GitHub] [carbondata] Karan980 opened a new pull request #3970: [CARBONDATA-4007] Fix multiple issues in SDK_IUD

[GitBox]

Karan980 commented on pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#issuecomment-708993120


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#issuecomment-709105593


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4461/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#issuecomment-709116721


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2707/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#issuecomment-712129726


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4513/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#issuecomment-712132333


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2759/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#issuecomment-712134800


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2760/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#issuecomment-712135326


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4514/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#issuecomment-712260963


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4515/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#issuecomment-712277239


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2761/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

Karan980 commented on pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#issuecomment-712318046


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#issuecomment-712386082


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4517/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#issuecomment-712405367


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2763/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

Karan980 commented on pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#issuecomment-712595181


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#issuecomment-712641816


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4524/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#issuecomment-712643240


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2770/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on a change in pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#discussion_r508306255



##########
File path: sdk/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonReader.java
##########
@@ -109,7 +116,39 @@ public boolean hasNext() throws IOException, InterruptedException {
    */
   public T readNextRow() throws IOException, InterruptedException {
     validateReader();
-    return currentReader.getCurrentValue();
+    return formatDateAndTimeStamp((Object[]) currentReader.getCurrentValue());
+  }
+
+  public T formatDateAndTimeStamp(Object[] row) {
+    List<ProjectionDimension> dimensions = ((AbstractRecordReader) currentReader)
+            .getQueryModel().getProjectionDimensions();
+    String carbonDateFormat = CarbonProperties.getInstance()
+            .getProperty(CarbonCommonConstants.CARBON_DATE_FORMAT);
+    if (carbonDateFormat == null) {
+      carbonDateFormat = CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT;
+    }
+    SimpleDateFormat dateFormat = new SimpleDateFormat(carbonDateFormat);
+    String carbonTimeStampFormat = CarbonProperties.getInstance()
+            .getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT);
+    if (carbonTimeStampFormat  == null) {
+      carbonTimeStampFormat  = CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT;
+    }
+    SimpleDateFormat timeStampFormat = new SimpleDateFormat(carbonTimeStampFormat);
+    for (ProjectionDimension dimension : dimensions) {
+      ColumnSchema columnSchema = dimension.getDimension().getColumnSchema();
+      if (columnSchema == null) {
+        continue;
+      }
+      DataType dataType = columnSchema.getDataType();
+      if (dataType == DataTypes.DATE) {
+        row[dimension.getOrdinal()] = dateFormat
+                .format(new Date(24L * 3600 * 1000 * (int)row[dimension.getOrdinal()]));

Review comment:
       use DateDirectDictionaryGenerator.MILLIS_PER_DAY instead of "24L * 3600 * 1000 *"




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on a change in pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#discussion_r508307792



##########
File path: sdk/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonIUDTest.java
##########
@@ -72,6 +74,48 @@ public void testDelete() throws Exception {
     FileUtils.deleteDirectory(new File(path));
   }
 
+  @Test
+  public void testUpdateOnDateType() throws Exception {
+    String path = "./testWriteFiles";
+    CarbonProperties.getInstance()
+            .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+                    CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT)
+            .addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT,
+                    CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT);
+    FileUtils.deleteDirectory(new File(path));
+    Field[] fields = new Field[3];
+    fields[0] = new Field("intField", DataTypes.INT);
+    fields[1] = new Field("dateField", DataTypes.DATE);
+    fields[2] = new Field("timeField", DataTypes.TIMESTAMP);
+    CarbonWriter writer = CarbonWriter.builder()
+            .outputPath(path)
+            .withCsvInput(new Schema(fields))
+            .writtenBy("IUDTest")
+            .build();
+    for (int i = 0; i < 10; i++) {
+      String[] row2 = new String[]{
+              String.valueOf(i % 10000),
+              "2019-03-02",
+              "2019-02-12 03:03:34",
+      };
+      writer.write(row2);
+    }
+    writer.close();
+    CarbonIUD.getInstance().update(path, "intField", "0", "intField", "20").commit();
+    CarbonReader reader =
+            CarbonReader.builder(path).projection(new String[] { "intField", "dateField", "timeField" })
+                    .build();
+    int i = 0;
+    while (reader.hasNext()) {
+      Object[] row = (Object[]) reader.readNextRow();
+      assert ((int) row[0] != 0);

Review comment:
       please assert the complete row instead of first value




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

kunal642 commented on a change in pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#discussion_r508308091



##########
File path: sdk/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java
##########
@@ -1260,8 +1260,8 @@ public int compare(Field o1, Field o2) {
       Assert.assertEquals((double) id / 2, row[4]);
       Assert.assertEquals(true, (boolean) row[5]);
       long day = 24L * 3600 * 1000;

Review comment:
       if these valriables are not used then remove it

##########
File path: sdk/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java
##########
@@ -1260,8 +1260,8 @@ public int compare(Field o1, Field o2) {
       Assert.assertEquals((double) id / 2, row[4]);
       Assert.assertEquals(true, (boolean) row[5]);
       long day = 24L * 3600 * 1000;

Review comment:
       if these variables are not used then remove it




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#issuecomment-713095278


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2792/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3970:
URL: https://github.com/apache/carbondata/pull/3970#issuecomment-713152841


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4546/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]