[GitHub] carbondata pull request #2792: [CARBONDATA-2981] Support read primitive data...

[qiuchenjian-2]

Github user KanakaKumar commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2792#discussion_r227836440
 
    --- Diff: docs/csdk-guide.md ---
    @@ -68,20 +68,42 @@ JNIEnv *initJVM() {
     bool readFromLocalWithoutProjection(JNIEnv *env) {
     
         CarbonReader carbonReaderClass;
    -    carbonReaderClass.builder(env, "../resources/carbondata", "test");
    +    carbonReaderClass.builder(env, "../resources/carbondata");
         carbonReaderClass.build();
     
    +    printf("\nRead data from local without projection:\n");
    +
    +    CarbonRow carbonRow(env);
         while (carbonReaderClass.hasNext()) {
    -        jobjectArray row = carbonReaderClass.readNextRow();
    -        jsize length = env->GetArrayLength(row);
    +        jobject row = carbonReaderClass.readNextCarbonRow();
    --- End diff --
   
    Need to correct readNextCarbonRow is no more present


---

[qiuchenjian-2]

Github user KanakaKumar commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2792#discussion_r227836644
 
    --- Diff: docs/csdk-guide.md ---
    @@ -106,20 +128,41 @@ bool readFromS3(JNIEnv *env, char *argv[]) {
         // "your endPoint"
         args[2] = argv[3];
     
    -    reader.builder(env, "s3a://sdk/WriterOutput", "test");
    -    reader.withHadoopConf(3, args);
    +    reader.builder(env, "s3a://sdk/WriterOutput/carbondata/", "test");
    +    reader.withHadoopConf("fs.s3a.access.key", argv[1]);
    +    reader.withHadoopConf("fs.s3a.secret.key", argv[2]);
    +    reader.withHadoopConf("fs.s3a.endpoint", argv[3]);
         reader.build();
         printf("\nRead data from S3:\n");
    +    CarbonRow carbonRow(env);
         while (reader.hasNext()) {
    -        jobjectArray row = reader.readNextRow();
    -        jsize length = env->GetArrayLength(row);
    -
    +        jobject row = reader.readNextCarbonRow();
    --- End diff --
   
    readNextCarbonRow is removed from this PR


---

[qiuchenjian-2]

Github user KanakaKumar commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2792#discussion_r227839131
 
    --- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java ---
    @@ -1522,4 +1522,204 @@ public boolean accept(File dir, String name) {
           e.printStackTrace();
         }
       }
    +
    +   @Test
    +  public void testReadNextRowWithRowUtil() {
    +    String path = "./carbondata";
    +    try {
    +      FileUtils.deleteDirectory(new File(path));
    +
    +      Field[] fields = new Field[12];
    +      fields[0] = new Field("stringField", DataTypes.STRING);
    +      fields[1] = new Field("shortField", DataTypes.SHORT);
    +      fields[2] = new Field("intField", DataTypes.INT);
    +      fields[3] = new Field("longField", DataTypes.LONG);
    +      fields[4] = new Field("doubleField", DataTypes.DOUBLE);
    +      fields[5] = new Field("boolField", DataTypes.BOOLEAN);
    +      fields[6] = new Field("dateField", DataTypes.DATE);
    +      fields[7] = new Field("timeField", DataTypes.TIMESTAMP);
    +      fields[8] = new Field("decimalField", DataTypes.createDecimalType(8, 2));
    +      fields[9] = new Field("varcharField", DataTypes.VARCHAR);
    +      fields[10] = new Field("arrayField", DataTypes.createArrayType(DataTypes.STRING));
    +      fields[11] = new Field("floatField", DataTypes.FLOAT);
    +      Map<String, String> map = new HashMap<>();
    +      map.put("complex_delimiter_level_1", "#");
    +      CarbonWriter writer = CarbonWriter.builder()
    +          .outputPath(path)
    +          .withLoadOptions(map)
    +          .withCsvInput(new Schema(fields)).build();
    +
    +      for (int i = 0; i < 10; i++) {
    +        String[] row2 = new String[]{
    +            "robot" + (i % 10),
    +            String.valueOf(i % 10000),
    +            String.valueOf(i),
    +            String.valueOf(Long.MAX_VALUE - i),
    +            String.valueOf((double) i / 2),
    +            String.valueOf(true),
    +            "2019-03-02",
    +            "2019-02-12 03:03:34",
    +            "12.345",
    +            "varchar",
    +            "Hello#World#From#Carbon",
    +            "1.23"
    +        };
    +        writer.write(row2);
    +      }
    +      writer.close();
    +
    +      File[] dataFiles = new File(path).listFiles(new FilenameFilter() {
    +        @Override
    +        public boolean accept(File dir, String name) {
    +          if (name == null) {
    +            return false;
    +          }
    +          return name.endsWith("carbonindex");
    +        }
    +      });
    +      if (dataFiles == null || dataFiles.length < 1) {
    +        throw new RuntimeException("Carbon index file not exists.");
    +      }
    +      Schema schema = CarbonSchemaReader
    +          .readSchemaInIndexFile(dataFiles[0].getAbsolutePath())
    +          .asOriginOrder();
    +      // Transform the schema
    +      int count = 0;
    +      for (int i = 0; i < schema.getFields().length; i++) {
    +        if (!((schema.getFields())[i].getFieldName().contains("."))) {
    +          count++;
    +        }
    +      }
    +      String[] strings = new String[count];
    +      int index = 0;
    +      for (int i = 0; i < schema.getFields().length; i++) {
    +        if (!((schema.getFields())[i].getFieldName().contains("."))) {
    +          strings[index] = (schema.getFields())[i].getFieldName();
    +          index++;
    +        }
    +      }
    +      // Read data
    +      CarbonReader reader = CarbonReader
    +          .builder(path, "_temp")
    +          .projection(strings)
    +          .build();
    +
    +      int i = 0;
    +      while (reader.hasNext()) {
    +        Object[] data = (Object[]) reader.readNextRow();
    +
    +        assert (RowUtil.getString(data, 0).equals("robot" + i));
    +        assertEquals(RowUtil.getShort(data, 1), i);
    +        assertEquals(RowUtil.getInt(data, 2), i);
    +        assertEquals(RowUtil.getLong(data, 3), Long.MAX_VALUE - i);
    +        assertEquals(RowUtil.getDouble(data, 4), ((double) i) / 2);
    +        assert (RowUtil.getBoolean(data, 5));
    +        assertEquals(RowUtil.getInt(data, 6), 17957);
    +        assert (RowUtil.getDecimal(data, 8).equals("12.35"));
    +        assert (RowUtil.getVarchar(data, 9).equals("varchar"));
    +
    +        Object[] arr = RowUtil.getArray(data, 10);
    +        assert (arr[0].equals("Hello"));
    +        assert (arr[1].equals("World"));
    +        assert (arr[2].equals("From"));
    +        assert (arr[3].equals("Carbon"));
    +
    +        assertEquals(RowUtil.getFloat(data, 11), (float) 1.23);
    +        i++;
    +      }
    +      reader.close();
    +    } catch (Throwable e) {
    +      e.printStackTrace();
    --- End diff --
   
    I think we should make test fail for any exception. Should not ignore


---

[qiuchenjian-2]

Github user KanakaKumar commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2792#discussion_r227839509
 
    --- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java ---
    @@ -1522,4 +1522,204 @@ public boolean accept(File dir, String name) {
           e.printStackTrace();
         }
       }
    +
    +   @Test
    +  public void testReadNextRowWithRowUtil() {
    +    String path = "./carbondata";
    +    try {
    +      FileUtils.deleteDirectory(new File(path));
    +
    +      Field[] fields = new Field[12];
    +      fields[0] = new Field("stringField", DataTypes.STRING);
    +      fields[1] = new Field("shortField", DataTypes.SHORT);
    +      fields[2] = new Field("intField", DataTypes.INT);
    +      fields[3] = new Field("longField", DataTypes.LONG);
    +      fields[4] = new Field("doubleField", DataTypes.DOUBLE);
    +      fields[5] = new Field("boolField", DataTypes.BOOLEAN);
    +      fields[6] = new Field("dateField", DataTypes.DATE);
    +      fields[7] = new Field("timeField", DataTypes.TIMESTAMP);
    +      fields[8] = new Field("decimalField", DataTypes.createDecimalType(8, 2));
    +      fields[9] = new Field("varcharField", DataTypes.VARCHAR);
    +      fields[10] = new Field("arrayField", DataTypes.createArrayType(DataTypes.STRING));
    +      fields[11] = new Field("floatField", DataTypes.FLOAT);
    +      Map<String, String> map = new HashMap<>();
    +      map.put("complex_delimiter_level_1", "#");
    +      CarbonWriter writer = CarbonWriter.builder()
    +          .outputPath(path)
    +          .withLoadOptions(map)
    +          .withCsvInput(new Schema(fields)).build();
    +
    +      for (int i = 0; i < 10; i++) {
    +        String[] row2 = new String[]{
    +            "robot" + (i % 10),
    +            String.valueOf(i % 10000),
    +            String.valueOf(i),
    +            String.valueOf(Long.MAX_VALUE - i),
    +            String.valueOf((double) i / 2),
    +            String.valueOf(true),
    +            "2019-03-02",
    +            "2019-02-12 03:03:34",
    +            "12.345",
    +            "varchar",
    +            "Hello#World#From#Carbon",
    +            "1.23"
    +        };
    +        writer.write(row2);
    +      }
    +      writer.close();
    +
    +      File[] dataFiles = new File(path).listFiles(new FilenameFilter() {
    +        @Override
    +        public boolean accept(File dir, String name) {
    +          if (name == null) {
    +            return false;
    +          }
    +          return name.endsWith("carbonindex");
    +        }
    +      });
    +      if (dataFiles == null || dataFiles.length < 1) {
    +        throw new RuntimeException("Carbon index file not exists.");
    +      }
    +      Schema schema = CarbonSchemaReader
    +          .readSchemaInIndexFile(dataFiles[0].getAbsolutePath())
    +          .asOriginOrder();
    +      // Transform the schema
    +      int count = 0;
    +      for (int i = 0; i < schema.getFields().length; i++) {
    +        if (!((schema.getFields())[i].getFieldName().contains("."))) {
    +          count++;
    +        }
    +      }
    +      String[] strings = new String[count];
    +      int index = 0;
    +      for (int i = 0; i < schema.getFields().length; i++) {
    +        if (!((schema.getFields())[i].getFieldName().contains("."))) {
    +          strings[index] = (schema.getFields())[i].getFieldName();
    +          index++;
    +        }
    +      }
    +      // Read data
    +      CarbonReader reader = CarbonReader
    +          .builder(path, "_temp")
    +          .projection(strings)
    +          .build();
    +
    +      int i = 0;
    +      while (reader.hasNext()) {
    +        Object[] data = (Object[]) reader.readNextRow();
    +
    +        assert (RowUtil.getString(data, 0).equals("robot" + i));
    +        assertEquals(RowUtil.getShort(data, 1), i);
    +        assertEquals(RowUtil.getInt(data, 2), i);
    +        assertEquals(RowUtil.getLong(data, 3), Long.MAX_VALUE - i);
    +        assertEquals(RowUtil.getDouble(data, 4), ((double) i) / 2);
    +        assert (RowUtil.getBoolean(data, 5));
    +        assertEquals(RowUtil.getInt(data, 6), 17957);
    +        assert (RowUtil.getDecimal(data, 8).equals("12.35"));
    +        assert (RowUtil.getVarchar(data, 9).equals("varchar"));
    +
    +        Object[] arr = RowUtil.getArray(data, 10);
    +        assert (arr[0].equals("Hello"));
    +        assert (arr[1].equals("World"));
    +        assert (arr[2].equals("From"));
    +        assert (arr[3].equals("Carbon"));
    +
    +        assertEquals(RowUtil.getFloat(data, 11), (float) 1.23);
    +        i++;
    +      }
    +      reader.close();
    +    } catch (Throwable e) {
    +      e.printStackTrace();
    +    } finally {
    +      try {
    +        FileUtils.deleteDirectory(new File(path));
    +      } catch (IOException e) {
    +        e.printStackTrace();
    +      }
    +    }
    +   }
    +
    +  @Test
    +  public void testReadNextRowWithProjectionAndRowUtil() {
    +    String path = "./carbondata";
    +    try {
    +      FileUtils.deleteDirectory(new File(path));
    +
    +      Field[] fields = new Field[12];
    +      fields[0] = new Field("stringField", DataTypes.STRING);
    +      fields[1] = new Field("shortField", DataTypes.SHORT);
    +      fields[2] = new Field("intField", DataTypes.INT);
    +      fields[3] = new Field("longField", DataTypes.LONG);
    +      fields[4] = new Field("doubleField", DataTypes.DOUBLE);
    +      fields[5] = new Field("boolField", DataTypes.BOOLEAN);
    +      fields[6] = new Field("dateField", DataTypes.DATE);
    +      fields[7] = new Field("timeField", DataTypes.TIMESTAMP);
    +      fields[8] = new Field("decimalField", DataTypes.createDecimalType(8, 2));
    +      fields[9] = new Field("varcharField", DataTypes.VARCHAR);
    +      fields[10] = new Field("arrayField", DataTypes.createArrayType(DataTypes.STRING));
    +      fields[11] = new Field("floatField", DataTypes.FLOAT);
    +      Map<String, String> map = new HashMap<>();
    +      map.put("complex_delimiter_level_1", "#");
    +      CarbonWriter writer = CarbonWriter.builder()
    +          .outputPath(path)
    +          .withLoadOptions(map)
    +          .withCsvInput(new Schema(fields)).build();
    +
    +      for (int i = 0; i < 10; i++) {
    +        String[] row2 = new String[]{
    +            "robot" + (i % 10),
    +            String.valueOf(i % 10000),
    +            String.valueOf(i),
    +            String.valueOf(Long.MAX_VALUE - i),
    +            String.valueOf((double) i / 2),
    +            String.valueOf(true),
    +            "2019-03-02",
    +            "2019-02-12 03:03:34",
    +            "12.345",
    +            "varchar",
    +            "Hello#World#From#Carbon",
    +            "1.23"
    +        };
    +        writer.write(row2);
    +      }
    +      writer.close();
    +
    +      // Read data
    +      CarbonReader reader = CarbonReader
    +          .builder(path, "_temp")
    +          .build();
    +
    +      int i = 0;
    +      while (reader.hasNext()) {
    +        Object[] data = (Object[]) reader.readNextRow();
    +
    +        assert (RowUtil.getString(data, 0).equals("robot" + i));
    +        assertEquals(RowUtil.getInt(data, 1), 17957);
    +        assert (RowUtil.getVarchar(data, 3).equals("varchar"));
    +        Object[] arr = RowUtil.getArray(data, 4);
    +        assert (arr[0].equals("Hello"));
    +        assert (arr[1].equals("World"));
    +        assert (arr[2].equals("From"));
    +        assert (arr[3].equals("Carbon"));
    +        assertEquals(RowUtil.getShort(data, 5), i);
    +        assertEquals(RowUtil.getInt(data, 6), i);
    +        assertEquals(RowUtil.getLong(data, 7), Long.MAX_VALUE - i);
    +        assertEquals(RowUtil.getDouble(data, 8), ((double) i) / 2);
    +        assert (RowUtil.getBoolean(data, 9));
    +        assert (RowUtil.getDecimal(data, 10).equals("12.35"));
    +        assertEquals(RowUtil.getFloat(data, 11), (float) 1.23);
    +        i++;
    +      }
    +      reader.close();
    +    } catch (Throwable e) {
    +      e.printStackTrace();
    --- End diff --
   
    add fail() for reader exception cases


---

[qiuchenjian-2]

Github user KanakaKumar commented on the issue:

    https://github.com/apache/carbondata/pull/2792
 
    LGTM apart from those nits in guide & ut.


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2792
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1209/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2792
 
    Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9262/



---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2792#discussion_r228009608
 
    --- Diff: docs/csdk-guide.md ---
    @@ -68,20 +68,42 @@ JNIEnv *initJVM() {
     bool readFromLocalWithoutProjection(JNIEnv *env) {
     
         CarbonReader carbonReaderClass;
    -    carbonReaderClass.builder(env, "../resources/carbondata", "test");
    +    carbonReaderClass.builder(env, "../resources/carbondata");
         carbonReaderClass.build();
     
    +    printf("\nRead data from local without projection:\n");
    +
    +    CarbonRow carbonRow(env);
         while (carbonReaderClass.hasNext()) {
    -        jobjectArray row = carbonReaderClass.readNextRow();
    -        jsize length = env->GetArrayLength(row);
    +        jobject row = carbonReaderClass.readNextCarbonRow();
    --- End diff --
   
    ok, done


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2792#discussion_r228009633
 
    --- Diff: docs/csdk-guide.md ---
    @@ -106,20 +128,41 @@ bool readFromS3(JNIEnv *env, char *argv[]) {
         // "your endPoint"
         args[2] = argv[3];
     
    -    reader.builder(env, "s3a://sdk/WriterOutput", "test");
    -    reader.withHadoopConf(3, args);
    +    reader.builder(env, "s3a://sdk/WriterOutput/carbondata/", "test");
    +    reader.withHadoopConf("fs.s3a.access.key", argv[1]);
    +    reader.withHadoopConf("fs.s3a.secret.key", argv[2]);
    +    reader.withHadoopConf("fs.s3a.endpoint", argv[3]);
         reader.build();
         printf("\nRead data from S3:\n");
    +    CarbonRow carbonRow(env);
         while (reader.hasNext()) {
    -        jobjectArray row = reader.readNextRow();
    -        jsize length = env->GetArrayLength(row);
    -
    +        jobject row = reader.readNextCarbonRow();
    --- End diff --
   
    ok, done


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2792#discussion_r228009906
 
    --- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java ---
    @@ -1522,4 +1522,204 @@ public boolean accept(File dir, String name) {
           e.printStackTrace();
         }
       }
    +
    +   @Test
    +  public void testReadNextRowWithRowUtil() {
    +    String path = "./carbondata";
    +    try {
    +      FileUtils.deleteDirectory(new File(path));
    +
    +      Field[] fields = new Field[12];
    +      fields[0] = new Field("stringField", DataTypes.STRING);
    +      fields[1] = new Field("shortField", DataTypes.SHORT);
    +      fields[2] = new Field("intField", DataTypes.INT);
    +      fields[3] = new Field("longField", DataTypes.LONG);
    +      fields[4] = new Field("doubleField", DataTypes.DOUBLE);
    +      fields[5] = new Field("boolField", DataTypes.BOOLEAN);
    +      fields[6] = new Field("dateField", DataTypes.DATE);
    +      fields[7] = new Field("timeField", DataTypes.TIMESTAMP);
    +      fields[8] = new Field("decimalField", DataTypes.createDecimalType(8, 2));
    +      fields[9] = new Field("varcharField", DataTypes.VARCHAR);
    +      fields[10] = new Field("arrayField", DataTypes.createArrayType(DataTypes.STRING));
    +      fields[11] = new Field("floatField", DataTypes.FLOAT);
    +      Map<String, String> map = new HashMap<>();
    +      map.put("complex_delimiter_level_1", "#");
    +      CarbonWriter writer = CarbonWriter.builder()
    +          .outputPath(path)
    +          .withLoadOptions(map)
    +          .withCsvInput(new Schema(fields)).build();
    +
    +      for (int i = 0; i < 10; i++) {
    +        String[] row2 = new String[]{
    +            "robot" + (i % 10),
    +            String.valueOf(i % 10000),
    +            String.valueOf(i),
    +            String.valueOf(Long.MAX_VALUE - i),
    +            String.valueOf((double) i / 2),
    +            String.valueOf(true),
    +            "2019-03-02",
    +            "2019-02-12 03:03:34",
    +            "12.345",
    +            "varchar",
    +            "Hello#World#From#Carbon",
    +            "1.23"
    +        };
    +        writer.write(row2);
    +      }
    +      writer.close();
    +
    +      File[] dataFiles = new File(path).listFiles(new FilenameFilter() {
    +        @Override
    +        public boolean accept(File dir, String name) {
    +          if (name == null) {
    +            return false;
    +          }
    +          return name.endsWith("carbonindex");
    +        }
    +      });
    +      if (dataFiles == null || dataFiles.length < 1) {
    +        throw new RuntimeException("Carbon index file not exists.");
    +      }
    +      Schema schema = CarbonSchemaReader
    +          .readSchemaInIndexFile(dataFiles[0].getAbsolutePath())
    +          .asOriginOrder();
    +      // Transform the schema
    +      int count = 0;
    +      for (int i = 0; i < schema.getFields().length; i++) {
    +        if (!((schema.getFields())[i].getFieldName().contains("."))) {
    +          count++;
    +        }
    +      }
    +      String[] strings = new String[count];
    +      int index = 0;
    +      for (int i = 0; i < schema.getFields().length; i++) {
    +        if (!((schema.getFields())[i].getFieldName().contains("."))) {
    +          strings[index] = (schema.getFields())[i].getFieldName();
    +          index++;
    +        }
    +      }
    +      // Read data
    +      CarbonReader reader = CarbonReader
    +          .builder(path, "_temp")
    +          .projection(strings)
    +          .build();
    +
    +      int i = 0;
    +      while (reader.hasNext()) {
    +        Object[] data = (Object[]) reader.readNextRow();
    +
    +        assert (RowUtil.getString(data, 0).equals("robot" + i));
    +        assertEquals(RowUtil.getShort(data, 1), i);
    +        assertEquals(RowUtil.getInt(data, 2), i);
    +        assertEquals(RowUtil.getLong(data, 3), Long.MAX_VALUE - i);
    +        assertEquals(RowUtil.getDouble(data, 4), ((double) i) / 2);
    +        assert (RowUtil.getBoolean(data, 5));
    +        assertEquals(RowUtil.getInt(data, 6), 17957);
    +        assert (RowUtil.getDecimal(data, 8).equals("12.35"));
    +        assert (RowUtil.getVarchar(data, 9).equals("varchar"));
    +
    +        Object[] arr = RowUtil.getArray(data, 10);
    +        assert (arr[0].equals("Hello"));
    +        assert (arr[1].equals("World"));
    +        assert (arr[2].equals("From"));
    +        assert (arr[3].equals("Carbon"));
    +
    +        assertEquals(RowUtil.getFloat(data, 11), (float) 1.23);
    +        i++;
    +      }
    +      reader.close();
    +    } catch (Throwable e) {
    +      e.printStackTrace();
    --- End diff --
   
    ok, done


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2792#discussion_r228010100
 
    --- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java ---
    @@ -1522,4 +1522,204 @@ public boolean accept(File dir, String name) {
           e.printStackTrace();
         }
       }
    +
    +   @Test
    +  public void testReadNextRowWithRowUtil() {
    +    String path = "./carbondata";
    +    try {
    +      FileUtils.deleteDirectory(new File(path));
    +
    +      Field[] fields = new Field[12];
    +      fields[0] = new Field("stringField", DataTypes.STRING);
    +      fields[1] = new Field("shortField", DataTypes.SHORT);
    +      fields[2] = new Field("intField", DataTypes.INT);
    +      fields[3] = new Field("longField", DataTypes.LONG);
    +      fields[4] = new Field("doubleField", DataTypes.DOUBLE);
    +      fields[5] = new Field("boolField", DataTypes.BOOLEAN);
    +      fields[6] = new Field("dateField", DataTypes.DATE);
    +      fields[7] = new Field("timeField", DataTypes.TIMESTAMP);
    +      fields[8] = new Field("decimalField", DataTypes.createDecimalType(8, 2));
    +      fields[9] = new Field("varcharField", DataTypes.VARCHAR);
    +      fields[10] = new Field("arrayField", DataTypes.createArrayType(DataTypes.STRING));
    +      fields[11] = new Field("floatField", DataTypes.FLOAT);
    +      Map<String, String> map = new HashMap<>();
    +      map.put("complex_delimiter_level_1", "#");
    +      CarbonWriter writer = CarbonWriter.builder()
    +          .outputPath(path)
    +          .withLoadOptions(map)
    +          .withCsvInput(new Schema(fields)).build();
    +
    +      for (int i = 0; i < 10; i++) {
    +        String[] row2 = new String[]{
    +            "robot" + (i % 10),
    +            String.valueOf(i % 10000),
    +            String.valueOf(i),
    +            String.valueOf(Long.MAX_VALUE - i),
    +            String.valueOf((double) i / 2),
    +            String.valueOf(true),
    +            "2019-03-02",
    +            "2019-02-12 03:03:34",
    +            "12.345",
    +            "varchar",
    +            "Hello#World#From#Carbon",
    +            "1.23"
    +        };
    +        writer.write(row2);
    +      }
    +      writer.close();
    +
    +      File[] dataFiles = new File(path).listFiles(new FilenameFilter() {
    +        @Override
    +        public boolean accept(File dir, String name) {
    +          if (name == null) {
    +            return false;
    +          }
    +          return name.endsWith("carbonindex");
    +        }
    +      });
    +      if (dataFiles == null || dataFiles.length < 1) {
    +        throw new RuntimeException("Carbon index file not exists.");
    +      }
    +      Schema schema = CarbonSchemaReader
    +          .readSchemaInIndexFile(dataFiles[0].getAbsolutePath())
    +          .asOriginOrder();
    +      // Transform the schema
    +      int count = 0;
    +      for (int i = 0; i < schema.getFields().length; i++) {
    +        if (!((schema.getFields())[i].getFieldName().contains("."))) {
    +          count++;
    +        }
    +      }
    +      String[] strings = new String[count];
    +      int index = 0;
    +      for (int i = 0; i < schema.getFields().length; i++) {
    +        if (!((schema.getFields())[i].getFieldName().contains("."))) {
    +          strings[index] = (schema.getFields())[i].getFieldName();
    +          index++;
    +        }
    +      }
    +      // Read data
    +      CarbonReader reader = CarbonReader
    +          .builder(path, "_temp")
    +          .projection(strings)
    +          .build();
    +
    +      int i = 0;
    +      while (reader.hasNext()) {
    +        Object[] data = (Object[]) reader.readNextRow();
    +
    +        assert (RowUtil.getString(data, 0).equals("robot" + i));
    +        assertEquals(RowUtil.getShort(data, 1), i);
    +        assertEquals(RowUtil.getInt(data, 2), i);
    +        assertEquals(RowUtil.getLong(data, 3), Long.MAX_VALUE - i);
    +        assertEquals(RowUtil.getDouble(data, 4), ((double) i) / 2);
    +        assert (RowUtil.getBoolean(data, 5));
    +        assertEquals(RowUtil.getInt(data, 6), 17957);
    +        assert (RowUtil.getDecimal(data, 8).equals("12.35"));
    +        assert (RowUtil.getVarchar(data, 9).equals("varchar"));
    +
    +        Object[] arr = RowUtil.getArray(data, 10);
    +        assert (arr[0].equals("Hello"));
    +        assert (arr[1].equals("World"));
    +        assert (arr[2].equals("From"));
    +        assert (arr[3].equals("Carbon"));
    +
    +        assertEquals(RowUtil.getFloat(data, 11), (float) 1.23);
    +        i++;
    +      }
    +      reader.close();
    +    } catch (Throwable e) {
    +      e.printStackTrace();
    +    } finally {
    +      try {
    +        FileUtils.deleteDirectory(new File(path));
    +      } catch (IOException e) {
    +        e.printStackTrace();
    +      }
    +    }
    +   }
    +
    +  @Test
    +  public void testReadNextRowWithProjectionAndRowUtil() {
    +    String path = "./carbondata";
    +    try {
    +      FileUtils.deleteDirectory(new File(path));
    +
    +      Field[] fields = new Field[12];
    +      fields[0] = new Field("stringField", DataTypes.STRING);
    +      fields[1] = new Field("shortField", DataTypes.SHORT);
    +      fields[2] = new Field("intField", DataTypes.INT);
    +      fields[3] = new Field("longField", DataTypes.LONG);
    +      fields[4] = new Field("doubleField", DataTypes.DOUBLE);
    +      fields[5] = new Field("boolField", DataTypes.BOOLEAN);
    +      fields[6] = new Field("dateField", DataTypes.DATE);
    +      fields[7] = new Field("timeField", DataTypes.TIMESTAMP);
    +      fields[8] = new Field("decimalField", DataTypes.createDecimalType(8, 2));
    +      fields[9] = new Field("varcharField", DataTypes.VARCHAR);
    +      fields[10] = new Field("arrayField", DataTypes.createArrayType(DataTypes.STRING));
    +      fields[11] = new Field("floatField", DataTypes.FLOAT);
    +      Map<String, String> map = new HashMap<>();
    +      map.put("complex_delimiter_level_1", "#");
    +      CarbonWriter writer = CarbonWriter.builder()
    +          .outputPath(path)
    +          .withLoadOptions(map)
    +          .withCsvInput(new Schema(fields)).build();
    +
    +      for (int i = 0; i < 10; i++) {
    +        String[] row2 = new String[]{
    +            "robot" + (i % 10),
    +            String.valueOf(i % 10000),
    +            String.valueOf(i),
    +            String.valueOf(Long.MAX_VALUE - i),
    +            String.valueOf((double) i / 2),
    +            String.valueOf(true),
    +            "2019-03-02",
    +            "2019-02-12 03:03:34",
    +            "12.345",
    +            "varchar",
    +            "Hello#World#From#Carbon",
    +            "1.23"
    +        };
    +        writer.write(row2);
    +      }
    +      writer.close();
    +
    +      // Read data
    +      CarbonReader reader = CarbonReader
    +          .builder(path, "_temp")
    +          .build();
    +
    +      int i = 0;
    +      while (reader.hasNext()) {
    +        Object[] data = (Object[]) reader.readNextRow();
    +
    +        assert (RowUtil.getString(data, 0).equals("robot" + i));
    +        assertEquals(RowUtil.getInt(data, 1), 17957);
    +        assert (RowUtil.getVarchar(data, 3).equals("varchar"));
    +        Object[] arr = RowUtil.getArray(data, 4);
    +        assert (arr[0].equals("Hello"));
    +        assert (arr[1].equals("World"));
    +        assert (arr[2].equals("From"));
    +        assert (arr[3].equals("Carbon"));
    +        assertEquals(RowUtil.getShort(data, 5), i);
    +        assertEquals(RowUtil.getInt(data, 6), i);
    +        assertEquals(RowUtil.getLong(data, 7), Long.MAX_VALUE - i);
    +        assertEquals(RowUtil.getDouble(data, 8), ((double) i) / 2);
    +        assert (RowUtil.getBoolean(data, 9));
    +        assert (RowUtil.getDecimal(data, 10).equals("12.35"));
    +        assertEquals(RowUtil.getFloat(data, 11), (float) 1.23);
    +        i++;
    +      }
    +      reader.close();
    +    } catch (Throwable e) {
    +      e.printStackTrace();
    --- End diff --
   
    ok, done


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2792
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/998/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2792
 
    Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1211/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2792
 
    Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9264/



---

[qiuchenjian-2]

Github user xubo245 commented on the issue:

    https://github.com/apache/carbondata/pull/2792
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2792
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1000/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2792
 
    Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9266/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2792
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1213/



---

[qiuchenjian-2]

Github user ajantha-bhat commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2792#discussion_r228039418
 
    --- Diff: docs/csdk-guide.md ---
    @@ -68,20 +68,42 @@ JNIEnv *initJVM() {
     bool readFromLocalWithoutProjection(JNIEnv *env) {
     
         CarbonReader carbonReaderClass;
    -    carbonReaderClass.builder(env, "../resources/carbondata", "test");
    +    carbonReaderClass.builder(env, "../resources/carbondata");
    --- End diff --
   
    Already main.cpp has these same examples. Please give a link to that file and remove from here.
    Because if any future change happens, we need not have to change at two places and keep duplicate code samples.


---

[qiuchenjian-2]

Github user ajantha-bhat commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2792#discussion_r228040609
 
    --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/RowUtil.java ---
    @@ -0,0 +1,146 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.sdk.file;
    --- End diff --
   
    I think we don't need this class at all. This is just a typecast. If user already knows data type, he can typecast instead of calling a method that does typecast.
   
    please remove this call


---