[GitHub] carbondata pull request #2265: Added Performance Optimization for Presto by ...

[qiuchenjian-2]

Github user bhavya411 commented on the issue:

    https://github.com/apache/carbondata/pull/2265
 
    Yes we have created a JIRA for the same CARBONDATA-2583


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2265
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6263/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2265
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5233/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2265
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5102/



---

[qiuchenjian-2]

Github user bhavya411 commented on the issue:

    https://github.com/apache/carbondata/pull/2265
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2265
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6276/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2265
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5114/



---

[qiuchenjian-2]

Github user bhavya411 commented on the issue:

    https://github.com/apache/carbondata/pull/2265
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2265
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6279/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2265
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5117/



---

[qiuchenjian-2]

Github user bhavya411 commented on the issue:

    https://github.com/apache/carbondata/pull/2265
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2265
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6296/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2265
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5134/



---

[qiuchenjian-2]

Github user chenliang613 commented on the issue:

    https://github.com/apache/carbondata/pull/2265
 
    please solve the conflict.


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2265
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6384/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2265
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5219/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2265
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5332/



---

[qiuchenjian-2]

Github user chenliang613 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2265#discussion_r196753301
 
    --- Diff: integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataMetadata.java ---
    @@ -152,19 +176,20 @@ private ConnectorTableMetadata getTableMetadata(SchemaTableName schemaTableName)
     
           Type spiType = carbonDataType2SpiMapper(cs);
           columnHandles.put(cs.getColumnName(),
    -          new CarbondataColumnHandle(connectorId, cs.getColumnName(), spiType, column.getSchemaOrdinal(),
    -              column.getKeyOrdinal(), column.getColumnGroupOrdinal(), false, cs.getColumnGroupId(),
    -              cs.getColumnUniqueId(), cs.isUseInvertedIndex(), cs.getPrecision(), cs.getScale()));
    +          new CarbondataColumnHandle(connectorId, cs.getColumnName(), spiType,
    +              column.getSchemaOrdinal(), column.getKeyOrdinal(), column.getColumnGroupOrdinal(),
    +              false, cs.getColumnGroupId(), cs.getColumnUniqueId(), cs.isUseInvertedIndex(),
    +              cs.getPrecision(), cs.getScale()));
         }
     
         for (CarbonMeasure measure : cb.getMeasureByTableName(tableName)) {
           ColumnSchema cs = measure.getColumnSchema();
    -
           Type spiType = carbonDataType2SpiMapper(cs);
           columnHandles.put(cs.getColumnName(),
    -          new CarbondataColumnHandle(connectorId, cs.getColumnName(), spiType, cs.getSchemaOrdinal(),
    -              measure.getOrdinal(), cs.getColumnGroupId(), true, cs.getColumnGroupId(),
    -              cs.getColumnUniqueId(), cs.isUseInvertedIndex(), cs.getPrecision(), cs.getScale()));
    +          new CarbondataColumnHandle(connectorId, cs.getColumnName(), spiType,
    +              cs.getSchemaOrdinal(), measure.getOrdinal(), cs.getColumnGroupId(), true,
    +              cs.getColumnGroupId(), cs.getColumnUniqueId(), cs.isUseInvertedIndex(),
    +              cs.getPrecision(), cs.getScale()));
         }
     
         //should i cache it?
    --- End diff --
   
    this comment, suggest optimizing it.


---

[qiuchenjian-2]

Github user chenliang613 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2265#discussion_r196768157
 
    --- Diff: integration/presto/README.md ---
    @@ -113,6 +116,10 @@ Please follow the below steps to query carbondata in presto
       enable.unsafe.in.query.processing property by default is true in CarbonData system, the carbon.unsafe.working.memory.in.mb
       property defines the limit for Unsafe Memory usage in Mega Bytes, the default value is 512 MB.
       If your tables are big you can increase the unsafe memory, or disable unsafe via setting enable.unsafe.in.query.processing=false.
    +  
    +  If you do not want to use unsafe memory at all please set the below properties to false as well.
    --- End diff --
   
    currently , carbon-presto integration doesn't support unsafe, so here the description should be : please disable unsafe feature for presto integration module.


---

[qiuchenjian-2]

Github user chenliang613 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2265#discussion_r196773919
 
    --- Diff: integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataPageSourceProvider.java ---
    @@ -129,23 +135,31 @@ private QueryModel createQueryModel(CarbondataSplit carbondataSplit,
           String carbonTablePath = carbonTable.getAbsoluteTableIdentifier().getTablePath();
     
           conf.set(CarbonTableInputFormat.INPUT_DIR, carbonTablePath);
    +      conf.set("query.id", queryId);
           JobConf jobConf = new JobConf(conf);
           CarbonTableInputFormat carbonTableInputFormat = createInputFormat(jobConf, carbonTable,
               PrestoFilterUtil.parseFilterExpression(carbondataSplit.getConstraints()),
               carbonProjection);
           TaskAttemptContextImpl hadoopAttemptContext =
               new TaskAttemptContextImpl(jobConf, new TaskAttemptID("", 1, TaskType.MAP, 0, 0));
    -      CarbonInputSplit carbonInputSplit =
    -          CarbonLocalInputSplit.convertSplit(carbondataSplit.getLocalInputSplit());
    +      CarbonMultiBlockSplit carbonInputSplit =
    +          CarbonLocalMultiBlockSplit.convertSplit(carbondataSplit.getLocalInputSplit());
           QueryModel queryModel =
               carbonTableInputFormat.createQueryModel(carbonInputSplit, hadoopAttemptContext);
    +      queryModel.setQueryId(queryId);
           queryModel.setVectorReader(true);
    +      queryModel.setStatisticsRecorder(
    +          CarbonTimeStatisticsFactory.createExecutorRecorder(queryModel.getQueryId()));
     
    +      /*
           List<CarbonInputSplit> splitList = new ArrayList<>(1);
    -      splitList.add(carbonInputSplit);
    -      List<TableBlockInfo> tableBlockInfoList = CarbonInputSplit.createBlocks(splitList);
    --- End diff --
   
    please remove these dummy code.


---