[GitHub] carbondata pull request #1583: [CARBONDATA-1822][Spark-Integration] Support ...

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1583
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2091/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1583
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2092/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1583
 
    Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/441/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1583
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1709/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1583
 
    retest sdv please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1583
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1711/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1583
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2096/



---

[qiuchenjian-2]

Github user mohammadshahidkhan commented on the issue:

    https://github.com/apache/carbondata/pull/1583
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1583
 
    Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/451/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1583
 
    retest this please


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1583#discussion_r154895427
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala ---
    @@ -876,5 +876,4 @@ object CommonUtil {
           }
         }
       }
    -
    --- End diff --
   
    remove unnecessary change


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1583
 
    Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/452/



---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1583#discussion_r154900160
 
    --- Diff: integration/spark2/src/main/scala/org/apache/carbondata/spark/util/CarbonSparkUtil.scala ---
    @@ -51,4 +55,90 @@ object CarbonSparkUtil {
           table)
       }
     
    +  /**
    +   * Below method will be used to get the fields object for pre aggregate table
    +   *
    +   * @param columnName
    +   * @param dataType
    +   * @param aggregateType
    +   * @param parentColumnId
    +   * @param parentTableName
    +   * @param parentDatabaseName
    +   * @param parentTableId
    +   * @return fields object
    +   */
    +  def getField(columnName: String,
    --- End diff --
   
    Why do you require this code? you ultimately require `rawSchema` , that you can anyway get from `dataType.catalogString`, just append the comment to it and use in createtablecommand


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1583#discussion_r154900365
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonCreateTableCommand.scala ---
    @@ -81,12 +90,10 @@ case class CarbonCreateTableCommand(
         val carbonSchemaString = catalog.generateTableSchemaString(tableInfo, tableIdentifier)
         if (createDSTable) {
           try {
    -        val fields = new Array[Field](cm.dimCols.size + cm.msrCols.size)
    -        cm.dimCols.foreach(f => fields(f.schemaOrdinal) = f)
    -        cm.msrCols.foreach(f => fields(f.schemaOrdinal) = f)
    -
    -        sparkSession.sparkContext.setLocalProperty(EXECUTION_ID_KEY, null)
             val tablePath = tableIdentifier.getTablePath
    +        val carbonRelation = CarbonSparkUtil.createCarbonRelation(tableInfo, tablePath)
    +        val fields: Array[Field] = CarbonSparkUtil.getColumnSchemaFields(carbonRelation)
    --- End diff --
   
    you don't require fields here, you just require rawSchema


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1583#discussion_r154900579
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala ---
    @@ -523,4 +523,25 @@ class CarbonFileMetastore extends CarbonMetaStore {
         val tableMetadataFile = tablePath.getSchemaFilePath
         CarbonUtil.readSchemaFile(tableMetadataFile)
       }
    +
    +  /**
    +   * the method returns the Wrapper TableInfo
    +   *
    +   * @param absoluteTableIdentifier
    +   * @param sparkSession
    +   * @return
    +   */
    +  override def getWrapperTableInfo(absoluteTableIdentifier: AbsoluteTableIdentifier)
    --- End diff --
   
    Just name name as `getTableInfo`


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1583
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1717/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1583
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1721/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1583
 
    Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/457/



---

[qiuchenjian-2]

Github user mohammadshahidkhan commented on the issue:

    https://github.com/apache/carbondata/pull/1583
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1583
 
    Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/465/



---