[GitHub] carbondata pull request #2861: [HOTFIX]handle passing spark appname for part...

[qiuchenjian-2]

Github user akashrn5 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2861#discussion_r229235332
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/datasources/SparkCarbonTableFormat.scala ---
    @@ -175,6 +174,12 @@ with Serializable {
               dataSchema: StructType,
               context: TaskAttemptContext): OutputWriter = {
             val model = CarbonTableOutputFormat.getLoadModel(context.getConfiguration)
    +        val appName = context.getConfiguration.get(CarbonCommonConstants.CARBON_WRITTEN_BY_APPNAME)
    +        if (null != appName) {
    +          CarbonProperties.getInstance()
    +            .addProperty(CarbonCommonConstants.CARBON_WRITTEN_BY_APPNAME,
    +              appName)
    --- End diff --
   
    done


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2861
 
    Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9419/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2861
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1156/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2861
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1367/



---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2861#discussion_r229285477
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/CarbonRDD.scala ---
    @@ -37,6 +37,11 @@ abstract class CarbonRDD[T: ClassTag](
         @transient private val ss: SparkSession,
         @transient private var deps: Seq[Dependency[_]]) extends RDD[T](ss.sparkContext, deps) {
     
    +  @transient val sparkAppName: String = ss.sparkContext.appName
    +  CarbonProperties.getInstance()
    +    .addProperty(CarbonCommonConstants.CARBON_WRITTEN_BY_APPNAME,
    +      sparkAppName)
    --- End diff --
   
    move to previous line


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2861#discussion_r229285529
 
    --- Diff: integration/spark-datasource/src/main/scala/org/apache/spark/sql/carbondata/execution/datasources/SparkCarbonFileFormat.scala ---
    @@ -121,15 +121,13 @@ class SparkCarbonFileFormat extends FileFormat
           dataSchema: StructType): OutputWriterFactory = {
     
         val conf = job.getConfiguration
    -
    +    conf
    +      .set(CarbonCommonConstants.CARBON_WRITTEN_BY_APPNAME,
    --- End diff --
   
    move to previous line


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2861#discussion_r229285778
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/datasources/SparkCarbonTableFormat.scala ---
    @@ -175,6 +172,11 @@ with Serializable {
               dataSchema: StructType,
               context: TaskAttemptContext): OutputWriter = {
             val model = CarbonTableOutputFormat.getLoadModel(context.getConfiguration)
    +        val appName = context.getConfiguration.get(CarbonCommonConstants.CARBON_WRITTEN_BY_APPNAME)
    +        if (null != appName) {
    --- End diff --
   
    Is there is no appName, I think we should construct one, the appName should be always write into the file


---

[qiuchenjian-2]

Github user akashrn5 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2861#discussion_r229294323
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/datasources/SparkCarbonTableFormat.scala ---
    @@ -175,6 +172,11 @@ with Serializable {
               dataSchema: StructType,
               context: TaskAttemptContext): OutputWriter = {
             val model = CarbonTableOutputFormat.getLoadModel(context.getConfiguration)
    +        val appName = context.getConfiguration.get(CarbonCommonConstants.CARBON_WRITTEN_BY_APPNAME)
    +        if (null != appName) {
    --- End diff --
   
    actually the appname will be always set ,spark will always set the appname, this check is added for one of the test cases , i will remove this


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2861
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1376/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2861
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1164/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2861
 
    Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9428/



---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/2861
 
    LGTM


---

[qiuchenjian-2]

Github user asfgit closed the pull request at:

    https://github.com/apache/carbondata/pull/2861


---