[GitHub] carbondata pull request #2981: [CARBONDATA-3154] Fix spark-2.1 test error

[qiuchenjian-2]

GitHub user xubo245 opened a pull request:

    https://github.com/apache/carbondata/pull/2981

    [CARBONDATA-3154] Fix spark-2.1 test error

    [CARBONDATA-3154] Fix spark-2.1 test error
    This PR fix  spark-2.1 test error, including:
    1. fix 6 errors of org.apache.spark.sql.carbondata.datasource.SparkCarbonDataSourceTest
   
    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
     - [ ] Any interfaces changed?
     No
     - [ ] Any backward compatibility impacted?
     No
     - [ ] Document update required?
    No
     - [ ] Testing done
           fix test code
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
    No


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/xubo245/carbondata CARBONDATA-3154_FixSpark2_1_0TestError

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2981.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2981
   
----
commit 82ce986f500e8412c7cc515814f0fffb84b26890
Author: xubo245 <xubo29@...>
Date:   2018-12-07T16:01:43Z

    [CARBONDATA-3154] Fix spark-2.1 test error

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2981
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1659/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2981
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1660/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2981
 
    Build Failed  with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9920/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2981
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1872/



---

[qiuchenjian-2]

Github user xubo245 commented on the issue:

    https://github.com/apache/carbondata/pull/2981
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2981
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1661/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2981
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1873/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2981
 
    Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9921/



---

[qiuchenjian-2]

Github user chenliang613 commented on the issue:

    https://github.com/apache/carbondata/pull/2981
 
    @kunal642   please review it


---

[qiuchenjian-2]

Github user kunal642 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2981#discussion_r240026585
 
    --- Diff: integration/spark-datasource/src/test/scala/org/apache/spark/sql/carbondata/datasource/SparkCarbonDataSourceTest.scala ---
    @@ -998,9 +999,19 @@ class SparkCarbonDataSourceTest extends FunSuite with BeforeAndAfterAll {
             i += 1
           }
           writer.close()
    -      spark.sql("create table complextable (stringfield string, structfield struct<bytefield: " +
    -                "byte, floatfield: float>) " +
    -                s"using carbon location '$path'")
    +      if (SparkUtil.isSparkVersionEqualTo("2.1")) {
    +        if (!FileFactory.isFileExist(path)) {
    +          FileFactory.createDirectoryAndSetPermission(path,
    +            new FsPermission(FsAction.ALL, FsAction.ALL, FsAction.ALL))
    +        }
    +        spark.sql("create table complextable (stringfield string, structfield struct<bytefield: " +
    +          "byte, floatfield: float>) " +
    +          s"using carbon options(path '$path')")
    +      } else if (SparkUtil.isSparkVersionXandAbove("2.2")) {
    --- End diff --
   
    change the else if to else so that the same command can run for 2.3 as well. Change the same for other tests as well


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2981#discussion_r240026676
 
    --- Diff: integration/spark-datasource/src/test/scala/org/apache/spark/sql/carbondata/datasource/SparkCarbonDataSourceTest.scala ---
    @@ -998,9 +999,19 @@ class SparkCarbonDataSourceTest extends FunSuite with BeforeAndAfterAll {
             i += 1
           }
           writer.close()
    -      spark.sql("create table complextable (stringfield string, structfield struct<bytefield: " +
    -                "byte, floatfield: float>) " +
    -                s"using carbon location '$path'")
    +      if (SparkUtil.isSparkVersionEqualTo("2.1")) {
    +        if (!FileFactory.isFileExist(path)) {
    +          FileFactory.createDirectoryAndSetPermission(path,
    +            new FsPermission(FsAction.ALL, FsAction.ALL, FsAction.ALL))
    +        }
    +        spark.sql("create table complextable (stringfield string, structfield struct<bytefield: " +
    +          "byte, floatfield: float>) " +
    +          s"using carbon options(path '$path')")
    +      } else if (SparkUtil.isSparkVersionXandAbove("2.2")) {
    --- End diff --
   
    it support 2.3, because it uses SparkUtil.isSparkVersionXandAbove.  


---

[qiuchenjian-2]

Github user kunal642 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2981#discussion_r240026697
 
    --- Diff: integration/spark-datasource/src/test/scala/org/apache/spark/sql/carbondata/datasource/SparkCarbonDataSourceTest.scala ---
    @@ -998,9 +999,19 @@ class SparkCarbonDataSourceTest extends FunSuite with BeforeAndAfterAll {
             i += 1
           }
           writer.close()
    -      spark.sql("create table complextable (stringfield string, structfield struct<bytefield: " +
    -                "byte, floatfield: float>) " +
    -                s"using carbon location '$path'")
    +      if (SparkUtil.isSparkVersionEqualTo("2.1")) {
    +        if (!FileFactory.isFileExist(path)) {
    +          FileFactory.createDirectoryAndSetPermission(path,
    +            new FsPermission(FsAction.ALL, FsAction.ALL, FsAction.ALL))
    +        }
    +        spark.sql("create table complextable (stringfield string, structfield struct<bytefield: " +
    +          "byte, floatfield: float>) " +
    +          s"using carbon options(path '$path')")
    +      } else if (SparkUtil.isSparkVersionXandAbove("2.2")) {
    --- End diff --
   
    yeah, right


---

[qiuchenjian-2]

Github user kunal642 commented on the issue:

    https://github.com/apache/carbondata/pull/2981
 
    LGTM


---

[qiuchenjian-2]

Github user asfgit closed the pull request at:

    https://github.com/apache/carbondata/pull/2981


---