[GitHub] carbondata pull request #2328: Support StreamSQL for streaming job

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2328
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6367/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2328
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5205/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2328
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5316/



---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2328#discussion_r196350110
 
    --- Diff: integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableOperation.scala ---
    @@ -123,12 +123,9 @@ class TestStreamingTableOperation extends QueryTest with BeforeAndAfterAll {
     
         createTable(tableName = "agg_table", streaming = true, withBatchLoad = false)
     
    -    var csvDataDir = new File("target/csvdatanew").getCanonicalPath
    +    val csvDataDir = integrationPath + "target/csvdatanew"
    --- End diff --
   
    it should be `integrationPath + "/spark2/target/csvdatanew`


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2328#discussion_r196350296
 
    --- Diff: integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableOperation.scala ---
    @@ -123,12 +123,9 @@ class TestStreamingTableOperation extends QueryTest with BeforeAndAfterAll {
     
         createTable(tableName = "agg_table", streaming = true, withBatchLoad = false)
     
    -    var csvDataDir = new File("target/csvdatanew").getCanonicalPath
    +    val csvDataDir = integrationPath + "target/csvdatanew"
         generateCSVDataFile(spark, idStart = 10, rowNums = 5, csvDataDir)
         generateCSVDataFile(spark, idStart = 10, rowNums = 5, csvDataDir, SaveMode.Append)
    -    csvDataDir = new File("target/csvdata").getCanonicalPath
    -    // streaming ingest 10 rows
    -    generateCSVDataFile(spark, idStart = 10, rowNums = 10, csvDataDir)
    --- End diff --
   
    why is this removed?


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2328#discussion_r196350452
 
    --- Diff: integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableOperation.scala ---
    @@ -195,7 +192,6 @@ class TestStreamingTableOperation extends QueryTest with BeforeAndAfterAll {
         sql("USE default")
         sql("DROP DATABASE IF EXISTS streaming CASCADE")
         new File("target/csvdatanew").delete()
    --- End diff --
   
    it should be integrationPath + "/spark2/target/csvdatanew


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2328#discussion_r196350482
 
    --- Diff: integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableOperation.scala ---
    @@ -1642,7 +1638,9 @@ class TestStreamingTableOperation extends QueryTest with BeforeAndAfterAll {
         var rows = sql("SHOW STREAMS").collect()
         assertResult(0)(rows.length)
     
    -    val csvDataDir = new File("target/csvdata").getCanonicalPath
    +    val csvDataDir = integrationPath + "/target/csvdata"
    --- End diff --
   
    it should be integrationPath + "/spark2/target/csvdatanew


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2328#discussion_r196444770
 
    --- Diff: integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableOperation.scala ---
    @@ -123,12 +123,9 @@ class TestStreamingTableOperation extends QueryTest with BeforeAndAfterAll {
     
         createTable(tableName = "agg_table", streaming = true, withBatchLoad = false)
     
    -    var csvDataDir = new File("target/csvdatanew").getCanonicalPath
    +    val csvDataDir = integrationPath + "target/csvdatanew"
         generateCSVDataFile(spark, idStart = 10, rowNums = 5, csvDataDir)
         generateCSVDataFile(spark, idStart = 10, rowNums = 5, csvDataDir, SaveMode.Append)
    -    csvDataDir = new File("target/csvdata").getCanonicalPath
    -    // streaming ingest 10 rows
    -    generateCSVDataFile(spark, idStart = 10, rowNums = 10, csvDataDir)
    --- End diff --
   
    I added it in individual test case


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2328#discussion_r196444812
 
    --- Diff: integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableOperation.scala ---
    @@ -195,7 +192,6 @@ class TestStreamingTableOperation extends QueryTest with BeforeAndAfterAll {
         sql("USE default")
         sql("DROP DATABASE IF EXISTS streaming CASCADE")
         new File("target/csvdatanew").delete()
    --- End diff --
   
    fixed


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2328#discussion_r196445223
 
    --- Diff: integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableOperation.scala ---
    @@ -1642,7 +1638,9 @@ class TestStreamingTableOperation extends QueryTest with BeforeAndAfterAll {
         var rows = sql("SHOW STREAMS").collect()
         assertResult(0)(rows.length)
     
    -    val csvDataDir = new File("target/csvdata").getCanonicalPath
    +    val csvDataDir = integrationPath + "/target/csvdata"
    --- End diff --
   
    fixed


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2328#discussion_r196445328
 
    --- Diff: integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableOperation.scala ---
    @@ -123,12 +123,9 @@ class TestStreamingTableOperation extends QueryTest with BeforeAndAfterAll {
     
         createTable(tableName = "agg_table", streaming = true, withBatchLoad = false)
     
    -    var csvDataDir = new File("target/csvdatanew").getCanonicalPath
    +    val csvDataDir = integrationPath + "target/csvdatanew"
    --- End diff --
   
    fixed


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2328
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6383/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2328
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5218/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2328
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6390/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2328
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5224/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2328
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5331/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2328
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5337/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2328
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6404/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2328
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5238/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2328
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5348/



---