[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...

[qiuchenjian-2]

GitHub user ajithme opened a pull request:

    https://github.com/apache/carbondata/pull/2522

    [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppelin support

    Apache Zeppelin is a popular open web-based notebook that enables interactive data analytics. This is one of the favored solutions for providing UI frontend as it can support solutions like Spark already. Carbon can leverage this to provide a UI for its operations. After CARBONDATA-2688 which provides a carbon REST server, we can add a UI support from zeppelin to provide a complete solution.
   
     - [ ] Document update required? YES, need to update usage guide for Zeppelin integration
   
     - [ ] Testing done
    1. Added UT
    2. Done Testing manually by integration with Zeppelin

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ajithme/carbondata zeppelinsupport

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2522.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2522
   
----
commit 16c69836c3903150875582f950f20cd1189fc69a
Author: Ajith <ajith2489@...>
Date:   2018-07-18T11:18:54Z

    CARBONDATA-2752 Zeppelin support

commit 13801259e5c46b1a4cc736fe94c4bf3678d75794
Author: Ajith <ajith2489@...>
Date:   2018-07-18T11:26:29Z

    update doc

----


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2522
 
    Can one of the admins verify this patch?


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2522
 
    Can one of the admins verify this patch?


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2522
 
    Can one of the admins verify this patch?


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2522#discussion_r203347064
 
    --- Diff: store/sql/src/main/java/org/apache/carbondata/horizon/rest/controller/SqlHorizonController.java ---
    @@ -42,20 +43,24 @@
       public ResponseEntity<SqlResponse> sql(@RequestBody SqlRequest request) throws StoreException {
         RequestValidator.validateSql(request);
         List<Row> rows;
    +    Dataset<Row> sqlDataFrame = null;
         try {
    -      rows = SparkSqlWrapper.sql(SqlHorizon.getSession(), request.getSqlStatement())
    +      sqlDataFrame = SparkSqlWrapper.sql(SqlHorizon.getSession(),
    +              request.getSqlStatement());
    +      rows = sqlDataFrame
               .collectAsList();
    --- End diff --
   
    move it to previous line


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2522#discussion_r203347146
 
    --- Diff: store/sql/src/main/java/org/apache/carbondata/horizon/rest/controller/SqlHorizonController.java ---
    @@ -42,20 +43,24 @@
       public ResponseEntity<SqlResponse> sql(@RequestBody SqlRequest request) throws StoreException {
         RequestValidator.validateSql(request);
         List<Row> rows;
    +    Dataset<Row> sqlDataFrame = null;
         try {
    -      rows = SparkSqlWrapper.sql(SqlHorizon.getSession(), request.getSqlStatement())
    +      sqlDataFrame = SparkSqlWrapper.sql(SqlHorizon.getSession(),
    +              request.getSqlStatement());
    +      rows = sqlDataFrame
               .collectAsList();
         } catch (AnalysisException e) {
           throw new StoreException(e.getSimpleMessage());
         } catch (Exception e) {
           throw new StoreException(e.getMessage());
         }
    -    Object[][] result = new Object[rows.size()][];
    +    Object[][] result = new Object[rows.size()+1][];
    --- End diff --
   
    add space before and after `+`


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2522#discussion_r203347726
 
    --- Diff: store/sql/src/main/java/org/apache/carbondata/horizon/rest/controller/SqlHorizonController.java ---
    @@ -42,20 +43,24 @@
       public ResponseEntity<SqlResponse> sql(@RequestBody SqlRequest request) throws StoreException {
         RequestValidator.validateSql(request);
         List<Row> rows;
    +    Dataset<Row> sqlDataFrame = null;
         try {
    -      rows = SparkSqlWrapper.sql(SqlHorizon.getSession(), request.getSqlStatement())
    +      sqlDataFrame = SparkSqlWrapper.sql(SqlHorizon.getSession(),
    +              request.getSqlStatement());
    +      rows = sqlDataFrame
               .collectAsList();
         } catch (AnalysisException e) {
           throw new StoreException(e.getSimpleMessage());
         } catch (Exception e) {
           throw new StoreException(e.getMessage());
         }
    -    Object[][] result = new Object[rows.size()][];
    +    Object[][] result = new Object[rows.size()+1][];
    +    result[0] = sqlDataFrame.schema().fieldNames();
         for (int i = 0; i < rows.size(); i++) {
           Row row = rows.get(i);
    -      result[i] = new Object[row.size()];
    +      result[i+1] = new Object[row.size()];
           for (int j = 0; j < row.size(); j++) {
    -        result[i][j] = row.get(j);
    +        result[i+1][j] = row.get(j);
    --- End diff --
   
    Can't use System.arraycopy?


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2522#discussion_r203347890
 
    --- Diff: store/sql/src/main/java/org/apache/carbondata/horizon/rest/controller/SqlHorizonController.java ---
    @@ -42,20 +43,24 @@
       public ResponseEntity<SqlResponse> sql(@RequestBody SqlRequest request) throws StoreException {
         RequestValidator.validateSql(request);
         List<Row> rows;
    +    Dataset<Row> sqlDataFrame = null;
         try {
    -      rows = SparkSqlWrapper.sql(SqlHorizon.getSession(), request.getSqlStatement())
    +      sqlDataFrame = SparkSqlWrapper.sql(SqlHorizon.getSession(),
    +              request.getSqlStatement());
    +      rows = sqlDataFrame
               .collectAsList();
         } catch (AnalysisException e) {
           throw new StoreException(e.getSimpleMessage());
         } catch (Exception e) {
           throw new StoreException(e.getMessage());
         }
    -    Object[][] result = new Object[rows.size()][];
    +    Object[][] result = new Object[rows.size()+1][];
    +    result[0] = sqlDataFrame.schema().fieldNames();
         for (int i = 0; i < rows.size(); i++) {
    --- End diff --
   
    `I ` can start from 1


---

[qiuchenjian-2]

Github user chenliang613 commented on the issue:

    https://github.com/apache/carbondata/pull/2522
 
    add to whitelist


---

[qiuchenjian-2]

Github user chenliang613 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2522#discussion_r203592254
 
    --- Diff: zeppelin/README.txt ---
    @@ -0,0 +1,18 @@
    +Please follow below steps to integrate with zeppelin
    --- End diff --
   
    1. please use .md format
    2. Please add apache license header also.


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2522
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7295/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2522
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7297/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2522
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6064/



---

[qiuchenjian-2]

Github user ajithme commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2522#discussion_r203618808
 
    --- Diff: zeppelin/README.txt ---
    @@ -0,0 +1,18 @@
    +Please follow below steps to integrate with zeppelin
    --- End diff --
   
    Done:
    https://github.com/ajithme/carbondata/tree/zeppelinsupport/integration/zeppelin


---

[qiuchenjian-2]

Github user ajithme commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2522#discussion_r203618873
 
    --- Diff: store/sql/src/main/java/org/apache/carbondata/horizon/rest/controller/SqlHorizonController.java ---
    @@ -42,20 +43,24 @@
       public ResponseEntity<SqlResponse> sql(@RequestBody SqlRequest request) throws StoreException {
         RequestValidator.validateSql(request);
         List<Row> rows;
    +    Dataset<Row> sqlDataFrame = null;
         try {
    -      rows = SparkSqlWrapper.sql(SqlHorizon.getSession(), request.getSqlStatement())
    +      sqlDataFrame = SparkSqlWrapper.sql(SqlHorizon.getSession(),
    +              request.getSqlStatement());
    +      rows = sqlDataFrame
               .collectAsList();
         } catch (AnalysisException e) {
           throw new StoreException(e.getSimpleMessage());
         } catch (Exception e) {
           throw new StoreException(e.getMessage());
         }
    -    Object[][] result = new Object[rows.size()][];
    +    Object[][] result = new Object[rows.size()+1][];
    +    result[0] = sqlDataFrame.schema().fieldNames();
         for (int i = 0; i < rows.size(); i++) {
    --- End diff --
   
    replaced with stream construct


---

[qiuchenjian-2]

Github user ajithme commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2522#discussion_r203618896
 
    --- Diff: store/sql/src/main/java/org/apache/carbondata/horizon/rest/controller/SqlHorizonController.java ---
    @@ -42,20 +43,24 @@
       public ResponseEntity<SqlResponse> sql(@RequestBody SqlRequest request) throws StoreException {
         RequestValidator.validateSql(request);
         List<Row> rows;
    +    Dataset<Row> sqlDataFrame = null;
         try {
    -      rows = SparkSqlWrapper.sql(SqlHorizon.getSession(), request.getSqlStatement())
    +      sqlDataFrame = SparkSqlWrapper.sql(SqlHorizon.getSession(),
    +              request.getSqlStatement());
    +      rows = sqlDataFrame
               .collectAsList();
         } catch (AnalysisException e) {
           throw new StoreException(e.getSimpleMessage());
         } catch (Exception e) {
           throw new StoreException(e.getMessage());
         }
    -    Object[][] result = new Object[rows.size()][];
    +    Object[][] result = new Object[rows.size()+1][];
    +    result[0] = sqlDataFrame.schema().fieldNames();
         for (int i = 0; i < rows.size(); i++) {
           Row row = rows.get(i);
    -      result[i] = new Object[row.size()];
    +      result[i+1] = new Object[row.size()];
           for (int j = 0; j < row.size(); j++) {
    -        result[i][j] = row.get(j);
    +        result[i+1][j] = row.get(j);
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user ajithme commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2522#discussion_r203618925
 
    --- Diff: store/sql/src/main/java/org/apache/carbondata/horizon/rest/controller/SqlHorizonController.java ---
    @@ -42,20 +43,24 @@
       public ResponseEntity<SqlResponse> sql(@RequestBody SqlRequest request) throws StoreException {
         RequestValidator.validateSql(request);
         List<Row> rows;
    +    Dataset<Row> sqlDataFrame = null;
         try {
    -      rows = SparkSqlWrapper.sql(SqlHorizon.getSession(), request.getSqlStatement())
    +      sqlDataFrame = SparkSqlWrapper.sql(SqlHorizon.getSession(),
    +              request.getSqlStatement());
    +      rows = sqlDataFrame
               .collectAsList();
         } catch (AnalysisException e) {
           throw new StoreException(e.getSimpleMessage());
         } catch (Exception e) {
           throw new StoreException(e.getMessage());
         }
    -    Object[][] result = new Object[rows.size()][];
    +    Object[][] result = new Object[rows.size()+1][];
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user ajithme commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2522#discussion_r203618953
 
    --- Diff: store/sql/src/main/java/org/apache/carbondata/horizon/rest/controller/SqlHorizonController.java ---
    @@ -42,20 +43,24 @@
       public ResponseEntity<SqlResponse> sql(@RequestBody SqlRequest request) throws StoreException {
         RequestValidator.validateSql(request);
         List<Row> rows;
    +    Dataset<Row> sqlDataFrame = null;
         try {
    -      rows = SparkSqlWrapper.sql(SqlHorizon.getSession(), request.getSqlStatement())
    +      sqlDataFrame = SparkSqlWrapper.sql(SqlHorizon.getSession(),
    +              request.getSqlStatement());
    +      rows = sqlDataFrame
               .collectAsList();
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2522
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7300/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2522
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6067/



---