[GitHub] carbondata pull request #2810: [WIP] Add CarbonSession Java Example

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2810: [WIP] Add CarbonSession Java Example

qiuchenjian-2
GitHub user lijiang2 opened a pull request:

    https://github.com/apache/carbondata/pull/2810

    [WIP] Add CarbonSession Java Example

    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
     - [ ] Any interfaces changed? NA
     
     - [ ] Any backward compatibility impacted? NA
     
     - [ ] Document update required? NA
   
     - [ ] Testing done
            Please provide details on
            - Whether new unit test cases have been added or why no new tests are required?
            - How it is tested? Please attach test report.
            - Is it a performance related change? Please attach the performance test report.
            - Any additional information to help reviewers in testing this change.
           
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/lijiang2/carbondata local

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2810.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2810
   
----
commit 50446bc711a2941c092cd5d30bf7e88a12b7cfa1
Author: Li Jiang <tudouxiaohei2@...>
Date:   2018-10-11T05:36:08Z

    [WIP] Add CarbonSession Java Example

commit c65e3514105bef870b3d7eeae17cf00f2238bd83
Author: Li Jiang <tudouxiaohei2@...>
Date:   2018-10-11T05:48:07Z

    [WIP] Add CarbonSession Java Example

----


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2810: [WIP] Add CarbonSession Java Example

qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2810
 
    Can one of the admins verify this patch?


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2810: [WIP] Add CarbonSession Java Example

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user lijiang2 commented on the issue:

    https://github.com/apache/carbondata/pull/2810
 
    @jackylk PTAL.


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2810: [WIP] Add CarbonSession Java Example

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user chenliang613 commented on the issue:

    https://github.com/apache/carbondata/pull/2810
 
    add to whitelist


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2810: [WIP] Add CarbonSession Java Example

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2810
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/780/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2810: [WIP] Add CarbonSession Java Example

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2810
 
    Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9045/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2810: [WIP] Add CarbonSession Java Example

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2810
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/977/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2810: [WIP] Add CarbonSession Java Example

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user xuchuanyin commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2810#discussion_r224658474
 
    --- Diff: examples/spark2/src/main/java/org/apache/carbondata/examples/sql/CarbonSessionExample.java ---
    @@ -0,0 +1,137 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.examples.sql;
    +
    +import org.apache.carbondata.core.constants.CarbonCommonConstants;
    +import org.apache.carbondata.core.util.CarbonProperties;
    +import org.apache.carbondata.examples.util.ExampleUtils;
    +import org.apache.log4j.PropertyConfigurator;
    +import org.apache.spark.sql.SparkSession;
    +
    +import java.io.File;
    +import java.io.IOException;
    +
    +public class CarbonSessionExample {
    +
    +  public static void main(String[] args) {
    +    File file = new File(CarbonSessionExample.class.getResource("/").getPath() + "../..");
    +    try {
    +      String rootPath = file.getCanonicalPath();
    +      System.setProperty("path.target", rootPath + "/examples/spark2/target");
    +      PropertyConfigurator.configure(rootPath + "/examples/spark2/src/main/resources/log4j.properties");
    +      CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_QUERY_STATISTICS, "true");
    +      SparkSession spark = ExampleUtils.createCarbonSession("JavaCarbonSessionExample", 1);
    --- End diff --
   
    I think you can change this variable name to 'carbon' to tell the user that it is not a raw SparkSession.
   
    Besides, I think it's better to add an example for DataFrame  (write dataframe to carbon table).


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2810: [WIP] Add CarbonSession Java Example

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2810#discussion_r226227374
 
    --- Diff: examples/spark2/src/main/java/org/apache/carbondata/examples/sql/CarbonSessionExample.java ---
    @@ -0,0 +1,137 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.examples.sql;
    +
    +import org.apache.carbondata.core.constants.CarbonCommonConstants;
    +import org.apache.carbondata.core.util.CarbonProperties;
    +import org.apache.carbondata.examples.util.ExampleUtils;
    +import org.apache.log4j.PropertyConfigurator;
    +import org.apache.spark.sql.SparkSession;
    +
    +import java.io.File;
    +import java.io.IOException;
    +
    +public class CarbonSessionExample {
    +
    +  public static void main(String[] args) {
    +    File file = new File(CarbonSessionExample.class.getResource("/").getPath() + "../..");
    +    try {
    +      String rootPath = file.getCanonicalPath();
    +      System.setProperty("path.target", rootPath + "/examples/spark2/target");
    +      PropertyConfigurator.configure(rootPath + "/examples/spark2/src/main/resources/log4j.properties");
    +      CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_QUERY_STATISTICS, "true");
    +      SparkSession spark = ExampleUtils.createCarbonSession("JavaCarbonSessionExample", 1);
    +      spark.sparkContext().setLogLevel("INFO");
    +
    +      spark.sql("DROP TABLE IF EXISTS carbonsession_table");
    +      spark.sql("DROP TABLE IF EXISTS stored_as_carbondata_table");
    +
    +      spark.sql(
    +              "CREATE TABLE carbonsession_table( " +
    +                      "shortField SHORT, " +
    +                      "intField INT, " +
    +                      "bigintField LONG, " +
    +                      "doubleField DOUBLE, " +
    +                      "stringField STRING, " +
    +                      "timestampField TIMESTAMP, " +
    +                      "decimalField DECIMAL(18,2), " +
    +                      "dateField DATE, " +
    +                      "charField CHAR(5), " +
    +                      "floatField FLOAT " +
    +                      ") " +
    +                      "STORED BY 'carbondata' " +
    --- End diff --
   
    use `STORED AS carbondata` instead


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2810: [WIP] Add CarbonSession Java Example

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/2810
 
    Hi, are you still working on this PR?


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2810: [WIP] Add CarbonSession Java Example

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user lijiang2 commented on the issue:

    https://github.com/apache/carbondata/pull/2810
 
    Thank you for the suggestion. Yes, I will work on it soon.


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2810: [WIP] Add CarbonSession Java Example

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/2810
 
    Thanks, and you can create a JIRA ticket by https://issues.apache.org/jira/projects/CARBONDATA/issues, then you can modify the PR title to include the JIRA number


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2810: [CARBONDATA-3071] Add CarbonSession Java Exam...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user lijiang2 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2810#discussion_r230574283
 
    --- Diff: examples/spark2/src/main/java/org/apache/carbondata/examples/sql/CarbonSessionExample.java ---
    @@ -0,0 +1,137 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.examples.sql;
    +
    +import org.apache.carbondata.core.constants.CarbonCommonConstants;
    +import org.apache.carbondata.core.util.CarbonProperties;
    +import org.apache.carbondata.examples.util.ExampleUtils;
    +import org.apache.log4j.PropertyConfigurator;
    +import org.apache.spark.sql.SparkSession;
    +
    +import java.io.File;
    +import java.io.IOException;
    +
    +public class CarbonSessionExample {
    +
    +  public static void main(String[] args) {
    +    File file = new File(CarbonSessionExample.class.getResource("/").getPath() + "../..");
    +    try {
    +      String rootPath = file.getCanonicalPath();
    +      System.setProperty("path.target", rootPath + "/examples/spark2/target");
    +      PropertyConfigurator.configure(rootPath + "/examples/spark2/src/main/resources/log4j.properties");
    +      CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_QUERY_STATISTICS, "true");
    +      SparkSession spark = ExampleUtils.createCarbonSession("JavaCarbonSessionExample", 1);
    --- End diff --
   
    Thank you for the suggestion. I updated the variable name. And I think I will start the DataFrame example in another pr.


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2810: [CARBONDATA-3071] Add CarbonSession Java Example

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2810
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1252/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2810: [CARBONDATA-3071] Add CarbonSession Java Example

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2810
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1467/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2810: [CARBONDATA-3071] Add CarbonSession Java Example

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2810
 
    Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9516/



---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata issue #2810: [CARBONDATA-3071] Add CarbonSession Java Example

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/2810
 
    LGTM


---
Reply | Threaded
Open this post in threaded view
|

[GitHub] carbondata pull request #2810: [CARBONDATA-3071] Add CarbonSession Java Exam...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user asfgit closed the pull request at:

    https://github.com/apache/carbondata/pull/2810


---