Apache CarbonData Dev Mailing List archive › Apache CarbonData JIRA issues

[GitHub] carbondata pull request #2274: [CARBONDATA-2440] default value of ENABLE_OFF...

classic Classic list List threaded Threaded
♦
♦
 71 messages Options
1234
qiuchenjian-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

[GitHub] carbondata pull request #2274: [CARBONDATA-2440] doc updated to set the prop...

qiuchenjian-2
Github user rahulforallp commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2274#discussion_r189576814
 
    --- Diff: docs/sdk-writer-guide.md ---
    @@ -13,25 +13,33 @@ These SDK writer output contains just a carbondata and carbonindex files. No met
     
      import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException;
      import org.apache.carbondata.core.metadata.datatype.DataTypes;
    + import org.apache.carbondata.core.util.CarbonProperties;
      import org.apache.carbondata.sdk.file.CarbonWriter;
      import org.apache.carbondata.sdk.file.CarbonWriterBuilder;
      import org.apache.carbondata.sdk.file.Field;
      import org.apache.carbondata.sdk.file.Schema;
     
      public class TestSdk {
    -
    +
    + // pass true or false whle executing the main to use offheap memory or not
        public static void main(String[] args) throws IOException, InvalidLoadOptionException {
    -     testSdkWriter();
    +     if(args[0] != null) {
    +     testSdkWriter(args[0]);
    --- End diff --
   
    done


---
qiuchenjian-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

[GitHub] carbondata pull request #2274: [CARBONDATA-2440] doc updated to set the prop...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user rahulforallp commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2274#discussion_r189576859
 
    --- Diff: docs/sdk-writer-guide.md ---
    @@ -13,25 +13,33 @@ These SDK writer output contains just a carbondata and carbonindex files. No met
     
      import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException;
      import org.apache.carbondata.core.metadata.datatype.DataTypes;
    + import org.apache.carbondata.core.util.CarbonProperties;
      import org.apache.carbondata.sdk.file.CarbonWriter;
      import org.apache.carbondata.sdk.file.CarbonWriterBuilder;
      import org.apache.carbondata.sdk.file.Field;
      import org.apache.carbondata.sdk.file.Schema;
     
      public class TestSdk {
    -
    +
    + // pass true or false whle executing the main to use offheap memory or not
        public static void main(String[] args) throws IOException, InvalidLoadOptionException {
    -     testSdkWriter();
    +     if(args[0] != null) {
    +     testSdkWriter(args[0]);
    +     } else {
    +     testSdkWriter("true");
    --- End diff --
   
    done


---
qiuchenjian-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

[GitHub] carbondata pull request #2274: [CARBONDATA-2440] doc updated to set the prop...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user rahulforallp commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2274#discussion_r189576886
 
    --- Diff: docs/sdk-writer-guide.md ---
    @@ -13,25 +13,33 @@ These SDK writer output contains just a carbondata and carbonindex files. No met
     
      import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException;
      import org.apache.carbondata.core.metadata.datatype.DataTypes;
    + import org.apache.carbondata.core.util.CarbonProperties;
      import org.apache.carbondata.sdk.file.CarbonWriter;
      import org.apache.carbondata.sdk.file.CarbonWriterBuilder;
      import org.apache.carbondata.sdk.file.Field;
      import org.apache.carbondata.sdk.file.Schema;
     
      public class TestSdk {
    -
    +
    + // pass true or false whle executing the main to use offheap memory or not
    --- End diff --
   
    done


---
qiuchenjian-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

[GitHub] carbondata issue #2274: [CARBONDATA-2440] doc updated to set the property fo...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2274
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6003/



---
qiuchenjian-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

[GitHub] carbondata pull request #2274: [CARBONDATA-2440] doc updated to set the prop...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2274#discussion_r189578770
 
    --- Diff: docs/sdk-writer-guide.md ---
    @@ -13,25 +13,33 @@ These SDK writer output contains just a carbondata and carbonindex files. No met
     
      import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException;
      import org.apache.carbondata.core.metadata.datatype.DataTypes;
    + import org.apache.carbondata.core.util.CarbonProperties;
      import org.apache.carbondata.sdk.file.CarbonWriter;
      import org.apache.carbondata.sdk.file.CarbonWriterBuilder;
      import org.apache.carbondata.sdk.file.Field;
      import org.apache.carbondata.sdk.file.Schema;
     
      public class TestSdk {
    -
    +
    +   // pass true or false while executing the main to use offheap memory or not
        public static void main(String[] args) throws IOException, InvalidLoadOptionException {
    -     testSdkWriter();
    +     if(args[0] != null) {
    --- End diff --
   
    It will throw "Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
    at org.apache.carbondata.examples.sdk.TestSDK.main(TestSDK.java:17)"


---
qiuchenjian-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

[GitHub] carbondata pull request #2274: [CARBONDATA-2440] doc updated to set the prop...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user rahulforallp commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2274#discussion_r189579501
 
    --- Diff: docs/sdk-writer-guide.md ---
    @@ -13,25 +13,33 @@ These SDK writer output contains just a carbondata and carbonindex files. No met
     
      import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException;
      import org.apache.carbondata.core.metadata.datatype.DataTypes;
    + import org.apache.carbondata.core.util.CarbonProperties;
      import org.apache.carbondata.sdk.file.CarbonWriter;
      import org.apache.carbondata.sdk.file.CarbonWriterBuilder;
      import org.apache.carbondata.sdk.file.Field;
      import org.apache.carbondata.sdk.file.Schema;
     
      public class TestSdk {
    -
    +
    +   // pass true or false while executing the main to use offheap memory or not
        public static void main(String[] args) throws IOException, InvalidLoadOptionException {
    -     testSdkWriter();
    +     if(args[0] != null) {
    --- End diff --
   
            BufferedReader reader =
                       new BufferedReader(new InputStreamReader(System.in));
            String enableOffheapSortVal = reader.readLine();


---
qiuchenjian-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

[GitHub] carbondata pull request #2274: [CARBONDATA-2440] doc updated to set the prop...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user rahulforallp commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2274#discussion_r189580805
 
    --- Diff: docs/sdk-writer-guide.md ---
    @@ -13,25 +13,33 @@ These SDK writer output contains just a carbondata and carbonindex files. No met
     
      import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException;
      import org.apache.carbondata.core.metadata.datatype.DataTypes;
    + import org.apache.carbondata.core.util.CarbonProperties;
      import org.apache.carbondata.sdk.file.CarbonWriter;
      import org.apache.carbondata.sdk.file.CarbonWriterBuilder;
      import org.apache.carbondata.sdk.file.Field;
      import org.apache.carbondata.sdk.file.Schema;
     
      public class TestSdk {
    -
    +
    +   // pass true or false while executing the main to use offheap memory or not
        public static void main(String[] args) throws IOException, InvalidLoadOptionException {
    -     testSdkWriter();
    +     if(args[0] != null) {
    --- End diff --
   
    is this ok?


---
qiuchenjian-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

[GitHub] carbondata pull request #2274: [CARBONDATA-2440] doc updated to set the prop...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2274#discussion_r189582940
 
    --- Diff: docs/sdk-writer-guide.md ---
    @@ -13,25 +13,33 @@ These SDK writer output contains just a carbondata and carbonindex files. No met
     
      import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException;
      import org.apache.carbondata.core.metadata.datatype.DataTypes;
    + import org.apache.carbondata.core.util.CarbonProperties;
      import org.apache.carbondata.sdk.file.CarbonWriter;
      import org.apache.carbondata.sdk.file.CarbonWriterBuilder;
      import org.apache.carbondata.sdk.file.Field;
      import org.apache.carbondata.sdk.file.Schema;
     
      public class TestSdk {
    -
    +
    +   // pass true or false while executing the main to use offheap memory or not
        public static void main(String[] args) throws IOException, InvalidLoadOptionException {
    -     testSdkWriter();
    +     if(args[0] != null) {
    --- End diff --
   
    Have you run this test case in you machine? I run it but failed


---
qiuchenjian-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

[GitHub] carbondata pull request #2274: [CARBONDATA-2440] doc updated to set the prop...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2274#discussion_r189583189
 
    --- Diff: docs/sdk-writer-guide.md ---
    @@ -13,25 +13,33 @@ These SDK writer output contains just a carbondata and carbonindex files. No met
     
      import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException;
      import org.apache.carbondata.core.metadata.datatype.DataTypes;
    + import org.apache.carbondata.core.util.CarbonProperties;
      import org.apache.carbondata.sdk.file.CarbonWriter;
      import org.apache.carbondata.sdk.file.CarbonWriterBuilder;
      import org.apache.carbondata.sdk.file.Field;
      import org.apache.carbondata.sdk.file.Schema;
     
      public class TestSdk {
    -
    +
    +   // pass true or false while executing the main to use offheap memory or not
        public static void main(String[] args) throws IOException, InvalidLoadOptionException {
    -     testSdkWriter();
    +     if(args[0] != null) {
    --- End diff --
   
    Didn't you rebase to the latest master?


---
qiuchenjian-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

[GitHub] carbondata pull request #2274: [CARBONDATA-2440] doc updated to set the prop...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2274#discussion_r189584123
 
    --- Diff: docs/sdk-writer-guide.md ---
    @@ -13,25 +13,33 @@ These SDK writer output contains just a carbondata and carbonindex files. No met
     
      import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException;
      import org.apache.carbondata.core.metadata.datatype.DataTypes;
    + import org.apache.carbondata.core.util.CarbonProperties;
      import org.apache.carbondata.sdk.file.CarbonWriter;
      import org.apache.carbondata.sdk.file.CarbonWriterBuilder;
      import org.apache.carbondata.sdk.file.Field;
      import org.apache.carbondata.sdk.file.Schema;
     
      public class TestSdk {
    -
    +
    +   // pass true or false while executing the main to use offheap memory or not
        public static void main(String[] args) throws IOException, InvalidLoadOptionException {
    -     testSdkWriter();
    +     if(args[0] != null) {
    +       testSdkWriter(args[0]);
    +     } else {
    +       testSdkWriter("true");
    +     }
        }
     
    -   public static void testSdkWriter() throws IOException, InvalidLoadOptionException {
    +   public static void testSdkWriter(String enableOffheap) throws IOException, InvalidLoadOptionException {
          String path = "/home/root1/Documents/ab/temp";
     
          Field[] fields = new Field[2];
          fields[0] = new Field("name", DataTypes.STRING);
          fields[1] = new Field("age", DataTypes.INT);
     
          Schema schema = new Schema(fields);
    +
    +     CarbonProperties.getInstance().addProperty("enable.offheap.sort", enableOffheap);
     
          CarbonWriterBuilder builder = CarbonWriter.builder().withSchema(schema).outputPath(path);
    --- End diff --
   
    Can you optimize the write build process?  some methods have been changed, like withSchema and some.


---
qiuchenjian-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

[GitHub] carbondata pull request #2274: [CARBONDATA-2440] doc updated to set the prop...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2274#discussion_r189586463
 
    --- Diff: docs/sdk-writer-guide.md ---
    @@ -13,25 +13,33 @@ These SDK writer output contains just a carbondata and carbonindex files. No met
     
      import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException;
      import org.apache.carbondata.core.metadata.datatype.DataTypes;
    + import org.apache.carbondata.core.util.CarbonProperties;
      import org.apache.carbondata.sdk.file.CarbonWriter;
      import org.apache.carbondata.sdk.file.CarbonWriterBuilder;
      import org.apache.carbondata.sdk.file.Field;
      import org.apache.carbondata.sdk.file.Schema;
     
      public class TestSdk {
    -
    +
    +   // pass true or false while executing the main to use offheap memory or not
        public static void main(String[] args) throws IOException, InvalidLoadOptionException {
    -     testSdkWriter();
    +     if(args[0] != null) {
    +       testSdkWriter(args[0]);
    +     } else {
    +       testSdkWriter("true");
    +     }
        }
     
    -   public static void testSdkWriter() throws IOException, InvalidLoadOptionException {
    +   public static void testSdkWriter(String enableOffheap) throws IOException, InvalidLoadOptionException {
          String path = "/home/root1/Documents/ab/temp";
    --- End diff --
   
    It will throw exception if the path not exists. Can you optimize it? create directory or change the path.


---
qiuchenjian-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

[GitHub] carbondata pull request #2274: [CARBONDATA-2440] doc updated to set the prop...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user rahulforallp commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2274#discussion_r189588091
 
    --- Diff: docs/sdk-writer-guide.md ---
    @@ -13,25 +13,33 @@ These SDK writer output contains just a carbondata and carbonindex files. No met
     
      import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException;
      import org.apache.carbondata.core.metadata.datatype.DataTypes;
    + import org.apache.carbondata.core.util.CarbonProperties;
      import org.apache.carbondata.sdk.file.CarbonWriter;
      import org.apache.carbondata.sdk.file.CarbonWriterBuilder;
      import org.apache.carbondata.sdk.file.Field;
      import org.apache.carbondata.sdk.file.Schema;
     
      public class TestSdk {
    -
    +
    +   // pass true or false while executing the main to use offheap memory or not
        public static void main(String[] args) throws IOException, InvalidLoadOptionException {
    -     testSdkWriter();
    +     if(args[0] != null) {
    +       testSdkWriter(args[0]);
    +     } else {
    +       testSdkWriter("true");
    +     }
        }
     
    -   public static void testSdkWriter() throws IOException, InvalidLoadOptionException {
    +   public static void testSdkWriter(String enableOffheap) throws IOException, InvalidLoadOptionException {
          String path = "/home/root1/Documents/ab/temp";
     
          Field[] fields = new Field[2];
          fields[0] = new Field("name", DataTypes.STRING);
          fields[1] = new Field("age", DataTypes.INT);
     
          Schema schema = new Schema(fields);
    +
    +     CarbonProperties.getInstance().addProperty("enable.offheap.sort", enableOffheap);
     
          CarbonWriterBuilder builder = CarbonWriter.builder().withSchema(schema).outputPath(path);
    --- End diff --
   
    it  may be failing because of invalid path given in testSdkWriter().
    ![image](https://user-images.githubusercontent.com/14244942/40309877-b1f88696-5d28-11e8-83c5-912de1aae70f.png)
   



---
qiuchenjian-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

[GitHub] carbondata pull request #2274: [CARBONDATA-2440] doc updated to set the prop...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user rahulforallp commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2274#discussion_r189588715
 
    --- Diff: docs/sdk-writer-guide.md ---
    @@ -13,25 +13,33 @@ These SDK writer output contains just a carbondata and carbonindex files. No met
     
      import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException;
      import org.apache.carbondata.core.metadata.datatype.DataTypes;
    + import org.apache.carbondata.core.util.CarbonProperties;
      import org.apache.carbondata.sdk.file.CarbonWriter;
      import org.apache.carbondata.sdk.file.CarbonWriterBuilder;
      import org.apache.carbondata.sdk.file.Field;
      import org.apache.carbondata.sdk.file.Schema;
     
      public class TestSdk {
    -
    +
    +   // pass true or false while executing the main to use offheap memory or not
        public static void main(String[] args) throws IOException, InvalidLoadOptionException {
    -     testSdkWriter();
    +     if(args[0] != null) {
    --- End diff --
   
    it may be failing because of invalid path given in testSdkWriter().
   
    ![image](https://user-images.githubusercontent.com/14244942/40309994-0ebc1d0c-5d29-11e8-8b1f-90898bdbe783.png)



---
qiuchenjian-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

[GitHub] carbondata pull request #2274: [CARBONDATA-2440] doc updated to set the prop...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2274#discussion_r189592810
 
    --- Diff: docs/sdk-writer-guide.md ---
    @@ -13,25 +13,33 @@ These SDK writer output contains just a carbondata and carbonindex files. No met
     
      import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException;
      import org.apache.carbondata.core.metadata.datatype.DataTypes;
    + import org.apache.carbondata.core.util.CarbonProperties;
      import org.apache.carbondata.sdk.file.CarbonWriter;
      import org.apache.carbondata.sdk.file.CarbonWriterBuilder;
      import org.apache.carbondata.sdk.file.Field;
      import org.apache.carbondata.sdk.file.Schema;
     
      public class TestSdk {
    -
    +
    +   // pass true or false while executing the main to use offheap memory or not
        public static void main(String[] args) throws IOException, InvalidLoadOptionException {
    -     testSdkWriter();
    +     if(args[0] != null) {
    --- End diff --
   
    yeah, this path also has problem. I raised a commit just now:https://github.com/xubo245/carbondata/commit/09f387717e5eabb01bf9f688c733e3342ce22799, it can run success, maybe you can refer


---
qiuchenjian-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

[GitHub] carbondata pull request #2274: [CARBONDATA-2440] doc updated to set the prop...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user rahulforallp commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2274#discussion_r189604495
 
    --- Diff: docs/sdk-writer-guide.md ---
    @@ -13,25 +13,33 @@ These SDK writer output contains just a carbondata and carbonindex files. No met
     
      import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException;
      import org.apache.carbondata.core.metadata.datatype.DataTypes;
    + import org.apache.carbondata.core.util.CarbonProperties;
      import org.apache.carbondata.sdk.file.CarbonWriter;
      import org.apache.carbondata.sdk.file.CarbonWriterBuilder;
      import org.apache.carbondata.sdk.file.Field;
      import org.apache.carbondata.sdk.file.Schema;
     
      public class TestSdk {
    -
    +
    +   // pass true or false while executing the main to use offheap memory or not
        public static void main(String[] args) throws IOException, InvalidLoadOptionException {
    -     testSdkWriter();
    +     if(args[0] != null) {
    +       testSdkWriter(args[0]);
    +     } else {
    +       testSdkWriter("true");
    +     }
        }
     
    -   public static void testSdkWriter() throws IOException, InvalidLoadOptionException {
    +   public static void testSdkWriter(String enableOffheap) throws IOException, InvalidLoadOptionException {
          String path = "/home/root1/Documents/ab/temp";
    --- End diff --
   
    done


---
qiuchenjian-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

[GitHub] carbondata issue #2274: [CARBONDATA-2440] doc updated to set the property fo...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2274
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6014/



---
qiuchenjian-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

[GitHub] carbondata issue #2274: [CARBONDATA-2440] doc updated to set the property fo...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2274
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4856/



---
qiuchenjian-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

[GitHub] carbondata issue #2274: [CARBONDATA-2440] doc updated to set the property fo...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2274
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6022/



---
qiuchenjian-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

[GitHub] carbondata issue #2274: [CARBONDATA-2440] doc updated to set the property fo...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2274
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5029/



---
qiuchenjian-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

[GitHub] carbondata issue #2274: [CARBONDATA-2440] doc updated to set the property fo...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2274
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4863/



---
1234
Free forum by Nabble Edit this page