[GitHub] carbondata pull request #2836: [CARBONDATA-3027] Increase unsafe working mem...

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2836#discussion_r229188244
 
    --- Diff: store/sdk/src/main/resources/log4j.properties ---
    @@ -0,0 +1,11 @@
    +# Root logger option
    +log4j.rootLogger=INFO,stdout
    +
    +
    +# Redirect log messages to console
    +log4j.appender.debug=org.apache.log4j.RollingFileAppender
    +log4j.appender.stdout=org.apache.log4j.ConsoleAppender
    +log4j.appender.stdout.Target=System.out
    +log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
    +log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
    --- End diff --
   
    there is already add  %c{1} to print class name:
    ```
    2018-10-30 14:26:23 INFO  CacheProvider:168 - Executor LRU cache size not configured. Initializing with driver LRU cache size.
    2018-10-30 14:26:23 INFO  CarbonLRUCache:76 - LRU cache size not configured. Therefore default behavior will be considered and no LRU based eviction of columns will be done
    2018-10-30 14:26:23 INFO  SegmentPropertiesAndSchemaHolder:117 - Constructing new SegmentProperties for table: null_null. Current size of segment properties holder list is: 1
    2018-10-30 14:26:23 INFO  QueryModelBuilder:94 - Projection Columns: [stringfield, datefield, timefield, varcharfield, arrayfield, shortfield, intfield, longfield, doublefield, boolfield, decimalfield, floatfield]
    2018-10-30 14:26:23 INFO  AbstractQueryExecutor:121 - Query will be executed on table: null
    2018-10-30 14:26:23 INFO  ResultCollectorFactory:79 - Row based dictionary collector is used to scan and collect the data
    2018-10-30 14:26:24 WARN  DataMapStoreManager:499 - failed to get carbon table from table PathFile does not exist: /Users/xubo/Desktop/xubo/git/carbondata1/store/sdk/carbondata/Metadata/schema
    2018-10-30 14:26:24 INFO  CarbonLRUCache:163 - Removed entry from InMemory lru cache :: /Users/xubo/Desktop/xubo/git/carbondata1/store/sdk/carbondata/632392937014916_batchno0-0-null-632392085265031.carbonindex
    2018-10-30 14:26:24 INFO  UnsafeMemoryManager:175 - Total offheap working memory used after task 14e4fd01-9df5-416a-9b0a-6970ea63c2bd is 0. Current running tasks are d26d0593-f4ce-4bb8-ac94-974638229901, a92307f4-9292-4645-87fb-2a772b421df4
   
    ```


---

[qiuchenjian-2]

Github user xubo245 commented on the issue:

    https://github.com/apache/carbondata/pull/2836
 
    @xuchuanyin updated doc


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2836
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1142/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2836
 
    Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1354/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2836
 
    Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9407/



---

[qiuchenjian-2]

Github user xubo245 commented on the issue:

    https://github.com/apache/carbondata/pull/2836
 
    @jackylk @xuchuanyin CI pass, please check it again.


---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2836#discussion_r229284826
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
    @@ -1234,7 +1234,7 @@
     
       @CarbonProperty
       public static final String UNSAFE_WORKING_MEMORY_IN_MB = "carbon.unsafe.working.memory.in.mb";
    -  public static final String UNSAFE_WORKING_MEMORY_IN_MB_DEFAULT = "512";
    +  public static final String UNSAFE_WORKING_MEMORY_IN_MB_DEFAULT = "1024";
    --- End diff --
   
    You can change the configuration in your application, there is no need to change the default value of this parameter


---

[qiuchenjian-2]

Github user KanakaKumar commented on the issue:

    https://github.com/apache/carbondata/pull/2836
 
    I think this PR may not be require any more after #2841.  @xubo245 , our intention is to make unsafe configuration flexible and even with default values we should  be able to continue with heap memory.


---

[qiuchenjian-2]

Github user xubo245 commented on the issue:

    https://github.com/apache/carbondata/pull/2836
 
    OK，I tested， it's work。 I  will close this PR.


---

[qiuchenjian-2]

Github user xubo245 closed the pull request at:

    https://github.com/apache/carbondata/pull/2836


---