[GitHub] carbondata pull request #2646: [WIP] Support SDK writer as thread safe api

[qiuchenjian-2]

GitHub user ajantha-bhat opened a pull request:

    https://github.com/apache/carbondata/pull/2646

    [WIP] Support SDK writer as thread safe api

    Be sure to do all of the following checklist to help us incorporate
    your contribution quickly and easily:
   
     - [ ] Any interfaces changed?
     
     - [ ] Any backward compatibility impacted?
     
     - [ ] Document update required?
   
     - [ ] Testing done
            Please provide details on
            - Whether new unit test cases have been added or why no new tests are required?
            - How it is tested? Please attach test report.
            - Is it a performance related change? Please attach the performance test report.
            - Any additional information to help reviewers in testing this change.
           
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ajantha-bhat/carbondata issue_fix

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2646.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2646
   
----
commit d239b6e379e6dd2f4627df33b93aa2217fb94558
Author: ajantha-bhat <ajanthabhat@...>
Date:   2018-08-21T05:22:35Z

    make sdk writer thread safe

commit f54bd7dc349b71a93c7343ab8aac12aacd90e737
Author: ajantha-bhat <ajanthabhat@...>
Date:   2018-08-21T10:05:31Z

    update sdk guide

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2646
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7962/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2646
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6686/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2646
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6313/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2646
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6314/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2646
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7963/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2646
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6687/



---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2646#discussion_r211841415
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/iterator/CarbonOutputIteratorWrapper.java ---
    @@ -45,17 +45,21 @@
     
       private RowBatch readBatch;
     
    +  private static Object lockObject = new Object();
    +
       private ArrayBlockingQueue<RowBatch> queue = new ArrayBlockingQueue<>(10);
     
       public void write(Object[] row) throws InterruptedException {
         if (close) {
           // already might be closed forcefully
           return;
         }
    -    if (!loadBatch.addRow(row)) {
    -      loadBatch.readyRead();
    -      queue.put(loadBatch);
    -      loadBatch = new RowBatch(batchSize);
    +    synchronized (lockObject) {
    --- End diff --
   
    why not use `this`


---

[qiuchenjian-2]

Github user ajantha-bhat commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2646#discussion_r211900027
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/iterator/CarbonOutputIteratorWrapper.java ---
    @@ -45,17 +45,21 @@
     
       private RowBatch readBatch;
     
    +  private static Object lockObject = new Object();
    +
       private ArrayBlockingQueue<RowBatch> queue = new ArrayBlockingQueue<>(10);
     
       public void write(Object[] row) throws InterruptedException {
         if (close) {
           // already might be closed forcefully
           return;
         }
    -    if (!loadBatch.addRow(row)) {
    -      loadBatch.readyRead();
    -      queue.put(loadBatch);
    -      loadBatch = new RowBatch(batchSize);
    +    synchronized (lockObject) {
    --- End diff --
   
    yes, can use this as this is object level synchronization


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2646
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7970/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2646
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6694/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2646
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6320/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2646
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6321/



---

[qiuchenjian-2]

Github user ajantha-bhat commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2646#discussion_r211914760
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/iterator/CarbonOutputIteratorWrapper.java ---
    @@ -45,17 +45,21 @@
     
       private RowBatch readBatch;
     
    +  private static Object lockObject = new Object();
    +
       private ArrayBlockingQueue<RowBatch> queue = new ArrayBlockingQueue<>(10);
     
       public void write(Object[] row) throws InterruptedException {
         if (close) {
           // already might be closed forcefully
           return;
         }
    -    if (!loadBatch.addRow(row)) {
    -      loadBatch.readyRead();
    -      queue.put(loadBatch);
    -      loadBatch = new RowBatch(batchSize);
    +    synchronized (lockObject) {
    --- End diff --
   
    if used 'this', findbugs thinks I need to synchronize hasNext() and close() method also. It says inconsistent synchronization.  Hence to avoid that this variable.



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2646
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6322/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2646
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7972/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2646
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6696/



---

[qiuchenjian-2]

Github user ajantha-bhat commented on the issue:

    https://github.com/apache/carbondata/pull/2646
 
    retest this please


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2646
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7974/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2646
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6698/



---