Apache CarbonData Dev Mailing List archive › Apache CarbonData JIRA issues

[jira] [Updated] (CARBONDATA-2875) Two different threads are overwriting the same carbondatafile

classic Classic list List threaded Threaded
♦
♦
 1 message Options
Akash R Nilugal (Jira)
Reply | Threaded
Open this post in threaded view
♦
♦
|

[jira] [Updated] (CARBONDATA-2875) Two different threads are overwriting the same carbondatafile

Akash R Nilugal (Jira)

     [ https://issues.apache.org/jira/browse/CARBONDATA-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Supritha updated CARBONDATA-2875:
---------------------------------
    Description:
Steps :

sparS.sql("create external table if not exists thread_x stored by 'carbondata' location 'D:/KernelTest_New/KernetTest/KernelTest_New/KernetTest/thread/thread'").show();
 sparS.sql("select count(*) from thread_x").show(); ----> 300000 record count

try{
 System.out.println("count called after write");
 Thread t1 = new Thread(new Trial(sparS));
 Thread t2 = new Thread(new Trial(sparS));
 //Each thread is holding 2 records each
 t1.setName("T1");
 t2.setName("T2");
 t1.start();

Thread.sleep(10);
 t2.start();
 t1.join();
 t2.join();

sparS.sql("select count(*) from thread_x").show();
 System.out.println("counrt called after load");

 

Actual Issue : sparS.sql("select count(*) from thread_x").show(); -----> 300002

Which implies Two different threads are overwriting the same carbondatafile.

Expected : 300004 record count should be displayed. 

(ie) A carbonfile has to be created for each thread. Concurrent load is failing.

 

From logs its is seen that two different threads are overwriting the same carbondatafile

Line 298: 18/08/22 16:58:10 INFO CarbonUtil: WriterForwardPool: thread_x Copying C:\Users\SWX593~1\AppData\Local\Temp\\carbon16096431985350_0\Fact\Part0\Segment_1534937289563\0\part-0-0_batchno0-0-1534937289563-1534937289561.carbondata --> D:/KernelTest_New/KernetTest/KernelTest_New/KernetTest/thread/thread
 Line 302: 18/08/22 16:58:10 INFO CarbonUtil: WriterForwardPool: thread_x Copying C:\Users\SWX593~1\AppData\Local\Temp\\carbon16096431985350_0\Fact\Part0\Segment_1534937289563\0\0_batchno0-0-1534937289563-1534937289561.carbonindex --> D:/KernelTest_New/KernetTest/KernelTest_New/KernetTest/thread/thread
 Line 374: 18/08/22 16:58:11 INFO CarbonUtil: WriterForwardPool: thread_x Copying C:\Users\SWX593~1\AppData\Local\Temp\\carbon16096980581610_0\Fact\Part0\Segment_1534937289563\0\part-0-0_batchno0-0-1534937289563-1534937289561.carbondata --> D:/KernelTest_New/KernetTest/KernelTest_New/KernetTest/thread/thread
 Line 378: 18/08/22 16:58:11 INFO CarbonUtil: WriterForwardPool: thread_x Copying C:\Users\SWX593~1\AppData\Local\Temp\\carbon16096980581610_0\Fact\Part0\Segment_1534937289563\0\0_batchno0-0-1534937289563-1534937289561.carbonindex --> D:/KernelTest_New/KernetTest/KernelTest_New/KernetTest/thread/thread

  was:
Steps :

sparS.sql("create external table if not exists thread_x stored by 'carbondata' location 'D:/KernelTest_New/KernetTest/KernelTest_New/KernetTest/thread/thread'").show();
 sparS.sql("select count(*) from thread_x").show(); ----> 300000 record count
 
 try{
 System.out.println("count called after write");
 Thread t1 = new Thread(new Trial(sparS));
 Thread t2 = new Thread(new Trial(sparS));
 //Each thread is holding 2 records each
 t1.setName("T1");
 t2.setName("T2");
 t1.start();
 
 Thread.sleep(10);
 t2.start();
 t1.join();
 t2.join();

sparS.sql("select count(*) from thread_x").show();
System.out.println("counrt called after load");

 

Actual Issue : sparS.sql("select count(*) from thread_x").show(); -----> 300002

Which implies Two different threads are overwriting the same carbondatafile.

Expected : 300004 record count should be displayed. 

ie A carbonfile has to be created for each thread.

 

From logs its is seen that two different threads are overwriting the same carbondatafile

Line 298: 18/08/22 16:58:10 INFO CarbonUtil: WriterForwardPool: thread_x Copying C:\Users\SWX593~1\AppData\Local\Temp\\carbon16096431985350_0\Fact\Part0\Segment_1534937289563\0\part-0-0_batchno0-0-1534937289563-1534937289561.carbondata --> D:/KernelTest_New/KernetTest/KernelTest_New/KernetTest/thread/thread
 Line 302: 18/08/22 16:58:10 INFO CarbonUtil: WriterForwardPool: thread_x Copying C:\Users\SWX593~1\AppData\Local\Temp\\carbon16096431985350_0\Fact\Part0\Segment_1534937289563\0\0_batchno0-0-1534937289563-1534937289561.carbonindex --> D:/KernelTest_New/KernetTest/KernelTest_New/KernetTest/thread/thread
 Line 374: 18/08/22 16:58:11 INFO CarbonUtil: WriterForwardPool: thread_x Copying C:\Users\SWX593~1\AppData\Local\Temp\\carbon16096980581610_0\Fact\Part0\Segment_1534937289563\0\part-0-0_batchno0-0-1534937289563-1534937289561.carbondata --> D:/KernelTest_New/KernetTest/KernelTest_New/KernetTest/thread/thread
 Line 378: 18/08/22 16:58:11 INFO CarbonUtil: WriterForwardPool: thread_x Copying C:\Users\SWX593~1\AppData\Local\Temp\\carbon16096980581610_0\Fact\Part0\Segment_1534937289563\0\0_batchno0-0-1534937289563-1534937289561.carbonindex --> D:/KernelTest_New/KernetTest/KernelTest_New/KernetTest/thread/thread


> Two different threads are overwriting the same carbondatafile
> -------------------------------------------------------------
>
>                 Key: CARBONDATA-2875
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-2875
>             Project: CarbonData
>          Issue Type: Bug
>          Components: data-load
>    Affects Versions: 1.5.0
>         Environment: Spark 2.2
>            Reporter: Supritha
>            Priority: Minor
>
> Steps :
> sparS.sql("create external table if not exists thread_x stored by 'carbondata' location 'D:/KernelTest_New/KernetTest/KernelTest_New/KernetTest/thread/thread'").show();
>  sparS.sql("select count(*) from thread_x").show(); ----> 300000 record count
> try{
>  System.out.println("count called after write");
>  Thread t1 = new Thread(new Trial(sparS));
>  Thread t2 = new Thread(new Trial(sparS));
>  //Each thread is holding 2 records each
>  t1.setName("T1");
>  t2.setName("T2");
>  t1.start();
> Thread.sleep(10);
>  t2.start();
>  t1.join();
>  t2.join();
> sparS.sql("select count(*) from thread_x").show();
>  System.out.println("counrt called after load");
>  
> Actual Issue : sparS.sql("select count(*) from thread_x").show(); -----> 300002
> Which implies Two different threads are overwriting the same carbondatafile.
> Expected : 300004 record count should be displayed. 
> (ie) A carbonfile has to be created for each thread. Concurrent load is failing.
>  
> From logs its is seen that two different threads are overwriting the same carbondatafile
> Line 298: 18/08/22 16:58:10 INFO CarbonUtil: WriterForwardPool: thread_x Copying C:\Users\SWX593~1\AppData\Local\Temp\\carbon16096431985350_0\Fact\Part0\Segment_1534937289563\0\part-0-0_batchno0-0-1534937289563-1534937289561.carbondata --> D:/KernelTest_New/KernetTest/KernelTest_New/KernetTest/thread/thread
>  Line 302: 18/08/22 16:58:10 INFO CarbonUtil: WriterForwardPool: thread_x Copying C:\Users\SWX593~1\AppData\Local\Temp\\carbon16096431985350_0\Fact\Part0\Segment_1534937289563\0\0_batchno0-0-1534937289563-1534937289561.carbonindex --> D:/KernelTest_New/KernetTest/KernelTest_New/KernetTest/thread/thread
>  Line 374: 18/08/22 16:58:11 INFO CarbonUtil: WriterForwardPool: thread_x Copying C:\Users\SWX593~1\AppData\Local\Temp\\carbon16096980581610_0\Fact\Part0\Segment_1534937289563\0\part-0-0_batchno0-0-1534937289563-1534937289561.carbondata --> D:/KernelTest_New/KernetTest/KernelTest_New/KernetTest/thread/thread
>  Line 378: 18/08/22 16:58:11 INFO CarbonUtil: WriterForwardPool: thread_x Copying C:\Users\SWX593~1\AppData\Local\Temp\\carbon16096980581610_0\Fact\Part0\Segment_1534937289563\0\0_batchno0-0-1534937289563-1534937289561.carbonindex --> D:/KernelTest_New/KernetTest/KernelTest_New/KernetTest/thread/thread



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Free forum by Nabble Edit this page