Dictionary file is locked for updation

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Dictionary file is locked for updation

李寅威
Hi all,


when I run the following script:
scala> cc.sql(s"load data inpath 'hdfs://master:9000/carbondata/sample.csv' into table test_table")


it turns out that:
WARN  27-12 12:37:58,044 - Lost task 1.3 in stage 2.0 (TID 13, slave1): java.lang.RuntimeException: Dictionary file name is locked for updation. Please try after some time


what I have done are:
1.in carbon.properties, set carbon.lock.type=HDFSLOCK
2.send carbon.properties & spark-defaults.conf to all nodes of the clusters


if any of you have any idea, looking forward to your replay, thx~
Reply | Threaded
Open this post in threaded view
|

Re: Dictionary file is locked for updation

kumarvishal09
Hi,
can you please find *"HDFS lock path"* string in executor log and let me
know the complete log message.

-Regards
Kumar Vishal

On Tue, Dec 27, 2016 at 12:45 PM, 251469031 <[hidden email]> wrote:

> Hi all,
>
>
> when I run the following script:
> scala> cc.sql(s"load data inpath 'hdfs://master:9000/carbondata/sample.csv'
> into table test_table")
>
>
> it turns out that:
> WARN  27-12 12:37:58,044 - Lost task 1.3 in stage 2.0 (TID 13, slave1):
> java.lang.RuntimeException: Dictionary file name is locked for updation.
> Please try after some time
>
>
> what I have done are:
> 1.in carbon.properties, set carbon.lock.type=HDFSLOCK
> 2.send carbon.properties & spark-defaults.conf to all nodes of the clusters
>
>
> if any of you have any idea, looking forward to your replay, thx~
kumar vishal
Reply | Threaded
Open this post in threaded view
|

回复: Dictionary file is locked for updation

李寅威
Hi Kumar,


  thx to your repley, the full logs is as follows:


16/12/27 12:30:17 INFO locks.HdfsFileLock: Executor task launch worker-0 HDFS lock path:hdfs://master:9000../carbon.store/default/test_table/2e9b7efa-2934-463a-9280-ff50c5129268.lock
16/12/27 12:30:17 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/12/27 12:30:17 INFO storage.ShuffleBlockFetcherIterator: Started 1 remote fetches in 1 ms
16/12/27 12:30:32 ERROR rdd.CarbonGlobalDictionaryGenerateRDD: Executor task launch worker-0
java.lang.RuntimeException: Dictionary file name is locked for updation. Please try after some time
        at scala.sys.package$.error(package.scala:27)
        at org.apache.carbondata.spark.rdd.CarbonGlobalDictionaryGenerateRDD$$anon$1.<init>(CarbonGlobalDictionaryRDD.scala:364)
        at org.apache.carbondata.spark.rdd.CarbonGlobalDictionaryGenerateRDD.compute(CarbonGlobalDictionaryRDD.scala:302)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
        at org.apache.spark.scheduler.Task.run(Task.scala:89)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)



as u see, the lock file path is:hdfs://master:9000../carbon.store/default/test_table/2e9b7efa-2934-463a-9280-ff50c5129268.lock




------------------ 原始邮件 ------------------
发件人: "Kumar Vishal";<[hidden email]>;
发送时间: 2016年12月27日(星期二) 下午3:25
收件人: "dev"<[hidden email]>;

主题: Re: Dictionary file is locked for updation



Hi,
can you please find *"HDFS lock path"* string in executor log and let me
know the complete log message.

-Regards
Kumar Vishal

On Tue, Dec 27, 2016 at 12:45 PM, 251469031 <[hidden email]> wrote:

> Hi all,
>
>
> when I run the following script:
> scala> cc.sql(s"load data inpath 'hdfs://master:9000/carbondata/sample.csv'
> into table test_table")
>
>
> it turns out that:
> WARN  27-12 12:37:58,044 - Lost task 1.3 in stage 2.0 (TID 13, slave1):
> java.lang.RuntimeException: Dictionary file name is locked for updation.
> Please try after some time
>
>
> what I have done are:
> 1.in carbon.properties, set carbon.lock.type=HDFSLOCK
> 2.send carbon.properties & spark-defaults.conf to all nodes of the clusters
>
>
> if any of you have any idea, looking forward to your replay, thx~
Reply | Threaded
Open this post in threaded view
|

Re: Dictionary file is locked for updation

ravipesala
Hi,

It seems the store path location is taking default location. Did you set
the store location properly? Which spark version you are using?

Regards,
Ravindra

On Tue, Dec 27, 2016, 1:38 PM 251469031 <[hidden email]> wrote:

> Hi Kumar,
>
>
>   thx to your repley, the full logs is as follows:
>
>
> 16/12/27 12:30:17 INFO locks.HdfsFileLock: Executor task launch worker-0
> HDFS lock
> path:hdfs://master:9000../carbon.store/default/test_table/2e9b7efa-2934-463a-9280-ff50c5129268.lock
> 16/12/27 12:30:17 INFO storage.ShuffleBlockFetcherIterator: Getting 1
> non-empty blocks out of 1 blocks
> 16/12/27 12:30:17 INFO storage.ShuffleBlockFetcherIterator: Started 1
> remote fetches in 1 ms
> 16/12/27 12:30:32 ERROR rdd.CarbonGlobalDictionaryGenerateRDD: Executor
> task launch worker-0
> java.lang.RuntimeException: Dictionary file name is locked for updation.
> Please try after some time
>         at scala.sys.package$.error(package.scala:27)
>         at
> org.apache.carbondata.spark.rdd.CarbonGlobalDictionaryGenerateRDD$$anon$1.<init>(CarbonGlobalDictionaryRDD.scala:364)
>         at
> org.apache.carbondata.spark.rdd.CarbonGlobalDictionaryGenerateRDD.compute(CarbonGlobalDictionaryRDD.scala:302)
>         at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>         at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>         at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
>         at org.apache.spark.scheduler.Task.run(Task.scala:89)
>         at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
>
>
>
> as u see, the lock file path
> is:hdfs://master:9000../carbon.store/default/test_table/2e9b7efa-2934-463a-9280-ff50c5129268.lock
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Kumar Vishal";<[hidden email]>;
> 发送时间: 2016年12月27日(星期二) 下午3:25
> 收件人: "dev"<[hidden email]>;
>
> 主题: Re: Dictionary file is locked for updation
>
>
>
> Hi,
> can you please find *"HDFS lock path"* string in executor log and let me
> know the complete log message.
>
> -Regards
> Kumar Vishal
>
> On Tue, Dec 27, 2016 at 12:45 PM, 251469031 <[hidden email]> wrote:
>
> > Hi all,
> >
> >
> > when I run the following script:
> > scala> cc.sql(s"load data inpath
> 'hdfs://master:9000/carbondata/sample.csv'
> > into table test_table")
> >
> >
> > it turns out that:
> > WARN  27-12 12:37:58,044 - Lost task 1.3 in stage 2.0 (TID 13, slave1):
> > java.lang.RuntimeException: Dictionary file name is locked for updation.
> > Please try after some time
> >
> >
> > what I have done are:
> > 1.in carbon.properties, set carbon.lock.type=HDFSLOCK
> > 2.send carbon.properties & spark-defaults.conf to all nodes of the
> clusters
> >
> >
> > if any of you have any idea, looking forward to your replay, thx~
Reply | Threaded
Open this post in threaded view
|

回复: Dictionary file is locked for updation

李寅威
I'm using spark 1.6.0, and the carbondata is the latest master branch in the github.


my carbon.properties is configured as :


carbon.ddl.base.hdfs.url=hdfs://master:9000/carbondata/data
carbon.badRecords.location=/opt/Carbon/Spark/badrecords
carbon.kettle.home=/opt/spark-1.6.0/carbonlib/carbonplugins

carbon.lock.type=HDFSLOCK
....


my spark-default.conf is configured as :


spark.master                                    spark://master:7077
spark.yarn.dist.files                           /opt/spark-1.6.0/conf/carbon.properties
spark.yarn.dist.archives                        /opt/spark-1.6.0/carbonlib/carbondata_2.10-1.0.0-incubating-SNAPSHOT-shade-hadoop2.7.2.jar
spark.executor.extraJavaOptions                 -Dcarbon.properties.filepath=carbon.properties
#spark.executor.extraClassPath                  /opt/spark-1.6.0/carbonlib/carbondata_2.10-1.0.0-incubating-SNAPSHOT-shade-hadoop2.2.0.jar
#spark.driver.extraClassPath                    /opt/spark-1.6.0/carbonlib/carbondata_2.10-1.0.0-incubating-SNAPSHOT-shade-hadoop2.2.0.jar
spark.driver.extraJavaOptions                   -Dcarbon.properties.filepath=/opt/spark-1.6.0/conf/carbon.properties
carbon.kettle.home                              /opt/spark-1.6.0/carbonlib/carbonplugins





------------------ 原始邮件 ------------------
发件人: "Ravindra Pesala";<[hidden email]>;
发送时间: 2016年12月27日(星期二) 下午4:15
收件人: "dev"<[hidden email]>;

主题: Re: Dictionary file is locked for updation



Hi,

It seems the store path location is taking default location. Did you set
the store location properly? Which spark version you are using?

Regards,
Ravindra

On Tue, Dec 27, 2016, 1:38 PM 251469031 <[hidden email]> wrote:

> Hi Kumar,
>
>
>   thx to your repley, the full logs is as follows:
>
>
> 16/12/27 12:30:17 INFO locks.HdfsFileLock: Executor task launch worker-0
> HDFS lock
> path:hdfs://master:9000../carbon.store/default/test_table/2e9b7efa-2934-463a-9280-ff50c5129268.lock
> 16/12/27 12:30:17 INFO storage.ShuffleBlockFetcherIterator: Getting 1
> non-empty blocks out of 1 blocks
> 16/12/27 12:30:17 INFO storage.ShuffleBlockFetcherIterator: Started 1
> remote fetches in 1 ms
> 16/12/27 12:30:32 ERROR rdd.CarbonGlobalDictionaryGenerateRDD: Executor
> task launch worker-0
> java.lang.RuntimeException: Dictionary file name is locked for updation.
> Please try after some time
>         at scala.sys.package$.error(package.scala:27)
>         at
> org.apache.carbondata.spark.rdd.CarbonGlobalDictionaryGenerateRDD$$anon$1.<init>(CarbonGlobalDictionaryRDD.scala:364)
>         at
> org.apache.carbondata.spark.rdd.CarbonGlobalDictionaryGenerateRDD.compute(CarbonGlobalDictionaryRDD.scala:302)
>         at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>         at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>         at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
>         at org.apache.spark.scheduler.Task.run(Task.scala:89)
>         at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
>
>
>
> as u see, the lock file path
> is:hdfs://master:9000../carbon.store/default/test_table/2e9b7efa-2934-463a-9280-ff50c5129268.lock
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Kumar Vishal";<[hidden email]>;
> 发送时间: 2016年12月27日(星期二) 下午3:25
> 收件人: "dev"<[hidden email]>;
>
> 主题: Re: Dictionary file is locked for updation
>
>
>
> Hi,
> can you please find *"HDFS lock path"* string in executor log and let me
> know the complete log message.
>
> -Regards
> Kumar Vishal
>
> On Tue, Dec 27, 2016 at 12:45 PM, 251469031 <[hidden email]> wrote:
>
> > Hi all,
> >
> >
> > when I run the following script:
> > scala> cc.sql(s"load data inpath
> 'hdfs://master:9000/carbondata/sample.csv'
> > into table test_table")
> >
> >
> > it turns out that:
> > WARN  27-12 12:37:58,044 - Lost task 1.3 in stage 2.0 (TID 13, slave1):
> > java.lang.RuntimeException: Dictionary file name is locked for updation.
> > Please try after some time
> >
> >
> > what I have done are:
> > 1.in carbon.properties, set carbon.lock.type=HDFSLOCK
> > 2.send carbon.properties & spark-defaults.conf to all nodes of the
> clusters
> >
> >
> > if any of you have any idea, looking forward to your replay, thx~
Reply | Threaded
Open this post in threaded view
|

Re: 回复: Dictionary file is locked for updation

David CaiQiang
Please try to add carbon.storelocation to carbon.properties file.
e.g.
carbon.storelocation=hdfs://master:9000/carbondata/store

You can have a look the following guide and pay attention to carbon.properties file.

https://cwiki.apache.org/confluence/display/CARBONDATA/Cluster+deployment+guide
Best Regards
David Cai
Reply | Threaded
Open this post in threaded view
|

Re: 回复: Dictionary file is locked for updation

李寅威
I'm sorry, carbon.storelocation has been configured in my cluster. I didn't copy it. the configuration is:


carbon.storelocation=hdfs://master:9000/carbondata




------------------ Original ------------------
From:  "QiangCai";<[hidden email]>;
Date:  Tue, Dec 27, 2016 05:29 PM
To:  "dev"<[hidden email]>;

Subject:  Re: 回复: Dictionary file is locked for updation



Please try to add carbon.storelocation to carbon.properties file.
e.g.
carbon.storelocation=hdfs://master:9000/carbondata/store

You can have a look the following guide and pay attention to
carbon.properties file.

https://cwiki.apache.org/confluence/display/CARBONDATA/Cluster+deployment+guide



--
View this message in context: http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/Dictionary-file-is-locked-for-updation-tp5076p5090.html
Sent from the Apache CarbonData Mailing List archive mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: 回复: Dictionary file is locked for updation

David CaiQiang
please correct the path of carbon.properties file.

spark.executor.extraJavaOptions                 -Dcarbon.properties.filepath=carbon.properties

Best Regards
David Cai
Reply | Threaded
Open this post in threaded view
|

Re: 回复: Dictionary file is locked for updation

李寅威
thx QiangCai, the problem is solved.


so, maybe it's better to correct the document at https://cwiki.apache.org/confluence/display/CARBONDATA/Cluster+deployment+guide, change the value of spark.executor.extraJavaOptions


from
-Dcarbon.properties.filepath=carbon.properties


to
-Dcarbon.properties.filepath=<YOUR_SPARK_HOME_PATH>"/conf/carbon.properties





------------------ Original ------------------
From:  "QiangCai";<[hidden email]>;
Date:  Tue, Dec 27, 2016 05:40 PM
To:  "dev"<[hidden email]>;

Subject:  Re:  回复: Dictionary file is locked for updation



please correct the path of carbon.properties file.

spark.executor.extraJavaOptions                
-Dcarbon.properties.filepath=carbon.properties





--
View this message in context: http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/Dictionary-file-is-locked-for-updation-tp5076p5092.html
Sent from the Apache CarbonData Mailing List archive mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: Dictionary file is locked for updation

kumarvishal09
In reply to this post by 李寅威
Hi,
Can you set the store location in carbon properties.

*carbon.storelocation=hdfs://master:9000/carbondata/<store_location>*

-Regards
Kumar Vishal

On Tue, Dec 27, 2016 at 4:27 PM, 251469031 <[hidden email]> wrote:

> I'm using spark 1.6.0, and the carbondata is the latest master branch in
> the github.
>
>
> my carbon.properties is configured as :
>
>
> carbon.ddl.base.hdfs.url=hdfs://master:9000/carbondata/data
> carbon.badRecords.location=/opt/Carbon/Spark/badrecords
> carbon.kettle.home=/opt/spark-1.6.0/carbonlib/carbonplugins
>
> carbon.lock.type=HDFSLOCK
> ....
>
>
> my spark-default.conf is configured as :
>
>
> spark.master                                    spark://master:7077
> spark.yarn.dist.files
>  /opt/spark-1.6.0/conf/carbon.properties
> spark.yarn.dist.archives                        /opt/spark-1.6.0/carbonlib/
> carbondata_2.10-1.0.0-incubating-SNAPSHOT-shade-hadoop2.7.2.jar
> spark.executor.extraJavaOptions
>  -Dcarbon.properties.filepath=carbon.properties
> #spark.executor.extraClassPath                  /opt/spark-1.6.0/carbonlib/
> carbondata_2.10-1.0.0-incubating-SNAPSHOT-shade-hadoop2.2.0.jar
> #spark.driver.extraClassPath                    /opt/spark-1.6.0/carbonlib/
> carbondata_2.10-1.0.0-incubating-SNAPSHOT-shade-hadoop2.2.0.jar
> spark.driver.extraJavaOptions
>  -Dcarbon.properties.filepath=/opt/spark-1.6.0/conf/carbon.properties
> carbon.kettle.home                              /opt/spark-1.6.0/carbonlib/
> carbonplugins
>
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Ravindra Pesala";<[hidden email]>;
> 发送时间: 2016年12月27日(星期二) 下午4:15
> 收件人: "dev"<[hidden email]>;
>
> 主题: Re: Dictionary file is locked for updation
>
>
>
> Hi,
>
> It seems the store path location is taking default location. Did you set
> the store location properly? Which spark version you are using?
>
> Regards,
> Ravindra
>
> On Tue, Dec 27, 2016, 1:38 PM 251469031 <[hidden email]> wrote:
>
> > Hi Kumar,
> >
> >
> >   thx to your repley, the full logs is as follows:
> >
> >
> > 16/12/27 12:30:17 INFO locks.HdfsFileLock: Executor task launch worker-0
> > HDFS lock
> > path:hdfs://master:9000../carbon.store/default/test_
> table/2e9b7efa-2934-463a-9280-ff50c5129268.lock
> > 16/12/27 12:30:17 INFO storage.ShuffleBlockFetcherIterator: Getting 1
> > non-empty blocks out of 1 blocks
> > 16/12/27 12:30:17 INFO storage.ShuffleBlockFetcherIterator: Started 1
> > remote fetches in 1 ms
> > 16/12/27 12:30:32 ERROR rdd.CarbonGlobalDictionaryGenerateRDD: Executor
> > task launch worker-0
> > java.lang.RuntimeException: Dictionary file name is locked for updation.
> > Please try after some time
> >         at scala.sys.package$.error(package.scala:27)
> >         at
> > org.apache.carbondata.spark.rdd.CarbonGlobalDictionaryGenerate
> RDD$$anon$1.<init>(CarbonGlobalDictionaryRDD.scala:364)
> >         at
> > org.apache.carbondata.spark.rdd.CarbonGlobalDictionaryGenerate
> RDD.compute(CarbonGlobalDictionaryRDD.scala:302)
> >         at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.
> scala:306)
> >         at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> >         at
> > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> >         at org.apache.spark.scheduler.Task.run(Task.scala:89)
> >         at
> > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
> >         at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> >         at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> >         at java.lang.Thread.run(Thread.java:745)
> >
> >
> >
> > as u see, the lock file path
> > is:hdfs://master:9000../carbon.store/default/test_
> table/2e9b7efa-2934-463a-9280-ff50c5129268.lock
> >
> >
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: "Kumar Vishal";<[hidden email]>;
> > 发送时间: 2016年12月27日(星期二) 下午3:25
> > 收件人: "dev"<[hidden email]>;
> >
> > 主题: Re: Dictionary file is locked for updation
> >
> >
> >
> > Hi,
> > can you please find *"HDFS lock path"* string in executor log and let me
> > know the complete log message.
> >
> > -Regards
> > Kumar Vishal
> >
> > On Tue, Dec 27, 2016 at 12:45 PM, 251469031 <[hidden email]> wrote:
> >
> > > Hi all,
> > >
> > >
> > > when I run the following script:
> > > scala> cc.sql(s"load data inpath
> > 'hdfs://master:9000/carbondata/sample.csv'
> > > into table test_table")
> > >
> > >
> > > it turns out that:
> > > WARN  27-12 12:37:58,044 - Lost task 1.3 in stage 2.0 (TID 13, slave1):
> > > java.lang.RuntimeException: Dictionary file name is locked for
> updation.
> > > Please try after some time
> > >
> > >
> > > what I have done are:
> > > 1.in carbon.properties, set carbon.lock.type=HDFSLOCK
> > > 2.send carbon.properties & spark-defaults.conf to all nodes of the
> > clusters
> > >
> > >
> > > if any of you have any idea, looking forward to your replay, thx~
>
kumar vishal
Reply | Threaded
Open this post in threaded view
|

Re: 回复: Dictionary file is locked for updation

Liang Chen
Administrator
In reply to this post by 李寅威
Hi

Updated ,thanks for you pointed out the issue.

Regards
Liang

李寅威 wrote
thx QiangCai, the problem is solved.


so, maybe it's better to correct the document at https://cwiki.apache.org/confluence/display/CARBONDATA/Cluster+deployment+guide, change the value of spark.executor.extraJavaOptions


from
-Dcarbon.properties.filepath=carbon.properties


to
-Dcarbon.properties.filepath=<YOUR_SPARK_HOME_PATH>"/conf/carbon.properties





------------------ Original ------------------
From:  "QiangCai";<[hidden email]>;
Date:  Tue, Dec 27, 2016 05:40 PM
To:  "dev"<[hidden email]>;

Subject:  Re:  回复: Dictionary file is locked for updation



please correct the path of carbon.properties file.

spark.executor.extraJavaOptions                
-Dcarbon.properties.filepath=carbon.properties





--
View this message in context: http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/Dictionary-file-is-locked-for-updation-tp5076p5092.html
Sent from the Apache CarbonData Mailing List archive mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: 回复: Dictionary file is locked for updation

Pallavi Singh
Hi all,

raised a jira issue :
https://issues.apache.org/jira/browse/CARBONDATA-585

On the execution of the following query :
LOAD DATA inpath 'hdfs://localhost:54310/csv/test.csv' INTO table employee
options('DELIMITER'=',', 'FILEHEADER'='id, firstname');

the table schema is a following :

-----------------------------+
col_name data_type comment

-----------------------------+
id bigint
firstname string

-----------------------------+

The load gets successful at times but we also end up often with the
following error :
Dictionary file is locked for Updation.

Following below are the logs :

AUDIT 02-01 18:17:07,009 - [knoldus][pallavi][Thread-110]Dataload failure
for default.employee. Please check the logs
INFO 02-01 18:17:07,020 - pool-30-thread-1 Successfully deleted the lock
file /tmp/default/employee/meta.lock
INFO 02-01 18:17:07,022 - Table MetaData Unlocked Successfully after data
load
ERROR 02-01 18:17:07,022 - Error executing query, currentState RUNNING,
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage
2.0 (TID 5, 192.168.2.188): java.lang.RuntimeException: Dictionary file
firstname is locked for updation. Please try after some time
at scala.sys.package$.error(package.scala:27)
at
org.apache.carbondata.spark.rdd.CarbonGlobalDictionaryGenerateRDD$$anon$1.<init>(CarbonGlobalDictionaryRDD.scala:364)
at
org.apache.carbondata.spark.rdd.CarbonGlobalDictionaryGenerateRDD.compute(CarbonGlobalDictionaryRDD.scala:302)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org
$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)
at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:927)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
at org.apache.spark.rdd.RDD.collect(RDD.scala:926)
at
org.apache.carbondata.spark.util.GlobalDictionaryUtil$.generateGlobalDictionary(GlobalDictionaryUtil.scala:769)

On Tue, Dec 27, 2016 at 8:10 PM, Liang Chen <[hidden email]> wrote:

> Hi
>
> Updated ,thanks for you pointed out the issue.
>
> Regards
> Liang
>
>
> 李寅威 wrote
> > thx QiangCai, the problem is solved.
> >
> >
> > so, maybe it's better to correct the document at
> > https://cwiki.apache.org/confluence/display/CARBONDATA/
> Cluster+deployment+guide,
> > change the value of spark.executor.extraJavaOptions
> >
> >
> > from
> > -Dcarbon.properties.filepath=carbon.properties
> >
> >
> > to
> > -Dcarbon.properties.filepath=
> > <YOUR_SPARK_HOME_PATH>
> > "/conf/carbon.properties
> >
> >
> >
> >
> >
> > ------------------ Original ------------------
> > From:  "QiangCai";&lt;
>
> > qiangcai@
>
> > &gt;;
> > Date:  Tue, Dec 27, 2016 05:40 PM
> > To:  "dev"&lt;
>
> > dev@.apache
>
> > &gt;;
> >
> > Subject:  Re:  回复: Dictionary file is locked for updation
> >
> >
> >
> > please correct the path of carbon.properties file.
> >
> > spark.executor.extraJavaOptions
> > -Dcarbon.properties.filepath=carbon.properties
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> > http://apache-carbondata-mailing-list-archive.1130556.
> n5.nabble.com/Dictionary-file-is-locked-for-updation-tp5076p5092.html
> > Sent from the Apache CarbonData Mailing List archive mailing list archive
> > at Nabble.com.
>
>
>
>
>
> --
> View this message in context: http://apache-carbondata-
> mailing-list-archive.1130556.n5.nabble.com/Dictionary-file-
> is-locked-for-updation-tp5076p5103.html
> Sent from the Apache CarbonData Mailing List archive mailing list archive
> at Nabble.com.
>



--
Regards | Pallavi Singh
Software Consultant
Knoldus Software LLP
[hidden email]
+91-9911235949