how to distribute cabon.properties file

[ZhuWilliam]

As we know, cabon.properties is the configuration file of carbondata. However ,for now, the driver and executors load this file from local disk which means we should distribute this file to all yarn nodes. At the same time ,we also should configure some items like follows:

--conf "spark.driver.extraJavaOptions=-Dcarbon.properties.filepath=/home/carbon/carbon.properties"  
--conf "spark.executor.extraJavaOptions=-Dcarbon.properties.filepath=/home/carbon/carbon.properties"

I think the best way to handle this is using "--files". Eg.

--files /home/carbon/carbon.properties

Then carbondata should try to load from classpath first.

Also,we hope we can using --conf to overwrite the configurations in carbon.properties. Eg.

--conf 'carbon.properties.filepath=HDFSLock'

It's easy to implements this function, we can just modify the the code like follows:

[David CaiQiang]

+1

[Venkata Gollamudi]

Hi ZhuWilliam,

Yes we need to distribute carbon.properties to all the executors. In
current carbon implementation you can achieve the same with below
configurations.

in spark-defaults.conf:(in driver)
   #this places carbon.properties in executor path
   spark.yarn.dist.files=/home/carbondriver/carbon.properties
   spark.executor.extraJavaOptions="-Dcarbon.properties.filepath=
carbon.properties"

Regards,
Ramana

On Wed, Apr 12, 2017 at 5:29 PM, QiangCai <[hidden email]> wrote:

> +1
>
>
>
> --
> View this message in context: http://apache-carbondata-
> mailing-list-archive.1130556.n5.nabble.com/how-to-
> distribute-cabon-properties-file-tp10687p10869.html
> Sent from the Apache CarbonData Mailing List archive mailing list archive
> at Nabble.com.
>