As we know, cabon.properties is the configuration file of carbondata. However ,for now, the driver and executors load this file from local disk which means we should distribute this file to all yarn nodes. At the same time ,we also should configure some items like follows:
--conf "spark.driver.extraJavaOptions=-Dcarbon.properties.filepath=/home/carbon/carbon.properties"
--conf "spark.executor.extraJavaOptions=-Dcarbon.properties.filepath=/home/carbon/carbon.properties"
I think the best way to handle this is using "--files". Eg.
--files /home/carbon/carbon.properties
Then carbondata should try to load from classpath first.
Also,we hope we can using --conf to overwrite the configurations in carbon.properties. Eg.
--conf 'carbon.properties.filepath=HDFSLock'
It's easy to implements this function, we can just modify the the code like follows: