Question about CarbonDataFrameWriter

Posted by xuchuanyin on Oct 17, 2017; 3:16pm
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Question-about-CarbonDataFrameWriter-tp24194.html

Hi, community:




When I go through the DataFrame.write related code in Carbondata, I find there is an option to control whether to save the dataframe's data to a temporary directory as CSV on disk.




My question is why we need this procedure which will consume more disk IO and why the option(tempCSV) is true by default?




Related code can be referred:

https://github.com/apache/carbondata/blob/master/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDataFrameWriter.scala#L45




https://github.com/apache/carbondata/blob/master/integration/spark-common/src/main/scala/org/apache/carbondata/spark/CarbonOption.scala#L43