Query About Carbon Write Process : why always 10 Task get created when we write dataframe or rdd in carbon format in a write job or save job

Posted by Anshul Jain on May 20, 2019; 5:04am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Query-About-Carbon-Write-Process-why-always-10-Task-get-created-when-we-write-dataframe-or-rdd-in-cab-tp79200.html

Hi Dev team ,


I am doing a test with carbondata to load a csv file of  600 gb and write it in carbon format in s3 location , while writing I can see only 10 task getting created in final step of execution job as I was using 10 nodes , while I have num-executor as 18 , so its degrading my job performance and How can I make change to let task no. equal to no. of executor for best performance.


Thanks & Regards,

Anshul Jain

Big Data Engineer

Impetus Infotech (India) Pvt. Ltd.

Tel: [phone]/3662



________________________________






NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.