Login  Register

Discussion about getting excution duration about a query when using sparkshell+carbondata

Posted by 李寅威 on Feb 07, 2017; 3:07am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Discussion-about-getting-excution-duration-about-a-query-when-using-sparkshell-carbondata-tp7379.html

Hi all,


  When we are using sparkshell + carbondata to send a query, how can we get the excution duration? Some topics are thrown as follows:


  1. One query can produce one or more jobs, and some of the jobs may have DAG dependence, thus we can't get the excution duration by sum up all the jobs' duration or get the max duration of the jobs roughly.


  2. In the spark shell console or spark application web ui, we can get each job's duration, but we can't get the carbondata-query directly, if some improvement would take by carbondata in the near future.


  3. Maybe we can use the following command to get a approximate result:


    scala > val begin = new Date();cc.sql("$SQL_COMMAND").show;val end = new Date();


  Any other opinions?