[GitHub] [carbondata] marchpure opened a new pull request #3997: [WIP]Add TPCDS TestCase

[GitBox]

marchpure opened a new pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997


    ### Why is this PR needed?
   
   
    ### What changes were proposed in this PR?
   
       
    ### Does this PR introduce any user interface change?
    - No
    - Yes. (please explain the change and update document)
   
    ### Is any new testcase added?
    - No
    - Yes
   
       
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997#issuecomment-716103357






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997#issuecomment-716166118


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2921/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997#issuecomment-716166185


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4678/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

marchpure commented on pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997#issuecomment-716167647


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997#issuecomment-716175045


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2923/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997#issuecomment-716175123


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4680/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997#issuecomment-716230413


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4681/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997#issuecomment-716230854


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2924/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997#issuecomment-716275742


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2927/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997#issuecomment-716276174


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4684/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997#issuecomment-716454832


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4690/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997#issuecomment-716455767


   Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2933/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997#issuecomment-716562736


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4693/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997#issuecomment-716563917


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2936/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997#issuecomment-717016610


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4697/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

CarbonDataQA1 commented on pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997#issuecomment-717017353


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2940/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

ajantha-bhat commented on pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997#issuecomment-717031195


   @marchpure : The reason why we don't have TPCH and TPCDS in UT is we need a huge data set, loading huge data takes time.
   
   What is the reason behind adding this? we can anyways have separate TPCH or TPCDS machines that can have automation script to give performance benchmark on every release
   
   Also no need to run TPCH and TPCDS on every PR builder.  Running once per release is enough.
   
   @QiangCai , @kunal642 : What's your opinion on this?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

marchpure commented on pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997#issuecomment-717045718


   > @marchpure : The reason why we don't have TPCH and TPCDS in UT is we need a huge data set, loading huge data takes time.
   >
   > What is the reason behind adding this? we can anyways have separate TPCH or TPCDS machines that can have automation script to give performance benchmark on every release
   >
   > Also no need to run TPCH and TPCDS on every PR builder. Running once per release is enough.
   >
   > @QiangCai , @kunal642 : What's your opinion on this?
   
   1. The TPCDS dataset in this PR is really small(totally 33KB).  it won"t took so much time to load and query.  It may help to avoid possible issues with accepted overhead.
   2. it also help us to debug tpcds.  explain plan ～analyse in local environment.
   
   the inspiration to add tpcds test case is CARBONDATA 4008.  Whose issue is Spark on CarbonData will fail in TPCDS Query 83. This issue seems has been there for a log time,  which implies that our UT is not enough.
   I believe that we can add a profile to turn on/off of TPCDS test in the future if the automatic TPCDS machine is ready.
   
   Maybe we can have a module name 'carbondata-integretion-test'?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]

[GitBox]

marchpure edited a comment on pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997#issuecomment-717045718


   > @marchpure : The reason why we don't have TPCH and TPCDS in UT is we need a huge data set, loading huge data takes time.
   >
   > What is the reason behind adding this? we can anyways have separate TPCH or TPCDS machines that can have automation script to give performance benchmark on every release
   >
   > Also no need to run TPCH and TPCDS on every PR builder. Running once per release is enough.
   >
   > @QiangCai , @kunal642 : What's your opinion on this?
   
   1. The TPCDS dataset in this PR is really small(totally 33KB).  it won"t took so much time to load and query.  It may help to avoid possible issues with accepted overhead.
   2. it also help us to debug tpcds.  explain plan ～analyse in local environment.
   
   the inspiration to add tpcds test case is CARBONDATA 4008.  Whose issue is Spark on CarbonData will fail in TPCDS Query 83. This issue seems has been there for a log time,  which implies that our UT is not enough.
   I believe that we can add a profile to turn on/off of TPCDS test in the future if the automatic TPCDS machine is ready.
   
   **Maybe we can have a module name 'carbondata-integretion-test'?**


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]