This post was updated on .
Hi community,
Recently, I viewed the implementation of carbon-spark-shell and tried to understand the function of this script. This script just wrap some steps and provide existing CarbonContext or CarbonSession for users to interact with Carbondata. I hold my opinion that we can remove this script because this script is useless except providing a existing CarbonContext or CarbonSession. Reasons as below: 1. Carbondata now has integration spark1.x and spark2.x, Carbondata should refactor carbon-spark-shell every time when spark update. 2. After run this script, it will generate redundant folders in project and user may forget to remove these folders. 3. The CarbonContext or CarbonSession may be created with store path and metastore path which user may not want. I just share my idea about this, we can discuss about whether we should keep this script. Thanks. Regards. Chenerlu. |
Hi
correct the file name, should be : ./bin/carbon-spark-sql and ./bin/carbon-spark-shell. Are you suggesting removing both file or only carbon-spark-shell ? Regards Liang 2017-06-07 0:24 GMT+08:00 Erlu Chen <[hidden email]>: > Hi community, > > Recently, I viewed the implementation of carbon-sql-shell and tried to > understand the function of this script. > > This script just wrap some steps and provide existing CarbonContext or > CarbonSession for users to interact with Carbondata. > > I hold my opinion that we can remove this script because this script is > useless except providing a existing CarbonContext or CarbonSession. > > Reasons as below: > 1. Carbondata now has integration spark1.x and spark2.x, Carbondata should > refactor carbon-spark-shell every time when spark update. > 2. After run this script, it will generate redundant folders in project and > user may forget to remove these folders. > 3. The CarbonContext or CarbonSession may be created with store path and > metastore path which user may not want. > > I just share my idea about this, we can discuss about whether we should > keep > this script. > > Thanks. > > Regards. > Chenerlu. > > > > > > > > -- > View this message in context: http://apache-carbondata-dev- > mailing-list-archive.1130556.n5.nabble.com/DISCUSSION- > Whether-Carbondata-should-keep-carbon-sql-shell-script-tp14077.html > Sent from the Apache CarbonData Dev Mailing List archive mailing list > archive at Nabble.com. > |
Thanks for correct my mistake.
Yes, Just carbon-spark-shell, I think carbon-spark-sql is more helpful than carbon-spark-shell, because it providing a method to interact with carbondata via sql command, rather than carbon api. Based on what I mentioned above, I think carbondata can still keep carbon-spark-sql. Regards. Chenerlu. |
Hi,
carbon-spark-shell was added to simplify the usage for first time users. I don't see any problem now of this feature to be removed. It is also simple code to be maintained. In case in future if this feature need to be modified because of spark compatible issues we can discuss again. Regards, Ravindra. On 7 June 2017 at 09:42, Erlu Chen <[hidden email]> wrote: > Thanks for correct my mistake. > > Yes, Just carbon-spark-shell, I think carbon-spark-sql is more helpful than > carbon-spark-shell, because it providing a method to interact with > carbondata via sql command, rather than carbon api. > > Based on what I mentioned above, I think carbondata can still keep > carbon-spark-sql. > > Regards. > Chenerlu. > > > > -- > View this message in context: http://apache-carbondata-dev- > mailing-list-archive.1130556.n5.nabble.com/DISCUSSION- > Whether-Carbondata-should-keep-carbon-spark-shell- > script-tp14077p14087.html > Sent from the Apache CarbonData Dev Mailing List archive mailing list > archive at Nabble.com. > -- Thanks & Regards, Ravi |
Hi, Ravindra.
users can learn how to use carbondata through QUICK START document. users should know how it works and this script just simply steps to get a existing CarbonSession. This is carbon API usage, I think community will send much time on maintenance this script which will do more harm than good. Now carbon-spark-shel has some problem when integrate spark2.1. What you said is suitable for carbon-spark-sql, rather than carbon-spark-shell. Because it provide a method of SQL command usage. Regards, Chenerlu. |
Hi,
+1 No problem, I think we can remove this feature and update the documentation as per your suggestion. Regards, Ravindra On 8 June 2017 at 14:28, Erlu Chen <[hidden email]> wrote: > Hi, Ravindra. > > users can learn how to use carbondata through QUICK START document. > > users should know how it works and this script just simply steps to get a > existing CarbonSession. > > This is carbon API usage, I think community will send much time on > maintenance this script > which will do more harm than good. > > Now carbon-spark-shel has some problem when integrate spark2.1. > > What you said is suitable for carbon-spark-sql, rather than > carbon-spark-shell. Because it provide a method of SQL command usage. > > > Regards, > Chenerlu. > > > > -- > View this message in context: http://apache-carbondata-dev- > mailing-list-archive.1130556.n5.nabble.com/DISCUSSION- > Whether-Carbondata-should-keep-carbon-spark-shell- > script-tp14077p14217.html > Sent from the Apache CarbonData Dev Mailing List archive mailing list > archive at Nabble.com. > -- Thanks & Regards, Ravi |
Free forum by Nabble | Edit this page |