[DISCUSSION] Whether Carbondata should keep carbon-spark-shell script

[Erlu Chen]

Hi community,

Recently, I viewed the implementation of carbon-spark-shell and tried to understand the function of this script.

This script just wrap some steps and provide existing CarbonContext or CarbonSession for users  to interact with Carbondata.

I hold my opinion that we can remove this script because this script is useless except providing a existing CarbonContext or CarbonSession.

Reasons as below:
1. Carbondata now has integration spark1.x and spark2.x, Carbondata should refactor carbon-spark-shell every time when spark update.
2. After run this script, it will generate redundant folders in project and user may forget to remove these folders.
3. The CarbonContext or CarbonSession may be created with store path and metastore path which user may not want.

I just share my idea about this, we can discuss about whether we should keep this script.

Thanks.

Regards.
Chenerlu.

[Liang Chen-2]

Hi

correct the file name, should be : ./bin/carbon-spark-sql and
./bin/carbon-spark-shell.

Are you suggesting removing both file or only carbon-spark-shell ?

Regards
Liang

2017-06-07 0:24 GMT+08:00 Erlu Chen <[hidden email]>:

> Hi community,
>
> Recently, I viewed the implementation of carbon-sql-shell and tried to
> understand the function of this script.
>
> This script just wrap some steps and provide existing CarbonContext or
> CarbonSession for users  to interact with Carbondata.
>
> I hold my opinion that we can remove this script because this script is
> useless except providing a existing CarbonContext or CarbonSession.
>
> Reasons as below:
> 1. Carbondata now has integration spark1.x and spark2.x, Carbondata should
> refactor carbon-spark-shell every time when spark update.
> 2. After run this script, it will generate redundant folders in project and
> user may forget to remove these folders.
> 3. The CarbonContext or CarbonSession may be created with store path and
> metastore path which user may not want.
>
> I just share my idea about this, we can discuss about whether we should
> keep
> this script.
>
> Thanks.
>
> Regards.
> Chenerlu.
>
>
>
>
>
>
>
> --
> View this message in context: http://apache-carbondata-dev-
> mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-
> Whether-Carbondata-should-keep-carbon-sql-shell-script-tp14077.html
> Sent from the Apache CarbonData Dev Mailing List archive mailing list
> archive at Nabble.com.
>

[Erlu Chen]

Thanks for correct my mistake.

Yes, Just carbon-spark-shell, I think carbon-spark-sql is more helpful than carbon-spark-shell, because it providing a method to interact with carbondata via sql command, rather than carbon api.

Based on what I mentioned above, I think carbondata can still keep carbon-spark-sql.

Regards.
Chenerlu.

[ravipesala]

Hi,

carbon-spark-shell was added to simplify the usage for first time users. I
don't see any problem now of this feature to be removed. It is also simple
code to be maintained. In case in future if this feature need to be
modified because of spark compatible issues we can discuss again.

Regards,
Ravindra.

On 7 June 2017 at 09:42, Erlu Chen <[hidden email]> wrote:

> Thanks for correct my mistake.
>
> Yes, Just carbon-spark-shell, I think carbon-spark-sql is more helpful than
> carbon-spark-shell, because it providing a method to interact with
> carbondata via sql command, rather than carbon api.
>
> Based on what I mentioned above, I think carbondata can still keep
> carbon-spark-sql.
>
> Regards.
> Chenerlu.
>
>
>
> --
> View this message in context: http://apache-carbondata-dev-
> mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-
> Whether-Carbondata-should-keep-carbon-spark-shell-
> script-tp14077p14087.html
> Sent from the Apache CarbonData Dev Mailing List archive mailing list
> archive at Nabble.com.
>

--
Thanks & Regards,
Ravi

[Erlu Chen]

Hi,  Ravindra.

users can learn how to use carbondata through QUICK START document.

users should know how it works and this script just simply steps to get a existing CarbonSession.

This is carbon API usage, I think community will send much time on maintenance this script
which will do more harm than good.

Now carbon-spark-shel has some problem when integrate spark2.1.

What you said is suitable for carbon-spark-sql, rather than carbon-spark-shell. Because  it provide a method of SQL command usage.


Regards,
Chenerlu.

[ravipesala]

Hi,

+1
No problem, I think we can remove this feature and update the documentation
as per your suggestion.

Regards,
Ravindra

On 8 June 2017 at 14:28, Erlu Chen <[hidden email]> wrote:

> Hi,  Ravindra.
>
> users can learn how to use carbondata through QUICK START document.
>
> users should know how it works and this script just simply steps to get a
> existing CarbonSession.
>
> This is carbon API usage, I think community will send much time on
> maintenance this script
> which will do more harm than good.
>
> Now carbon-spark-shel has some problem when integrate spark2.1.
>
> What you said is suitable for carbon-spark-sql, rather than
> carbon-spark-shell. Because  it provide a method of SQL command usage.
>
>
> Regards,
> Chenerlu.
>
>
>
> --
> View this message in context: http://apache-carbondata-dev-
> mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-
> Whether-Carbondata-should-keep-carbon-spark-shell-
> script-tp14077p14217.html
> Sent from the Apache CarbonData Dev Mailing List archive mailing list
> archive at Nabble.com.
>

--
Thanks & Regards,
Ravi

[Erlu Chen]

Hi community

Any comments on this topic ?

If others have no idea, I will raise a PR to remove this feature.


Regards
Chenerlu.