[GitHub] incubator-carbondata pull request #494: [CARBONDATA-218]Using CSVInputFormat...

[qiuchenjian-2]

GitHub user QiangCai opened a pull request:

    https://github.com/apache/incubator-carbondata/pull/494

    [CARBONDATA-218]Using CSVInputFormat instead of spark-csv during dictionary geneartion

    1. Using CSVInputFormat instead of spark-csv during dictionary geneartion
   
    2. Remove spark-csv dependency from whole project.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/QiangCai/incubator-carbondata unifyCsvReader

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-carbondata/pull/494.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #494
   
----
commit 34f55769d74969cff5d9a322ad8ae0cbf14befdc
Author: QiangCai <[hidden email]>
Date:   2017-01-03T08:28:06Z

    unify csv reader

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/494
 
    Build Success with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/429/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/494#discussion_r94544073
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/GlobalDictionaryUtil.scala ---
    @@ -356,37 +363,49 @@ object GlobalDictionaryUtil {
        */
       def loadDataFrame(sqlContext: SQLContext,
           carbonLoadModel: CarbonLoadModel): DataFrame = {
    -    val df = sqlContext.read
    -      .format("com.databricks.spark.csv.newapi")
    -      .option("header", {
    -        if (StringUtils.isEmpty(carbonLoadModel.getCsvHeader)) {
    -          "true"
    -        } else {
    -          "false"
    -        }
    -      })
    -      .option("delimiter", {
    -        if (StringUtils.isEmpty(carbonLoadModel.getCsvDelimiter)) {
    -          "" + DEFAULT_SEPARATOR
    -        } else {
    -          carbonLoadModel.getCsvDelimiter
    +      val hadoopConfiguration = new Configuration()
    +      CommonUtil.configureCSVInputFormat(hadoopConfiguration, carbonLoadModel)
    +      hadoopConfiguration.set(FileInputFormat.INPUT_DIR, carbonLoadModel.getFactFilePath)
    --- End diff --
   
    use `FileInputFormat.addInput`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user QiangCai commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/494#discussion_r94545504
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/GlobalDictionaryUtil.scala ---
    @@ -356,37 +363,49 @@ object GlobalDictionaryUtil {
        */
       def loadDataFrame(sqlContext: SQLContext,
           carbonLoadModel: CarbonLoadModel): DataFrame = {
    -    val df = sqlContext.read
    -      .format("com.databricks.spark.csv.newapi")
    -      .option("header", {
    -        if (StringUtils.isEmpty(carbonLoadModel.getCsvHeader)) {
    -          "true"
    -        } else {
    -          "false"
    -        }
    -      })
    -      .option("delimiter", {
    -        if (StringUtils.isEmpty(carbonLoadModel.getCsvDelimiter)) {
    -          "" + DEFAULT_SEPARATOR
    -        } else {
    -          carbonLoadModel.getCsvDelimiter
    +      val hadoopConfiguration = new Configuration()
    +      CommonUtil.configureCSVInputFormat(hadoopConfiguration, carbonLoadModel)
    +      hadoopConfiguration.set(FileInputFormat.INPUT_DIR, carbonLoadModel.getFactFilePath)
    --- End diff --
   
    FileInputFormat.addInputPath method need a Job type paramter.
    In addition, this FactFilePath already consist of all file path, we can directly set input path, no need to separate path and add path again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/494#discussion_r94552530
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/GlobalDictionaryUtil.scala ---
    @@ -356,37 +363,49 @@ object GlobalDictionaryUtil {
        */
       def loadDataFrame(sqlContext: SQLContext,
           carbonLoadModel: CarbonLoadModel): DataFrame = {
    -    val df = sqlContext.read
    -      .format("com.databricks.spark.csv.newapi")
    -      .option("header", {
    -        if (StringUtils.isEmpty(carbonLoadModel.getCsvHeader)) {
    -          "true"
    -        } else {
    -          "false"
    -        }
    -      })
    -      .option("delimiter", {
    -        if (StringUtils.isEmpty(carbonLoadModel.getCsvDelimiter)) {
    -          "" + DEFAULT_SEPARATOR
    -        } else {
    -          carbonLoadModel.getCsvDelimiter
    +      val hadoopConfiguration = new Configuration()
    +      CommonUtil.configureCSVInputFormat(hadoopConfiguration, carbonLoadModel)
    +      hadoopConfiguration.set(FileInputFormat.INPUT_DIR, carbonLoadModel.getFactFilePath)
    --- End diff --
   
    ok


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/494#discussion_r94552544
 
    --- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/GlobalDictionaryUtil.scala ---
    @@ -356,37 +363,49 @@ object GlobalDictionaryUtil {
        */
       def loadDataFrame(sqlContext: SQLContext,
           carbonLoadModel: CarbonLoadModel): DataFrame = {
    -    val df = sqlContext.read
    -      .format("com.databricks.spark.csv.newapi")
    -      .option("header", {
    -        if (StringUtils.isEmpty(carbonLoadModel.getCsvHeader)) {
    -          "true"
    -        } else {
    -          "false"
    -        }
    -      })
    -      .option("delimiter", {
    -        if (StringUtils.isEmpty(carbonLoadModel.getCsvDelimiter)) {
    -          "" + DEFAULT_SEPARATOR
    -        } else {
    -          carbonLoadModel.getCsvDelimiter
    +      val hadoopConfiguration = new Configuration()
    +      CommonUtil.configureCSVInputFormat(hadoopConfiguration, carbonLoadModel)
    +      hadoopConfiguration.set(FileInputFormat.INPUT_DIR, carbonLoadModel.getFactFilePath)
    --- End diff --
   
    ok


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/494
 
    LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-carbondata/pull/494


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---