[jira] [Created] (CARBONDATA-465) Spark streaming dataframe support

[Akash R Nilugal (Jira)]

WilliamZhu created CARBONDATA-465:
-------------------------------------

             Summary: Spark streaming dataframe support
                 Key: CARBONDATA-465
                 URL: https://issues.apache.org/jira/browse/CARBONDATA-465
             Project: CarbonData
          Issue Type: Improvement
          Components: data-load
    Affects Versions: 0.3.0-incubating
            Reporter: WilliamZhu
            Assignee: WilliamZhu
            Priority: Minor
             Fix For: 0.3.0-incubating


Carbondata-0.3.0 support load data with spark data frame api. There is a limit that kettle is still required since DataFrameLoaderRDD still depends on kettle. We provide NewDataFrameLoaderRDD  to load data with new flow .

Also,we discovered some bugs:

1. CarbonMetastoreCatalog.createTableFromThrift

```
/**
     * schemaFilePath starts with file:// will not create meta files successfully
     * while thriftWriter will have no complains.
     * This will cause some weired error eg. No table found.
     */
    val thriftWriter = new ThriftWriter(schemaFilePath, false)
    thriftWriter.open()
    thriftWriter.write(thriftTableInfo)
    thriftWriter.close()
```

2. There are some exceptions raised  even when you have set useKettle to false.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)