[GitHub] carbondata pull request #1440: [WIP][CARBONDATA-1581][CARBONDATA-1582] Imple...

[qiuchenjian-2]

GitHub user QiangCai opened a pull request:

    https://github.com/apache/carbondata/pull/1440

    [WIP][CARBONDATA-1581][CARBONDATA-1582] Implement StreamSinkProvider and stream file writer

    1. Change hadoop.version to 2.7.2 as default
    Require using truncate operation of the filesystem.
   
    2. CarbonSource extend StreamSinkProvider
    Provide stream sink to support streaming ingest
   
    3. Implement CarbonStreamOutputFormat and CarbonStreamRecordWriter
    CarbonStreamRecordWriter write input data to CarbonData stream file.
   
    4. Avoid Small file issue
    Append new blocklet to old file to avoid small file issue
   
    5. Support fault tolerant
    Stream segment has a CarbonIndex file, this index file record the information of the CarbonData files.
    We can recover data to last successful commit.
   


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/QiangCai/carbondata streaming

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1440.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1440
   
----
commit 6c94c9311ea1b260e75bf576eec75aea17ce8984
Author: QiangCai <[hidden email]>
Date:   2017-10-18T03:13:00Z

    support streaming ingest

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/1440
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/694/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/1440
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1329/



---

[qiuchenjian-2]

Github user jackylk commented on the issue:

    https://github.com/apache/carbondata/pull/1440
 
    Can you tag the PR title with [Streaming]


---

[qiuchenjian-2]

Github user QiangCai commented on the issue:

    https://github.com/apache/carbondata/pull/1440
 
    close this pr
    I will raise new pr.


---

[qiuchenjian-2]

Github user QiangCai closed the pull request at:

    https://github.com/apache/carbondata/pull/1440


---