[jira] [Commented] (CARBONDATA-297) 2. Add interfaces for data loading.

[Akash R Nilugal (Jira)]

    [ https://issues.apache.org/jira/browse/CARBONDATA-297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569219#comment-15569219 ]

ASF GitHub Bot commented on CARBONDATA-297:
-------------------------------------------

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/229#discussion_r83049479
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/newflow/DataLoadProcessorStep.java ---
    @@ -0,0 +1,40 @@
    +package org.apache.carbondata.processing.newflow;
    +
    +import java.util.Iterator;
    +
    +import org.apache.carbondata.processing.newflow.exception.CarbonDataLoadingException;
    +
    +/**
    + * This base interface for data loading. It can do transformation jobs as per the implementation.
    + *
    + */
    +public interface DataLoadProcessorStep {
    +
    +  /**
    +   * The output meta for this step. The data returns from this step is as per this meta.
    +   * @return
    +   */
    +  DataField[] getOutput();
    +
    +  /**
    +   * Intialization process for this step.
    +   * @param configuration
    +   * @param child
    +   * @throws CarbonDataLoadingException
    +   */
    +  void intialize(CarbonDataLoadConfiguration configuration, DataLoadProcessorStep child) throws
    +      CarbonDataLoadingException;
    +
    +  /**
    +   * Tranform the data as per the implemetation.
    +   * @return Iterator of data
    +   * @throws CarbonDataLoadingException
    +   */
    +  Iterator<Object[]> execute() throws CarbonDataLoadingException;
    --- End diff --
   
    For suppose if we are loading 50GB of csv files and each HDFS block size is 256MB then total number of partitions are 200. If we allow one task per partition then it would be 200 tasks. In carbondata one btree is created for each task. So if we allow all 200 tasks then it would be massively 200 btrees and it is not effective both in performance and memory wise.
    That is the reason why we pool multiple blocks per task in the current kettle implementation. And these blocks are processed parallely. We can take the same way and use iterator for each thread and returns array of iterator.
   
    What do you mean by datanode-scope sorting? how to synchronize between multiple tasks?

> 2. Add interfaces for data loading.
> -----------------------------------
>
>                 Key: CARBONDATA-297
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-297
>             Project: CarbonData
>          Issue Type: Sub-task
>            Reporter: Ravindra Pesala
>            Assignee: Ravindra Pesala
>             Fix For: 0.2.0-incubating
>
>
> Add the major interface classes for data loading so that the following jiras can use this interfaces to implement it.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)