[jira] [Commented] (CARBONDATA-302) 7. Add DataWriterProcessorStep which reads the data from sort temp files and creates carbondata files.

[Akash R Nilugal (Jira)]

    [ https://issues.apache.org/jira/browse/CARBONDATA-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610793#comment-15610793 ]

ASF GitHub Bot commented on CARBONDATA-302:
-------------------------------------------

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/251#discussion_r85270229
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/util/CarbonDataProcessorUtil.java ---
    @@ -304,4 +311,92 @@ public static String getLocalDataFolderLocation(String databaseName, String tabl
         return ArrayUtils
             .toPrimitive(noDictionaryMapping.toArray(new Boolean[noDictionaryMapping.size()]));
       }
    +
    +  /**
    +   * Preparing the boolean [] to map whether the dimension use inverted index or not.
    +   */
    +  public static boolean[] getIsUseInvertedIndex(DataField[] fields) {
    +    List<Boolean> isUseInvertedIndexList = new ArrayList<Boolean>();
    +    for (DataField field : fields) {
    +      if (field.getColumn().isUseInvertedIndnex() && field.getColumn().isDimesion()) {
    +        isUseInvertedIndexList.add(true);
    +      } else if(field.getColumn().isDimesion()){
    +        isUseInvertedIndexList.add(false);
    +      }
    +    }
    +    return ArrayUtils
    +        .toPrimitive(isUseInvertedIndexList.toArray(new Boolean[isUseInvertedIndexList.size()]));
    +  }
    +
    +  private static String getComplexTypeString(DataField[] dataFields) {
    +    StringBuilder dimString = new StringBuilder();
    +    for (int i = 0; i < dataFields.length; i++) {
    +      DataField dataField = dataFields[i];
    +      if (dataField.getColumn().getDataType().equals(DataType.ARRAY) || dataField.getColumn()
    +          .getDataType().equals(DataType.STRUCT)) {
    +        addAllComplexTypeChildren((CarbonDimension) dataField.getColumn(), dimString, "");
    +        dimString.append(CarbonCommonConstants.SEMICOLON_SPC_CHARACTER);
    +      }
    +    }
    +    return dimString.toString();
    +  }
    +
    +  /**
    +   * This method will return all the child dimensions under complex dimension
    +   *
    +   */
    +  private static void addAllComplexTypeChildren(CarbonDimension dimension, StringBuilder dimString,
    +      String parent) {
    +    dimString.append(
    +        dimension.getColName() + CarbonCommonConstants.COLON_SPC_CHARACTER + dimension.getDataType()
    --- End diff --
   
    ok

> 7. Add DataWriterProcessorStep which reads the data from sort temp files and creates carbondata files.
> ------------------------------------------------------------------------------------------------------
>
>                 Key: CARBONDATA-302
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-302
>             Project: CarbonData
>          Issue Type: Sub-task
>            Reporter: Ravindra Pesala
>            Assignee: Ravindra Pesala
>             Fix For: 0.3.0-incubating
>
>
> Add DataWriterProcessorStep which reads the data from sort temp files and merge sort it, and apply mdk generator on key and creates carbondata files.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)