[GitHub] incubator-carbondata pull request #641: [WIP] Alter table support for carbon...

classic Classic list List threaded Threaded
73 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata pull request #641: [WIP] Alter table support for carbon...

qiuchenjian-2
GitHub user manishgupta88 opened a pull request:

    https://github.com/apache/incubator-carbondata/pull/641

    [WIP] Alter table support for carbondata

    As part of this PR following features will be supported.
   
    1. Support alter table result preparation.
    2. Support reading data with different block key generators.
    3. Support addition of a new column.
    4. Support deletion of a column.
    5. Support change in data type form INT to BIGINT
    6. Support Change of decimal datatype from lower to higher precision.
    7. Support filtering on newly added columns.
    8. Support rename table
    9. Parsing support for the new DDL commands added.
   
    Not supported features as part of this PR.
    1. Complex type
    2. Column groups
    3. Alter table parsing support on spark-1.6


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/manishgupta88/incubator-carbondata nm_restructure_with_v3_format

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-carbondata/pull/641.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #641
   
----
commit ffb0631be2c73c453262912ef13f2f7ce9fa33ed
Author: manishgupta88 <[hidden email]>
Date:   2017-03-10T05:23:44Z

    Following changes are done as part of this commit.
   
    1. Support alter table result preparation.
    2. Support reading data with different block key generators.
    3. Support addition of a new column.
    4. Support deletion of a column.
    5. Support change in data type form INT to BIGINT
    6. Support Change of decimal datatype from lower to higher precision.
    7. Support filtering on newly added columns.

commit 52a365b5e38d4b5a8f8ad6a40701c5c99889b829
Author: nareshpr <[hidden email]>
Date:   2017-03-10T06:00:51Z

    Restructure requirement related spark-integration changes

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #641: [WIP] Alter table support for carbondata

qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/641
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1068/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #641: [WIP] Alter table support for carbondata

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/641
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1071/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #641: [WIP] Alter table support for carbondata

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/641
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1072/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #641: [WIP] Alter table support for carbondata

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/641
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1073/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #641: [WIP] Alter table support for carbondata

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/641
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1076/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #641: [WIP] Alter table support for carbondata

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/641
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1083/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #641: [WIP] Alter table support for carbondata

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/641
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1102/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #641: [WIP] Alter table support for carbondata

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/641
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1105/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #641: [WIP] Alter table support for carbondata

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/641
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1110/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #641: [WIP] Alter table support for carbondata

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/641
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1113/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #641: [CARBONDATA-767] Alter table support for ca...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/641
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1114/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #641: [CARBONDATA-767] Alter table support for ca...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/641
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1115/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #641: [CARBONDATA-767] Alter table support for ca...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/641
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1116/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #641: [CARBONDATA-767] Alter table support for ca...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/641
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1126/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata issue #641: [CARBONDATA-767] Alter table support for ca...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/641
 
    Build Failed  with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1129/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata pull request #641: [CARBONDATA-767] Alter table support...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/641#discussion_r105879703
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/collector/impl/RestructureBasedVectorResultCollector.java ---
    @@ -0,0 +1,321 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.carbondata.core.scan.collector.impl;
    +
    +import java.math.BigDecimal;
    +import java.util.ArrayList;
    +import java.util.Arrays;
    +import java.util.List;
    +
    +import org.apache.carbondata.core.keygenerator.directdictionary.DirectDictionaryKeyGeneratorFactory;
    +import org.apache.carbondata.core.metadata.encoder.Encoding;
    +import org.apache.carbondata.core.metadata.schema.table.column.CarbonDimension;
    +import org.apache.carbondata.core.metadata.schema.table.column.CarbonMeasure;
    +import org.apache.carbondata.core.scan.executor.infos.BlockExecutionInfo;
    +import org.apache.carbondata.core.scan.model.QueryDimension;
    +import org.apache.carbondata.core.scan.model.QueryMeasure;
    +import org.apache.carbondata.core.scan.result.AbstractScannedResult;
    +import org.apache.carbondata.core.scan.result.vector.CarbonColumnVector;
    +import org.apache.carbondata.core.scan.result.vector.CarbonColumnarBatch;
    +import org.apache.carbondata.core.scan.result.vector.ColumnVectorInfo;
    +import org.apache.carbondata.core.scan.result.vector.MeasureDataVectorProcessor;
    +
    +import org.apache.spark.sql.types.Decimal;
    +
    +/**
    + * It is not a collector it is just a scanned result holder.
    + */
    +public class RestructureBasedVectorResultCollector extends AbstractScannedResultCollector {
    +
    +  private ColumnVectorInfo[] dictionaryInfo;
    +
    +  private ColumnVectorInfo[] noDictionaryInfo;
    +
    +  private ColumnVectorInfo[] complexInfo;
    +
    +  private ColumnVectorInfo[] measureColumnInfo;
    +
    +  private ColumnVectorInfo[] allColumnInfo;
    +
    +  public RestructureBasedVectorResultCollector(BlockExecutionInfo blockExecutionInfos) {
    +    super(blockExecutionInfos);
    +    QueryDimension[] queryDimensions = tableBlockExecutionInfos.getActualQueryDimensions();
    +    QueryMeasure[] queryMeasures = tableBlockExecutionInfos.getActualQueryMeasures();
    +    measureColumnInfo = new ColumnVectorInfo[queryMeasures.length];
    +    allColumnInfo = new ColumnVectorInfo[queryDimensions.length + queryMeasures.length];
    +    List<ColumnVectorInfo> dictInfoList = new ArrayList<>();
    +    List<ColumnVectorInfo> noDictInfoList = new ArrayList<>();
    +    List<ColumnVectorInfo> complexList = new ArrayList<>();
    +    int dimensionExistIndex = 0;
    +    for (int i = 0; i < queryDimensions.length; i++) {
    +      if (!dimensionInfo.getDimensionExists()[i]) {
    +        // add a dummy column vector result collector object
    +        ColumnVectorInfo columnVectorInfo = new ColumnVectorInfo();
    +        allColumnInfo[queryDimensions[i].getQueryOrder()] = columnVectorInfo;
    +        continue;
    +      }
    +      // get the current block dimension and fetch the required information from it
    +      QueryDimension currentBlockDimension =
    +          tableBlockExecutionInfos.getQueryDimensions()[dimensionExistIndex++];
    +      if (!queryDimensions[i].getDimension().hasEncoding(Encoding.DICTIONARY)) {
    +        ColumnVectorInfo columnVectorInfo = new ColumnVectorInfo();
    +        noDictInfoList.add(columnVectorInfo);
    +        columnVectorInfo.dimension = currentBlockDimension;
    +        columnVectorInfo.ordinal = currentBlockDimension.getDimension().getOrdinal();
    +        allColumnInfo[queryDimensions[i].getQueryOrder()] = columnVectorInfo;
    +      } else if (queryDimensions[i].getDimension().hasEncoding(Encoding.DIRECT_DICTIONARY)) {
    +        ColumnVectorInfo columnVectorInfo = new ColumnVectorInfo();
    +        dictInfoList.add(columnVectorInfo);
    +        columnVectorInfo.dimension = currentBlockDimension;
    +        columnVectorInfo.directDictionaryGenerator = DirectDictionaryKeyGeneratorFactory
    +            .getDirectDictionaryGenerator(currentBlockDimension.getDimension().getDataType());
    +        columnVectorInfo.ordinal = currentBlockDimension.getDimension().getOrdinal();
    +        allColumnInfo[queryDimensions[i].getQueryOrder()] = columnVectorInfo;
    +      } else if (queryDimensions[i].getDimension().isComplex()) {
    +        ColumnVectorInfo columnVectorInfo = new ColumnVectorInfo();
    +        complexList.add(columnVectorInfo);
    +        columnVectorInfo.dimension = currentBlockDimension;
    +        columnVectorInfo.ordinal = currentBlockDimension.getDimension().getOrdinal();
    +        columnVectorInfo.genericQueryType =
    +            tableBlockExecutionInfos.getComlexDimensionInfoMap().get(columnVectorInfo.ordinal);
    +        allColumnInfo[queryDimensions[i].getQueryOrder()] = columnVectorInfo;
    +      } else {
    +        ColumnVectorInfo columnVectorInfo = new ColumnVectorInfo();
    +        dictInfoList.add(columnVectorInfo);
    +        columnVectorInfo.dimension = currentBlockDimension;
    +        columnVectorInfo.ordinal = currentBlockDimension.getDimension().getOrdinal();
    +        allColumnInfo[queryDimensions[i].getQueryOrder()] = columnVectorInfo;
    +      }
    +    }
    +    int measureExistIndex = 0;
    +    for (int i = 0; i < queryMeasures.length; i++) {
    +      if (!measureInfo.getMeasureExists()[i]) {
    +        // add a dummy column vector result collector object
    +        ColumnVectorInfo columnVectorInfo = new ColumnVectorInfo();
    +        allColumnInfo[queryMeasures[i].getQueryOrder()] = columnVectorInfo;
    +        continue;
    +      }
    +      QueryMeasure currentBlockMeasure =
    +          tableBlockExecutionInfos.getQueryMeasures()[measureExistIndex++];
    +      ColumnVectorInfo columnVectorInfo = new ColumnVectorInfo();
    +      columnVectorInfo.measureVectorFiller = MeasureDataVectorProcessor.MeasureVectorFillerFactory
    +          .getMeasureVectorFiller(currentBlockMeasure.getMeasure().getDataType());
    +      columnVectorInfo.ordinal = currentBlockMeasure.getMeasure().getOrdinal();
    +      columnVectorInfo.measure = currentBlockMeasure;
    +      this.measureColumnInfo[i] = columnVectorInfo;
    +      allColumnInfo[queryMeasures[i].getQueryOrder()] = columnVectorInfo;
    +    }
    +    dictionaryInfo = dictInfoList.toArray(new ColumnVectorInfo[dictInfoList.size()]);
    +    noDictionaryInfo = noDictInfoList.toArray(new ColumnVectorInfo[noDictInfoList.size()]);
    +    complexInfo = complexList.toArray(new ColumnVectorInfo[complexList.size()]);
    +    Arrays.sort(dictionaryInfo);
    +    Arrays.sort(noDictionaryInfo);
    +    Arrays.sort(complexInfo);
    +  }
    +
    +  @Override public List<Object[]> collectData(AbstractScannedResult scannedResult, int batchSize) {
    +    throw new UnsupportedOperationException("collectData is not supported here");
    +  }
    +
    +  @Override public void collectVectorBatch(AbstractScannedResult scannedResult,
    +      CarbonColumnarBatch columnarBatch) {
    +    int numberOfPages = scannedResult.numberOfpages();
    +    while (scannedResult.getCurrentPageCounter() < numberOfPages) {
    +      int currentPageRowCount = scannedResult.getCurrentPageRowCount();
    +      if (currentPageRowCount == 0) {
    +        scannedResult.incrementPageCounter();
    +        continue;
    +      }
    +      int rowCounter = scannedResult.getRowCounter();
    +      int availableRows = currentPageRowCount - rowCounter;
    +      int requiredRows = columnarBatch.getBatchSize() - columnarBatch.getActualSize();
    +      requiredRows = Math.min(requiredRows, availableRows);
    +      if (requiredRows < 1) {
    +        return;
    +      }
    +      for (int i = 0; i < allColumnInfo.length; i++) {
    +        allColumnInfo[i].size = requiredRows;
    +        allColumnInfo[i].offset = rowCounter;
    +        allColumnInfo[i].vectorOffset = columnarBatch.getRowCounter();
    +        allColumnInfo[i].vector = columnarBatch.columnVectors[i];
    +      }
    +
    +      scannedResult.fillColumnarDictionaryBatch(dictionaryInfo);
    +      scannedResult.fillColumnarNoDictionaryBatch(noDictionaryInfo);
    +      scannedResult.fillColumnarMeasureBatch(measureColumnInfo, measureInfo.getMeasureOrdinals());
    +      scannedResult.fillColumnarComplexBatch(complexInfo);
    +      // fill default values for non existing dimensions and measures
    +      fillDataForNonExistingDimensions();
    +      fillDataForNonExistingMeasures();
    +      // it means fetched all data out of page so increment the page counter
    +      if (availableRows == requiredRows) {
    +        scannedResult.incrementPageCounter();
    +      } else {
    +        // Or set the row counter.
    +        scannedResult.setRowCounter(rowCounter + requiredRows);
    +      }
    +      columnarBatch.setActualSize(columnarBatch.getActualSize() + requiredRows);
    +      columnarBatch.setRowCounter(columnarBatch.getRowCounter() + requiredRows);
    +    }
    +  }
    +
    +  /**
    +   * This method will fill the default values of non existing dimensions in the current block
    +   */
    +  private void fillDataForNonExistingDimensions() {
    +    for (int i = 0; i < tableBlockExecutionInfos.getActualQueryDimensions().length; i++) {
    +      if (!dimensionInfo.getDimensionExists()[i]) {
    +        CarbonDimension dimension =
    +            tableBlockExecutionInfos.getActualQueryDimensions()[i].getDimension();
    +        if (dimension.hasEncoding(Encoding.DIRECT_DICTIONARY)) {
    +          // fill direct dictionary column data
    +          fillDirectDictionaryData(allColumnInfo[i].vector, allColumnInfo[i],
    +              dimensionInfo.getDefaultValues()[i]);
    +        } else if (dimension.hasEncoding(Encoding.DICTIONARY)) {
    +          // fill dictionary column data
    +          fillDictionaryData(allColumnInfo[i].vector, allColumnInfo[i],
    +              dimensionInfo.getDefaultValues()[i]);
    +        } else {
    +          // fill no dictionary data
    +          fillNoDictionaryData(allColumnInfo[i].vector, allColumnInfo[i],
    +              dimension.getDefaultValue());
    +        }
    +      }
    +    }
    +  }
    +
    +  /**
    +   * This method will fill the dictionary column data
    +   *
    +   * @param vector
    +   * @param columnVectorInfo
    +   *
    +   * @param defaultValue
    +   */
    +  private void fillDictionaryData(CarbonColumnVector vector, ColumnVectorInfo columnVectorInfo,
    +      Object defaultValue) {
    +    int offset = columnVectorInfo.offset;
    +    int vectorOffset = columnVectorInfo.vectorOffset;
    +    int len = columnVectorInfo.size + offset;
    +    for (int j = offset; j < len; j++) {
    +      vector.putInt(vectorOffset++, (int) defaultValue);
    +    }
    +  }
    +
    +  /**
    +   * This method will fill the direct dictionary column data
    +   *
    +   * @param vector
    +   * @param columnVectorInfo
    +   * @param defaultValue
    +   */
    +  private void fillDirectDictionaryData(CarbonColumnVector vector,
    +      ColumnVectorInfo columnVectorInfo, Object defaultValue) {
    +    int offset = columnVectorInfo.offset;
    +    int vectorOffset = columnVectorInfo.vectorOffset;
    +    int len = columnVectorInfo.size + offset;
    +    for (int j = offset; j < len; j++) {
    --- End diff --
   
    Directly use default value fill method of ColumnVector to put at once.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata pull request #641: [CARBONDATA-767] Alter table support...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/641#discussion_r105883959
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/executor/impl/AbstractQueryExecutor.java ---
    @@ -429,9 +396,10 @@ protected BlockExecutionInfo getBlockExecutionInfoForBlock(QueryModel queryModel
        * @param blockMetadataInfo block metadata info
        * @return key size
        */
    -  private int getKeySize(List<QueryDimension> queryDimension, SegmentProperties blockMetadataInfo) {
    -    List<Integer> fixedLengthDimensionOrdinal =
    -        new ArrayList<Integer>(CarbonCommonConstants.DEFAULT_COLLECTION_SIZE);
    +  private int getKeySize(List<QueryDimension> queryDimension,
    +      SegmentProperties blockMetadataInfo) {
    +    Set<Integer> fixedLengthDimensionOrdinal =
    --- End diff --
   
    Comment above to tell its purpose


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata pull request #641: [CARBONDATA-767] Alter table support...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/641#discussion_r105883296
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/executor/impl/AbstractQueryExecutor.java ---
    @@ -353,12 +312,15 @@ protected BlockExecutionInfo getBlockExecutionInfoForBlock(QueryModel queryModel
         } else {
           blockExecutionInfo.setAllSelectedDimensionBlocksIndexes(new int[0][0]);
         }
    +    // get the list of updated filter measures present in the current block
    +    Set<CarbonMeasure> updatedFilterMeasures = QueryUtil
    +        .getUpdatedFilterMeasures(queryProperties.filterMeasures, segmentProperties.getMeasures());
         // list of measures to be projected
    -    List<Integer> allProjectionListMeasureIdexes = new ArrayList<>();
    +    List<Integer> allProjectionListMeasureIndexes = new ArrayList<>();
         int[] measureBlockIndexes = QueryUtil
    -        .getMeasureBlockIndexes(queryModel.getQueryMeasures(), expressionMeasures,
    -            segmentProperties.getMeasuresOrdinalToBlockMapping(), queryProperties.filterMeasures,
    -            allProjectionListMeasureIdexes);
    +        .getMeasureBlockIndexes(updatedQueryMeasures, expressionMeasures,
    --- End diff --
   
    Change name to currentBlockMeasures and Dimentions


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-carbondata pull request #641: [CARBONDATA-767] Alter table support...

qiuchenjian-2
In reply to this post by qiuchenjian-2
Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/641#discussion_r105887002
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/executor/util/QueryUtil.java ---
    @@ -249,6 +248,25 @@
       }
     
       /**
    +   * This method will return the key ordinal of the query dimension from the current block
    +   *
    +   * @param blockDimensions
    +   * @param queryDimension
    +   * @return
    +   */
    +  public static int getKeyOrdinalOfDimensionFromCurrentBlock(List<CarbonDimension> blockDimensions,
    --- End diff --
   
    Use CarbonUtil.getDimentionFromCurrentBlock, add getter to SegmentProperties


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
1234