[GitHub] carbondata pull request #2209: [WIP][Non Transactional Table]Avro Record Com...

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2209#discussion_r184962372
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java ---
    @@ -171,9 +175,9 @@ public void fillCardinality(List<Integer> dimCardWithComplex) {
       /**
        * parse byte array and bit pack
        */
    -  @Override
    -  public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
    -      KeyGenerator[] generator) throws IOException, KeyGenException {
    +  @Override public void parseAndBitPack(ByteBuffer byteArrayInput,
    --- End diff --
   
    Removed


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2209#discussion_r184962411
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java ---
    @@ -183,7 +187,27 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
         for (int i = 0; i < dataLength; i++) {
           children.parseAndBitPack(byteArrayInput, dataOutputStream, generator);
         }
    +  }
    +
    +  @Override
    +  public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
    +      KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
    --- End diff --
   
    Removed


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2209#discussion_r184963324
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java ---
    @@ -183,7 +187,27 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
         for (int i = 0; i < dataLength; i++) {
           children.parseAndBitPack(byteArrayInput, dataOutputStream, generator);
         }
    +  }
    +
    +  @Override
    +  public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
    +      KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
    +      throws IOException, KeyGenException {
    +    int dataLength = byteArrayInput.getInt();
    +    startOffset += Integer.SIZE / Byte.SIZE;
     
    +    dataOutputStream.writeInt(dataLength);
    +    if (children instanceof PrimitiveDataType) {
    +      if (children.getIsColumnDictionary()) {
    +        dataOutputStream.writeInt(generator[children.getSurrogateIndex()].getKeySizeInBytes());
    +        startOffset += Integer.SIZE / Byte.SIZE;
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2209#discussion_r184963673
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/GenericDataType.java ---
    @@ -71,6 +71,8 @@ void writeByteArray(T input, DataOutputStream dataOutputStream)
        */
       void setSurrogateIndex(int surrIndex);
     
    +  Boolean getIsColumnDictionary();
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2209#discussion_r184963740
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/GenericDataType.java ---
    @@ -80,7 +82,13 @@ void writeByteArray(T input, DataOutputStream dataOutputStream)
        * @throws KeyGenException
        */
       void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
    -      KeyGenerator[] generator) throws IOException, KeyGenException;
    +      KeyGenerator[] generator)
    +      throws IOException, KeyGenException;
    +
    +
    +  int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
    +      KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
    +      throws IOException, KeyGenException;
    --- End diff --
   
    complexDictionaryIndentification is removed but StartOffset is used.


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2209#discussion_r184964228
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
    @@ -94,6 +99,11 @@
     
       private CarbonDimension carbonDimension;
     
    +  private Boolean isDictionary;
    --- End diff --
   
    done


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2209#discussion_r184964754
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
    @@ -94,6 +99,11 @@
     
       private CarbonDimension carbonDimension;
     
    +  private Boolean isDictionary;
    +
    +  private FieldConverter fieldConverterForNoDictionary;
    --- End diff --
   
    Removed


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2209#discussion_r184968935
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
    @@ -245,6 +312,28 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
         dataOutputStream.write(v);
       }
     
    +  @Override
    +  public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
    +      KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
    --- End diff --
   
    Removed


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2209#discussion_r184968977
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
    @@ -245,6 +312,28 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
         dataOutputStream.write(v);
       }
     
    +  @Override
    +  public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
    +      KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
    --- End diff --
   
    Currently no required. Removed


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2209#discussion_r184969127
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/CarbonDataLoadConfiguration.java ---
    @@ -348,6 +356,33 @@ public void setCardinalityFinder(DictionaryCardinalityFinder cardinalityFinder)
         return complexKeyGenerators;
       }
     
    +  public Boolean[][] createComplexDictionaryFieldIdentification() {
    --- End diff --
   
    removed


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2209#discussion_r184975951
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/CarbonDataLoadConfiguration.java ---
    @@ -348,6 +356,33 @@ public void setCardinalityFinder(DictionaryCardinalityFinder cardinalityFinder)
         return complexKeyGenerators;
       }
     
    +  public Boolean[][] createComplexDictionaryFieldIdentification() {
    --- End diff --
   
    Removed


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2209#discussion_r184975993
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
    @@ -245,6 +312,28 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
         dataOutputStream.write(v);
       }
     
    +  @Override
    +  public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
    +      KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
    --- End diff --
   
    Removed


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2209#discussion_r184976034
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/store/CarbonFactDataHandlerModel.java ---
    @@ -609,5 +616,13 @@ public short getWritingCoresCount() {
       public DataMapWriterListener getDataMapWriterlistener() {
         return dataMapWriterlistener;
       }
    +
    +  public Boolean[][] getComplexDictionaryFields() {
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2209#discussion_r184976110
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
    @@ -245,6 +312,28 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
         dataOutputStream.write(v);
       }
     
    +  @Override
    +  public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
    +      KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
    +      throws IOException, KeyGenException {
    +    if (!this.isDictionary) {
    +      int sizeOfData = byteArrayInput.getInt();
    +      startOffset += Integer.SIZE / Byte.SIZE;
    --- End diff --
   
    Added the constant


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2209#discussion_r184976155
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
    @@ -245,6 +312,28 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
         dataOutputStream.write(v);
       }
     
    +  @Override
    +  public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
    +      KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
    --- End diff --
   
    Removed


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2209#discussion_r184976193
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java ---
    @@ -183,7 +187,27 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
         for (int i = 0; i < dataLength; i++) {
           children.parseAndBitPack(byteArrayInput, dataOutputStream, generator);
         }
    +  }
    +
    +  @Override
    +  public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
    +      KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
    +      throws IOException, KeyGenException {
    +    int dataLength = byteArrayInput.getInt();
    +    startOffset += Integer.SIZE / Byte.SIZE;
     
    +    dataOutputStream.writeInt(dataLength);
    +    if (children instanceof PrimitiveDataType) {
    +      if (children.getIsColumnDictionary()) {
    +        dataOutputStream.writeInt(generator[children.getSurrogateIndex()].getKeySizeInBytes());
    +        startOffset += Integer.SIZE / Byte.SIZE;
    --- End diff --
   
    Added a constant


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2209#discussion_r184976246
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java ---
    @@ -183,7 +187,27 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
         for (int i = 0; i < dataLength; i++) {
           children.parseAndBitPack(byteArrayInput, dataOutputStream, generator);
         }
    +  }
    +
    +  @Override
    +  public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
    +      KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
    +      throws IOException, KeyGenException {
    +    int dataLength = byteArrayInput.getInt();
    +    startOffset += Integer.SIZE / Byte.SIZE;
    --- End diff --
   
    Added CarbonCommonConstant


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2209#discussion_r184976615
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java ---
    @@ -183,7 +187,27 @@ public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutp
         for (int i = 0; i < dataLength; i++) {
           children.parseAndBitPack(byteArrayInput, dataOutputStream, generator);
         }
    +  }
    +
    +  @Override
    +  public int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
    +      KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
    --- End diff --
   
    removed complexDictionaryIndentification


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2209#discussion_r184976629
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/GenericDataType.java ---
    @@ -71,6 +71,8 @@ void writeByteArray(T input, DataOutputStream dataOutputStream)
        */
       void setSurrogateIndex(int surrIndex);
     
    +  Boolean getIsColumnDictionary();
    --- End diff --
   
    Done


---

[qiuchenjian-2]

Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2209#discussion_r184976649
 
    --- Diff: processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java ---
    @@ -94,6 +99,11 @@
     
       private CarbonDimension carbonDimension;
     
    +  private Boolean isDictionary;
    --- End diff --
   
    Done


---