ravipesala commented on a change in pull request #3126: [WIP][CARBONDATA-3293] Prune datamaps improvement
URL:
https://github.com/apache/carbondata/pull/3126#discussion_r258032385
##########
File path: core/src/main/java/org/apache/carbondata/core/indexstore/UnsafeMemoryDMStore.java
##########
@@ -94,87 +101,110 @@ public void addIndexRow(CarbonRowSchema[] schema, DataMapRow indexRow) throws Me
// Check whether allocated memory is sufficient or not.
ensureSize(rowSize);
int pointer = runningLength;
-
+ List<Object> dataPos = new ArrayList<>();
for (int i = 0; i < schema.length; i++) {
- addToUnsafe(schema[i], indexRow, i);
+ addToUnsafe(schema[i], indexRow, i, dataPos);
}
+ dataPosition.put(rowCount, dataPos);
pointers[rowCount++] = pointer;
}
- private void addToUnsafe(CarbonRowSchema schema, DataMapRow row, int index) {
+ private void addToUnsafe(CarbonRowSchema schema, DataMapRow row, int index,
+ List<Object> dataPos) {
switch (schema.getSchemaType()) {
case FIXED:
DataType dataType = schema.getDataType();
if (dataType == DataTypes.BYTE) {
getUnsafe()
.putByte(memoryBlock.getBaseObject(), memoryBlock.getBaseOffset() + runningLength,
row.getByte(index));
- runningLength += row.getSizeInBytes(index);
+ int sizeInBytes = row.getSizeInBytes(index);
+ dataPos.add(sizeInBytes);
Review comment:
Storing in another list is not a good choice it will increase the heap size a lot. We need to come up with better storage layout to traverse row to anywhere. May we store the positions at starting of the key can be better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[hidden email]
With regards,
Apache Git Services
Locked
