Vectorized reader exceptions

Posted by Swapnil Shinde on Jul 20, 2017; 4:10pm
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Vectorized-reader-exceptions-tp18603.html

Hi All

    I am not sure if this is random exception but this is what I have observed -

Create and load carbondata table from Spark dataframe - Without dictionary_include on two INT columns - Works fine. I can "select *" on it.

Create and load carbondata table from same Spark datafrmae - With dictionary_include on two INT columns - Throws NullPointerException when I run "select *".

Stacktrace throws error here -

https://github.com/apache/carbondata/blob/branch-1.1/core/src/main/java/org/apache/carbondata/core/datastore/chunk/store/impl/safe/SafeVariableLengthDimensionDataChunkStore.java#L140

I found similar error fixed in this PR but not available in 1.1.0 version - https://github.com/apache/carbondata/pull/970

I kind of tried adding above PR's changes with additional necessary changes to build successfully. (Not sure if it was a good idea)

If I load again and run "select *"  it throws different error -

I am not much familiar with vectorized reader source code so not sure how to fix this. However, due to above error I have to load that dataset without creating dimensions on columns I need for filtration.

As always, your help is very much appreciated!

Thanks

Swapnil