akashrn5 commented on a change in pull request #3410: [CARBONDATA-3552]Fix dataload failure for column added in 1.1 which is a sort column in latest version.
URL:
https://github.com/apache/carbondata/pull/3410#discussion_r337456556
##########
File path: processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/holder/UnsafeSortTempFileChunkHolder.java
##########
@@ -105,18 +106,24 @@
* Constructor to initialize
*/
public UnsafeSortTempFileChunkHolder(File tempFile, SortParameters parameters,
- boolean convertNoSortFields) {
+ boolean convertNoSortFields, TableFieldStat tableFieldStat) {
// set temp file
this.tempFile = tempFile;
this.readBufferSize = parameters.getBufferSize();
this.compressorName = parameters.getSortTempCompressorName();
- this.tableFieldStat = new TableFieldStat(parameters);
+ this.tableFieldStat = tableFieldStat;
this.sortStepRowHandler = new SortStepRowHandler(tableFieldStat);
this.executorService = Executors.newFixedThreadPool(1);
- comparator = new IntermediateSortTempRowComparator(parameters.getNoDictionarySortColumn(),
- parameters.getNoDictDataType());
this.convertNoSortFields = convertNoSortFields;
this.sortTempRowUpdater = tableFieldStat.getSortTempRowUpdater();
+ if (!this.convertNoSortFields) {
Review comment:
new comparator comes into picture when the carbonrow is updated to schema order, so the sort column went to last, so only in case of final merging only this comparator is used, else it should go to current comparator as before final merge the carbon row has all sort columns at beginning only .(This is done with the help of method updateDataFieldsBasedOnSortColumns)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[hidden email]
With regards,
Apache Git Services
Locked
