Apache CarbonData Dev Mailing List archive › Apache CarbonData JIRA issues

[GitHub] carbondata pull request #3029: [CARBONDATA-3200] No-Sort compaction

Classic

List

Threaded

84 messages Options

12345

qiuchenjian-2

[GitHub] carbondata issue #3029: [CARBONDATA-3200] No-Sort compaction

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/3029

Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10383/

---

qiuchenjian-2

[GitHub] carbondata issue #3029: [CARBONDATA-3200] No-Sort compaction

In reply to this post by qiuchenjian-2

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/3029

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2335/

---

qiuchenjian-2

[GitHub] carbondata issue #3029: [CARBONDATA-3200] No-Sort compaction

In reply to this post by qiuchenjian-2

qiuchenjian-2

[GitHub] carbondata issue #3029: [CARBONDATA-3200] No-Sort compaction

In reply to this post by qiuchenjian-2

qiuchenjian-2

[GitHub] carbondata issue #3029: [CARBONDATA-3200] No-Sort compaction

In reply to this post by qiuchenjian-2

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/3029

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2137/

---

qiuchenjian-2

[GitHub] carbondata issue #3029: [CARBONDATA-3200] No-Sort compaction

In reply to this post by qiuchenjian-2

qiuchenjian-2

[GitHub] carbondata issue #3029: [CARBONDATA-3200] No-Sort compaction

In reply to this post by qiuchenjian-2

qiuchenjian-2

[GitHub] carbondata pull request #3029: [CARBONDATA-3200] No-Sort compaction

In reply to this post by qiuchenjian-2

Github user kumarvishal09 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/3029#discussion_r244921991

--- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/CarbonMergerRDD.scala ---
@@ -180,10 +180,10 @@ class CarbonMergerRDD[K, V](
context.addTaskCompletionListener { _ =>
close()
}
- try {
--- End diff --

Revert this change

---

qiuchenjian-2

[GitHub] carbondata issue #3029: [CARBONDATA-3200] No-Sort compaction

In reply to this post by qiuchenjian-2

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/3029

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2138/

---

qiuchenjian-2

[GitHub] carbondata pull request #3029: [CARBONDATA-3200] No-Sort compaction

In reply to this post by qiuchenjian-2

Github user kumarvishal09 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/3029#discussion_r244923003

--- Diff: core/src/main/java/org/apache/carbondata/core/util/DataFileFooterConverterV3.java ---
@@ -73,6 +73,11 @@ public DataFileFooter convertDataFileFooter(FileHeader fileHeader, FileFooter3 f
dataFileFooter.setNumberOfRows(footer.getNum_rows());
dataFileFooter.setSegmentInfo(getSegmentInfo(footer.getSegment_info()));
dataFileFooter.setSchemaUpdatedTimeStamp(fileHeader.getTime_stamp());
+ if (footer.isSetIs_sort()) {
+ dataFileFooter.setIsSorted(footer.isIs_sort());
+ } else {
+ dataFileFooter.setIsSorted(false);
--- End diff --

for older file if you are setting false then use table sort scope

---

qiuchenjian-2

[GitHub] carbondata issue #3029: [CARBONDATA-3200] No-Sort compaction

In reply to this post by qiuchenjian-2

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/3029

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2344/

---

qiuchenjian-2

[GitHub] carbondata issue #3029: [CARBONDATA-3200] No-Sort compaction

In reply to this post by qiuchenjian-2

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/3029

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10392/

---

qiuchenjian-2

[GitHub] carbondata issue #3029: [CARBONDATA-3200] No-Sort compaction

In reply to this post by qiuchenjian-2

qiuchenjian-2

[GitHub] carbondata pull request #3029: [CARBONDATA-3200] No-Sort compaction

In reply to this post by qiuchenjian-2

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/3029#discussion_r244959393

--- Diff: processing/src/main/java/org/apache/carbondata/processing/sort/sortdata/SingleThreadFinalSortFilesMerger.java ---
@@ -114,6 +113,31 @@ public void startFinalMerge() throws CarbonDataWriterException {
startSorting(filesToMerge);
}

+ /**
+ * Below method will be used to add in memory raw result iterator to priority queue.
+ * This will be called in case of compaction, when it is compacting sorted and unsorted
+ * both type of carbon data file
+ * This method will add sorted file's RawResultIterator to priority queue using
+ * InMemorySortTempChunkHolder as wrapper
+ *
+ * @param sortedRawResultMergerList
+ * @param segmentProperties
+ * @param noDicAndComplexColumns
+ * @throws CarbonSortKeyAndGroupByException
+ */
+ public void addInMemoryRawResultIterator(List<RawResultIterator> sortedRawResultMergerList,
+ SegmentProperties segmentProperties, CarbonColumn[] noDicAndComplexColumns,
+ DataType[] measureDataType)
+ throws CarbonSortKeyAndGroupByException {
+ for (RawResultIterator rawResultIterator : sortedRawResultMergerList) {
+ InMemorySortTempChunkHolder inMemorySortTempChunkHolder =
+ new InMemorySortTempChunkHolder(rawResultIterator, segmentProperties,
+ noDicAndComplexColumns, sortParameters, measureDataType);
+ inMemorySortTempChunkHolder.readRow();
--- End diff --

Don't need to check hasNext here before reading the row first time?

---

qiuchenjian-2