Apache CarbonData Dev Mailing List archive › Apache CarbonData JIRA issues

[GitHub] carbondata pull request #2504: [CARBONDATA-2740]flat folder handling for imp...

Classic

List

Threaded

10 messages Options

qiuchenjian-2

[GitHub] carbondata pull request #2504: [CARBONDATA-2740]flat folder handling for imp...

GitHub user akashrn5 opened a pull request:

https://github.com/apache/carbondata/pull/2504

[CARBONDATA-2740]flat folder handling for implicit column and other bug fix

### Problem
1) When flat folder is enabled for table, pruning is not happening for implicit column.
2) error message is wrong when complex column which does not have any string datatype column is given in local dictionary and exclude column.
3) when data load is failed, corresponding segment file is not getting deleted.

### Solution:
1) when flat folder is enabled, based on the segment number present in the carbondata file name, pruning is taken care
2) error message is corrected
3) when data load is failed, segment file will be deleted

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/akashrn5/incubator-carbondata implicit

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2504.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2504

----
commit 037769a4ccedd9a5a76cf1c9992a36b99cb4926d
Author: akashrn5 <akashnilugal@...>
Date: 2018-07-13T09:04:42Z

flat folder handling for implicit column and other bug fix

----

---

qiuchenjian-2

[GitHub] carbondata pull request #2504: [CARBONDATA-2740]flat folder handling for imp...

Github user KanakaKumar commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2504#discussion_r202329272

--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonRelation.scala ---
@@ -173,15 +172,38 @@ case class CarbonRelation(
.getValidAndInvalidSegments.getValidSegments.asScala
var size = 0L
// for each segment calculate the size
- segments.foreach {validSeg =>
- // for older store
- if (null != validSeg.getLoadMetadataDetails.getDataSize &&
- null != validSeg.getLoadMetadataDetails.getIndexSize) {
- size = size + validSeg.getLoadMetadataDetails.getDataSize.toLong +
- validSeg.getLoadMetadataDetails.getIndexSize.toLong
- } else {
- size = size + FileFactory.getDirectorySize(
- CarbonTablePath.getSegmentPath(tablePath, validSeg.getSegmentNo))
+ if (carbonTable.getTableInfo.getFactTable.getTableProperties.asScala
+ .get(CarbonCommonConstants.FLAT_FOLDER).isDefined &&
+ carbonTable.getTableInfo.getFactTable.getTableProperties.asScala
+ (CarbonCommonConstants.FLAT_FOLDER).toBoolean) {
+ val tableDirectorySize = FileFactory.getDirectorySize(carbonTable.getTablePath)
+ val metaDirectorySize = FileFactory.getDirectorySize(carbonTable.getMetadataPath)
+ val factDirectorySize = FileFactory
+ .getDirectorySize(CarbonTablePath.getFactDir(carbonTable.getTablePath))
+ val lockDirSize = FileFactory
+ .getDirectorySize(CarbonTablePath.getLockFilesDirPath(carbonTable.getTablePath))
+ val datamaps = DataMapStoreManager.getInstance().getAllDataMap(carbonTable)
+ var datamapsDirectorySize = 0L
+ if (datamaps.size() > 0) {
+ datamaps.asScala.foreach { datamap =>
+ datamapsDirectorySize = datamapsDirectorySize + FileFactory
+ .getDirectorySize(
+ carbonTable.getTablePath + datamap.getDataMapSchema.getDataMapName)
--- End diff --

Please use a utility method to make to dm path like org.apache.carbondata.core.util.path.CarbonTablePath#getDataMapStorePath.

---

qiuchenjian-2