Posted by
Akash R Nilugal (Jira) on
Sep 28, 2016; 2:31pm
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/jira-Commented-CARBONDATA-279-DataLoading-Save-a-DataFrame-to-CarbonData-file-without-writing-CSV-fie-tp1545.html
[
https://issues.apache.org/jira/browse/CARBONDATA-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15529818#comment-15529818 ]
ASF GitHub Bot commented on CARBONDATA-279:
-------------------------------------------
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/203#discussion_r80915025
--- Diff: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataLoadRDD.scala ---
@@ -145,92 +232,28 @@ class CarbonDataLoadRDD[K, V](
override def compute(theSplit: Partition, context: TaskContext): Iterator[(K, V)] = {
val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
val iter = new Iterator[(K, V)] {
- var dataloadStatus = CarbonCommonConstants.STORE_LOADSTATUS_FAILURE
var partitionID = "0"
+ val loadMetadataDetails = new LoadMetadataDetails()
var model: CarbonLoadModel = _
var uniqueLoadStatusId = carbonLoadModel.getTableName + CarbonCommonConstants.UNDERSCORE +
theSplit.index
try {
- val carbonPropertiesFilePath = System.getProperty("carbon.properties.filepath", null)
- if (null == carbonPropertiesFilePath) {
- System.setProperty("carbon.properties.filepath",
- System.getProperty("user.dir") + '/' + "conf" + '/' + "carbon.properties")
- }
+ loadMetadataDetails.setPartitionCount(partitionID)
+ loadMetadataDetails.setLoadStatus(CarbonCommonConstants.STORE_LOADSTATUS_FAILURE)
+
carbonLoadModel.setSegmentId(String.valueOf(loadCount))
setModelAndBlocksInfo()
- CarbonTimeStatisticsFactory.getLoadStatisticsInstance.initPartitonInfo(model.getPartitionId)
- CarbonProperties.getInstance().addProperty("carbon.is.columnar.storage", "true")
- CarbonProperties.getInstance().addProperty("carbon.dimension.split.value.in.columnar", "1")
- CarbonProperties.getInstance().addProperty("carbon.is.fullyfilled.bits", "true")
- CarbonProperties.getInstance().addProperty("is.int.based.indexer", "true")
- CarbonProperties.getInstance().addProperty("aggregate.columnar.keyblock", "true")
- CarbonProperties.getInstance().addProperty("high.cardinality.value", "100000")
- CarbonProperties.getInstance().addProperty("is.compressed.keyblock", "false")
- CarbonProperties.getInstance().addProperty("carbon.leaf.node.size", "120000")
-
- // this property is used to determine whether temp location for carbon is inside
- // container temp dir or is yarn application directory.
- val carbonUseLocalDir = CarbonProperties.getInstance()
- .getProperty("carbon.use.local.dir", "false")
-
- if(carbonUseLocalDir.equalsIgnoreCase("true")) {
- val storeLocations = CarbonLoaderUtil.getConfiguredLocalDirs(SparkEnv.get.conf)
- if (null != storeLocations && storeLocations.nonEmpty) {
- storeLocation = storeLocations(Random.nextInt(storeLocations.length))
- }
- if (storeLocation == null) {
- storeLocation = System.getProperty("java.io.tmpdir")
- }
- }
- else {
- storeLocation = System.getProperty("java.io.tmpdir")
- }
- storeLocation = storeLocation + '/' + System.nanoTime() + '/' + theSplit.index
- dataloadStatus = CarbonCommonConstants.STORE_LOADSTATUS_SUCCESS
-
+ storeLocation = CarbonDataLoadRDD.initialize(model, theSplit.index)
+ loadMetadataDetails.setLoadStatus(CarbonCommonConstants.STORE_LOADSTATUS_SUCCESS)
if (model.isRetentionRequest) {
recreateAggregationTableForRetention
}
else if (model.isAggLoadRequest) {
--- End diff --
Is this required? If it is not required, remove it.
> [DataLoading]Save a DataFrame to CarbonData file without writing CSV file
> -------------------------------------------------------------------------
>
> Key: CARBONDATA-279
> URL:
https://issues.apache.org/jira/browse/CARBONDATA-279> Project: CarbonData
> Issue Type: Improvement
> Affects Versions: 0.1.0-incubating
> Reporter: QiangCai
> Assignee: QiangCai
> Priority: Minor
> Fix For: 0.2.0-incubating
>
>
> Directly save a DataFrame to CarbonData file without writing CSV file
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)