GitHub user QiangCai opened a pull request:
https://github.com/apache/carbondata/pull/1572 [CARBONDATA-1812] Provide API to get table dynamic information(table size and last modified time) provide API to get table dynamic information(table size and last modified time) - [x] Any interfaces changed? new interface, don't change old interface - [x] Any backward compatibility impacted? no - [x] Document update required? no - [x] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? added - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. small change You can merge this pull request into a Git repository by running: $ git pull https://github.com/QiangCai/carbondata get_table_info Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1572.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1572 ---- commit 4f1e95e84417bc0d266371980aa6cdb5fa95c35c Author: QiangCai <[hidden email]> Date: 2017-11-27T02:01:44Z implement GetTableInfoCommand ---- --- |
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1572#discussion_r153136866 --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/GetTableDetailCommand.scala --- @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.execution.command.schema + +import org.apache.spark.sql.{CarbonEnv, Row, SparkSession} +import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference} +import org.apache.spark.sql.execution.command.{DataProcessCommand, RunnableCommand} +import org.apache.spark.sql.types.{LongType, StringType} + +import org.apache.carbondata.core.constants.CarbonCommonConstants +import org.apache.carbondata.core.metadata.schema.table.CarbonTable +import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier +import org.apache.carbondata.core.statusmanager.SegmentStatusManager +import org.apache.carbondata.core.util.{CarbonProperties, CarbonUtil} + +/** + * collect dynamic detail information of the table, including table size, last modified time, etc. + */ +case class GetTableDetailCommand( --- End diff -- Add Carbon prefix --- |
In reply to this post by qiuchenjian-2
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1572#discussion_r153137101 --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/GetTableDetailCommand.scala --- @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.execution.command.schema + +import org.apache.spark.sql.{CarbonEnv, Row, SparkSession} +import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference} +import org.apache.spark.sql.execution.command.{DataProcessCommand, RunnableCommand} +import org.apache.spark.sql.types.{LongType, StringType} + +import org.apache.carbondata.core.constants.CarbonCommonConstants +import org.apache.carbondata.core.metadata.schema.table.CarbonTable +import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier +import org.apache.carbondata.core.statusmanager.SegmentStatusManager +import org.apache.carbondata.core.util.{CarbonProperties, CarbonUtil} + +/** + * collect dynamic detail information of the table, including table size, last modified time, etc. + */ +case class GetTableDetailCommand( + databaseName: String, + tableNames: Seq[String]) + extends RunnableCommand with DataProcessCommand { + + override def run(sparkSession: SparkSession): Seq[Row] = { + processData(sparkSession) + } + + override def processData(sparkSession: SparkSession): Seq[Row] = { + val storePath = CarbonProperties.getStorePath + tableNames.map { tablename => + val absoluteTableIdentifier = + AbsoluteTableIdentifier.from(storePath, databaseName.toLowerCase, tablename.toLowerCase) + val carbonTableIdentifier = absoluteTableIdentifier.getCarbonTableIdentifier + val carbonTable = CarbonEnv.getCarbonTable(Option(carbonTableIdentifier.getDatabaseName), + carbonTableIdentifier.getTableName)(sparkSession) + + Row( + tablename, + getSizeInBytes(sparkSession, carbonTable), + getLastModifiedTime(absoluteTableIdentifier) + ) + } + } + + private def getLastModifiedTime(absoluteTableIdentifier: AbsoluteTableIdentifier): Long = { + SegmentStatusManager.getTableStatusLastModifiedTime(absoluteTableIdentifier) + } + + private def getSizeInBytes( --- End diff -- Add this in CarbonTable class --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1572 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1901/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1572 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1484/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1572 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1487/ --- |
In reply to this post by qiuchenjian-2
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1572 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1491/ --- |
In reply to this post by qiuchenjian-2
|
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1572 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1905/ --- |
In reply to this post by qiuchenjian-2
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1572 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1909/ --- |
Free forum by Nabble | Edit this page |