[GitHub] carbondata pull request #2954: [CARBONDATA-3128]HiveExample has some excepti...

[qiuchenjian-2]

GitHub user SteNicholas opened a pull request:

    https://github.com/apache/carbondata/pull/2954

    [CARBONDATA-3128]HiveExample has some exception

    [CARBONDATA-3128]HiveExample has some exception
    This pull request fix HiveExample exception with different spark versions.When through intellij idea running HiveExample exception with spark-2.1 or spark-2.3 profile,just update pom.xml of parent,integration/spark2,integration/spark-datasource.
     - [ ] Any interfaces changed?
           No
     - [ ] Any backward compatibility impacted?
           No
     - [ ] Document update required?
           No
     - [ ] Testing done
           No
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
           No


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/SteNicholas/carbondata CARBONDATA-3128

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2954.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2954
   
----
commit b53a09c3e76442ebfb868104478a31cca8b23c52
Author: Nicholas Jiang <programgeek@...>
Date:   2018-11-27T06:51:56Z

    update maven profile

commit 421c244f9197276b994c36bd2927eda88f2b5c34
Author: Nicholas Jiang <programgeek@...>
Date:   2018-11-27T07:00:09Z

    update maven profile

----


---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2954
 
    Can one of the admins verify this patch?


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2954#discussion_r236544211
 
    --- Diff: pom.xml ---
    @@ -545,16 +557,14 @@
         </profile>
         <profile>
           <id>spark-2.2</id>
    -      <activation>
    --- End diff --
   
    If user haven't selected any spark profile after you remove this, what will happen？


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2954#discussion_r236545789
 
    --- Diff: pom.xml ---
    @@ -100,7 +100,6 @@
         <module>processing</module>
         <module>hadoop</module>
         <module>integration/spark-common</module>
    -    <module>integration/spark-datasource</module>
    --- End diff --
   
    Can you describe in PR content why do you need remove this and add it to spark-2.1,2.2,2.3 profile?  


---

[qiuchenjian-2]

Github user xubo245 commented on the issue:

    https://github.com/apache/carbondata/pull/2954
 
    Can you add HiveExample to CI test? please refer examples/spark2/src/test/scala/org/apache/carbondata/examplesCI/RunExamples.scala


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2954#discussion_r236547455
 
    --- Diff: pom.xml ---
    @@ -545,16 +557,14 @@
         </profile>
         <profile>
           <id>spark-2.2</id>
    -      <activation>
    --- End diff --
   
    It seems don't support "mvn clean package" command if you remove this.


---

[qiuchenjian-2]

Github user SteNicholas commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2954#discussion_r236550005
 
    --- Diff: pom.xml ---
    @@ -545,16 +557,14 @@
         </profile>
         <profile>
           <id>spark-2.2</id>
    -      <activation>
    --- End diff --
   
    Because activeByDefault setting limit maven profile only spark-2.2.If there is activeByDefault setting,Intellij idea perform switch profile only add active profile.The default active profile dependencies always exists.But this is not necessary that removing activeByDefault setting is helpful for switching profile in Intellij idea.


---

[qiuchenjian-2]

Github user SteNicholas commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2954#discussion_r236551091
 
    --- Diff: pom.xml ---
    @@ -100,7 +100,6 @@
         <module>processing</module>
         <module>hadoop</module>
         <module>integration/spark-common</module>
    -    <module>integration/spark-datasource</module>
    --- End diff --
   
    Remoing module and adding to spark 2.1,2.2,2.3 is adaption to switching profile performance. This module's pom.xml profiles corresponds to the profile of spark version.


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2954#discussion_r236555834
 
    --- Diff: pom.xml ---
    @@ -100,7 +100,6 @@
         <module>processing</module>
         <module>hadoop</module>
         <module>integration/spark-common</module>
    -    <module>integration/spark-datasource</module>
    --- End diff --
   
    please add it to PR content


---

[qiuchenjian-2]

Github user SteNicholas commented on the issue:

    https://github.com/apache/carbondata/pull/2954
 
    Add RunHiveExample into CI test.


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2954#discussion_r236622117
 
    --- Diff: integration/hive/src/test/scala/org/apache/carbondata/hiveexampleCI/RunHiveExample.scala ---
    @@ -0,0 +1,51 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.hiveexampleCI
    +
    +import org.apache.carbondata.core.constants.CarbonCommonConstants
    +import org.apache.carbondata.core.util.CarbonProperties
    +import org.apache.carbondata.hiveexample.HiveExample
    +import org.apache.spark.sql.test.util.QueryTest
    +import org.scalatest.BeforeAndAfterAll
    +
    +/**
    + * Test suite for examples
    + */
    +
    --- End diff --
   
    Remove this line



---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2954#discussion_r236622183
 
    --- Diff: integration/hive/src/test/scala/org/apache/carbondata/hiveexampleCI/RunHiveExample.scala ---
    @@ -0,0 +1,51 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.hiveexampleCI
    +
    +import org.apache.carbondata.core.constants.CarbonCommonConstants
    +import org.apache.carbondata.core.util.CarbonProperties
    +import org.apache.carbondata.hiveexample.HiveExample
    +import org.apache.spark.sql.test.util.QueryTest
    +import org.scalatest.BeforeAndAfterAll
    +
    +/**
    + * Test suite for examples
    + */
    +
    +class RunHiveExample extends QueryTest with BeforeAndAfterAll {
    +
    +  private val spark = sqlContext.sparkSession
    +
    +  override def beforeAll: Unit = {
    +    CarbonProperties.getInstance().addProperty(
    +      CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
    +      CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT)
    +    CarbonProperties.getInstance().addProperty(
    +      CarbonCommonConstants.CARBON_DATE_FORMAT,
    +      CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT)
    +  }
    +
    +  override def afterAll {
    +    sql("USE default")
    +
    --- End diff --
   
    please remove this line


---

[qiuchenjian-2]

Github user xubo245 commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2954#discussion_r236623474
 
    --- Diff: integration/hive/src/test/scala/org/apache/carbondata/hiveexampleCI/RunHiveExample.scala ---
    @@ -0,0 +1,51 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.hiveexampleCI
    +
    +import org.apache.carbondata.core.constants.CarbonCommonConstants
    +import org.apache.carbondata.core.util.CarbonProperties
    +import org.apache.carbondata.hiveexample.HiveExample
    +import org.apache.spark.sql.test.util.QueryTest
    --- End diff --
   
    Please move the two line to before org.apache.carbondata.*


---

[qiuchenjian-2]

Github user xubo245 commented on the issue:

    https://github.com/apache/carbondata/pull/2954
 
    retest this please


---

[qiuchenjian-2]

Github user xubo245 commented on the issue:

    https://github.com/apache/carbondata/pull/2954
 
    Please optimize the title, like： Fix the HiveExample exception


---

[qiuchenjian-2]

Github user SteNicholas commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2954#discussion_r236627192
 
    --- Diff: integration/hive/src/test/scala/org/apache/carbondata/hiveexampleCI/RunHiveExample.scala ---
    @@ -0,0 +1,51 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.hiveexampleCI
    +
    +import org.apache.carbondata.core.constants.CarbonCommonConstants
    +import org.apache.carbondata.core.util.CarbonProperties
    +import org.apache.carbondata.hiveexample.HiveExample
    +import org.apache.spark.sql.test.util.QueryTest
    --- End diff --
   
    Do it


---

[qiuchenjian-2]

Github user xubo245 commented on the issue:

    https://github.com/apache/carbondata/pull/2954
 
    add to whitelist


---

[qiuchenjian-2]

Github user zzcclp commented on the issue:

    https://github.com/apache/carbondata/pull/2954
 
    add to whitelist


---

[qiuchenjian-2]

Github user zzcclp commented on the issue:

    https://github.com/apache/carbondata/pull/2954
 
    retest this please


---

[qiuchenjian-2]

Github user zzcclp commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2954#discussion_r236634194
 
    --- Diff: integration/hive/src/test/scala/org/apache/carbondata/hiveexampleCI/RunHiveExample.scala ---
    @@ -0,0 +1,43 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.hiveexampleCI
    +
    +import org.apache.spark.sql.test.util.QueryTest
    +import org.scalatest.BeforeAndAfterAll
    +
    +import org.apache.carbondata.core.constants.CarbonCommonConstants
    +import org.apache.carbondata.core.util.CarbonProperties
    +import org.apache.carbondata.hiveexample.HiveExample
    +
    +class RunHiveExample extends QueryTest with BeforeAndAfterAll {
    +
    +  private val spark = sqlContext.sparkSession
    +
    +  override def beforeAll: Unit = {
    +    CarbonProperties.getInstance().addProperty(
    +      CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
    +      CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT)
    +    CarbonProperties.getInstance().addProperty(
    +      CarbonCommonConstants.CARBON_DATE_FORMAT,
    +      CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT)
    +  }
    +
    +  test("HiveExample") {
    +    HiveExample.main(null)
    --- End diff --
   
    Don't call HiveExample.main directly, it better write some test cases.


---