Apache CarbonData Dev Mailing List archive › Apache CarbonData JIRA issues

[GitHub] carbondata pull request #2787: [HOTFIX]fixed filter mismatch issue

Classic

List

Threaded

25 messages Options

qiuchenjian-2

[GitHub] carbondata pull request #2787: [HOTFIX]fixed filter mismatch issue

GitHub user kumarvishal09 opened a pull request:

https://github.com/apache/carbondata/pull/2787

[HOTFIX]fixed filter mismatch issue

**Problem:**
Between query is giving wrong result.
**Root cause:**
For timestamp time when filter is given in yyyy-mm-dd format instead of yyyy-mm-dd HH:MM:SS format it will add cast, In CastExpressionOptimization it is using SimpleDateFormat object to parse the filter value which is failing as filter values is not same.
**Solution:**
Use SPARK:DateTimeUtils.stringToTime method as spark is handling for above scenario.

- [ ] Any interfaces changed?

- [ ] Any backward compatibility impacted?

- [ ] Document update required?

- [ ] Testing done
Added UT

- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kumarvishal09/incubator-carbondata filterfix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2787.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2787

----
commit e14820f87b316282edcd9d510a99539cfcfbd52b
Author: kumarvishal09 <kumarvishal1802@...>
Date: 2018-09-28T13:03:29Z

fixed filter mismatch issue

----

---

qiuchenjian-2

[GitHub] carbondata pull request #2787: [HOTFIX] Fixed Between Query Data Mismatch is...

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2787#discussion_r221258708

--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/directdictionary/TimestampNoDictionaryColumnCastTestCase.scala ---
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.spark.testsuite.directdictionary
+
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+
+/**
+ * Test Class for detailed query on timestamp datatypes
+ */
+class TimestampNoDictionaryColumnCastTestCase extends QueryTest with BeforeAndAfterAll {
+
+ override def beforeAll {
+ CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+ CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT)
+
+ sql("drop table if exists timestamp_nodictionary")
+ sql(
+ """
+ CREATE TABLE IF NOT EXISTS timestamp_nodictionary
--- End diff --

Since you unified for timestamp and date, please add testcase for date as well.

---

qiuchenjian-2