[GitHub] incubator-carbondata pull request #653: [CARBONDATA-769] Added codegen suppo...

[qiuchenjian-2]

GitHub user ravipesala opened a pull request:

    https://github.com/apache/incubator-carbondata/pull/653

    [CARBONDATA-769] Added codegen support to spark2.1 carbon dictionary decoder.

   

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ravipesala/incubator-carbondata measure-filter

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-carbondata/pull/653.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #653
   
----
commit 91104245c482b93b8bdf90775a083a401544f26e
Author: ravipesala <[hidden email]>
Date:   2017-03-14T07:44:38Z

    Added codegen support to spark2.1 carbon decoder.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/653
 
    Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1127/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

GitHub user ravipesala reopened a pull request:

    https://github.com/apache/incubator-carbondata/pull/653

    [CARBONDATA-769] Added codegen support to spark2.1 carbon dictionary decoder.

   

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ravipesala/incubator-carbondata measure-filter

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-carbondata/pull/653.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #653
   
----
commit 91104245c482b93b8bdf90775a083a401544f26e
Author: ravipesala <[hidden email]>
Date:   2017-03-14T07:44:38Z

    Added codegen support to spark2.1 carbon decoder.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala closed the pull request at:

    https://github.com/apache/incubator-carbondata/pull/653


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/653#discussion_r105920583
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala ---
    @@ -200,6 +201,123 @@ case class CarbonDictionaryDecoder(
         }
       }
     
    +  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
    +
    +    val storePath = CarbonEnv.get.carbonMetastore.storePath
    +    val absoluteTableIdentifiers = relations.map { relation =>
    +      val carbonTable = relation.carbonRelation.carbonRelation.metaData.carbonTable
    +      (carbonTable.getFactTableName, carbonTable.getAbsoluteTableIdentifier)
    +    }.toMap
    +
    +    if (isRequiredToDecode) {
    +      val cacheProvider: CacheProvider = CacheProvider.getInstance
    +      val forwardDictionaryCache: Cache[DictionaryColumnUniqueIdentifier, Dictionary] =
    +        cacheProvider.createCache(CacheType.FORWARD_DICTIONARY, storePath)
    +      val dicts: Seq[ForwardDictionaryWrapper] = getDictionaryWrapper(absoluteTableIdentifiers,
    +        forwardDictionaryCache, storePath)
    +
    +      val exprs = child.output.map(x =>
    +        ExpressionCanonicalizer.execute(BindReferences.bindReference(x, child.output)))
    +      ctx.currentVars = input
    +      val resultVars = exprs.zipWithIndex.map { e =>
    +        if (dicts(e._2) != null) {
    +          val ev = e._1.genCode(ctx)
    +          val value = ctx.freshName("value")
    +          val valueIntern = ctx.freshName("valueIntern")
    +          val isNull = ctx.freshName("isNull")
    +          val dictsRef = ctx.addReferenceObj("dictsRef", dicts(e._2))
    +          var code =
    +            s"""
    +               |${ev.code}
    +             """.stripMargin
    +          code +=
    +            s"""
    +             |boolean $isNull = false;
    +             |byte[] $valueIntern = $dictsRef.getDictionaryValueForKeyInBytes(${ ev.value });
    +             |if (java.util.Arrays.equals(org.apache.carbondata.core.constants
    +             |.CarbonCommonConstants.MEMBER_DEFAULT_VAL_ARRAY, $valueIntern)) {
    +             |  $isNull = true;
    +             |}
    +             """.stripMargin
    +
    +            val caseCode = getDictionaryColumnIds(e._2)._3 match {
    +              case DataType.INT =>
    +                s"""
    +                   |int $value = Integer.parseInt(new String($valueIntern,
    +                   |org.apache.carbondata.core.constants.CarbonCommonConstants
    +                   |.DEFAULT_CHARSET_CLASS));
    +                 """.stripMargin
    +              case DataType.SHORT =>
    +                s"""
    +                   |short $value =
    +                   |Short.parseShort(new String($valueIntern,
    +                   |org.apache.carbondata.core.constants.CarbonCommonConstants
    +                   |.DEFAULT_CHARSET_CLASS));
    +                 """.stripMargin
    +              case DataType.DOUBLE =>
    +                s"""
    +                   |double $value =
    +                   |Double.parseDouble(new String($valueIntern,
    +                   |org.apache.carbondata.core.constants.CarbonCommonConstants
    +                   |.DEFAULT_CHARSET_CLASS));
    +                 """.stripMargin
    +              case DataType.LONG =>
    +                s"""
    +                   |long $value =
    +                   |Long.parseLong(new String($valueIntern,
    +                   |org.apache.carbondata.core.constants.CarbonCommonConstants
    +                   |.DEFAULT_CHARSET_CLASS));
    +                 """.stripMargin
    +              case DataType.DECIMAL =>
    +                s"""
    +                   |org.apache.spark.sql.types.Decimal $value =
    +                   |Decimal.apply(new java.math.BigDecimal(
    +                   |new String($valueIntern, org.apache.carbondata.core.constants
    +                   |.CarbonCommonConstants.DEFAULT_CHARSET_CLASS)));
    +                 """.stripMargin
    +              case _ =>
    +                s"""
    +                   | UTF8String $value = UTF8String.fromBytes($valueIntern);
    +                 """.stripMargin
    +            }
    +          code +=
    +            s"""
    +               |$caseCode
    +             """.stripMargin
    --- End diff --
   
    simplify it in one line


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/653#discussion_r105921062
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala ---
    @@ -200,6 +201,123 @@ case class CarbonDictionaryDecoder(
         }
       }
     
    +  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
    +
    +    val storePath = CarbonEnv.get.carbonMetastore.storePath
    +    val absoluteTableIdentifiers = relations.map { relation =>
    +      val carbonTable = relation.carbonRelation.carbonRelation.metaData.carbonTable
    +      (carbonTable.getFactTableName, carbonTable.getAbsoluteTableIdentifier)
    +    }.toMap
    +
    +    if (isRequiredToDecode) {
    +      val cacheProvider: CacheProvider = CacheProvider.getInstance
    +      val forwardDictionaryCache: Cache[DictionaryColumnUniqueIdentifier, Dictionary] =
    +        cacheProvider.createCache(CacheType.FORWARD_DICTIONARY, storePath)
    +      val dicts: Seq[ForwardDictionaryWrapper] = getDictionaryWrapper(absoluteTableIdentifiers,
    +        forwardDictionaryCache, storePath)
    +
    +      val exprs = child.output.map(x =>
    +        ExpressionCanonicalizer.execute(BindReferences.bindReference(x, child.output)))
    +      ctx.currentVars = input
    +      val resultVars = exprs.zipWithIndex.map { e =>
    --- End diff --
   
    change `e` to `case (expr, index)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/653#discussion_r105921333
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala ---
    @@ -200,6 +201,123 @@ case class CarbonDictionaryDecoder(
         }
       }
     
    +  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
    +
    +    val storePath = CarbonEnv.get.carbonMetastore.storePath
    +    val absoluteTableIdentifiers = relations.map { relation =>
    +      val carbonTable = relation.carbonRelation.carbonRelation.metaData.carbonTable
    +      (carbonTable.getFactTableName, carbonTable.getAbsoluteTableIdentifier)
    +    }.toMap
    +
    +    if (isRequiredToDecode) {
    +      val cacheProvider: CacheProvider = CacheProvider.getInstance
    +      val forwardDictionaryCache: Cache[DictionaryColumnUniqueIdentifier, Dictionary] =
    +        cacheProvider.createCache(CacheType.FORWARD_DICTIONARY, storePath)
    +      val dicts: Seq[ForwardDictionaryWrapper] = getDictionaryWrapper(absoluteTableIdentifiers,
    +        forwardDictionaryCache, storePath)
    +
    +      val exprs = child.output.map(x =>
    --- End diff --
   
    change `map(x` to `map { x`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/653#discussion_r105921624
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala ---
    @@ -225,6 +343,25 @@ case class CarbonDictionaryDecoder(
         dicts
       }
     
    +  private def getDictionaryWrapper(atiMap: Map[String, AbsoluteTableIdentifier],
    +      cache: Cache[DictionaryColumnUniqueIdentifier, Dictionary], storePath: String) = {
    +    val dicts: Seq[ForwardDictionaryWrapper] = getDictionaryColumnIds.map { f =>
    --- End diff --
   
    suggest to change `f` to a more meaningful variable


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/653#discussion_r106149905
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala ---
    @@ -200,6 +201,123 @@ case class CarbonDictionaryDecoder(
         }
       }
     
    +  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
    +
    +    val storePath = CarbonEnv.get.carbonMetastore.storePath
    +    val absoluteTableIdentifiers = relations.map { relation =>
    +      val carbonTable = relation.carbonRelation.carbonRelation.metaData.carbonTable
    +      (carbonTable.getFactTableName, carbonTable.getAbsoluteTableIdentifier)
    +    }.toMap
    +
    +    if (isRequiredToDecode) {
    +      val cacheProvider: CacheProvider = CacheProvider.getInstance
    +      val forwardDictionaryCache: Cache[DictionaryColumnUniqueIdentifier, Dictionary] =
    +        cacheProvider.createCache(CacheType.FORWARD_DICTIONARY, storePath)
    +      val dicts: Seq[ForwardDictionaryWrapper] = getDictionaryWrapper(absoluteTableIdentifiers,
    +        forwardDictionaryCache, storePath)
    +
    +      val exprs = child.output.map(x =>
    --- End diff --
   
    ok


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/653#discussion_r106149917
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala ---
    @@ -225,6 +343,25 @@ case class CarbonDictionaryDecoder(
         dicts
       }
     
    +  private def getDictionaryWrapper(atiMap: Map[String, AbsoluteTableIdentifier],
    +      cache: Cache[DictionaryColumnUniqueIdentifier, Dictionary], storePath: String) = {
    +    val dicts: Seq[ForwardDictionaryWrapper] = getDictionaryColumnIds.map { f =>
    --- End diff --
   
    ok


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/653
 
    Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1159/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/653#discussion_r106162467
 
    --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala ---
    @@ -200,6 +201,123 @@ case class CarbonDictionaryDecoder(
         }
       }
     
    +  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
    +
    +    val storePath = CarbonEnv.get.carbonMetastore.storePath
    +    val absoluteTableIdentifiers = relations.map { relation =>
    +      val carbonTable = relation.carbonRelation.carbonRelation.metaData.carbonTable
    +      (carbonTable.getFactTableName, carbonTable.getAbsoluteTableIdentifier)
    +    }.toMap
    +
    +    if (isRequiredToDecode) {
    +      val cacheProvider: CacheProvider = CacheProvider.getInstance
    +      val forwardDictionaryCache: Cache[DictionaryColumnUniqueIdentifier, Dictionary] =
    +        cacheProvider.createCache(CacheType.FORWARD_DICTIONARY, storePath)
    +      val dicts: Seq[ForwardDictionaryWrapper] = getDictionaryWrapper(absoluteTableIdentifiers,
    +        forwardDictionaryCache, storePath)
    +
    +      val exprs = child.output.map(x =>
    +        ExpressionCanonicalizer.execute(BindReferences.bindReference(x, child.output)))
    +      ctx.currentVars = input
    +      val resultVars = exprs.zipWithIndex.map { e =>
    --- End diff --
   
    ok


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/incubator-carbondata/pull/653
 
    Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1161/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---

[qiuchenjian-2]

Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-carbondata/pull/653


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---