[GitHub] [carbondata] shenh062326 opened a new pull request #3546: [CARBONDATA-3642] Add column idx in error msg when string length exceed 32000

[GitBox]

CarbonDataQA1 commented on issue #3546: [CARBONDATA-3642] Add column idx in error msg when string length exceed 32000
URL: https://github.com/apache/carbondata/pull/3546#issuecomment-570115862
 
 
   Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1397/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3546: [CARBONDATA-3642] Add column idx in error msg when string length exceed 32000
URL: https://github.com/apache/carbondata/pull/3546#issuecomment-570118242
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1409/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3546: [CARBONDATA-3642] Add column idx in error msg when string length exceed 32000
URL: https://github.com/apache/carbondata/pull/3546#discussion_r363074161
 
 

 ##########
 File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala
 ##########
 @@ -60,17 +60,27 @@ object CarbonScalaUtil {
 
   private val LOGGER: Logger = LogServiceFactory.getLogService(this.getClass.getCanonicalName)
 
-  def getString(value: Any,
+  def getString(row: Row,
+      idx: Int,
       serializationNullFormat: String,
       complexDelimiters: util.ArrayList[String],
       timeStampFormat: SimpleDateFormat,
       dateFormat: SimpleDateFormat,
       isVarcharType: Boolean = false,
       isComplexType: Boolean = false,
       level: Int = 0): String = {
-    FieldConverter.objectToString(value, serializationNullFormat, complexDelimiters,
-      timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
-      level)
+    try {
+      FieldConverter.objectToString(row.get(idx), serializationNullFormat, complexDelimiters,
+        timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
+        level)
+    } catch {
+      case e: Exception =>
+        if (e.getMessage.startsWith(FieldConverter.exceedErrorMsg)) {
+          throw new Exception("Column idx " + idx + " too long", e)
 
 Review comment:
   ok, beside the column index, maybe you want to log the column name also. You can get it by `carbonLoadModel.getCarbonDataLoadSchema.getCarbonTable.getCreateOrderColumn.get(i)`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

shenh062326 commented on a change in pull request #3546: [CARBONDATA-3642] Add column idx in error msg when string length exceed 32000
URL: https://github.com/apache/carbondata/pull/3546#discussion_r363150752
 
 

 ##########
 File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala
 ##########
 @@ -60,17 +60,27 @@ object CarbonScalaUtil {
 
   private val LOGGER: Logger = LogServiceFactory.getLogService(this.getClass.getCanonicalName)
 
-  def getString(value: Any,
+  def getString(row: Row,
+      idx: Int,
       serializationNullFormat: String,
       complexDelimiters: util.ArrayList[String],
       timeStampFormat: SimpleDateFormat,
       dateFormat: SimpleDateFormat,
       isVarcharType: Boolean = false,
       isComplexType: Boolean = false,
       level: Int = 0): String = {
-    FieldConverter.objectToString(value, serializationNullFormat, complexDelimiters,
-      timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
-      level)
+    try {
+      FieldConverter.objectToString(row.get(idx), serializationNullFormat, complexDelimiters,
+        timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
+        level)
+    } catch {
+      case e: Exception =>
+        if (e.getMessage.startsWith(FieldConverter.exceedErrorMsg)) {
+          throw new Exception("Column idx " + idx + " too long", e)
 
 Review comment:
   Thanks, your suggestion is better. I will use column name instand of column idx.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

shenh062326 commented on a change in pull request #3546: [CARBONDATA-3642] Add column idx in error msg when string length exceed 32000
URL: https://github.com/apache/carbondata/pull/3546#discussion_r363150752
 
 

 ##########
 File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala
 ##########
 @@ -60,17 +60,27 @@ object CarbonScalaUtil {
 
   private val LOGGER: Logger = LogServiceFactory.getLogService(this.getClass.getCanonicalName)
 
-  def getString(value: Any,
+  def getString(row: Row,
+      idx: Int,
       serializationNullFormat: String,
       complexDelimiters: util.ArrayList[String],
       timeStampFormat: SimpleDateFormat,
       dateFormat: SimpleDateFormat,
       isVarcharType: Boolean = false,
       isComplexType: Boolean = false,
       level: Int = 0): String = {
-    FieldConverter.objectToString(value, serializationNullFormat, complexDelimiters,
-      timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
-      level)
+    try {
+      FieldConverter.objectToString(row.get(idx), serializationNullFormat, complexDelimiters,
+        timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
+        level)
+    } catch {
+      case e: Exception =>
+        if (e.getMessage.startsWith(FieldConverter.exceedErrorMsg)) {
+          throw new Exception("Column idx " + idx + " too long", e)
 
 Review comment:
   Thanks, your suggestion is better. I will use column name instead of column idx.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3546: [CARBONDATA-3642] Add column idx in error msg when string length exceed 32000
URL: https://github.com/apache/carbondata/pull/3546#issuecomment-570994390
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1484/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3546: [CARBONDATA-3642] Add column name in error msg when string length exceed 32000
URL: https://github.com/apache/carbondata/pull/3546#discussion_r363158667
 
 

 ##########
 File path: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/longstring/VarcharDataTypesBasicTestCase.scala
 ##########
 @@ -196,7 +196,7 @@ class VarcharDataTypesBasicTestCase extends QueryTest with BeforeAndAfterEach wi
     val e = intercept[Exception] {
       sql(s""" insert into testlongstring select 1, 'abc', '$longChar'""")
     }
-    assert(e.getMessage.contains("Dataload failed, String length cannot exceed 32000 characters"))
+    assert(e.getMessage.contains("DataLoad failure: Column idx 2 too long"))
 
 Review comment:
   please add column name in error message than column index

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3546: [CARBONDATA-3642] Add column name in error msg when string length exceed 32000
URL: https://github.com/apache/carbondata/pull/3546#discussion_r363159229
 
 

 ##########
 File path: streaming/src/main/scala/org/apache/carbondata/streaming/parser/FieldConverter.scala
 ##########
 @@ -24,6 +24,7 @@ import java.util
 import org.apache.carbondata.core.constants.CarbonCommonConstants
 
 object FieldConverter {
+  val exceedErrorMsg = "Dataload failed, String length cannot exceed "
 
 Review comment:
   Better to rename `exceedErrorMsg` to `stringLengthExceedErrorMsg`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3546: [CARBONDATA-3642] Add column name in error msg when string length exceed 32000
URL: https://github.com/apache/carbondata/pull/3546#discussion_r363158993
 
 

 ##########
 File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala
 ##########
 @@ -53,24 +53,38 @@ import org.apache.carbondata.core.util.DataTypeUtil
 import org.apache.carbondata.processing.exception.DataLoadingException
 import org.apache.carbondata.processing.loading.FailureCauses
 import org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
 import org.apache.carbondata.processing.util.CarbonDataProcessorUtil
 import org.apache.carbondata.streaming.parser.FieldConverter
 
 object CarbonScalaUtil {
 
   private val LOGGER: Logger = LogServiceFactory.getLogService(this.getClass.getCanonicalName)
 
-  def getString(value: Any,
+  def getString(
+      row: Row,
+      idx: Int,
+      carbonLoadModel: CarbonLoadModel,
       serializationNullFormat: String,
       complexDelimiters: util.ArrayList[String],
       timeStampFormat: SimpleDateFormat,
       dateFormat: SimpleDateFormat,
       isVarcharType: Boolean = false,
       isComplexType: Boolean = false,
       level: Int = 0): String = {
-    FieldConverter.objectToString(value, serializationNullFormat, complexDelimiters,
-      timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
-      level)
+    try {
+      FieldConverter.objectToString(row.get(idx), serializationNullFormat, complexDelimiters,
+        timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
 
 Review comment:
   just pass `isVarcharType` and `isComplexType`, no need of equals

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3546: [CARBONDATA-3642] Add column name in error msg when string length exceed 32000
URL: https://github.com/apache/carbondata/pull/3546#discussion_r363161173
 
 

 ##########
 File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala
 ##########
 @@ -53,24 +53,38 @@ import org.apache.carbondata.core.util.DataTypeUtil
 import org.apache.carbondata.processing.exception.DataLoadingException
 import org.apache.carbondata.processing.loading.FailureCauses
 import org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
 import org.apache.carbondata.processing.util.CarbonDataProcessorUtil
 import org.apache.carbondata.streaming.parser.FieldConverter
 
 object CarbonScalaUtil {
 
   private val LOGGER: Logger = LogServiceFactory.getLogService(this.getClass.getCanonicalName)
 
-  def getString(value: Any,
+  def getString(
+      row: Row,
+      idx: Int,
+      carbonLoadModel: CarbonLoadModel,
       serializationNullFormat: String,
       complexDelimiters: util.ArrayList[String],
       timeStampFormat: SimpleDateFormat,
       dateFormat: SimpleDateFormat,
       isVarcharType: Boolean = false,
       isComplexType: Boolean = false,
       level: Int = 0): String = {
-    FieldConverter.objectToString(value, serializationNullFormat, complexDelimiters,
-      timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
-      level)
+    try {
+      FieldConverter.objectToString(row.get(idx), serializationNullFormat, complexDelimiters,
+        timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
+        level)
+    } catch {
+      case e: Exception =>
+        if (e.getMessage.startsWith(FieldConverter.exceedErrorMsg)) {
+          throw new Exception(s"Column ${carbonLoadModel.getCarbonDataLoadSchema.getCarbonTable
 
 Review comment:
   please change to this, i have added is in message `Column ${carbonLoadModel.getCarbonDataLoadSchema.getCarbonTable.getCreateOrderColumn.get(idx).getColName} is too long`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

akashrn5 commented on a change in pull request #3546: [CARBONDATA-3642] Add column name in error msg when string length exceed 32000
URL: https://github.com/apache/carbondata/pull/3546#discussion_r363161309
 
 

 ##########
 File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala
 ##########
 @@ -53,24 +53,38 @@ import org.apache.carbondata.core.util.DataTypeUtil
 import org.apache.carbondata.processing.exception.DataLoadingException
 import org.apache.carbondata.processing.loading.FailureCauses
 import org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
 import org.apache.carbondata.processing.util.CarbonDataProcessorUtil
 import org.apache.carbondata.streaming.parser.FieldConverter
 
 object CarbonScalaUtil {
 
   private val LOGGER: Logger = LogServiceFactory.getLogService(this.getClass.getCanonicalName)
 
-  def getString(value: Any,
+  def getString(
+      row: Row,
+      idx: Int,
+      carbonLoadModel: CarbonLoadModel,
       serializationNullFormat: String,
       complexDelimiters: util.ArrayList[String],
       timeStampFormat: SimpleDateFormat,
       dateFormat: SimpleDateFormat,
       isVarcharType: Boolean = false,
       isComplexType: Boolean = false,
       level: Int = 0): String = {
-    FieldConverter.objectToString(value, serializationNullFormat, complexDelimiters,
-      timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
-      level)
+    try {
+      FieldConverter.objectToString(row.get(idx), serializationNullFormat, complexDelimiters,
+        timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
+        level)
+    } catch {
+      case e: Exception =>
+        if (e.getMessage.startsWith(FieldConverter.exceedErrorMsg)) {
 
 Review comment:
   Better to add a Logger.error() and then throw the exception

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3546: [CARBONDATA-3642] Add column name in error msg when string length exceed 32000
URL: https://github.com/apache/carbondata/pull/3546#issuecomment-571008476
 
 
   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1485/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

shenh062326 commented on a change in pull request #3546: [CARBONDATA-3642] Add column name in error msg when string length exceed 32000
URL: https://github.com/apache/carbondata/pull/3546#discussion_r363172880
 
 

 ##########
 File path: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/longstring/VarcharDataTypesBasicTestCase.scala
 ##########
 @@ -196,7 +196,7 @@ class VarcharDataTypesBasicTestCase extends QueryTest with BeforeAndAfterEach wi
     val e = intercept[Exception] {
       sql(s""" insert into testlongstring select 1, 'abc', '$longChar'""")
     }
-    assert(e.getMessage.contains("Dataload failed, String length cannot exceed 32000 characters"))
+    assert(e.getMessage.contains("DataLoad failure: Column idx 2 too long"))
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

shenh062326 commented on a change in pull request #3546: [CARBONDATA-3642] Add column name in error msg when string length exceed 32000
URL: https://github.com/apache/carbondata/pull/3546#discussion_r363172900
 
 

 ##########
 File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala
 ##########
 @@ -53,24 +53,38 @@ import org.apache.carbondata.core.util.DataTypeUtil
 import org.apache.carbondata.processing.exception.DataLoadingException
 import org.apache.carbondata.processing.loading.FailureCauses
 import org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
 import org.apache.carbondata.processing.util.CarbonDataProcessorUtil
 import org.apache.carbondata.streaming.parser.FieldConverter
 
 object CarbonScalaUtil {
 
   private val LOGGER: Logger = LogServiceFactory.getLogService(this.getClass.getCanonicalName)
 
-  def getString(value: Any,
+  def getString(
+      row: Row,
+      idx: Int,
+      carbonLoadModel: CarbonLoadModel,
       serializationNullFormat: String,
       complexDelimiters: util.ArrayList[String],
       timeStampFormat: SimpleDateFormat,
       dateFormat: SimpleDateFormat,
       isVarcharType: Boolean = false,
       isComplexType: Boolean = false,
       level: Int = 0): String = {
-    FieldConverter.objectToString(value, serializationNullFormat, complexDelimiters,
-      timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
-      level)
+    try {
+      FieldConverter.objectToString(row.get(idx), serializationNullFormat, complexDelimiters,
+        timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

shenh062326 commented on a change in pull request #3546: [CARBONDATA-3642] Add column name in error msg when string length exceed 32000
URL: https://github.com/apache/carbondata/pull/3546#discussion_r363172928
 
 

 ##########
 File path: streaming/src/main/scala/org/apache/carbondata/streaming/parser/FieldConverter.scala
 ##########
 @@ -24,6 +24,7 @@ import java.util
 import org.apache.carbondata.core.constants.CarbonCommonConstants
 
 object FieldConverter {
+  val exceedErrorMsg = "Dataload failed, String length cannot exceed "
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

shenh062326 commented on a change in pull request #3546: [CARBONDATA-3642] Add column name in error msg when string length exceed 32000
URL: https://github.com/apache/carbondata/pull/3546#discussion_r363172951
 
 

 ##########
 File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala
 ##########
 @@ -53,24 +53,38 @@ import org.apache.carbondata.core.util.DataTypeUtil
 import org.apache.carbondata.processing.exception.DataLoadingException
 import org.apache.carbondata.processing.loading.FailureCauses
 import org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
 import org.apache.carbondata.processing.util.CarbonDataProcessorUtil
 import org.apache.carbondata.streaming.parser.FieldConverter
 
 object CarbonScalaUtil {
 
   private val LOGGER: Logger = LogServiceFactory.getLogService(this.getClass.getCanonicalName)
 
-  def getString(value: Any,
+  def getString(
+      row: Row,
+      idx: Int,
+      carbonLoadModel: CarbonLoadModel,
       serializationNullFormat: String,
       complexDelimiters: util.ArrayList[String],
       timeStampFormat: SimpleDateFormat,
       dateFormat: SimpleDateFormat,
       isVarcharType: Boolean = false,
       isComplexType: Boolean = false,
       level: Int = 0): String = {
-    FieldConverter.objectToString(value, serializationNullFormat, complexDelimiters,
-      timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
-      level)
+    try {
+      FieldConverter.objectToString(row.get(idx), serializationNullFormat, complexDelimiters,
+        timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
+        level)
+    } catch {
+      case e: Exception =>
+        if (e.getMessage.startsWith(FieldConverter.exceedErrorMsg)) {
+          throw new Exception(s"Column ${carbonLoadModel.getCarbonDataLoadSchema.getCarbonTable
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

shenh062326 commented on a change in pull request #3546: [CARBONDATA-3642] Add column name in error msg when string length exceed 32000
URL: https://github.com/apache/carbondata/pull/3546#discussion_r363172977
 
 

 ##########
 File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala
 ##########
 @@ -53,24 +53,38 @@ import org.apache.carbondata.core.util.DataTypeUtil
 import org.apache.carbondata.processing.exception.DataLoadingException
 import org.apache.carbondata.processing.loading.FailureCauses
 import org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
 import org.apache.carbondata.processing.util.CarbonDataProcessorUtil
 import org.apache.carbondata.streaming.parser.FieldConverter
 
 object CarbonScalaUtil {
 
   private val LOGGER: Logger = LogServiceFactory.getLogService(this.getClass.getCanonicalName)
 
-  def getString(value: Any,
+  def getString(
+      row: Row,
+      idx: Int,
+      carbonLoadModel: CarbonLoadModel,
       serializationNullFormat: String,
       complexDelimiters: util.ArrayList[String],
       timeStampFormat: SimpleDateFormat,
       dateFormat: SimpleDateFormat,
       isVarcharType: Boolean = false,
       isComplexType: Boolean = false,
       level: Int = 0): String = {
-    FieldConverter.objectToString(value, serializationNullFormat, complexDelimiters,
-      timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
-      level)
+    try {
+      FieldConverter.objectToString(row.get(idx), serializationNullFormat, complexDelimiters,
+        timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
+        level)
+    } catch {
+      case e: Exception =>
+        if (e.getMessage.startsWith(FieldConverter.exceedErrorMsg)) {
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3546: [CARBONDATA-3642] Add column name in error msg when string length exceed 32000
URL: https://github.com/apache/carbondata/pull/3546#issuecomment-571041350
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1486/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3546: [CARBONDATA-3642] Add column name in error msg when string length exceed 32000
URL: https://github.com/apache/carbondata/pull/3546#discussion_r363301088
 
 

 ##########
 File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala
 ##########
 @@ -53,24 +53,39 @@ import org.apache.carbondata.core.util.DataTypeUtil
 import org.apache.carbondata.processing.exception.DataLoadingException
 import org.apache.carbondata.processing.loading.FailureCauses
 import org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
 import org.apache.carbondata.processing.util.CarbonDataProcessorUtil
 import org.apache.carbondata.streaming.parser.FieldConverter
 
 object CarbonScalaUtil {
 
   private val LOGGER: Logger = LogServiceFactory.getLogService(this.getClass.getCanonicalName)
 
-  def getString(value: Any,
+  def getString(
+      row: Row,
+      idx: Int,
+      carbonLoadModel: CarbonLoadModel,
       serializationNullFormat: String,
       complexDelimiters: util.ArrayList[String],
       timeStampFormat: SimpleDateFormat,
       dateFormat: SimpleDateFormat,
       isVarcharType: Boolean = false,
       isComplexType: Boolean = false,
       level: Int = 0): String = {
-    FieldConverter.objectToString(value, serializationNullFormat, complexDelimiters,
-      timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
-      level)
+    try {
+      FieldConverter.objectToString(row.get(idx), serializationNullFormat, complexDelimiters,
+        timeStampFormat, dateFormat, isVarcharType, isComplexType, level)
+    } catch {
+      case e: Exception =>
+        if (e.getMessage.startsWith(FieldConverter.stringLengthExceedErrorMsg)) {
+          val msg = s"Column ${carbonLoadModel.getCarbonDataLoadSchema.getCarbonTable
+            .getCreateOrderColumn.get(idx).getColName} is too long"
 
 Review comment:
   suggest to add "consider to use 'long_string_columns' table property" in the log to indicate user how to solve it

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

shenh062326 commented on a change in pull request #3546: [CARBONDATA-3642] Add column name in error msg when string length exceed 32000
URL: https://github.com/apache/carbondata/pull/3546#discussion_r363564933
 
 

 ##########
 File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala
 ##########
 @@ -53,24 +53,39 @@ import org.apache.carbondata.core.util.DataTypeUtil
 import org.apache.carbondata.processing.exception.DataLoadingException
 import org.apache.carbondata.processing.loading.FailureCauses
 import org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
 import org.apache.carbondata.processing.util.CarbonDataProcessorUtil
 import org.apache.carbondata.streaming.parser.FieldConverter
 
 object CarbonScalaUtil {
 
   private val LOGGER: Logger = LogServiceFactory.getLogService(this.getClass.getCanonicalName)
 
-  def getString(value: Any,
+  def getString(
+      row: Row,
+      idx: Int,
+      carbonLoadModel: CarbonLoadModel,
       serializationNullFormat: String,
       complexDelimiters: util.ArrayList[String],
       timeStampFormat: SimpleDateFormat,
       dateFormat: SimpleDateFormat,
       isVarcharType: Boolean = false,
       isComplexType: Boolean = false,
       level: Int = 0): String = {
-    FieldConverter.objectToString(value, serializationNullFormat, complexDelimiters,
-      timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
-      level)
+    try {
+      FieldConverter.objectToString(row.get(idx), serializationNullFormat, complexDelimiters,
+        timeStampFormat, dateFormat, isVarcharType, isComplexType, level)
+    } catch {
+      case e: Exception =>
+        if (e.getMessage.startsWith(FieldConverter.stringLengthExceedErrorMsg)) {
+          val msg = s"Column ${carbonLoadModel.getCarbonDataLoadSchema.getCarbonTable
+            .getCreateOrderColumn.get(idx).getColName} is too long"
 
 Review comment:
   done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services