Problem with with writing the loadStartTime in "dd-MM-yyyy HH:mm:ss:SSS" format

Posted by mohdshahidkhan on Dec 15, 2017; 12:04pm
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Problem-with-with-writing-the-loadStartTime-in-dd-MM-yyyy-HH-mm-ss-SSS-format-tp30971.html

Hi All,
Please find the details.
Please response as earliest as possible.
*Problem*:

if we move the table to environment having different timezone or we change
the system current timezone
then after IUD operation some of the blocks are not treated as valid blocks.

[{"timestamp":"15-12-2017
16:50:31:703","loadStatus":"Success","loadName":"0","partitionCount":"0","isDeleted":"FALSE","dataSize":"912","indexSize":"700","updateDeltaEndTimestamp":"","updateDeltaStartTimestamp":"","updateStatusFileName":"",*"loadStartTime":"15-12-2017
16:50:27:493"*,"visibility":"true","fileFormat":"COLUMNAR_V3"}]

part-0-0_batchno0-0-*1513336827493*.carbondata

If timezone is different than at the time load was done, the value
calculated from *loadStartTime *15-12-2017 16:50:27:493 will not match to
the time stamp extracted from block file name.

*Solution*:

We should stop writing the loadStartTime and timestamp in "dd-MM-yyyy
HH:mm:ss:SSS" format.
*We should write the long value of the timestamp.*
*like below:*
[{"timestamp":"*1513336827593*
","loadStatus":"Success","loadName":"0","partitionCount":"0","isDeleted":"FALSE","dataSize":"912","indexSize":"700","updateDeltaEndTimestamp":"","updateDeltaStartTimestamp":"","updateStatusFileName":"",
*"loadStartTime":"**1513336827493**"*,
"visibility":"true","fileFormat":"COLUMNAR_V3"}]

Backward compatibility:
If string to Long parse fail, we can fall back for date parsing.

private long convertTimeStampToLong(String factTimeStamp) {
    try {
      // for new loads the factTimeStamp will be long string
      // but for the old store the it will be in form of date string
      *return Long.parseLong(factTimeStamp);*
    } catch (NumberFormatException nf) {
      SimpleDateFormat parser = new
SimpleDateFormat(CarbonCommonConstants.CARBON_TIMESTAMP_MILLIS);
      Date dateToStr = null;
      try {
        dateToStr = parser.parse(factTimeStamp);
        return dateToStr.getTime();
      } catch (ParseException e) {
        LOGGER
            .error("Cannot convert" + factTimeStamp + " to Time/Long type
value" + e.getMessage());
        parser = new
SimpleDateFormat(CarbonCommonConstants.CARBON_TIMESTAMP);
        try {
          // if the load is in progress, factTimeStamp will be null, so use
current time
          if (null == factTimeStamp) {
            return System.currentTimeMillis();
          }
          dateToStr = parser.parse(factTimeStamp);
          return dateToStr.getTime();
        } catch (ParseException e1) {
          LOGGER.error(
              "Cannot convert" + factTimeStamp + " to Time/Long type value"
+ e1.getMessage());
          return 0;
        }
      }
    }
  }


Regards,
Mohammad Shahid Khan