Problem with with writing the loadStartTime in "dd-MM-yyyy HH:mm:ss:SSS" format

[mohdshahidkhan]

Hi All,
Please find the details.
Please response as earliest as possible.
*Problem*:

if we move the table to environment having different timezone or we change
the system current timezone
then after IUD operation some of the blocks are not treated as valid blocks.

[{"timestamp":"15-12-2017
16:50:31:703","loadStatus":"Success","loadName":"0","partitionCount":"0","isDeleted":"FALSE","dataSize":"912","indexSize":"700","updateDeltaEndTimestamp":"","updateDeltaStartTimestamp":"","updateStatusFileName":"",*"loadStartTime":"15-12-2017
16:50:27:493"*,"visibility":"true","fileFormat":"COLUMNAR_V3"}]

part-0-0_batchno0-0-*1513336827493*.carbondata

If timezone is different than at the time load was done, the value
calculated from *loadStartTime *15-12-2017 16:50:27:493 will not match to
the time stamp extracted from block file name.

*Solution*:

We should stop writing the loadStartTime and timestamp in "dd-MM-yyyy
HH:mm:ss:SSS" format.
*We should write the long value of the timestamp.*
*like below:*
[{"timestamp":"*1513336827593*
","loadStatus":"Success","loadName":"0","partitionCount":"0","isDeleted":"FALSE","dataSize":"912","indexSize":"700","updateDeltaEndTimestamp":"","updateDeltaStartTimestamp":"","updateStatusFileName":"",
*"loadStartTime":"**1513336827493**"*,
"visibility":"true","fileFormat":"COLUMNAR_V3"}]

Backward compatibility:
If string to Long parse fail, we can fall back for date parsing.

private long convertTimeStampToLong(String factTimeStamp) {
    try {
      // for new loads the factTimeStamp will be long string
      // but for the old store the it will be in form of date string
      *return Long.parseLong(factTimeStamp);*
    } catch (NumberFormatException nf) {
      SimpleDateFormat parser = new
SimpleDateFormat(CarbonCommonConstants.CARBON_TIMESTAMP_MILLIS);
      Date dateToStr = null;
      try {
        dateToStr = parser.parse(factTimeStamp);
        return dateToStr.getTime();
      } catch (ParseException e) {
        LOGGER
            .error("Cannot convert" + factTimeStamp + " to Time/Long type
value" + e.getMessage());
        parser = new
SimpleDateFormat(CarbonCommonConstants.CARBON_TIMESTAMP);
        try {
          // if the load is in progress, factTimeStamp will be null, so use
current time
          if (null == factTimeStamp) {
            return System.currentTimeMillis();
          }
          dateToStr = parser.parse(factTimeStamp);
          return dateToStr.getTime();
        } catch (ParseException e1) {
          LOGGER.error(
              "Cannot convert" + factTimeStamp + " to Time/Long type value"
+ e1.getMessage());
          return 0;
        }
      }
    }
  }


Regards,
Mohammad Shahid Khan

[sraghunandan]

+1
On Fri, 15 Dec 2017 at 5:34 PM, Mohammad Shahid Khan <
[hidden email]> wrote:

> Hi All,
> Please find the details.
> Please response as earliest as possible.
> *Problem*:
>
> if we move the table to environment having different timezone or we change
> the system current timezone
> then after IUD operation some of the blocks are not treated as valid
> blocks.
>
> [{"timestamp":"15-12-2017
>
> 16:50:31:703","loadStatus":"Success","loadName":"0","partitionCount":"0","isDeleted":"FALSE","dataSize":"912","indexSize":"700","updateDeltaEndTimestamp":"","updateDeltaStartTimestamp":"","updateStatusFileName":"",*"loadStartTime":"15-12-2017
> 16:50:27:493"*,"visibility":"true","fileFormat":"COLUMNAR_V3"}]
>
> part-0-0_batchno0-0-*1513336827493*.carbondata
>
> If timezone is different than at the time load was done, the value
> calculated from *loadStartTime *15-12-2017 16:50:27:493 will not match to
> the time stamp extracted from block file name.
>
> *Solution*:
>
> We should stop writing the loadStartTime and timestamp in "dd-MM-yyyy
> HH:mm:ss:SSS" format.
> *We should write the long value of the timestamp.*
> *like below:*
> [{"timestamp":"*1513336827593*
>
> ","loadStatus":"Success","loadName":"0","partitionCount":"0","isDeleted":"FALSE","dataSize":"912","indexSize":"700","updateDeltaEndTimestamp":"","updateDeltaStartTimestamp":"","updateStatusFileName":"",
> *"loadStartTime":"**1513336827493**"*,
> "visibility":"true","fileFormat":"COLUMNAR_V3"}]
>
> Backward compatibility:
> If string to Long parse fail, we can fall back for date parsing.
>
> private long convertTimeStampToLong(String factTimeStamp) {
>     try {
>       // for new loads the factTimeStamp will be long string
>       // but for the old store the it will be in form of date string
>       *return Long.parseLong(factTimeStamp);*
>     } catch (NumberFormatException nf) {
>       SimpleDateFormat parser = new
> SimpleDateFormat(CarbonCommonConstants.CARBON_TIMESTAMP_MILLIS);
>       Date dateToStr = null;
>       try {
>         dateToStr = parser.parse(factTimeStamp);
>         return dateToStr.getTime();
>       } catch (ParseException e) {
>         LOGGER
>             .error("Cannot convert" + factTimeStamp + " to Time/Long type
> value" + e.getMessage());
>         parser = new
> SimpleDateFormat(CarbonCommonConstants.CARBON_TIMESTAMP);
>         try {
>           // if the load is in progress, factTimeStamp will be null, so use
> current time
>           if (null == factTimeStamp) {
>             return System.currentTimeMillis();
>           }
>           dateToStr = parser.parse(factTimeStamp);
>           return dateToStr.getTime();
>         } catch (ParseException e1) {
>           LOGGER.error(
>               "Cannot convert" + factTimeStamp + " to Time/Long type value"
> + e1.getMessage());
>           return 0;
>         }
>       }
>     }
>   }
>
>
> Regards,
> Mohammad Shahid Khan
>

[David CaiQiang]

+1



-----
Best Regards
David Cai
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/