Re: Problem with with writing the loadStartTime in "dd-MM-yyyy HH:mm:ss:SSS" format

Posted by sraghunandan on Dec 15, 2017; 2:44pm
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/Problem-with-with-writing-the-loadStartTime-in-dd-MM-yyyy-HH-mm-ss-SSS-format-tp30971p30994.html

+1
On Fri, 15 Dec 2017 at 5:34 PM, Mohammad Shahid Khan <
[hidden email]> wrote:

> Hi All,
> Please find the details.
> Please response as earliest as possible.
> *Problem*:
>
> if we move the table to environment having different timezone or we change
> the system current timezone
> then after IUD operation some of the blocks are not treated as valid
> blocks.
>
> [{"timestamp":"15-12-2017
>
> 16:50:31:703","loadStatus":"Success","loadName":"0","partitionCount":"0","isDeleted":"FALSE","dataSize":"912","indexSize":"700","updateDeltaEndTimestamp":"","updateDeltaStartTimestamp":"","updateStatusFileName":"",*"loadStartTime":"15-12-2017
> 16:50:27:493"*,"visibility":"true","fileFormat":"COLUMNAR_V3"}]
>
> part-0-0_batchno0-0-*1513336827493*.carbondata
>
> If timezone is different than at the time load was done, the value
> calculated from *loadStartTime *15-12-2017 16:50:27:493 will not match to
> the time stamp extracted from block file name.
>
> *Solution*:
>
> We should stop writing the loadStartTime and timestamp in "dd-MM-yyyy
> HH:mm:ss:SSS" format.
> *We should write the long value of the timestamp.*
> *like below:*
> [{"timestamp":"*1513336827593*
>
> ","loadStatus":"Success","loadName":"0","partitionCount":"0","isDeleted":"FALSE","dataSize":"912","indexSize":"700","updateDeltaEndTimestamp":"","updateDeltaStartTimestamp":"","updateStatusFileName":"",
> *"loadStartTime":"**1513336827493**"*,
> "visibility":"true","fileFormat":"COLUMNAR_V3"}]
>
> Backward compatibility:
> If string to Long parse fail, we can fall back for date parsing.
>
> private long convertTimeStampToLong(String factTimeStamp) {
>     try {
>       // for new loads the factTimeStamp will be long string
>       // but for the old store the it will be in form of date string
>       *return Long.parseLong(factTimeStamp);*
>     } catch (NumberFormatException nf) {
>       SimpleDateFormat parser = new
> SimpleDateFormat(CarbonCommonConstants.CARBON_TIMESTAMP_MILLIS);
>       Date dateToStr = null;
>       try {
>         dateToStr = parser.parse(factTimeStamp);
>         return dateToStr.getTime();
>       } catch (ParseException e) {
>         LOGGER
>             .error("Cannot convert" + factTimeStamp + " to Time/Long type
> value" + e.getMessage());
>         parser = new
> SimpleDateFormat(CarbonCommonConstants.CARBON_TIMESTAMP);
>         try {
>           // if the load is in progress, factTimeStamp will be null, so use
> current time
>           if (null == factTimeStamp) {
>             return System.currentTimeMillis();
>           }
>           dateToStr = parser.parse(factTimeStamp);
>           return dateToStr.getTime();
>         } catch (ParseException e1) {
>           LOGGER.error(
>               "Cannot convert" + factTimeStamp + " to Time/Long type value"
> + e1.getMessage());
>           return 0;
>         }
>       }
>     }
>   }
>
>
> Regards,
> Mohammad Shahid Khan
>