[DISCUSSION] Support Time Series for MV datamap and autodatamap loading of timeseries datamaps

Hi Community,

Timeseries data are simply measurements or events that are tracked,monitored, downsampled, and aggregated over time.

Basicallytimeseries data analysis helps in analyzing or monitoring theaggregated data over period of time to take better decision forbusiness.

So since carbondata supports olap datamap like preaggregate, MV and since time series is of atmost importance,

we can supporttimeseries for carbondata over MV datamap model. 

Currentlycarbondata supports timeseries on preaggregate datamap, but its analpha feature and there are so many limitations when we compare and 

analyze the existing timeseries database or projects which supportstime series like apache druid or influxdb. So, in this feature we can support timeseries

by avoiding the limitations in the current system. After doing the analysis on the current existing timeseries database like influxdb, and the apache druid,

i have  prepared a solution/design document. Any inputs, improvements or suggestion are most welcome.

I have created jira https://issues.apache.org/jira/browse/CARBONDATA-3525 for this. Later i will create sub jiras for tracking.

Regards,

Akash R Nilugal

[chetdb]

Hi Akash,

1. For preaggregate table deprecation as part of subtask 7 can specific details be provided in the design doc.
2. Will alter table add partition supported now for table that has timeseries MV datamap ?If not supported it can be updated in the design doc.
3. Would Complex datatypes for timeseries MV datamap be supported ? If not supported it can be updated in the design doc.

Regards
Chetan

On 2019/09/23 13:42:48, Akash Nilugal <[hidden email]> wrote:

> Hi Community,
>
> Timeseries data are simply measurements or events that are
> tracked,monitored, downsampled, and aggregated over time.
> Basicallytimeseries data analysis helps in analyzing or monitoring
> theaggregated data over period of time to take better decision forbusiness.
> So since carbondata supports olap datamap like preaggregate, MV and since
> time series is of atmost importance,
> we can supporttimeseries for carbondata over MV datamap model.
>
> Currentlycarbondata supports timeseries on preaggregate datamap, but its
> analpha feature and there are so many limitations when we compare and
> analyze the existing timeseries database or projects which supportstime
> series like apache druid or influxdb. So, in this feature we can support
> timeseries
> by avoiding the limitations in the current system. After doing the analysis
> on the current existing timeseries database like influxdb, and the apache
> druid,
> i have  prepared a solution/design document. Any inputs, improvements or
> suggestion are most welcome.
>
> I have created jira https://issues.apache.org/jira/browse/CARBONDATA-3525 for
> this. Later i will create sub jiras for tracking.
>
>
> Regards,
> Akash R Nilugal
>

[Manhua-2]

Hi Akash

Can user specific the granularity? Such as 5minutes, 15 minutes

Is there any constraint on timestamp_column's datatype?  Including DATE,  TIMESTAMP,  BIGINT(Unix timestamp)

On 2019/09/23 13:42:48, Akash Nilugal <[hidden email]> wrote:

> Hi Community,
>
> Timeseries data are simply measurements or events that are
> tracked,monitored, downsampled, and aggregated over time.
> Basicallytimeseries data analysis helps in analyzing or monitoring
> theaggregated data over period of time to take better decision forbusiness.
> So since carbondata supports olap datamap like preaggregate, MV and since
> time series is of atmost importance,
> we can supporttimeseries for carbondata over MV datamap model.
>
> Currentlycarbondata supports timeseries on preaggregate datamap, but its
> analpha feature and there are so many limitations when we compare and
> analyze the existing timeseries database or projects which supportstime
> series like apache druid or influxdb. So, in this feature we can support
> timeseries
> by avoiding the limitations in the current system. After doing the analysis
> on the current existing timeseries database like influxdb, and the apache
> druid,
> i have  prepared a solution/design document. Any inputs, improvements or
> suggestion are most welcome.
>
> I have created jira https://issues.apache.org/jira/browse/CARBONDATA-3525 for
> this. Later i will create sub jiras for tracking.
>
>
> Regards,
> Akash R Nilugal
>

Hi Manhua,

Thanks for the questions, Please find the comments
1. User can specify the granularity as minute, so we take as 1 unit, (1 minute in this case) and store the data, then during query in UDF he can mention for minute level query, how many minutes data he wants.

2. Since we will be loading data to datamap from fact, there is no restriction on it.

Regards,
Akash

On 2019/09/24 08:00:55, Manhua <[hidden email]> wrote:

> Hi Akash
>
> Can user specific the granularity? Such as 5minutes, 15 minutes
>
> Is there any constraint on timestamp_column's datatype?  Including DATE,  TIMESTAMP,  BIGINT(Unix timestamp)
>
> On 2019/09/23 13:42:48, Akash Nilugal <[hidden email]> wrote:
> > Hi Community,
> >
> > Timeseries data are simply measurements or events that are
> > tracked,monitored, downsampled, and aggregated over time.
> > Basicallytimeseries data analysis helps in analyzing or monitoring
> > theaggregated data over period of time to take better decision forbusiness.
> > So since carbondata supports olap datamap like preaggregate, MV and since
> > time series is of atmost importance,
> > we can supporttimeseries for carbondata over MV datamap model.
> >
> > Currentlycarbondata supports timeseries on preaggregate datamap, but its
> > analpha feature and there are so many limitations when we compare and
> > analyze the existing timeseries database or projects which supportstime
> > series like apache druid or influxdb. So, in this feature we can support
> > timeseries
> > by avoiding the limitations in the current system. After doing the analysis
> > on the current existing timeseries database like influxdb, and the apache
> > druid,
> > i have  prepared a solution/design document. Any inputs, improvements or
> > suggestion are most welcome.
> >
> > I have created jira https://issues.apache.org/jira/browse/CARBONDATA-3525 for
> > this. Later i will create sub jiras for tracking.
> >
> >
> > Regards,
> > Akash R Nilugal
> >
>

Hi chetan,

1. Deprecate means, not recommended to use, so i think no further details are required for that
2. complex and add partition not supported. it wil be updated in document.


On 2019/09/23 14:43:51, chetan bhat <[hidden email]> wrote:

> Hi Akash,
>
> 1. For preaggregate table deprecation as part of subtask 7 can specific details be provided in the design doc.
> 2. Will alter table add partition supported now for table that has timeseries MV datamap ?If not supported it can be updated in the design doc.
> 3. Would Complex datatypes for timeseries MV datamap be supported ? If not supported it can be updated in the design doc.
>
> Regards
> Chetan
>
> On 2019/09/23 13:42:48, Akash Nilugal <[hidden email]> wrote:
> > Hi Community,
> >
> > Timeseries data are simply measurements or events that are
> > tracked,monitored, downsampled, and aggregated over time.
> > Basicallytimeseries data analysis helps in analyzing or monitoring
> > theaggregated data over period of time to take better decision forbusiness.
> > So since carbondata supports olap datamap like preaggregate, MV and since
> > time series is of atmost importance,
> > we can supporttimeseries for carbondata over MV datamap model.
> >
> > Currentlycarbondata supports timeseries on preaggregate datamap, but its
> > analpha feature and there are so many limitations when we compare and
> > analyze the existing timeseries database or projects which supportstime
> > series like apache druid or influxdb. So, in this feature we can support
> > timeseries
> > by avoiding the limitations in the current system. After doing the analysis
> > on the current existing timeseries database like influxdb, and the apache
> > druid,
> > i have  prepared a solution/design document. Any inputs, improvements or
> > suggestion are most welcome.
> >
> > I have created jira https://issues.apache.org/jira/browse/CARBONDATA-3525 for
> > this. Later i will create sub jiras for tracking.
> >
> >
> > Regards,
> > Akash R Nilugal
> >
>

[xuchuanyin]

Hi akash, glad to see the feature proposed and I have some questions about
this. Please notice that some of the following descriptions are comments
followed by '===' described in the design document attached in the
corresponding jira.

1.
"Currently carbondata supports timeseries on preaggregate datamap, but its
an alpha feature"
===
It has been some time since the preaggregate datamap was introduced and it
is still **alpha**, why it is still not product-ready? Will the new feature
also come into the similar situation?

2.
"there are so many limitations when we compare and analyze the existing
timeseries database or projects which supports time series like apache druid
or influxdb"
===
What are the actual limitations? Besides, please give an example of this.

3.
"Segment_Timestamp_Min"
===
Suggest using camel-case style like 'segmentTimestampMin'

4.
"RP is way of telling the system, for how long the data should be kept"
===
Since the function is simple, I'd suggest using 'retentionTime'=15 and
'timeUnit'='day' instead of 'RP'='15_days'

5.
"When the data load is called for main table, use an spark accumulator to
get the maximum value of timestamp in that load and return to the load."
===
How can you get the spark accumulator? The load is launched using
loading-by-dataframe not using global-sort-by-spark.

6.
For the rest of the content, still reading.




--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Hi xuchuanyin,

Thanks for the comments/Suggestions

1. Preaggregate is productized, but not the timeseries with preaggregate, i think you  got confused with that, if im right.
2. Limitations like, auto sampling or rollup, which we will be supporting now. Retention policies. etc
3. segmentTimestampMin, this i will consider in design.
4. RP is added as a separate task, i thought instead of maintaining two variables better to maintabin one and parse it. But i will consider your point based on feasibility during implementation.
5. We use an accumulator which takes list, so before writing index files we take the min max of the timestamp column and fill in accumulator and then we can access accumulator.value in driver after load is finished.

Regards,
Akash R Nilugal

On 2019/09/28 10:46:31, xuchuanyin <[hidden email]> wrote:

> Hi akash, glad to see the feature proposed and I have some questions about
> this. Please notice that some of the following descriptions are comments
> followed by '===' described in the design document attached in the
> corresponding jira.
>
> 1.
> "Currently carbondata supports timeseries on preaggregate datamap, but its
> an alpha feature"
> ===
> It has been some time since the preaggregate datamap was introduced and it
> is still **alpha**, why it is still not product-ready? Will the new feature
> also come into the similar situation?
>
> 2.
> "there are so many limitations when we compare and analyze the existing
> timeseries database or projects which supports time series like apache druid
> or influxdb"
> ===
> What are the actual limitations? Besides, please give an example of this.
>
> 3.
> "Segment_Timestamp_Min"
> ===
> Suggest using camel-case style like 'segmentTimestampMin'
>
> 4.
> "RP is way of telling the system, for how long the data should be kept"
> ===
> Since the function is simple, I'd suggest using 'retentionTime'=15 and
> 'timeUnit'='day' instead of 'RP'='15_days'
>
> 5.
> "When the data load is called for main table, use an spark accumulator to
> get the maximum value of timestamp in that load and return to the load."
> ===
> How can you get the spark accumulator? The load is launched using
> loading-by-dataframe not using global-sort-by-spark.
>
> 6.
> For the rest of the content, still reading.
>
>
>
>
> --
> Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>

[Ajantha Bhat]

+ 1 ,

I have some suggestions and questions.

1. In DMPROPERTIES, instead of 'timestamp_column' suggest to use
'timeseries_column'.
 so that it won't give an impression that only time stamp datatype is
supported and update the document with all the datatype supported.

2. Querying on this datamap table is also supported right ? supporting
changing plan for main table to refer datamap table is for user to avoid
changing his query or any other reason ?

3. If user has not created day granularity datamap, but just created hour
granularity datamap. When query has day granularity, data will be fetched
form hour granularity datamap and aggregated ? or data is fetched from main
table ?

Thanks,
Ajantha

On Mon, Sep 30, 2019 at 11:46 AM Akash Nilugal <[hidden email]>
wrote:

> Hi xuchuanyin,
>
> Thanks for the comments/Suggestions
>
> 1. Preaggregate is productized, but not the timeseries with preaggregate,
> i think you  got confused with that, if im right.
> 2. Limitations like, auto sampling or rollup, which we will be supporting
> now. Retention policies. etc
> 3. segmentTimestampMin, this i will consider in design.
> 4. RP is added as a separate task, i thought instead of maintaining two
> variables better to maintabin one and parse it. But i will consider your
> point based on feasibility during implementation.
> 5. We use an accumulator which takes list, so before writing index files
> we take the min max of the timestamp column and fill in accumulator and
> then we can access accumulator.value in driver after load is finished.
>
> Regards,
> Akash R Nilugal
>
> On 2019/09/28 10:46:31, xuchuanyin <[hidden email]> wrote:
> > Hi akash, glad to see the feature proposed and I have some questions
> about
> > this. Please notice that some of the following descriptions are comments
> > followed by '===' described in the design document attached in the
> > corresponding jira.
> >
> > 1.
> > "Currently carbondata supports timeseries on preaggregate datamap, but
> its
> > an alpha feature"
> > ===
> > It has been some time since the preaggregate datamap was introduced and
> it
> > is still **alpha**, why it is still not product-ready? Will the new
> feature
> > also come into the similar situation?
> >
> > 2.
> > "there are so many limitations when we compare and analyze the existing
> > timeseries database or projects which supports time series like apache
> druid
> > or influxdb"
> > ===
> > What are the actual limitations? Besides, please give an example of this.
> >
> > 3.
> > "Segment_Timestamp_Min"
> > ===
> > Suggest using camel-case style like 'segmentTimestampMin'
> >
> > 4.
> > "RP is way of telling the system, for how long the data should be kept"
> > ===
> > Since the function is simple, I'd suggest using 'retentionTime'=15 and
> > 'timeUnit'='day' instead of 'RP'='15_days'
> >
> > 5.
> > "When the data load is called for main table, use an spark accumulator to
> > get the maximum value of timestamp in that load and return to the load."
> > ===
> > How can you get the spark accumulator? The load is launched using
> > loading-by-dataframe not using global-sort-by-spark.
> >
> > 6.
> > For the rest of the content, still reading.
> >
> >
> >
> >
> > --
> > Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> >
>

Hi Ajantha,

Thanks for the queries and suggestions

1. Yes, this is a good suggestion, i ll include this change. Both date and timestamp columns are supported, will be updated in document.
2. yes, you are right.
3. you are right, if the day level is not available, then we will try to get the whole day data from hour level, if not availaible, as explained in design document, we will get the data from datamap UNION data from main table based on user query.

Regards,
Akash R Nilugal


On 2019/09/30 06:56:45, Ajantha Bhat <[hidden email]> wrote:

> + 1 ,
>
> I have some suggestions and questions.
>
> 1. In DMPROPERTIES, instead of 'timestamp_column' suggest to use
> 'timeseries_column'.
>  so that it won't give an impression that only time stamp datatype is
> supported and update the document with all the datatype supported.
>
> 2. Querying on this datamap table is also supported right ? supporting
> changing plan for main table to refer datamap table is for user to avoid
> changing his query or any other reason ?
>
> 3. If user has not created day granularity datamap, but just created hour
> granularity datamap. When query has day granularity, data will be fetched
> form hour granularity datamap and aggregated ? or data is fetched from main
> table ?
>
> Thanks,
> Ajantha
>
> On Mon, Sep 30, 2019 at 11:46 AM Akash Nilugal <[hidden email]>
> wrote:
>
> > Hi xuchuanyin,
> >
> > Thanks for the comments/Suggestions
> >
> > 1. Preaggregate is productized, but not the timeseries with preaggregate,
> > i think you  got confused with that, if im right.
> > 2. Limitations like, auto sampling or rollup, which we will be supporting
> > now. Retention policies. etc
> > 3. segmentTimestampMin, this i will consider in design.
> > 4. RP is added as a separate task, i thought instead of maintaining two
> > variables better to maintabin one and parse it. But i will consider your
> > point based on feasibility during implementation.
> > 5. We use an accumulator which takes list, so before writing index files
> > we take the min max of the timestamp column and fill in accumulator and
> > then we can access accumulator.value in driver after load is finished.
> >
> > Regards,
> > Akash R Nilugal
> >
> > On 2019/09/28 10:46:31, xuchuanyin <[hidden email]> wrote:
> > > Hi akash, glad to see the feature proposed and I have some questions
> > about
> > > this. Please notice that some of the following descriptions are comments
> > > followed by '===' described in the design document attached in the
> > > corresponding jira.
> > >
> > > 1.
> > > "Currently carbondata supports timeseries on preaggregate datamap, but
> > its
> > > an alpha feature"
> > > ===
> > > It has been some time since the preaggregate datamap was introduced and
> > it
> > > is still **alpha**, why it is still not product-ready? Will the new
> > feature
> > > also come into the similar situation?
> > >
> > > 2.
> > > "there are so many limitations when we compare and analyze the existing
> > > timeseries database or projects which supports time series like apache
> > druid
> > > or influxdb"
> > > ===
> > > What are the actual limitations? Besides, please give an example of this.
> > >
> > > 3.
> > > "Segment_Timestamp_Min"
> > > ===
> > > Suggest using camel-case style like 'segmentTimestampMin'
> > >
> > > 4.
> > > "RP is way of telling the system, for how long the data should be kept"
> > > ===
> > > Since the function is simple, I'd suggest using 'retentionTime'=15 and
> > > 'timeUnit'='day' instead of 'RP'='15_days'
> > >
> > > 5.
> > > "When the data load is called for main table, use an spark accumulator to
> > > get the maximum value of timestamp in that load and return to the load."
> > > ===
> > > How can you get the spark accumulator? The load is launched using
> > > loading-by-dataframe not using global-sort-by-spark.
> > >
> > > 6.
> > > For the rest of the content, still reading.
> > >
> > >
> > >
> > >
> > > --
> > > Sent from:
> > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> > >
> >
>

[kumarvishal09]

Hi Akash,

In this desing document you haven't mentioned how to handle data loading
for timeseries datamap for older segments[Existing table].
If the customer's main table data is also stored based on time[increasing
time] in different segments,he can use this feature as well.

We can discuss and finalize the solution.

-Regards
Kumar Vishal

On Mon, Sep 30, 2019 at 2:42 PM Akash Nilugal <[hidden email]>
wrote:

> Hi Ajantha,
>
> Thanks for the queries and suggestions
>
> 1. Yes, this is a good suggestion, i ll include this change. Both date and
> timestamp columns are supported, will be updated in document.
> 2. yes, you are right.
> 3. you are right, if the day level is not available, then we will try to
> get the whole day data from hour level, if not availaible, as explained in
> design document, we will get the data from datamap UNION data from main
> table based on user query.
>
> Regards,
> Akash R Nilugal
>
>
> On 2019/09/30 06:56:45, Ajantha Bhat <[hidden email]> wrote:
> > + 1 ,
> >
> > I have some suggestions and questions.
> >
> > 1. In DMPROPERTIES, instead of 'timestamp_column' suggest to use
> > 'timeseries_column'.
> >  so that it won't give an impression that only time stamp datatype is
> > supported and update the document with all the datatype supported.
> >
> > 2. Querying on this datamap table is also supported right ? supporting
> > changing plan for main table to refer datamap table is for user to avoid
> > changing his query or any other reason ?
> >
> > 3. If user has not created day granularity datamap, but just created hour
> > granularity datamap. When query has day granularity, data will be fetched
> > form hour granularity datamap and aggregated ? or data is fetched from
> main
> > table ?
> >
> > Thanks,
> > Ajantha
> >
> > On Mon, Sep 30, 2019 at 11:46 AM Akash Nilugal <[hidden email]>
> > wrote:
> >
> > > Hi xuchuanyin,
> > >
> > > Thanks for the comments/Suggestions
> > >
> > > 1. Preaggregate is productized, but not the timeseries with
> preaggregate,
> > > i think you  got confused with that, if im right.
> > > 2. Limitations like, auto sampling or rollup, which we will be
> supporting
> > > now. Retention policies. etc
> > > 3. segmentTimestampMin, this i will consider in design.
> > > 4. RP is added as a separate task, i thought instead of maintaining two
> > > variables better to maintabin one and parse it. But i will consider
> your
> > > point based on feasibility during implementation.
> > > 5. We use an accumulator which takes list, so before writing index
> files
> > > we take the min max of the timestamp column and fill in accumulator and
> > > then we can access accumulator.value in driver after load is finished.
> > >
> > > Regards,
> > > Akash R Nilugal
> > >
> > > On 2019/09/28 10:46:31, xuchuanyin <[hidden email]> wrote:
> > > > Hi akash, glad to see the feature proposed and I have some questions
> > > about
> > > > this. Please notice that some of the following descriptions are
> comments
> > > > followed by '===' described in the design document attached in the
> > > > corresponding jira.
> > > >
> > > > 1.
> > > > "Currently carbondata supports timeseries on preaggregate datamap,
> but
> > > its
> > > > an alpha feature"
> > > > ===
> > > > It has been some time since the preaggregate datamap was introduced
> and
> > > it
> > > > is still **alpha**, why it is still not product-ready? Will the new
> > > feature
> > > > also come into the similar situation?
> > > >
> > > > 2.
> > > > "there are so many limitations when we compare and analyze the
> existing
> > > > timeseries database or projects which supports time series like
> apache
> > > druid
> > > > or influxdb"
> > > > ===
> > > > What are the actual limitations? Besides, please give an example of
> this.
> > > >
> > > > 3.
> > > > "Segment_Timestamp_Min"
> > > > ===
> > > > Suggest using camel-case style like 'segmentTimestampMin'
> > > >
> > > > 4.
> > > > "RP is way of telling the system, for how long the data should be
> kept"
> > > > ===
> > > > Since the function is simple, I'd suggest using 'retentionTime'=15
> and
> > > > 'timeUnit'='day' instead of 'RP'='15_days'
> > > >
> > > > 5.
> > > > "When the data load is called for main table, use an spark
> accumulator to
> > > > get the maximum value of timestamp in that load and return to the
> load."
> > > > ===
> > > > How can you get the spark accumulator? The load is launched using
> > > > loading-by-dataframe not using global-sort-by-spark.
> > > >
> > > > 6.
> > > > For the rest of the content, still reading.
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Sent from:
> > >
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> > > >
> > >
> >
>

Hi vishal,

In the design document, in the impacted analysis section, there is a topic compatibility/legacy stores, so basically For old tables when the datamap is created, we load all the timeseries datamaps with different granularity. I think this should do fine, please let me know for further suggestions/comments.

Regards,
Akash R Nilugal

On 2019/09/30 17:09:44, Kumar Vishal <[hidden email]> wrote:

> Hi Akash,
>
> In this desing document you haven't mentioned how to handle data loading
> for timeseries datamap for older segments[Existing table].
> If the customer's main table data is also stored based on time[increasing
> time] in different segments,he can use this feature as well.
>
> We can discuss and finalize the solution.
>
> -Regards
> Kumar Vishal
>
> On Mon, Sep 30, 2019 at 2:42 PM Akash Nilugal <[hidden email]>
> wrote:
>
> > Hi Ajantha,
> >
> > Thanks for the queries and suggestions
> >
> > 1. Yes, this is a good suggestion, i ll include this change. Both date and
> > timestamp columns are supported, will be updated in document.
> > 2. yes, you are right.
> > 3. you are right, if the day level is not available, then we will try to
> > get the whole day data from hour level, if not availaible, as explained in
> > design document, we will get the data from datamap UNION data from main
> > table based on user query.
> >
> > Regards,
> > Akash R Nilugal
> >
> >
> > On 2019/09/30 06:56:45, Ajantha Bhat <[hidden email]> wrote:
> > > + 1 ,
> > >
> > > I have some suggestions and questions.
> > >
> > > 1. In DMPROPERTIES, instead of 'timestamp_column' suggest to use
> > > 'timeseries_column'.
> > >  so that it won't give an impression that only time stamp datatype is
> > > supported and update the document with all the datatype supported.
> > >
> > > 2. Querying on this datamap table is also supported right ? supporting
> > > changing plan for main table to refer datamap table is for user to avoid
> > > changing his query or any other reason ?
> > >
> > > 3. If user has not created day granularity datamap, but just created hour
> > > granularity datamap. When query has day granularity, data will be fetched
> > > form hour granularity datamap and aggregated ? or data is fetched from
> > main
> > > table ?
> > >
> > > Thanks,
> > > Ajantha
> > >
> > > On Mon, Sep 30, 2019 at 11:46 AM Akash Nilugal <[hidden email]>
> > > wrote:
> > >
> > > > Hi xuchuanyin,
> > > >
> > > > Thanks for the comments/Suggestions
> > > >
> > > > 1. Preaggregate is productized, but not the timeseries with
> > preaggregate,
> > > > i think you  got confused with that, if im right.
> > > > 2. Limitations like, auto sampling or rollup, which we will be
> > supporting
> > > > now. Retention policies. etc
> > > > 3. segmentTimestampMin, this i will consider in design.
> > > > 4. RP is added as a separate task, i thought instead of maintaining two
> > > > variables better to maintabin one and parse it. But i will consider
> > your
> > > > point based on feasibility during implementation.
> > > > 5. We use an accumulator which takes list, so before writing index
> > files
> > > > we take the min max of the timestamp column and fill in accumulator and
> > > > then we can access accumulator.value in driver after load is finished.
> > > >
> > > > Regards,
> > > > Akash R Nilugal
> > > >
> > > > On 2019/09/28 10:46:31, xuchuanyin <[hidden email]> wrote:
> > > > > Hi akash, glad to see the feature proposed and I have some questions
> > > > about
> > > > > this. Please notice that some of the following descriptions are
> > comments
> > > > > followed by '===' described in the design document attached in the
> > > > > corresponding jira.
> > > > >
> > > > > 1.
> > > > > "Currently carbondata supports timeseries on preaggregate datamap,
> > but
> > > > its
> > > > > an alpha feature"
> > > > > ===
> > > > > It has been some time since the preaggregate datamap was introduced
> > and
> > > > it
> > > > > is still **alpha**, why it is still not product-ready? Will the new
> > > > feature
> > > > > also come into the similar situation?
> > > > >
> > > > > 2.
> > > > > "there are so many limitations when we compare and analyze the
> > existing
> > > > > timeseries database or projects which supports time series like
> > apache
> > > > druid
> > > > > or influxdb"
> > > > > ===
> > > > > What are the actual limitations? Besides, please give an example of
> > this.
> > > > >
> > > > > 3.
> > > > > "Segment_Timestamp_Min"
> > > > > ===
> > > > > Suggest using camel-case style like 'segmentTimestampMin'
> > > > >
> > > > > 4.
> > > > > "RP is way of telling the system, for how long the data should be
> > kept"
> > > > > ===
> > > > > Since the function is simple, I'd suggest using 'retentionTime'=15
> > and
> > > > > 'timeUnit'='day' instead of 'RP'='15_days'
> > > > >
> > > > > 5.
> > > > > "When the data load is called for main table, use an spark
> > accumulator to
> > > > > get the maximum value of timestamp in that load and return to the
> > load."
> > > > > ===
> > > > > How can you get the spark accumulator? The load is launched using
> > > > > loading-by-dataframe not using global-sort-by-spark.
> > > > >
> > > > > 6.
> > > > > For the rest of the content, still reading.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sent from:
> > > >
> > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> > > > >
> > > >
> > >
> >
>

[BabuLal]

Hi Akash, Thanks for Time Series DataMap proposal.
Please check below Points.

1. During Query Planing Change Union to Union All , Otherwise will loose
row if same value appears.
2. Whether system start load for next granularity level table as soon it
matches the data condition or next granularity level table has to wait till
current  granularity level table is finished ? please handle if possible.
3. Add Configuration to load multiple Ranges at a time(across granularity
tables).
4. Please check if Current data loading min ,max is enough to find current
load . No need to refer the DataMap's min,max because data loading Range
prepration can go wrong if loading happens from multiple driver . i think
below rules are enough for loading.
    4.a. Create MV should should sync data.   On any failure Rebuild should
sync again till than MV will be disabled.
    4.b.  Each load has independent Ranges and should load only those
ranges. Any failure MV may go in disable state(only if intermediate ranges
load is failed ,last loads failure will NOT make MV disable).
5. We can make Data loading sync because anyway queries can be served from
fact table if any segments is in-progress in  Datamap.
6. In Data loading Pipleline ,failures in intermediate time series datamap,
still we can continue loading next level data. (ignore if already handled).
   For Example.
    DataMaps:- Hour,Day,Month Level
    Load Data(10 day):- 2018-01-01 01:00:00 to 2018-01-10 01:00:00
      Failure in hour level during below range
        2018-01-06 01:00:00 to 2018-01-06 01:00:00
     This point of time Hour level has 5 day data.so start loading on day
level .
7. Add SubTask to support loading of in-between missing time.(Incremental
but old records if timeseries device stopped working for some time).

On Tue, Oct 1, 2019 at 10:41 AM Akash Nilugal <[hidden email]>
wrote:

> Hi vishal,
>
> In the design document, in the impacted analysis section, there is a topic
> compatibility/legacy stores, so basically For old tables when the datamap
> is created, we load all the timeseries datamaps with different granularity.
> I think this should do fine, please let me know for further
> suggestions/comments.
>
> Regards,
> Akash R Nilugal
>
> On 2019/09/30 17:09:44, Kumar Vishal <[hidden email]> wrote:
> > Hi Akash,
> >
> > In this desing document you haven't mentioned how to handle data loading
> > for timeseries datamap for older segments[Existing table].
> > If the customer's main table data is also stored based on time[increasing
> > time] in different segments,he can use this feature as well.
> >
> > We can discuss and finalize the solution.
> >
> > -Regards
> > Kumar Vishal
> >
> > On Mon, Sep 30, 2019 at 2:42 PM Akash Nilugal <[hidden email]>
> > wrote:
> >
> > > Hi Ajantha,
> > >
> > > Thanks for the queries and suggestions
> > >
> > > 1. Yes, this is a good suggestion, i ll include this change. Both date
> and
> > > timestamp columns are supported, will be updated in document.
> > > 2. yes, you are right.
> > > 3. you are right, if the day level is not available, then we will try
> to
> > > get the whole day data from hour level, if not availaible, as
> explained in
> > > design document, we will get the data from datamap UNION data from main
> > > table based on user query.
> > >
> > > Regards,
> > > Akash R Nilugal
> > >
> > >
> > > On 2019/09/30 06:56:45, Ajantha Bhat <[hidden email]> wrote:
> > > > + 1 ,
> > > >
> > > > I have some suggestions and questions.
> > > >
> > > > 1. In DMPROPERTIES, instead of 'timestamp_column' suggest to use
> > > > 'timeseries_column'.
> > > >  so that it won't give an impression that only time stamp datatype is
> > > > supported and update the document with all the datatype supported.
> > > >
> > > > 2. Querying on this datamap table is also supported right ?
> supporting
> > > > changing plan for main table to refer datamap table is for user to
> avoid
> > > > changing his query or any other reason ?
> > > >
> > > > 3. If user has not created day granularity datamap, but just created
> hour
> > > > granularity datamap. When query has day granularity, data will be
> fetched
> > > > form hour granularity datamap and aggregated ? or data is fetched
> from
> > > main
> > > > table ?
> > > >
> > > > Thanks,
> > > > Ajantha
> > > >
> > > > On Mon, Sep 30, 2019 at 11:46 AM Akash Nilugal <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > Hi xuchuanyin,
> > > > >
> > > > > Thanks for the comments/Suggestions
> > > > >
> > > > > 1. Preaggregate is productized, but not the timeseries with
> > > preaggregate,
> > > > > i think you  got confused with that, if im right.
> > > > > 2. Limitations like, auto sampling or rollup, which we will be
> > > supporting
> > > > > now. Retention policies. etc
> > > > > 3. segmentTimestampMin, this i will consider in design.
> > > > > 4. RP is added as a separate task, i thought instead of
> maintaining two
> > > > > variables better to maintabin one and parse it. But i will consider
> > > your
> > > > > point based on feasibility during implementation.
> > > > > 5. We use an accumulator which takes list, so before writing index
> > > files
> > > > > we take the min max of the timestamp column and fill in
> accumulator and
> > > > > then we can access accumulator.value in driver after load is
> finished.
> > > > >
> > > > > Regards,
> > > > > Akash R Nilugal
> > > > >
> > > > > On 2019/09/28 10:46:31, xuchuanyin <[hidden email]> wrote:
> > > > > > Hi akash, glad to see the feature proposed and I have some
> questions
> > > > > about
> > > > > > this. Please notice that some of the following descriptions are
> > > comments
> > > > > > followed by '===' described in the design document attached in
> the
> > > > > > corresponding jira.
> > > > > >
> > > > > > 1.
> > > > > > "Currently carbondata supports timeseries on preaggregate
> datamap,
> > > but
> > > > > its
> > > > > > an alpha feature"
> > > > > > ===
> > > > > > It has been some time since the preaggregate datamap was
> introduced
> > > and
> > > > > it
> > > > > > is still **alpha**, why it is still not product-ready? Will the
> new
> > > > > feature
> > > > > > also come into the similar situation?
> > > > > >
> > > > > > 2.
> > > > > > "there are so many limitations when we compare and analyze the
> > > existing
> > > > > > timeseries database or projects which supports time series like
> > > apache
> > > > > druid
> > > > > > or influxdb"
> > > > > > ===
> > > > > > What are the actual limitations? Besides, please give an example
> of
> > > this.
> > > > > >
> > > > > > 3.
> > > > > > "Segment_Timestamp_Min"
> > > > > > ===
> > > > > > Suggest using camel-case style like 'segmentTimestampMin'
> > > > > >
> > > > > > 4.
> > > > > > "RP is way of telling the system, for how long the data should be
> > > kept"
> > > > > > ===
> > > > > > Since the function is simple, I'd suggest using
> 'retentionTime'=15
> > > and
> > > > > > 'timeUnit'='day' instead of 'RP'='15_days'
> > > > > >
> > > > > > 5.
> > > > > > "When the data load is called for main table, use an spark
> > > accumulator to
> > > > > > get the maximum value of timestamp in that load and return to the
> > > load."
> > > > > > ===
> > > > > > How can you get the spark accumulator? The load is launched using
> > > > > > loading-by-dataframe not using global-sort-by-spark.
> > > > > >
> > > > > > 6.
> > > > > > For the rest of the content, still reading.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Sent from:
> > > > >
> > >
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> > > > > >
> > > > >
> > > >
> > >
> >
>

Hi vishal,

I got your point, i have changed accordingly and updated the document in jira, please check

Regards,
Akash R Nilugal

On 2019/09/30 17:09:44, Kumar Vishal <[hidden email]> wrote:

> Hi Akash,
>
> In this desing document you haven't mentioned how to handle data loading
> for timeseries datamap for older segments[Existing table].
> If the customer's main table data is also stored based on time[increasing
> time] in different segments,he can use this feature as well.
>
> We can discuss and finalize the solution.
>
> -Regards
> Kumar Vishal
>
> On Mon, Sep 30, 2019 at 2:42 PM Akash Nilugal <[hidden email]>
> wrote:
>
> > Hi Ajantha,
> >
> > Thanks for the queries and suggestions
> >
> > 1. Yes, this is a good suggestion, i ll include this change. Both date and
> > timestamp columns are supported, will be updated in document.
> > 2. yes, you are right.
> > 3. you are right, if the day level is not available, then we will try to
> > get the whole day data from hour level, if not availaible, as explained in
> > design document, we will get the data from datamap UNION data from main
> > table based on user query.
> >
> > Regards,
> > Akash R Nilugal
> >
> >
> > On 2019/09/30 06:56:45, Ajantha Bhat <[hidden email]> wrote:
> > > + 1 ,
> > >
> > > I have some suggestions and questions.
> > >
> > > 1. In DMPROPERTIES, instead of 'timestamp_column' suggest to use
> > > 'timeseries_column'.
> > >  so that it won't give an impression that only time stamp datatype is
> > > supported and update the document with all the datatype supported.
> > >
> > > 2. Querying on this datamap table is also supported right ? supporting
> > > changing plan for main table to refer datamap table is for user to avoid
> > > changing his query or any other reason ?
> > >
> > > 3. If user has not created day granularity datamap, but just created hour
> > > granularity datamap. When query has day granularity, data will be fetched
> > > form hour granularity datamap and aggregated ? or data is fetched from
> > main
> > > table ?
> > >
> > > Thanks,
> > > Ajantha
> > >
> > > On Mon, Sep 30, 2019 at 11:46 AM Akash Nilugal <[hidden email]>
> > > wrote:
> > >
> > > > Hi xuchuanyin,
> > > >
> > > > Thanks for the comments/Suggestions
> > > >
> > > > 1. Preaggregate is productized, but not the timeseries with
> > preaggregate,
> > > > i think you  got confused with that, if im right.
> > > > 2. Limitations like, auto sampling or rollup, which we will be
> > supporting
> > > > now. Retention policies. etc
> > > > 3. segmentTimestampMin, this i will consider in design.
> > > > 4. RP is added as a separate task, i thought instead of maintaining two
> > > > variables better to maintabin one and parse it. But i will consider
> > your
> > > > point based on feasibility during implementation.
> > > > 5. We use an accumulator which takes list, so before writing index
> > files
> > > > we take the min max of the timestamp column and fill in accumulator and
> > > > then we can access accumulator.value in driver after load is finished.
> > > >
> > > > Regards,
> > > > Akash R Nilugal
> > > >
> > > > On 2019/09/28 10:46:31, xuchuanyin <[hidden email]> wrote:
> > > > > Hi akash, glad to see the feature proposed and I have some questions
> > > > about
> > > > > this. Please notice that some of the following descriptions are
> > comments
> > > > > followed by '===' described in the design document attached in the
> > > > > corresponding jira.
> > > > >
> > > > > 1.
> > > > > "Currently carbondata supports timeseries on preaggregate datamap,
> > but
> > > > its
> > > > > an alpha feature"
> > > > > ===
> > > > > It has been some time since the preaggregate datamap was introduced
> > and
> > > > it
> > > > > is still **alpha**, why it is still not product-ready? Will the new
> > > > feature
> > > > > also come into the similar situation?
> > > > >
> > > > > 2.
> > > > > "there are so many limitations when we compare and analyze the
> > existing
> > > > > timeseries database or projects which supports time series like
> > apache
> > > > druid
> > > > > or influxdb"
> > > > > ===
> > > > > What are the actual limitations? Besides, please give an example of
> > this.
> > > > >
> > > > > 3.
> > > > > "Segment_Timestamp_Min"
> > > > > ===
> > > > > Suggest using camel-case style like 'segmentTimestampMin'
> > > > >
> > > > > 4.
> > > > > "RP is way of telling the system, for how long the data should be
> > kept"
> > > > > ===
> > > > > Since the function is simple, I'd suggest using 'retentionTime'=15
> > and
> > > > > 'timeUnit'='day' instead of 'RP'='15_days'
> > > > >
> > > > > 5.
> > > > > "When the data load is called for main table, use an spark
> > accumulator to
> > > > > get the maximum value of timestamp in that load and return to the
> > load."
> > > > > ===
> > > > > How can you get the spark accumulator? The load is launched using
> > > > > loading-by-dataframe not using global-sort-by-spark.
> > > > >
> > > > > 6.
> > > > > For the rest of the content, still reading.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sent from:
> > > >
> > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> > > > >
> > > >
> > >
> >
>

Hi Babu,

Thanks for the inputs. Please find the comments
1. I will change from Union to UnionAll
2. For auto datamap loading, once the data is loaded to lower level granularity datamap, then we load the higher level datamap from the lower level datamap. But as per your point, i think you are telling to load from main table itself.
3. similar to 2nd point, whether to need configuration or not we can decide i think.
4. a. I think the max of the datamap is required to decide the range for the load, because in case of failure case, we may need.
b. This point will be taken care.
5. Yes, dataload is sync based on current design, as it is non lazy, it will happen with main table load only.
6. Yes, this will be handled.
7. Already added a task in jira.
On 2019/10/01 08:50:05, babu lal jangir <[hidden email]> wrote:

> Hi Akash, Thanks for Time Series DataMap proposal.
> Please check below Points.
>
> 1. During Query Planing Change Union to Union All , Otherwise will loose
> row if same value appears.
> 2. Whether system start load for next granularity level table as soon it
> matches the data condition or next granularity level table has to wait till
> current  granularity level table is finished ? please handle if possible.
> 3. Add Configuration to load multiple Ranges at a time(across granularity
> tables).
> 4. Please check if Current data loading min ,max is enough to find current
> load . No need to refer the DataMap's min,max because data loading Range
> prepration can go wrong if loading happens from multiple driver . i think
> below rules are enough for loading.
>     4.a. Create MV should should sync data.   On any failure Rebuild should
> sync again till than MV will be disabled.
>     4.b.  Each load has independent Ranges and should load only those
> ranges. Any failure MV may go in disable state(only if intermediate ranges
> load is failed ,last loads failure will NOT make MV disable).
> 5. We can make Data loading sync because anyway queries can be served from
> fact table if any segments is in-progress in  Datamap.
> 6. In Data loading Pipleline ,failures in intermediate time series datamap,
> still we can continue loading next level data. (ignore if already handled).
>    For Example.
>     DataMaps:- Hour,Day,Month Level
>     Load Data(10 day):- 2018-01-01 01:00:00 to 2018-01-10 01:00:00
>       Failure in hour level during below range
>         2018-01-06 01:00:00 to 2018-01-06 01:00:00
>      This point of time Hour level has 5 day data.so start loading on day
> level .
> 7. Add SubTask to support loading of in-between missing time.(Incremental
> but old records if timeseries device stopped working for some time).
>
> On Tue, Oct 1, 2019 at 10:41 AM Akash Nilugal <[hidden email]>
> wrote:
>
> > Hi vishal,
> >
> > In the design document, in the impacted analysis section, there is a topic
> > compatibility/legacy stores, so basically For old tables when the datamap
> > is created, we load all the timeseries datamaps with different granularity.
> > I think this should do fine, please let me know for further
> > suggestions/comments.
> >
> > Regards,
> > Akash R Nilugal
> >
> > On 2019/09/30 17:09:44, Kumar Vishal <[hidden email]> wrote:
> > > Hi Akash,
> > >
> > > In this desing document you haven't mentioned how to handle data loading
> > > for timeseries datamap for older segments[Existing table].
> > > If the customer's main table data is also stored based on time[increasing
> > > time] in different segments,he can use this feature as well.
> > >
> > > We can discuss and finalize the solution.
> > >
> > > -Regards
> > > Kumar Vishal
> > >
> > > On Mon, Sep 30, 2019 at 2:42 PM Akash Nilugal <[hidden email]>
> > > wrote:
> > >
> > > > Hi Ajantha,
> > > >
> > > > Thanks for the queries and suggestions
> > > >
> > > > 1. Yes, this is a good suggestion, i ll include this change. Both date
> > and
> > > > timestamp columns are supported, will be updated in document.
> > > > 2. yes, you are right.
> > > > 3. you are right, if the day level is not available, then we will try
> > to
> > > > get the whole day data from hour level, if not availaible, as
> > explained in
> > > > design document, we will get the data from datamap UNION data from main
> > > > table based on user query.
> > > >
> > > > Regards,
> > > > Akash R Nilugal
> > > >
> > > >
> > > > On 2019/09/30 06:56:45, Ajantha Bhat <[hidden email]> wrote:
> > > > > + 1 ,
> > > > >
> > > > > I have some suggestions and questions.
> > > > >
> > > > > 1. In DMPROPERTIES, instead of 'timestamp_column' suggest to use
> > > > > 'timeseries_column'.
> > > > >  so that it won't give an impression that only time stamp datatype is
> > > > > supported and update the document with all the datatype supported.
> > > > >
> > > > > 2. Querying on this datamap table is also supported right ?
> > supporting
> > > > > changing plan for main table to refer datamap table is for user to
> > avoid
> > > > > changing his query or any other reason ?
> > > > >
> > > > > 3. If user has not created day granularity datamap, but just created
> > hour
> > > > > granularity datamap. When query has day granularity, data will be
> > fetched
> > > > > form hour granularity datamap and aggregated ? or data is fetched
> > from
> > > > main
> > > > > table ?
> > > > >
> > > > > Thanks,
> > > > > Ajantha
> > > > >
> > > > > On Mon, Sep 30, 2019 at 11:46 AM Akash Nilugal <
> > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Hi xuchuanyin,
> > > > > >
> > > > > > Thanks for the comments/Suggestions
> > > > > >
> > > > > > 1. Preaggregate is productized, but not the timeseries with
> > > > preaggregate,
> > > > > > i think you  got confused with that, if im right.
> > > > > > 2. Limitations like, auto sampling or rollup, which we will be
> > > > supporting
> > > > > > now. Retention policies. etc
> > > > > > 3. segmentTimestampMin, this i will consider in design.
> > > > > > 4. RP is added as a separate task, i thought instead of
> > maintaining two
> > > > > > variables better to maintabin one and parse it. But i will consider
> > > > your
> > > > > > point based on feasibility during implementation.
> > > > > > 5. We use an accumulator which takes list, so before writing index
> > > > files
> > > > > > we take the min max of the timestamp column and fill in
> > accumulator and
> > > > > > then we can access accumulator.value in driver after load is
> > finished.
> > > > > >
> > > > > > Regards,
> > > > > > Akash R Nilugal
> > > > > >
> > > > > > On 2019/09/28 10:46:31, xuchuanyin <[hidden email]> wrote:
> > > > > > > Hi akash, glad to see the feature proposed and I have some
> > questions
> > > > > > about
> > > > > > > this. Please notice that some of the following descriptions are
> > > > comments
> > > > > > > followed by '===' described in the design document attached in
> > the
> > > > > > > corresponding jira.
> > > > > > >
> > > > > > > 1.
> > > > > > > "Currently carbondata supports timeseries on preaggregate
> > datamap,
> > > > but
> > > > > > its
> > > > > > > an alpha feature"
> > > > > > > ===
> > > > > > > It has been some time since the preaggregate datamap was
> > introduced
> > > > and
> > > > > > it
> > > > > > > is still **alpha**, why it is still not product-ready? Will the
> > new
> > > > > > feature
> > > > > > > also come into the similar situation?
> > > > > > >
> > > > > > > 2.
> > > > > > > "there are so many limitations when we compare and analyze the
> > > > existing
> > > > > > > timeseries database or projects which supports time series like
> > > > apache
> > > > > > druid
> > > > > > > or influxdb"
> > > > > > > ===
> > > > > > > What are the actual limitations? Besides, please give an example
> > of
> > > > this.
> > > > > > >
> > > > > > > 3.
> > > > > > > "Segment_Timestamp_Min"
> > > > > > > ===
> > > > > > > Suggest using camel-case style like 'segmentTimestampMin'
> > > > > > >
> > > > > > > 4.
> > > > > > > "RP is way of telling the system, for how long the data should be
> > > > kept"
> > > > > > > ===
> > > > > > > Since the function is simple, I'd suggest using
> > 'retentionTime'=15
> > > > and
> > > > > > > 'timeUnit'='day' instead of 'RP'='15_days'
> > > > > > >
> > > > > > > 5.
> > > > > > > "When the data load is called for main table, use an spark
> > > > accumulator to
> > > > > > > get the maximum value of timestamp in that load and return to the
> > > > load."
> > > > > > > ===
> > > > > > > How can you get the spark accumulator? The load is launched using
> > > > > > > loading-by-dataframe not using global-sort-by-spark.
> > > > > > >
> > > > > > > 6.
> > > > > > > For the rest of the content, still reading.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Sent from:
> > > > > >
> > > >
> > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[ravipesala]

Hi Akash,

I have following suggestions.

1. I think it is redundant to use granularity inside create datamap, user can use the respective granularity UDF in his query like time(1h) or time(1d) etc.

2. Better create separate RP commands and let user add the RP on the datamap or even on the main table also. It would be more manageable if you independent feature for RP instead of including in datamap.

3. I am not getting why exactly we need accumulator instead of using index min/max? Can you explain with some scenario

4. Why to store min/max at segment level? We can get from datamap also right?

4.  Union with high granularity tables to low granularity tables are really needed? Any other time series DB is doing it? Or any known use case we have?

Regards,
Ravindra.

> On 1 Oct 2019, at 5:49 PM, Akash Nilugal <[hidden email]> wrote:
>
> Hi Babu,
>
> Thanks for the inputs. Please find the comments
> 1. I will change from Union to UnionAll
> 2. For auto datamap loading, once the data is loaded to lower level granularity datamap, then we load the higher level datamap from the lower level datamap. But as per your point, i think you are telling to load from main table itself.
> 3. similar to 2nd point, whether to need configuration or not we can decide i think.
> 4. a. I think the max of the datamap is required to decide the range for the load, because in case of failure case, we may need.
> b. This point will be taken care.
> 5. Yes, dataload is sync based on current design, as it is non lazy, it will happen with main table load only.
> 6. Yes, this will be handled.
> 7. Already added a task in jira.
> On 2019/10/01 08:50:05, babu lal jangir <[hidden email]> wrote:
>> Hi Akash, Thanks for Time Series DataMap proposal.
>> Please check below Points.
>>
>> 1. During Query Planing Change Union to Union All , Otherwise will loose
>> row if same value appears.
>> 2. Whether system start load for next granularity level table as soon it
>> matches the data condition or next granularity level table has to wait till
>> current  granularity level table is finished ? please handle if possible.
>> 3. Add Configuration to load multiple Ranges at a time(across granularity
>> tables).
>> 4. Please check if Current data loading min ,max is enough to find current
>> load . No need to refer the DataMap's min,max because data loading Range
>> prepration can go wrong if loading happens from multiple driver . i think
>> below rules are enough for loading.
>>    4.a. Create MV should should sync data.   On any failure Rebuild should
>> sync again till than MV will be disabled.
>>    4.b.  Each load has independent Ranges and should load only those
>> ranges. Any failure MV may go in disable state(only if intermediate ranges
>> load is failed ,last loads failure will NOT make MV disable).
>> 5. We can make Data loading sync because anyway queries can be served from
>> fact table if any segments is in-progress in  Datamap.
>> 6. In Data loading Pipleline ,failures in intermediate time series datamap,
>> still we can continue loading next level data. (ignore if already handled).
>>   For Example.
>>    DataMaps:- Hour,Day,Month Level
>>    Load Data(10 day):- 2018-01-01 01:00:00 to 2018-01-10 01:00:00
>>      Failure in hour level during below range
>>        2018-01-06 01:00:00 to 2018-01-06 01:00:00
>>     This point of time Hour level has 5 day data.so start loading on day
>> level .
>> 7. Add SubTask to support loading of in-between missing time.(Incremental
>> but old records if timeseries device stopped working for some time).
>>
>> On Tue, Oct 1, 2019 at 10:41 AM Akash Nilugal <[hidden email]>
>> wrote:
>>
>>> Hi vishal,
>>>
>>> In the design document, in the impacted analysis section, there is a topic
>>> compatibility/legacy stores, so basically For old tables when the datamap
>>> is created, we load all the timeseries datamaps with different granularity.
>>> I think this should do fine, please let me know for further
>>> suggestions/comments.
>>>
>>> Regards,
>>> Akash R Nilugal
>>>
>>> On 2019/09/30 17:09:44, Kumar Vishal <[hidden email]> wrote:
>>>> Hi Akash,
>>>>
>>>> In this desing document you haven't mentioned how to handle data loading
>>>> for timeseries datamap for older segments[Existing table].
>>>> If the customer's main table data is also stored based on time[increasing
>>>> time] in different segments,he can use this feature as well.
>>>>
>>>> We can discuss and finalize the solution.
>>>>
>>>> -Regards
>>>> Kumar Vishal
>>>>
>>>> On Mon, Sep 30, 2019 at 2:42 PM Akash Nilugal <[hidden email]>
>>>> wrote:
>>>>
>>>>> Hi Ajantha,
>>>>>
>>>>> Thanks for the queries and suggestions
>>>>>
>>>>> 1. Yes, this is a good suggestion, i ll include this change. Both date
>>> and
>>>>> timestamp columns are supported, will be updated in document.
>>>>> 2. yes, you are right.
>>>>> 3. you are right, if the day level is not available, then we will try
>>> to
>>>>> get the whole day data from hour level, if not availaible, as
>>> explained in
>>>>> design document, we will get the data from datamap UNION data from main
>>>>> table based on user query.
>>>>>
>>>>> Regards,
>>>>> Akash R Nilugal
>>>>>
>>>>>
>>>>> On 2019/09/30 06:56:45, Ajantha Bhat <[hidden email]> wrote:
>>>>>> + 1 ,
>>>>>>
>>>>>> I have some suggestions and questions.
>>>>>>
>>>>>> 1. In DMPROPERTIES, instead of 'timestamp_column' suggest to use
>>>>>> 'timeseries_column'.
>>>>>> so that it won't give an impression that only time stamp datatype is
>>>>>> supported and update the document with all the datatype supported.
>>>>>>
>>>>>> 2. Querying on this datamap table is also supported right ?
>>> supporting
>>>>>> changing plan for main table to refer datamap table is for user to
>>> avoid
>>>>>> changing his query or any other reason ?
>>>>>>
>>>>>> 3. If user has not created day granularity datamap, but just created
>>> hour
>>>>>> granularity datamap. When query has day granularity, data will be
>>> fetched
>>>>>> form hour granularity datamap and aggregated ? or data is fetched
>>> from
>>>>> main
>>>>>> table ?
>>>>>>
>>>>>> Thanks,
>>>>>> Ajantha
>>>>>>
>>>>>> On Mon, Sep 30, 2019 at 11:46 AM Akash Nilugal <
>>> [hidden email]>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi xuchuanyin,
>>>>>>>
>>>>>>> Thanks for the comments/Suggestions
>>>>>>>
>>>>>>> 1. Preaggregate is productized, but not the timeseries with
>>>>> preaggregate,
>>>>>>> i think you  got confused with that, if im right.
>>>>>>> 2. Limitations like, auto sampling or rollup, which we will be
>>>>> supporting
>>>>>>> now. Retention policies. etc
>>>>>>> 3. segmentTimestampMin, this i will consider in design.
>>>>>>> 4. RP is added as a separate task, i thought instead of
>>> maintaining two
>>>>>>> variables better to maintabin one and parse it. But i will consider
>>>>> your
>>>>>>> point based on feasibility during implementation.
>>>>>>> 5. We use an accumulator which takes list, so before writing index
>>>>> files
>>>>>>> we take the min max of the timestamp column and fill in
>>> accumulator and
>>>>>>> then we can access accumulator.value in driver after load is
>>> finished.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Akash R Nilugal
>>>>>>>
>>>>>>> On 2019/09/28 10:46:31, xuchuanyin <[hidden email]> wrote:
>>>>>>>> Hi akash, glad to see the feature proposed and I have some
>>> questions
>>>>>>> about
>>>>>>>> this. Please notice that some of the following descriptions are
>>>>> comments
>>>>>>>> followed by '===' described in the design document attached in
>>> the
>>>>>>>> corresponding jira.
>>>>>>>>
>>>>>>>> 1.
>>>>>>>> "Currently carbondata supports timeseries on preaggregate
>>> datamap,
>>>>> but
>>>>>>> its
>>>>>>>> an alpha feature"
>>>>>>>> ===
>>>>>>>> It has been some time since the preaggregate datamap was
>>> introduced
>>>>> and
>>>>>>> it
>>>>>>>> is still **alpha**, why it is still not product-ready? Will the
>>> new
>>>>>>> feature
>>>>>>>> also come into the similar situation?
>>>>>>>>
>>>>>>>> 2.
>>>>>>>> "there are so many limitations when we compare and analyze the
>>>>> existing
>>>>>>>> timeseries database or projects which supports time series like
>>>>> apache
>>>>>>> druid
>>>>>>>> or influxdb"
>>>>>>>> ===
>>>>>>>> What are the actual limitations? Besides, please give an example
>>> of
>>>>> this.
>>>>>>>>
>>>>>>>> 3.
>>>>>>>> "Segment_Timestamp_Min"
>>>>>>>> ===
>>>>>>>> Suggest using camel-case style like 'segmentTimestampMin'
>>>>>>>>
>>>>>>>> 4.
>>>>>>>> "RP is way of telling the system, for how long the data should be
>>>>> kept"
>>>>>>>> ===
>>>>>>>> Since the function is simple, I'd suggest using
>>> 'retentionTime'=15
>>>>> and
>>>>>>>> 'timeUnit'='day' instead of 'RP'='15_days'
>>>>>>>>
>>>>>>>> 5.
>>>>>>>> "When the data load is called for main table, use an spark
>>>>> accumulator to
>>>>>>>> get the maximum value of timestamp in that load and return to the
>>>>> load."
>>>>>>>> ===
>>>>>>>> How can you get the spark accumulator? The load is launched using
>>>>>>>> loading-by-dataframe not using global-sort-by-spark.
>>>>>>>>
>>>>>>>> 6.
>>>>>>>> For the rest of the content, still reading.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Sent from:
>>>>>>>
>>>>>
>>> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>

Hi Ravi,

1. I forgot to mention the CTAS query in the create datamap statement, i have updated the document, during create datamap user can give granularity, during query just the UDF. That should be fine right.
2. I think may be we can mention the RP policy in DM properties also, and then may be we provide add RP, drop RP, alter RP for existing and older datamaps. RP will be taken as a separate subtask and will be handled in later part. That should be fine i tink.
3. Actually consider a scenario when datamap is already created, then load happened to main table, then i use accumulator to get all the min max to driver, so that i can avoid reading index file in driver in order to load to datamap.
         other scenario is when main table already has segments and then datamap is created, the we will read index files from each segments to decide the min max of timestamp column.
4. We are not storing min max in main table  table status. We are storing in datamap table's table status file, so that it will be used to prepare the plan during the query phase.

5. Other timeseries db supports only getting the data present in hour or day .. aggregated data. Since we cannot miss the data, plan is to get the data like higher to lower. May be it does not make much difference when its from minute to second, but it makes difference from year to month , so that we cannot avoid aggregations from main table.


Regards,
Akash R Nilugal

On 2019/10/04 11:35:46, Ravindra Pesala <[hidden email]> wrote:

> Hi Akash,
>
> I have following suggestions.
>
> 1. I think it is redundant to use granularity inside create datamap, user can use the respective granularity UDF in his query like time(1h) or time(1d) etc.
>
> 2. Better create separate RP commands and let user add the RP on the datamap or even on the main table also. It would be more manageable if you independent feature for RP instead of including in datamap.
>
> 3. I am not getting why exactly we need accumulator instead of using index min/max? Can you explain with some scenario
>
> 4. Why to store min/max at segment level? We can get from datamap also right?
>
> 4.  Union with high granularity tables to low granularity tables are really needed? Any other time series DB is doing it? Or any known use case we have?
>
> Regards,
> Ravindra.
>
> > On 1 Oct 2019, at 5:49 PM, Akash Nilugal <[hidden email]> wrote:
> >
> > Hi Babu,
> >
> > Thanks for the inputs. Please find the comments
> > 1. I will change from Union to UnionAll
> > 2. For auto datamap loading, once the data is loaded to lower level granularity datamap, then we load the higher level datamap from the lower level datamap. But as per your point, i think you are telling to load from main table itself.
> > 3. similar to 2nd point, whether to need configuration or not we can decide i think.
> > 4. a. I think the max of the datamap is required to decide the range for the load, because in case of failure case, we may need.
> > b. This point will be taken care.
> > 5. Yes, dataload is sync based on current design, as it is non lazy, it will happen with main table load only.
> > 6. Yes, this will be handled.
> > 7. Already added a task in jira.
> > On 2019/10/01 08:50:05, babu lal jangir <[hidden email]> wrote:
> >> Hi Akash, Thanks for Time Series DataMap proposal.
> >> Please check below Points.
> >>
> >> 1. During Query Planing Change Union to Union All , Otherwise will loose
> >> row if same value appears.
> >> 2. Whether system start load for next granularity level table as soon it
> >> matches the data condition or next granularity level table has to wait till
> >> current  granularity level table is finished ? please handle if possible.
> >> 3. Add Configuration to load multiple Ranges at a time(across granularity
> >> tables).
> >> 4. Please check if Current data loading min ,max is enough to find current
> >> load . No need to refer the DataMap's min,max because data loading Range
> >> prepration can go wrong if loading happens from multiple driver . i think
> >> below rules are enough for loading.
> >>    4.a. Create MV should should sync data.   On any failure Rebuild should
> >> sync again till than MV will be disabled.
> >>    4.b.  Each load has independent Ranges and should load only those
> >> ranges. Any failure MV may go in disable state(only if intermediate ranges
> >> load is failed ,last loads failure will NOT make MV disable).
> >> 5. We can make Data loading sync because anyway queries can be served from
> >> fact table if any segments is in-progress in  Datamap.
> >> 6. In Data loading Pipleline ,failures in intermediate time series datamap,
> >> still we can continue loading next level data. (ignore if already handled).
> >>   For Example.
> >>    DataMaps:- Hour,Day,Month Level
> >>    Load Data(10 day):- 2018-01-01 01:00:00 to 2018-01-10 01:00:00
> >>      Failure in hour level during below range
> >>        2018-01-06 01:00:00 to 2018-01-06 01:00:00
> >>     This point of time Hour level has 5 day data.so start loading on day
> >> level .
> >> 7. Add SubTask to support loading of in-between missing time.(Incremental
> >> but old records if timeseries device stopped working for some time).
> >>
> >> On Tue, Oct 1, 2019 at 10:41 AM Akash Nilugal <[hidden email]>
> >> wrote:
> >>
> >>> Hi vishal,
> >>>
> >>> In the design document, in the impacted analysis section, there is a topic
> >>> compatibility/legacy stores, so basically For old tables when the datamap
> >>> is created, we load all the timeseries datamaps with different granularity.
> >>> I think this should do fine, please let me know for further
> >>> suggestions/comments.
> >>>
> >>> Regards,
> >>> Akash R Nilugal
> >>>
> >>> On 2019/09/30 17:09:44, Kumar Vishal <[hidden email]> wrote:
> >>>> Hi Akash,
> >>>>
> >>>> In this desing document you haven't mentioned how to handle data loading
> >>>> for timeseries datamap for older segments[Existing table].
> >>>> If the customer's main table data is also stored based on time[increasing
> >>>> time] in different segments,he can use this feature as well.
> >>>>
> >>>> We can discuss and finalize the solution.
> >>>>
> >>>> -Regards
> >>>> Kumar Vishal
> >>>>
> >>>> On Mon, Sep 30, 2019 at 2:42 PM Akash Nilugal <[hidden email]>
> >>>> wrote:
> >>>>
> >>>>> Hi Ajantha,
> >>>>>
> >>>>> Thanks for the queries and suggestions
> >>>>>
> >>>>> 1. Yes, this is a good suggestion, i ll include this change. Both date
> >>> and
> >>>>> timestamp columns are supported, will be updated in document.
> >>>>> 2. yes, you are right.
> >>>>> 3. you are right, if the day level is not available, then we will try
> >>> to
> >>>>> get the whole day data from hour level, if not availaible, as
> >>> explained in
> >>>>> design document, we will get the data from datamap UNION data from main
> >>>>> table based on user query.
> >>>>>
> >>>>> Regards,
> >>>>> Akash R Nilugal
> >>>>>
> >>>>>
> >>>>> On 2019/09/30 06:56:45, Ajantha Bhat <[hidden email]> wrote:
> >>>>>> + 1 ,
> >>>>>>
> >>>>>> I have some suggestions and questions.
> >>>>>>
> >>>>>> 1. In DMPROPERTIES, instead of 'timestamp_column' suggest to use
> >>>>>> 'timeseries_column'.
> >>>>>> so that it won't give an impression that only time stamp datatype is
> >>>>>> supported and update the document with all the datatype supported.
> >>>>>>
> >>>>>> 2. Querying on this datamap table is also supported right ?
> >>> supporting
> >>>>>> changing plan for main table to refer datamap table is for user to
> >>> avoid
> >>>>>> changing his query or any other reason ?
> >>>>>>
> >>>>>> 3. If user has not created day granularity datamap, but just created
> >>> hour
> >>>>>> granularity datamap. When query has day granularity, data will be
> >>> fetched
> >>>>>> form hour granularity datamap and aggregated ? or data is fetched
> >>> from
> >>>>> main
> >>>>>> table ?
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Ajantha
> >>>>>>
> >>>>>> On Mon, Sep 30, 2019 at 11:46 AM Akash Nilugal <
> >>> [hidden email]>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hi xuchuanyin,
> >>>>>>>
> >>>>>>> Thanks for the comments/Suggestions
> >>>>>>>
> >>>>>>> 1. Preaggregate is productized, but not the timeseries with
> >>>>> preaggregate,
> >>>>>>> i think you  got confused with that, if im right.
> >>>>>>> 2. Limitations like, auto sampling or rollup, which we will be
> >>>>> supporting
> >>>>>>> now. Retention policies. etc
> >>>>>>> 3. segmentTimestampMin, this i will consider in design.
> >>>>>>> 4. RP is added as a separate task, i thought instead of
> >>> maintaining two
> >>>>>>> variables better to maintabin one and parse it. But i will consider
> >>>>> your
> >>>>>>> point based on feasibility during implementation.
> >>>>>>> 5. We use an accumulator which takes list, so before writing index
> >>>>> files
> >>>>>>> we take the min max of the timestamp column and fill in
> >>> accumulator and
> >>>>>>> then we can access accumulator.value in driver after load is
> >>> finished.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Akash R Nilugal
> >>>>>>>
> >>>>>>> On 2019/09/28 10:46:31, xuchuanyin <[hidden email]> wrote:
> >>>>>>>> Hi akash, glad to see the feature proposed and I have some
> >>> questions
> >>>>>>> about
> >>>>>>>> this. Please notice that some of the following descriptions are
> >>>>> comments
> >>>>>>>> followed by '===' described in the design document attached in
> >>> the
> >>>>>>>> corresponding jira.
> >>>>>>>>
> >>>>>>>> 1.
> >>>>>>>> "Currently carbondata supports timeseries on preaggregate
> >>> datamap,
> >>>>> but
> >>>>>>> its
> >>>>>>>> an alpha feature"
> >>>>>>>> ===
> >>>>>>>> It has been some time since the preaggregate datamap was
> >>> introduced
> >>>>> and
> >>>>>>> it
> >>>>>>>> is still **alpha**, why it is still not product-ready? Will the
> >>> new
> >>>>>>> feature
> >>>>>>>> also come into the similar situation?
> >>>>>>>>
> >>>>>>>> 2.
> >>>>>>>> "there are so many limitations when we compare and analyze the
> >>>>> existing
> >>>>>>>> timeseries database or projects which supports time series like
> >>>>> apache
> >>>>>>> druid
> >>>>>>>> or influxdb"
> >>>>>>>> ===
> >>>>>>>> What are the actual limitations? Besides, please give an example
> >>> of
> >>>>> this.
> >>>>>>>>
> >>>>>>>> 3.
> >>>>>>>> "Segment_Timestamp_Min"
> >>>>>>>> ===
> >>>>>>>> Suggest using camel-case style like 'segmentTimestampMin'
> >>>>>>>>
> >>>>>>>> 4.
> >>>>>>>> "RP is way of telling the system, for how long the data should be
> >>>>> kept"
> >>>>>>>> ===
> >>>>>>>> Since the function is simple, I'd suggest using
> >>> 'retentionTime'=15
> >>>>> and
> >>>>>>>> 'timeUnit'='day' instead of 'RP'='15_days'
> >>>>>>>>
> >>>>>>>> 5.
> >>>>>>>> "When the data load is called for main table, use an spark
> >>>>> accumulator to
> >>>>>>>> get the maximum value of timestamp in that load and return to the
> >>>>> load."
> >>>>>>>> ===
> >>>>>>>> How can you get the spark accumulator? The load is launched using
> >>>>>>>> loading-by-dataframe not using global-sort-by-spark.
> >>>>>>>>
> >>>>>>>> 6.
> >>>>>>>> For the rest of the content, still reading.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Sent from:
> >>>>>>>
> >>>>>
> >>> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>

[ravipesala]

HI Akash,

1. I feel user providing granularity is redundant, he can just provide respective udf in select query should be enough.

2. I think it is better to add the RP management now itself, otherwise if you start adding to DM properties as temporary then it will never be moved. Better put little more effort to decouple it from datamaps.

3. I feel accumulator is a added cost, we already have feature in development to load datamap immediately after load happens, why not use that? If the datamap is already in memory why we need min/max at segment level?

4. I feel there must be some reason why other timeseries db does not support union of data.  Consider a scenario that we have data from 1pm to 4.30 pm , it means 4 to 5pm data is still loading.  when user asks the data at hour level I feel it is safe to give data for 1,2,3 hours data, because providing 4pm is actually not a complete data. So atleast user comes to know that 4 pm data is not available and starts querying the low level data if he needs it.
I think better get some real uses how user wants this time series data.

Regards,
Ravindra.

> On 4 Oct 2019, at 9:39 PM, Akash Nilugal <[hidden email]> wrote:
>
> Hi Ravi,
>
> 1. I forgot to mention the CTAS query in the create datamap statement, i have updated the document, during create datamap user can give granularity, during query just the UDF. That should be fine right.
> 2. I think may be we can mention the RP policy in DM properties also, and then may be we provide add RP, drop RP, alter RP for existing and older datamaps. RP will be taken as a separate subtask and will be handled in later part. That should be fine i tink.
> 3. Actually consider a scenario when datamap is already created, then load happened to main table, then i use accumulator to get all the min max to driver, so that i can avoid reading index file in driver in order to load to datamap.
>         other scenario is when main table already has segments and then datamap is created, the we will read index files from each segments to decide the min max of timestamp column.
> 4. We are not storing min max in main table  table status. We are storing in datamap table's table status file, so that it will be used to prepare the plan during the query phase.
>
> 5. Other timeseries db supports only getting the data present in hour or day .. aggregated data. Since we cannot miss the data, plan is to get the data like higher to lower. May be it does not make much difference when its from minute to second, but it makes difference from year to month , so that we cannot avoid aggregations from main table.
>
>
> Regards,
> Akash R Nilugal
>
> On 2019/10/04 11:35:46, Ravindra Pesala <[hidden email]> wrote:
>> Hi Akash,
>>
>> I have following suggestions.
>>
>> 1. I think it is redundant to use granularity inside create datamap, user can use the respective granularity UDF in his query like time(1h) or time(1d) etc.
>>
>> 2. Better create separate RP commands and let user add the RP on the datamap or even on the main table also. It would be more manageable if you independent feature for RP instead of including in datamap.
>>
>> 3. I am not getting why exactly we need accumulator instead of using index min/max? Can you explain with some scenario
>>
>> 4. Why to store min/max at segment level? We can get from datamap also right?
>>
>> 4.  Union with high granularity tables to low granularity tables are really needed? Any other time series DB is doing it? Or any known use case we have?
>>
>> Regards,
>> Ravindra.
>>
>>> On 1 Oct 2019, at 5:49 PM, Akash Nilugal <[hidden email]> wrote:
>>>
>>> Hi Babu,
>>>
>>> Thanks for the inputs. Please find the comments
>>> 1. I will change from Union to UnionAll
>>> 2. For auto datamap loading, once the data is loaded to lower level granularity datamap, then we load the higher level datamap from the lower level datamap. But as per your point, i think you are telling to load from main table itself.
>>> 3. similar to 2nd point, whether to need configuration or not we can decide i think.
>>> 4. a. I think the max of the datamap is required to decide the range for the load, because in case of failure case, we may need.
>>> b. This point will be taken care.
>>> 5. Yes, dataload is sync based on current design, as it is non lazy, it will happen with main table load only.
>>> 6. Yes, this will be handled.
>>> 7. Already added a task in jira.
>>> On 2019/10/01 08:50:05, babu lal jangir <[hidden email]> wrote:
>>>> Hi Akash, Thanks for Time Series DataMap proposal.
>>>> Please check below Points.
>>>>
>>>> 1. During Query Planing Change Union to Union All , Otherwise will loose
>>>> row if same value appears.
>>>> 2. Whether system start load for next granularity level table as soon it
>>>> matches the data condition or next granularity level table has to wait till
>>>> current  granularity level table is finished ? please handle if possible.
>>>> 3. Add Configuration to load multiple Ranges at a time(across granularity
>>>> tables).
>>>> 4. Please check if Current data loading min ,max is enough to find current
>>>> load . No need to refer the DataMap's min,max because data loading Range
>>>> prepration can go wrong if loading happens from multiple driver . i think
>>>> below rules are enough for loading.
>>>>   4.a. Create MV should should sync data.   On any failure Rebuild should
>>>> sync again till than MV will be disabled.
>>>>   4.b.  Each load has independent Ranges and should load only those
>>>> ranges. Any failure MV may go in disable state(only if intermediate ranges
>>>> load is failed ,last loads failure will NOT make MV disable).
>>>> 5. We can make Data loading sync because anyway queries can be served from
>>>> fact table if any segments is in-progress in  Datamap.
>>>> 6. In Data loading Pipleline ,failures in intermediate time series datamap,
>>>> still we can continue loading next level data. (ignore if already handled).
>>>>  For Example.
>>>>   DataMaps:- Hour,Day,Month Level
>>>>   Load Data(10 day):- 2018-01-01 01:00:00 to 2018-01-10 01:00:00
>>>>     Failure in hour level during below range
>>>>       2018-01-06 01:00:00 to 2018-01-06 01:00:00
>>>>    This point of time Hour level has 5 day data.so start loading on day
>>>> level .
>>>> 7. Add SubTask to support loading of in-between missing time.(Incremental
>>>> but old records if timeseries device stopped working for some time).
>>>>
>>>> On Tue, Oct 1, 2019 at 10:41 AM Akash Nilugal <[hidden email]>
>>>> wrote:
>>>>
>>>>> Hi vishal,
>>>>>
>>>>> In the design document, in the impacted analysis section, there is a topic
>>>>> compatibility/legacy stores, so basically For old tables when the datamap
>>>>> is created, we load all the timeseries datamaps with different granularity.
>>>>> I think this should do fine, please let me know for further
>>>>> suggestions/comments.
>>>>>
>>>>> Regards,
>>>>> Akash R Nilugal
>>>>>
>>>>> On 2019/09/30 17:09:44, Kumar Vishal <[hidden email]> wrote:
>>>>>> Hi Akash,
>>>>>>
>>>>>> In this desing document you haven't mentioned how to handle data loading
>>>>>> for timeseries datamap for older segments[Existing table].
>>>>>> If the customer's main table data is also stored based on time[increasing
>>>>>> time] in different segments,he can use this feature as well.
>>>>>>
>>>>>> We can discuss and finalize the solution.
>>>>>>
>>>>>> -Regards
>>>>>> Kumar Vishal
>>>>>>
>>>>>> On Mon, Sep 30, 2019 at 2:42 PM Akash Nilugal <[hidden email]>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Ajantha,
>>>>>>>
>>>>>>> Thanks for the queries and suggestions
>>>>>>>
>>>>>>> 1. Yes, this is a good suggestion, i ll include this change. Both date
>>>>> and
>>>>>>> timestamp columns are supported, will be updated in document.
>>>>>>> 2. yes, you are right.
>>>>>>> 3. you are right, if the day level is not available, then we will try
>>>>> to
>>>>>>> get the whole day data from hour level, if not availaible, as
>>>>> explained in
>>>>>>> design document, we will get the data from datamap UNION data from main
>>>>>>> table based on user query.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Akash R Nilugal
>>>>>>>
>>>>>>>
>>>>>>> On 2019/09/30 06:56:45, Ajantha Bhat <[hidden email]> wrote:
>>>>>>>> + 1 ,
>>>>>>>>
>>>>>>>> I have some suggestions and questions.
>>>>>>>>
>>>>>>>> 1. In DMPROPERTIES, instead of 'timestamp_column' suggest to use
>>>>>>>> 'timeseries_column'.
>>>>>>>> so that it won't give an impression that only time stamp datatype is
>>>>>>>> supported and update the document with all the datatype supported.
>>>>>>>>
>>>>>>>> 2. Querying on this datamap table is also supported right ?
>>>>> supporting
>>>>>>>> changing plan for main table to refer datamap table is for user to
>>>>> avoid
>>>>>>>> changing his query or any other reason ?
>>>>>>>>
>>>>>>>> 3. If user has not created day granularity datamap, but just created
>>>>> hour
>>>>>>>> granularity datamap. When query has day granularity, data will be
>>>>> fetched
>>>>>>>> form hour granularity datamap and aggregated ? or data is fetched
>>>>> from
>>>>>>> main
>>>>>>>> table ?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Ajantha
>>>>>>>>
>>>>>>>> On Mon, Sep 30, 2019 at 11:46 AM Akash Nilugal <
>>>>> [hidden email]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi xuchuanyin,
>>>>>>>>>
>>>>>>>>> Thanks for the comments/Suggestions
>>>>>>>>>
>>>>>>>>> 1. Preaggregate is productized, but not the timeseries with
>>>>>>> preaggregate,
>>>>>>>>> i think you  got confused with that, if im right.
>>>>>>>>> 2. Limitations like, auto sampling or rollup, which we will be
>>>>>>> supporting
>>>>>>>>> now. Retention policies. etc
>>>>>>>>> 3. segmentTimestampMin, this i will consider in design.
>>>>>>>>> 4. RP is added as a separate task, i thought instead of
>>>>> maintaining two
>>>>>>>>> variables better to maintabin one and parse it. But i will consider
>>>>>>> your
>>>>>>>>> point based on feasibility during implementation.
>>>>>>>>> 5. We use an accumulator which takes list, so before writing index
>>>>>>> files
>>>>>>>>> we take the min max of the timestamp column and fill in
>>>>> accumulator and
>>>>>>>>> then we can access accumulator.value in driver after load is
>>>>> finished.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Akash R Nilugal
>>>>>>>>>
>>>>>>>>> On 2019/09/28 10:46:31, xuchuanyin <[hidden email]> wrote:
>>>>>>>>>> Hi akash, glad to see the feature proposed and I have some
>>>>> questions
>>>>>>>>> about
>>>>>>>>>> this. Please notice that some of the following descriptions are
>>>>>>> comments
>>>>>>>>>> followed by '===' described in the design document attached in
>>>>> the
>>>>>>>>>> corresponding jira.
>>>>>>>>>>
>>>>>>>>>> 1.
>>>>>>>>>> "Currently carbondata supports timeseries on preaggregate
>>>>> datamap,
>>>>>>> but
>>>>>>>>> its
>>>>>>>>>> an alpha feature"
>>>>>>>>>> ===
>>>>>>>>>> It has been some time since the preaggregate datamap was
>>>>> introduced
>>>>>>> and
>>>>>>>>> it
>>>>>>>>>> is still **alpha**, why it is still not product-ready? Will the
>>>>> new
>>>>>>>>> feature
>>>>>>>>>> also come into the similar situation?
>>>>>>>>>>
>>>>>>>>>> 2.
>>>>>>>>>> "there are so many limitations when we compare and analyze the
>>>>>>> existing
>>>>>>>>>> timeseries database or projects which supports time series like
>>>>>>> apache
>>>>>>>>> druid
>>>>>>>>>> or influxdb"
>>>>>>>>>> ===
>>>>>>>>>> What are the actual limitations? Besides, please give an example
>>>>> of
>>>>>>> this.
>>>>>>>>>>
>>>>>>>>>> 3.
>>>>>>>>>> "Segment_Timestamp_Min"
>>>>>>>>>> ===
>>>>>>>>>> Suggest using camel-case style like 'segmentTimestampMin'
>>>>>>>>>>
>>>>>>>>>> 4.
>>>>>>>>>> "RP is way of telling the system, for how long the data should be
>>>>>>> kept"
>>>>>>>>>> ===
>>>>>>>>>> Since the function is simple, I'd suggest using
>>>>> 'retentionTime'=15
>>>>>>> and
>>>>>>>>>> 'timeUnit'='day' instead of 'RP'='15_days'
>>>>>>>>>>
>>>>>>>>>> 5.
>>>>>>>>>> "When the data load is called for main table, use an spark
>>>>>>> accumulator to
>>>>>>>>>> get the maximum value of timestamp in that load and return to the
>>>>>>> load."
>>>>>>>>>> ===
>>>>>>>>>> How can you get the spark accumulator? The load is launched using
>>>>>>>>>> loading-by-dataframe not using global-sort-by-spark.
>>>>>>>>>>
>>>>>>>>>> 6.
>>>>>>>>>> For the rest of the content, still reading.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Sent from:
>>>>>>>>>
>>>>>>>
>>>>> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>
>>

Hi Ravi,

1. i) During create datamap, in ctas query, user does not mention udf, so if granularity is present in DM properties, then internally we rewrite the ctas query with udf and then load the data to datamap according to current design.
   ii) but if we say user to give ctas query with udf only, then internally no need to rewite the query, we can just load data to it and avoid giving the granularity in DMproperties.
        Currently im planning to do first one. Please give your input on this.

2. Ok, we will not use the RP management in DMProperties, we will use as separate command and do proper decoupling.

3. I think you are referring to the cache pre-priming in index server. Problem with this is that, we wil be not sure whether the cache loaded for the segment or not, because as per pre-priming design, if loading to cache fails after data load to main table, we ignore it as query takes care of it. So we cannot completely rely on this feature for min max.
So for accumulator, im not calculating again, i just take the minmax before writing index file in dataload and use that in driver to prepare the dataload ranges for datamaps.

The reason to keep the segment min max in the table status of datamap is that, it will be helful in RP scenarios, second is we will not be missing any data from loading to datamap from main table[if 1st time data came from 1 to 4:15 , then next we get data 5:10 to 6, then there might be chance that we can miss 15minutes of data from 4 to 4:15]. It will be helpful in querying also. So that we can avoid the problem i mentioned above with datamaps loaded in cache.

4. I agree, your point is valid one. I will do more abalysis on this based on the user use cases and then we can decide finally. That would be better.

Please give your inputs/suggestions on the above points.

regards,
Akash R Nilugal

On 2019/10/07 03:03:35, Ravindra Pesala <[hidden email]> wrote:

> HI Akash,
>
> 1. I feel user providing granularity is redundant, he can just provide respective udf in select query should be enough.
>
> 2. I think it is better to add the RP management now itself, otherwise if you start adding to DM properties as temporary then it will never be moved. Better put little more effort to decouple it from datamaps.
>
> 3. I feel accumulator is a added cost, we already have feature in development to load datamap immediately after load happens, why not use that? If the datamap is already in memory why we need min/max at segment level?
>
> 4. I feel there must be some reason why other timeseries db does not support union of data.  Consider a scenario that we have data from 1pm to 4.30 pm , it means 4 to 5pm data is still loading.  when user asks the data at hour level I feel it is safe to give data for 1,2,3 hours data, because providing 4pm is actually not a complete data. So atleast user comes to know that 4 pm data is not available and starts querying the low level data if he needs it.
> I think better get some real uses how user wants this time series data.
>
> Regards,
> Ravindra.
>
> > On 4 Oct 2019, at 9:39 PM, Akash Nilugal <[hidden email]> wrote:
> >
> > Hi Ravi,
> >
> > 1. I forgot to mention the CTAS query in the create datamap statement, i have updated the document, during create datamap user can give granularity, during query just the UDF. That should be fine right.
> > 2. I think may be we can mention the RP policy in DM properties also, and then may be we provide add RP, drop RP, alter RP for existing and older datamaps. RP will be taken as a separate subtask and will be handled in later part. That should be fine i tink.
> > 3. Actually consider a scenario when datamap is already created, then load happened to main table, then i use accumulator to get all the min max to driver, so that i can avoid reading index file in driver in order to load to datamap.
> >         other scenario is when main table already has segments and then datamap is created, the we will read index files from each segments to decide the min max of timestamp column.
> > 4. We are not storing min max in main table  table status. We are storing in datamap table's table status file, so that it will be used to prepare the plan during the query phase.
> >
> > 5. Other timeseries db supports only getting the data present in hour or day .. aggregated data. Since we cannot miss the data, plan is to get the data like higher to lower. May be it does not make much difference when its from minute to second, but it makes difference from year to month , so that we cannot avoid aggregations from main table.
> >
> >
> > Regards,
> > Akash R Nilugal
> >
> > On 2019/10/04 11:35:46, Ravindra Pesala <[hidden email]> wrote:
> >> Hi Akash,
> >>
> >> I have following suggestions.
> >>
> >> 1. I think it is redundant to use granularity inside create datamap, user can use the respective granularity UDF in his query like time(1h) or time(1d) etc.
> >>
> >> 2. Better create separate RP commands and let user add the RP on the datamap or even on the main table also. It would be more manageable if you independent feature for RP instead of including in datamap.
> >>
> >> 3. I am not getting why exactly we need accumulator instead of using index min/max? Can you explain with some scenario
> >>
> >> 4. Why to store min/max at segment level? We can get from datamap also right?
> >>
> >> 4.  Union with high granularity tables to low granularity tables are really needed? Any other time series DB is doing it? Or any known use case we have?
> >>
> >> Regards,
> >> Ravindra.
> >>
> >>> On 1 Oct 2019, at 5:49 PM, Akash Nilugal <[hidden email]> wrote:
> >>>
> >>> Hi Babu,
> >>>
> >>> Thanks for the inputs. Please find the comments
> >>> 1. I will change from Union to UnionAll
> >>> 2. For auto datamap loading, once the data is loaded to lower level granularity datamap, then we load the higher level datamap from the lower level datamap. But as per your point, i think you are telling to load from main table itself.
> >>> 3. similar to 2nd point, whether to need configuration or not we can decide i think.
> >>> 4. a. I think the max of the datamap is required to decide the range for the load, because in case of failure case, we may need.
> >>> b. This point will be taken care.
> >>> 5. Yes, dataload is sync based on current design, as it is non lazy, it will happen with main table load only.
> >>> 6. Yes, this will be handled.
> >>> 7. Already added a task in jira.
> >>> On 2019/10/01 08:50:05, babu lal jangir <[hidden email]> wrote:
> >>>> Hi Akash, Thanks for Time Series DataMap proposal.
> >>>> Please check below Points.
> >>>>
> >>>> 1. During Query Planing Change Union to Union All , Otherwise will loose
> >>>> row if same value appears.
> >>>> 2. Whether system start load for next granularity level table as soon it
> >>>> matches the data condition or next granularity level table has to wait till
> >>>> current  granularity level table is finished ? please handle if possible.
> >>>> 3. Add Configuration to load multiple Ranges at a time(across granularity
> >>>> tables).
> >>>> 4. Please check if Current data loading min ,max is enough to find current
> >>>> load . No need to refer the DataMap's min,max because data loading Range
> >>>> prepration can go wrong if loading happens from multiple driver . i think
> >>>> below rules are enough for loading.
> >>>>   4.a. Create MV should should sync data.   On any failure Rebuild should
> >>>> sync again till than MV will be disabled.
> >>>>   4.b.  Each load has independent Ranges and should load only those
> >>>> ranges. Any failure MV may go in disable state(only if intermediate ranges
> >>>> load is failed ,last loads failure will NOT make MV disable).
> >>>> 5. We can make Data loading sync because anyway queries can be served from
> >>>> fact table if any segments is in-progress in  Datamap.
> >>>> 6. In Data loading Pipleline ,failures in intermediate time series datamap,
> >>>> still we can continue loading next level data. (ignore if already handled).
> >>>>  For Example.
> >>>>   DataMaps:- Hour,Day,Month Level
> >>>>   Load Data(10 day):- 2018-01-01 01:00:00 to 2018-01-10 01:00:00
> >>>>     Failure in hour level during below range
> >>>>       2018-01-06 01:00:00 to 2018-01-06 01:00:00
> >>>>    This point of time Hour level has 5 day data.so start loading on day
> >>>> level .
> >>>> 7. Add SubTask to support loading of in-between missing time.(Incremental
> >>>> but old records if timeseries device stopped working for some time).
> >>>>
> >>>> On Tue, Oct 1, 2019 at 10:41 AM Akash Nilugal <[hidden email]>
> >>>> wrote:
> >>>>
> >>>>> Hi vishal,
> >>>>>
> >>>>> In the design document, in the impacted analysis section, there is a topic
> >>>>> compatibility/legacy stores, so basically For old tables when the datamap
> >>>>> is created, we load all the timeseries datamaps with different granularity.
> >>>>> I think this should do fine, please let me know for further
> >>>>> suggestions/comments.
> >>>>>
> >>>>> Regards,
> >>>>> Akash R Nilugal
> >>>>>
> >>>>> On 2019/09/30 17:09:44, Kumar Vishal <[hidden email]> wrote:
> >>>>>> Hi Akash,
> >>>>>>
> >>>>>> In this desing document you haven't mentioned how to handle data loading
> >>>>>> for timeseries datamap for older segments[Existing table].
> >>>>>> If the customer's main table data is also stored based on time[increasing
> >>>>>> time] in different segments,he can use this feature as well.
> >>>>>>
> >>>>>> We can discuss and finalize the solution.
> >>>>>>
> >>>>>> -Regards
> >>>>>> Kumar Vishal
> >>>>>>
> >>>>>> On Mon, Sep 30, 2019 at 2:42 PM Akash Nilugal <[hidden email]>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hi Ajantha,
> >>>>>>>
> >>>>>>> Thanks for the queries and suggestions
> >>>>>>>
> >>>>>>> 1. Yes, this is a good suggestion, i ll include this change. Both date
> >>>>> and
> >>>>>>> timestamp columns are supported, will be updated in document.
> >>>>>>> 2. yes, you are right.
> >>>>>>> 3. you are right, if the day level is not available, then we will try
> >>>>> to
> >>>>>>> get the whole day data from hour level, if not availaible, as
> >>>>> explained in
> >>>>>>> design document, we will get the data from datamap UNION data from main
> >>>>>>> table based on user query.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Akash R Nilugal
> >>>>>>>
> >>>>>>>
> >>>>>>> On 2019/09/30 06:56:45, Ajantha Bhat <[hidden email]> wrote:
> >>>>>>>> + 1 ,
> >>>>>>>>
> >>>>>>>> I have some suggestions and questions.
> >>>>>>>>
> >>>>>>>> 1. In DMPROPERTIES, instead of 'timestamp_column' suggest to use
> >>>>>>>> 'timeseries_column'.
> >>>>>>>> so that it won't give an impression that only time stamp datatype is
> >>>>>>>> supported and update the document with all the datatype supported.
> >>>>>>>>
> >>>>>>>> 2. Querying on this datamap table is also supported right ?
> >>>>> supporting
> >>>>>>>> changing plan for main table to refer datamap table is for user to
> >>>>> avoid
> >>>>>>>> changing his query or any other reason ?
> >>>>>>>>
> >>>>>>>> 3. If user has not created day granularity datamap, but just created
> >>>>> hour
> >>>>>>>> granularity datamap. When query has day granularity, data will be
> >>>>> fetched
> >>>>>>>> form hour granularity datamap and aggregated ? or data is fetched
> >>>>> from
> >>>>>>> main
> >>>>>>>> table ?
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Ajantha
> >>>>>>>>
> >>>>>>>> On Mon, Sep 30, 2019 at 11:46 AM Akash Nilugal <
> >>>>> [hidden email]>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi xuchuanyin,
> >>>>>>>>>
> >>>>>>>>> Thanks for the comments/Suggestions
> >>>>>>>>>
> >>>>>>>>> 1. Preaggregate is productized, but not the timeseries with
> >>>>>>> preaggregate,
> >>>>>>>>> i think you  got confused with that, if im right.
> >>>>>>>>> 2. Limitations like, auto sampling or rollup, which we will be
> >>>>>>> supporting
> >>>>>>>>> now. Retention policies. etc
> >>>>>>>>> 3. segmentTimestampMin, this i will consider in design.
> >>>>>>>>> 4. RP is added as a separate task, i thought instead of
> >>>>> maintaining two
> >>>>>>>>> variables better to maintabin one and parse it. But i will consider
> >>>>>>> your
> >>>>>>>>> point based on feasibility during implementation.
> >>>>>>>>> 5. We use an accumulator which takes list, so before writing index
> >>>>>>> files
> >>>>>>>>> we take the min max of the timestamp column and fill in
> >>>>> accumulator and
> >>>>>>>>> then we can access accumulator.value in driver after load is
> >>>>> finished.
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Akash R Nilugal
> >>>>>>>>>
> >>>>>>>>> On 2019/09/28 10:46:31, xuchuanyin <[hidden email]> wrote:
> >>>>>>>>>> Hi akash, glad to see the feature proposed and I have some
> >>>>> questions
> >>>>>>>>> about
> >>>>>>>>>> this. Please notice that some of the following descriptions are
> >>>>>>> comments
> >>>>>>>>>> followed by '===' described in the design document attached in
> >>>>> the
> >>>>>>>>>> corresponding jira.
> >>>>>>>>>>
> >>>>>>>>>> 1.
> >>>>>>>>>> "Currently carbondata supports timeseries on preaggregate
> >>>>> datamap,
> >>>>>>> but
> >>>>>>>>> its
> >>>>>>>>>> an alpha feature"
> >>>>>>>>>> ===
> >>>>>>>>>> It has been some time since the preaggregate datamap was
> >>>>> introduced
> >>>>>>> and
> >>>>>>>>> it
> >>>>>>>>>> is still **alpha**, why it is still not product-ready? Will the
> >>>>> new
> >>>>>>>>> feature
> >>>>>>>>>> also come into the similar situation?
> >>>>>>>>>>
> >>>>>>>>>> 2.
> >>>>>>>>>> "there are so many limitations when we compare and analyze the
> >>>>>>> existing
> >>>>>>>>>> timeseries database or projects which supports time series like
> >>>>>>> apache
> >>>>>>>>> druid
> >>>>>>>>>> or influxdb"
> >>>>>>>>>> ===
> >>>>>>>>>> What are the actual limitations? Besides, please give an example
> >>>>> of
> >>>>>>> this.
> >>>>>>>>>>
> >>>>>>>>>> 3.
> >>>>>>>>>> "Segment_Timestamp_Min"
> >>>>>>>>>> ===
> >>>>>>>>>> Suggest using camel-case style like 'segmentTimestampMin'
> >>>>>>>>>>
> >>>>>>>>>> 4.
> >>>>>>>>>> "RP is way of telling the system, for how long the data should be
> >>>>>>> kept"
> >>>>>>>>>> ===
> >>>>>>>>>> Since the function is simple, I'd suggest using
> >>>>> 'retentionTime'=15
> >>>>>>> and
> >>>>>>>>>> 'timeUnit'='day' instead of 'RP'='15_days'
> >>>>>>>>>>
> >>>>>>>>>> 5.
> >>>>>>>>>> "When the data load is called for main table, use an spark
> >>>>>>> accumulator to
> >>>>>>>>>> get the maximum value of timestamp in that load and return to the
> >>>>>>> load."
> >>>>>>>>>> ===
> >>>>>>>>>> How can you get the spark accumulator? The load is launched using
> >>>>>>>>>> loading-by-dataframe not using global-sort-by-spark.
> >>>>>>>>>>
> >>>>>>>>>> 6.
> >>>>>>>>>> For the rest of the content, still reading.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Sent from:
> >>>>>>>>>
> >>>>>>>
> >>>>> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
>
>

[ravipesala]

Hi Akash,

1. It is better to make it simple and let user provide the udf he wants in the query. So no need to rewrite the query and no need provide extra granularity property.

3. I got your point why you want to use accumulator to get min/max. But why I am worried is it should not add complexity to generate min/max as we already has this information available. I don’t think we should be so bothered about reading min/max on data loading phase as it is already heavy duty job and adding few more mills does not do any harm. But as you mentioned it is easier to do so we can go a head your way.


Regards,
Ravindra.

> On 7 Oct 2019, at 5:38 PM, Akash Nilugal <[hidden email]> wrote:
>
> Hi Ravi,
>
> 1. i) During create datamap, in ctas query, user does not mention udf, so if granularity is present in DM properties, then internally we rewrite the ctas query with udf and then load the data to datamap according to current design.
>   ii) but if we say user to give ctas query with udf only, then internally no need to rewite the query, we can just load data to it and avoid giving the granularity in DMproperties.
> Currently im planning to do first one. Please give your input on this.
>
> 2. Ok, we will not use the RP management in DMProperties, we will use as separate command and do proper decoupling.
>
> 3. I think you are referring to the cache pre-priming in index server. Problem with this is that, we wil be not sure whether the cache loaded for the segment or not, because as per pre-priming design, if loading to cache fails after data load to main table, we ignore it as query takes care of it. So we cannot completely rely on this feature for min max.
> So for accumulator, im not calculating again, i just take the minmax before writing index file in dataload and use that in driver to prepare the dataload ranges for datamaps.
>
> The reason to keep the segment min max in the table status of datamap is that, it will be helful in RP scenarios, second is we will not be missing any data from loading to datamap from main table[if 1st time data came from 1 to 4:15 , then next we get data 5:10 to 6, then there might be chance that we can miss 15minutes of data from 4 to 4:15]. It will be helpful in querying also. So that we can avoid the problem i mentioned above with datamaps loaded in cache.
>
> 4. I agree, your point is valid one. I will do more abalysis on this based on the user use cases and then we can decide finally. That would be better.
>
> Please give your inputs/suggestions on the above points.
>
> regards,
> Akash R Nilugal
>
> On 2019/10/07 03:03:35, Ravindra Pesala <[hidden email]> wrote:
>> HI Akash,
>>
>> 1. I feel user providing granularity is redundant, he can just provide respective udf in select query should be enough.
>>
>> 2. I think it is better to add the RP management now itself, otherwise if you start adding to DM properties as temporary then it will never be moved. Better put little more effort to decouple it from datamaps.
>>
>> 3. I feel accumulator is a added cost, we already have feature in development to load datamap immediately after load happens, why not use that? If the datamap is already in memory why we need min/max at segment level?
>>
>> 4. I feel there must be some reason why other timeseries db does not support union of data.  Consider a scenario that we have data from 1pm to 4.30 pm , it means 4 to 5pm data is still loading.  when user asks the data at hour level I feel it is safe to give data for 1,2,3 hours data, because providing 4pm is actually not a complete data. So atleast user comes to know that 4 pm data is not available and starts querying the low level data if he needs it.
>> I think better get some real uses how user wants this time series data.
>>
>> Regards,
>> Ravindra.
>>
>>> On 4 Oct 2019, at 9:39 PM, Akash Nilugal <[hidden email]> wrote:
>>>
>>> Hi Ravi,
>>>
>>> 1. I forgot to mention the CTAS query in the create datamap statement, i have updated the document, during create datamap user can give granularity, during query just the UDF. That should be fine right.
>>> 2. I think may be we can mention the RP policy in DM properties also, and then may be we provide add RP, drop RP, alter RP for existing and older datamaps. RP will be taken as a separate subtask and will be handled in later part. That should be fine i tink.
>>> 3. Actually consider a scenario when datamap is already created, then load happened to main table, then i use accumulator to get all the min max to driver, so that i can avoid reading index file in driver in order to load to datamap.
>>>        other scenario is when main table already has segments and then datamap is created, the we will read index files from each segments to decide the min max of timestamp column.
>>> 4. We are not storing min max in main table  table status. We are storing in datamap table's table status file, so that it will be used to prepare the plan during the query phase.
>>>
>>> 5. Other timeseries db supports only getting the data present in hour or day .. aggregated data. Since we cannot miss the data, plan is to get the data like higher to lower. May be it does not make much difference when its from minute to second, but it makes difference from year to month , so that we cannot avoid aggregations from main table.
>>>
>>>
>>> Regards,
>>> Akash R Nilugal
>>>
>>> On 2019/10/04 11:35:46, Ravindra Pesala <[hidden email]> wrote:
>>>> Hi Akash,
>>>>
>>>> I have following suggestions.
>>>>
>>>> 1. I think it is redundant to use granularity inside create datamap, user can use the respective granularity UDF in his query like time(1h) or time(1d) etc.
>>>>
>>>> 2. Better create separate RP commands and let user add the RP on the datamap or even on the main table also. It would be more manageable if you independent feature for RP instead of including in datamap.
>>>>
>>>> 3. I am not getting why exactly we need accumulator instead of using index min/max? Can you explain with some scenario
>>>>
>>>> 4. Why to store min/max at segment level? We can get from datamap also right?
>>>>
>>>> 4.  Union with high granularity tables to low granularity tables are really needed? Any other time series DB is doing it? Or any known use case we have?
>>>>
>>>> Regards,
>>>> Ravindra.
>>>>
>>>>> On 1 Oct 2019, at 5:49 PM, Akash Nilugal <[hidden email]> wrote:
>>>>>
>>>>> Hi Babu,
>>>>>
>>>>> Thanks for the inputs. Please find the comments
>>>>> 1. I will change from Union to UnionAll
>>>>> 2. For auto datamap loading, once the data is loaded to lower level granularity datamap, then we load the higher level datamap from the lower level datamap. But as per your point, i think you are telling to load from main table itself.
>>>>> 3. similar to 2nd point, whether to need configuration or not we can decide i think.
>>>>> 4. a. I think the max of the datamap is required to decide the range for the load, because in case of failure case, we may need.
>>>>> b. This point will be taken care.
>>>>> 5. Yes, dataload is sync based on current design, as it is non lazy, it will happen with main table load only.
>>>>> 6. Yes, this will be handled.
>>>>> 7. Already added a task in jira.
>>>>> On 2019/10/01 08:50:05, babu lal jangir <[hidden email]> wrote:
>>>>>> Hi Akash, Thanks for Time Series DataMap proposal.
>>>>>> Please check below Points.
>>>>>>
>>>>>> 1. During Query Planing Change Union to Union All , Otherwise will loose
>>>>>> row if same value appears.
>>>>>> 2. Whether system start load for next granularity level table as soon it
>>>>>> matches the data condition or next granularity level table has to wait till
>>>>>> current  granularity level table is finished ? please handle if possible.
>>>>>> 3. Add Configuration to load multiple Ranges at a time(across granularity
>>>>>> tables).
>>>>>> 4. Please check if Current data loading min ,max is enough to find current
>>>>>> load . No need to refer the DataMap's min,max because data loading Range
>>>>>> prepration can go wrong if loading happens from multiple driver . i think
>>>>>> below rules are enough for loading.
>>>>>>  4.a. Create MV should should sync data.   On any failure Rebuild should
>>>>>> sync again till than MV will be disabled.
>>>>>>  4.b.  Each load has independent Ranges and should load only those
>>>>>> ranges. Any failure MV may go in disable state(only if intermediate ranges
>>>>>> load is failed ,last loads failure will NOT make MV disable).
>>>>>> 5. We can make Data loading sync because anyway queries can be served from
>>>>>> fact table if any segments is in-progress in  Datamap.
>>>>>> 6. In Data loading Pipleline ,failures in intermediate time series datamap,
>>>>>> still we can continue loading next level data. (ignore if already handled).
>>>>>> For Example.
>>>>>>  DataMaps:- Hour,Day,Month Level
>>>>>>  Load Data(10 day):- 2018-01-01 01:00:00 to 2018-01-10 01:00:00
>>>>>>    Failure in hour level during below range
>>>>>>      2018-01-06 01:00:00 to 2018-01-06 01:00:00
>>>>>>   This point of time Hour level has 5 day data.so start loading on day
>>>>>> level .
>>>>>> 7. Add SubTask to support loading of in-between missing time.(Incremental
>>>>>> but old records if timeseries device stopped working for some time).
>>>>>>
>>>>>> On Tue, Oct 1, 2019 at 10:41 AM Akash Nilugal <[hidden email]>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi vishal,
>>>>>>>
>>>>>>> In the design document, in the impacted analysis section, there is a topic
>>>>>>> compatibility/legacy stores, so basically For old tables when the datamap
>>>>>>> is created, we load all the timeseries datamaps with different granularity.
>>>>>>> I think this should do fine, please let me know for further
>>>>>>> suggestions/comments.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Akash R Nilugal
>>>>>>>
>>>>>>> On 2019/09/30 17:09:44, Kumar Vishal <[hidden email]> wrote:
>>>>>>>> Hi Akash,
>>>>>>>>
>>>>>>>> In this desing document you haven't mentioned how to handle data loading
>>>>>>>> for timeseries datamap for older segments[Existing table].
>>>>>>>> If the customer's main table data is also stored based on time[increasing
>>>>>>>> time] in different segments,he can use this feature as well.
>>>>>>>>
>>>>>>>> We can discuss and finalize the solution.
>>>>>>>>
>>>>>>>> -Regards
>>>>>>>> Kumar Vishal
>>>>>>>>
>>>>>>>> On Mon, Sep 30, 2019 at 2:42 PM Akash Nilugal <[hidden email]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Ajantha,
>>>>>>>>>
>>>>>>>>> Thanks for the queries and suggestions
>>>>>>>>>
>>>>>>>>> 1. Yes, this is a good suggestion, i ll include this change. Both date
>>>>>>> and
>>>>>>>>> timestamp columns are supported, will be updated in document.
>>>>>>>>> 2. yes, you are right.
>>>>>>>>> 3. you are right, if the day level is not available, then we will try
>>>>>>> to
>>>>>>>>> get the whole day data from hour level, if not availaible, as
>>>>>>> explained in
>>>>>>>>> design document, we will get the data from datamap UNION data from main
>>>>>>>>> table based on user query.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Akash R Nilugal
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2019/09/30 06:56:45, Ajantha Bhat <[hidden email]> wrote:
>>>>>>>>>> + 1 ,
>>>>>>>>>>
>>>>>>>>>> I have some suggestions and questions.
>>>>>>>>>>
>>>>>>>>>> 1. In DMPROPERTIES, instead of 'timestamp_column' suggest to use
>>>>>>>>>> 'timeseries_column'.
>>>>>>>>>> so that it won't give an impression that only time stamp datatype is
>>>>>>>>>> supported and update the document with all the datatype supported.
>>>>>>>>>>
>>>>>>>>>> 2. Querying on this datamap table is also supported right ?
>>>>>>> supporting
>>>>>>>>>> changing plan for main table to refer datamap table is for user to
>>>>>>> avoid
>>>>>>>>>> changing his query or any other reason ?
>>>>>>>>>>
>>>>>>>>>> 3. If user has not created day granularity datamap, but just created
>>>>>>> hour
>>>>>>>>>> granularity datamap. When query has day granularity, data will be
>>>>>>> fetched
>>>>>>>>>> form hour granularity datamap and aggregated ? or data is fetched
>>>>>>> from
>>>>>>>>> main
>>>>>>>>>> table ?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Ajantha
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 30, 2019 at 11:46 AM Akash Nilugal <
>>>>>>> [hidden email]>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi xuchuanyin,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the comments/Suggestions
>>>>>>>>>>>
>>>>>>>>>>> 1. Preaggregate is productized, but not the timeseries with
>>>>>>>>> preaggregate,
>>>>>>>>>>> i think you  got confused with that, if im right.
>>>>>>>>>>> 2. Limitations like, auto sampling or rollup, which we will be
>>>>>>>>> supporting
>>>>>>>>>>> now. Retention policies. etc
>>>>>>>>>>> 3. segmentTimestampMin, this i will consider in design.
>>>>>>>>>>> 4. RP is added as a separate task, i thought instead of
>>>>>>> maintaining two
>>>>>>>>>>> variables better to maintabin one and parse it. But i will consider
>>>>>>>>> your
>>>>>>>>>>> point based on feasibility during implementation.
>>>>>>>>>>> 5. We use an accumulator which takes list, so before writing index
>>>>>>>>> files
>>>>>>>>>>> we take the min max of the timestamp column and fill in
>>>>>>> accumulator and
>>>>>>>>>>> then we can access accumulator.value in driver after load is
>>>>>>> finished.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Akash R Nilugal
>>>>>>>>>>>
>>>>>>>>>>> On 2019/09/28 10:46:31, xuchuanyin <[hidden email]> wrote:
>>>>>>>>>>>> Hi akash, glad to see the feature proposed and I have some
>>>>>>> questions
>>>>>>>>>>> about
>>>>>>>>>>>> this. Please notice that some of the following descriptions are
>>>>>>>>> comments
>>>>>>>>>>>> followed by '===' described in the design document attached in
>>>>>>> the
>>>>>>>>>>>> corresponding jira.
>>>>>>>>>>>>
>>>>>>>>>>>> 1.
>>>>>>>>>>>> "Currently carbondata supports timeseries on preaggregate
>>>>>>> datamap,
>>>>>>>>> but
>>>>>>>>>>> its
>>>>>>>>>>>> an alpha feature"
>>>>>>>>>>>> ===
>>>>>>>>>>>> It has been some time since the preaggregate datamap was
>>>>>>> introduced
>>>>>>>>> and
>>>>>>>>>>> it
>>>>>>>>>>>> is still **alpha**, why it is still not product-ready? Will the
>>>>>>> new
>>>>>>>>>>> feature
>>>>>>>>>>>> also come into the similar situation?
>>>>>>>>>>>>
>>>>>>>>>>>> 2.
>>>>>>>>>>>> "there are so many limitations when we compare and analyze the
>>>>>>>>> existing
>>>>>>>>>>>> timeseries database or projects which supports time series like
>>>>>>>>> apache
>>>>>>>>>>> druid
>>>>>>>>>>>> or influxdb"
>>>>>>>>>>>> ===
>>>>>>>>>>>> What are the actual limitations? Besides, please give an example
>>>>>>> of
>>>>>>>>> this.
>>>>>>>>>>>>
>>>>>>>>>>>> 3.
>>>>>>>>>>>> "Segment_Timestamp_Min"
>>>>>>>>>>>> ===
>>>>>>>>>>>> Suggest using camel-case style like 'segmentTimestampMin'
>>>>>>>>>>>>
>>>>>>>>>>>> 4.
>>>>>>>>>>>> "RP is way of telling the system, for how long the data should be
>>>>>>>>> kept"
>>>>>>>>>>>> ===
>>>>>>>>>>>> Since the function is simple, I'd suggest using
>>>>>>> 'retentionTime'=15
>>>>>>>>> and
>>>>>>>>>>>> 'timeUnit'='day' instead of 'RP'='15_days'
>>>>>>>>>>>>
>>>>>>>>>>>> 5.
>>>>>>>>>>>> "When the data load is called for main table, use an spark
>>>>>>>>> accumulator to
>>>>>>>>>>>> get the maximum value of timestamp in that load and return to the
>>>>>>>>> load."
>>>>>>>>>>>> ===
>>>>>>>>>>>> How can you get the spark accumulator? The load is launched using
>>>>>>>>>>>> loading-by-dataframe not using global-sort-by-spark.
>>>>>>>>>>>>
>>>>>>>>>>>> 6.
>>>>>>>>>>>> For the rest of the content, still reading.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Sent from:
>>>>>>>>>>>
>>>>>>>>>
>>>>>>> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>
>>
>>

[Manhua-2]

UDF might have performance problem: Spark built-in UDF  vs  Spark UDF  vs  Hive UDF have some different


On 2019/10/07 10:26:07, Ravindra Pesala <[hidden email]> wrote:

> Hi Akash,
>
> 1. It is better to make it simple and let user provide the udf he wants in the query. So no need to rewrite the query and no need provide extra granularity property.
>
> 3. I got your point why you want to use accumulator to get min/max. But why I am worried is it should not add complexity to generate min/max as we already has this information available. I don’t think we should be so bothered about reading min/max on data loading phase as it is already heavy duty job and adding few more mills does not do any harm. But as you mentioned it is easier to do so we can go a head your way.
>
>
> Regards,
> Ravindra.
>
> > On 7 Oct 2019, at 5:38 PM, Akash Nilugal <[hidden email]> wrote:
> >
> > Hi Ravi,
> >
> > 1. i) During create datamap, in ctas query, user does not mention udf, so if granularity is present in DM properties, then internally we rewrite the ctas query with udf and then load the data to datamap according to current design.
> >   ii) but if we say user to give ctas query with udf only, then internally no need to rewite the query, we can just load data to it and avoid giving the granularity in DMproperties.
> > Currently im planning to do first one. Please give your input on this.
> >
> > 2. Ok, we will not use the RP management in DMProperties, we will use as separate command and do proper decoupling.
> >
> > 3. I think you are referring to the cache pre-priming in index server. Problem with this is that, we wil be not sure whether the cache loaded for the segment or not, because as per pre-priming design, if loading to cache fails after data load to main table, we ignore it as query takes care of it. So we cannot completely rely on this feature for min max.
> > So for accumulator, im not calculating again, i just take the minmax before writing index file in dataload and use that in driver to prepare the dataload ranges for datamaps.
> >
> > The reason to keep the segment min max in the table status of datamap is that, it will be helful in RP scenarios, second is we will not be missing any data from loading to datamap from main table[if 1st time data came from 1 to 4:15 , then next we get data 5:10 to 6, then there might be chance that we can miss 15minutes of data from 4 to 4:15]. It will be helpful in querying also. So that we can avoid the problem i mentioned above with datamaps loaded in cache.
> >
> > 4. I agree, your point is valid one. I will do more abalysis on this based on the user use cases and then we can decide finally. That would be better.
> >
> > Please give your inputs/suggestions on the above points.
> >
> > regards,
> > Akash R Nilugal
> >
> > On 2019/10/07 03:03:35, Ravindra Pesala <[hidden email]> wrote:
> >> HI Akash,
> >>
> >> 1. I feel user providing granularity is redundant, he can just provide respective udf in select query should be enough.
> >>
> >> 2. I think it is better to add the RP management now itself, otherwise if you start adding to DM properties as temporary then it will never be moved. Better put little more effort to decouple it from datamaps.
> >>
> >> 3. I feel accumulator is a added cost, we already have feature in development to load datamap immediately after load happens, why not use that? If the datamap is already in memory why we need min/max at segment level?
> >>
> >> 4. I feel there must be some reason why other timeseries db does not support union of data.  Consider a scenario that we have data from 1pm to 4.30 pm , it means 4 to 5pm data is still loading.  when user asks the data at hour level I feel it is safe to give data for 1,2,3 hours data, because providing 4pm is actually not a complete data. So atleast user comes to know that 4 pm data is not available and starts querying the low level data if he needs it.
> >> I think better get some real uses how user wants this time series data.
> >>
> >> Regards,
> >> Ravindra.
> >>
> >>> On 4 Oct 2019, at 9:39 PM, Akash Nilugal <[hidden email]> wrote:
> >>>
> >>> Hi Ravi,
> >>>
> >>> 1. I forgot to mention the CTAS query in the create datamap statement, i have updated the document, during create datamap user can give granularity, during query just the UDF. That should be fine right.
> >>> 2. I think may be we can mention the RP policy in DM properties also, and then may be we provide add RP, drop RP, alter RP for existing and older datamaps. RP will be taken as a separate subtask and will be handled in later part. That should be fine i tink.
> >>> 3. Actually consider a scenario when datamap is already created, then load happened to main table, then i use accumulator to get all the min max to driver, so that i can avoid reading index file in driver in order to load to datamap.
> >>>        other scenario is when main table already has segments and then datamap is created, the we will read index files from each segments to decide the min max of timestamp column.
> >>> 4. We are not storing min max in main table  table status. We are storing in datamap table's table status file, so that it will be used to prepare the plan during the query phase.
> >>>
> >>> 5. Other timeseries db supports only getting the data present in hour or day .. aggregated data. Since we cannot miss the data, plan is to get the data like higher to lower. May be it does not make much difference when its from minute to second, but it makes difference from year to month , so that we cannot avoid aggregations from main table.
> >>>
> >>>
> >>> Regards,
> >>> Akash R Nilugal
> >>>
> >>> On 2019/10/04 11:35:46, Ravindra Pesala <[hidden email]> wrote:
> >>>> Hi Akash,
> >>>>
> >>>> I have following suggestions.
> >>>>
> >>>> 1. I think it is redundant to use granularity inside create datamap, user can use the respective granularity UDF in his query like time(1h) or time(1d) etc.
> >>>>
> >>>> 2. Better create separate RP commands and let user add the RP on the datamap or even on the main table also. It would be more manageable if you independent feature for RP instead of including in datamap.
> >>>>
> >>>> 3. I am not getting why exactly we need accumulator instead of using index min/max? Can you explain with some scenario
> >>>>
> >>>> 4. Why to store min/max at segment level? We can get from datamap also right?
> >>>>
> >>>> 4.  Union with high granularity tables to low granularity tables are really needed? Any other time series DB is doing it? Or any known use case we have?
> >>>>
> >>>> Regards,
> >>>> Ravindra.
> >>>>
> >>>>> On 1 Oct 2019, at 5:49 PM, Akash Nilugal <[hidden email]> wrote:
> >>>>>
> >>>>> Hi Babu,
> >>>>>
> >>>>> Thanks for the inputs. Please find the comments
> >>>>> 1. I will change from Union to UnionAll
> >>>>> 2. For auto datamap loading, once the data is loaded to lower level granularity datamap, then we load the higher level datamap from the lower level datamap. But as per your point, i think you are telling to load from main table itself.
> >>>>> 3. similar to 2nd point, whether to need configuration or not we can decide i think.
> >>>>> 4. a. I think the max of the datamap is required to decide the range for the load, because in case of failure case, we may need.
> >>>>> b. This point will be taken care.
> >>>>> 5. Yes, dataload is sync based on current design, as it is non lazy, it will happen with main table load only.
> >>>>> 6. Yes, this will be handled.
> >>>>> 7. Already added a task in jira.
> >>>>> On 2019/10/01 08:50:05, babu lal jangir <[hidden email]> wrote:
> >>>>>> Hi Akash, Thanks for Time Series DataMap proposal.
> >>>>>> Please check below Points.
> >>>>>>
> >>>>>> 1. During Query Planing Change Union to Union All , Otherwise will loose
> >>>>>> row if same value appears.
> >>>>>> 2. Whether system start load for next granularity level table as soon it
> >>>>>> matches the data condition or next granularity level table has to wait till
> >>>>>> current  granularity level table is finished ? please handle if possible.
> >>>>>> 3. Add Configuration to load multiple Ranges at a time(across granularity
> >>>>>> tables).
> >>>>>> 4. Please check if Current data loading min ,max is enough to find current
> >>>>>> load . No need to refer the DataMap's min,max because data loading Range
> >>>>>> prepration can go wrong if loading happens from multiple driver . i think
> >>>>>> below rules are enough for loading.
> >>>>>>  4.a. Create MV should should sync data.   On any failure Rebuild should
> >>>>>> sync again till than MV will be disabled.
> >>>>>>  4.b.  Each load has independent Ranges and should load only those
> >>>>>> ranges. Any failure MV may go in disable state(only if intermediate ranges
> >>>>>> load is failed ,last loads failure will NOT make MV disable).
> >>>>>> 5. We can make Data loading sync because anyway queries can be served from
> >>>>>> fact table if any segments is in-progress in  Datamap.
> >>>>>> 6. In Data loading Pipleline ,failures in intermediate time series datamap,
> >>>>>> still we can continue loading next level data. (ignore if already handled).
> >>>>>> For Example.
> >>>>>>  DataMaps:- Hour,Day,Month Level
> >>>>>>  Load Data(10 day):- 2018-01-01 01:00:00 to 2018-01-10 01:00:00
> >>>>>>    Failure in hour level during below range
> >>>>>>      2018-01-06 01:00:00 to 2018-01-06 01:00:00
> >>>>>>   This point of time Hour level has 5 day data.so start loading on day
> >>>>>> level .
> >>>>>> 7. Add SubTask to support loading of in-between missing time.(Incremental
> >>>>>> but old records if timeseries device stopped working for some time).
> >>>>>>
> >>>>>> On Tue, Oct 1, 2019 at 10:41 AM Akash Nilugal <[hidden email]>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hi vishal,
> >>>>>>>
> >>>>>>> In the design document, in the impacted analysis section, there is a topic
> >>>>>>> compatibility/legacy stores, so basically For old tables when the datamap
> >>>>>>> is created, we load all the timeseries datamaps with different granularity.
> >>>>>>> I think this should do fine, please let me know for further
> >>>>>>> suggestions/comments.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Akash R Nilugal
> >>>>>>>
> >>>>>>> On 2019/09/30 17:09:44, Kumar Vishal <[hidden email]> wrote:
> >>>>>>>> Hi Akash,
> >>>>>>>>
> >>>>>>>> In this desing document you haven't mentioned how to handle data loading
> >>>>>>>> for timeseries datamap for older segments[Existing table].
> >>>>>>>> If the customer's main table data is also stored based on time[increasing
> >>>>>>>> time] in different segments,he can use this feature as well.
> >>>>>>>>
> >>>>>>>> We can discuss and finalize the solution.
> >>>>>>>>
> >>>>>>>> -Regards
> >>>>>>>> Kumar Vishal
> >>>>>>>>
> >>>>>>>> On Mon, Sep 30, 2019 at 2:42 PM Akash Nilugal <[hidden email]>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Ajantha,
> >>>>>>>>>
> >>>>>>>>> Thanks for the queries and suggestions
> >>>>>>>>>
> >>>>>>>>> 1. Yes, this is a good suggestion, i ll include this change. Both date
> >>>>>>> and
> >>>>>>>>> timestamp columns are supported, will be updated in document.
> >>>>>>>>> 2. yes, you are right.
> >>>>>>>>> 3. you are right, if the day level is not available, then we will try
> >>>>>>> to
> >>>>>>>>> get the whole day data from hour level, if not availaible, as
> >>>>>>> explained in
> >>>>>>>>> design document, we will get the data from datamap UNION data from main
> >>>>>>>>> table based on user query.
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Akash R Nilugal
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 2019/09/30 06:56:45, Ajantha Bhat <[hidden email]> wrote:
> >>>>>>>>>> + 1 ,
> >>>>>>>>>>
> >>>>>>>>>> I have some suggestions and questions.
> >>>>>>>>>>
> >>>>>>>>>> 1. In DMPROPERTIES, instead of 'timestamp_column' suggest to use
> >>>>>>>>>> 'timeseries_column'.
> >>>>>>>>>> so that it won't give an impression that only time stamp datatype is
> >>>>>>>>>> supported and update the document with all the datatype supported.
> >>>>>>>>>>
> >>>>>>>>>> 2. Querying on this datamap table is also supported right ?
> >>>>>>> supporting
> >>>>>>>>>> changing plan for main table to refer datamap table is for user to
> >>>>>>> avoid
> >>>>>>>>>> changing his query or any other reason ?
> >>>>>>>>>>
> >>>>>>>>>> 3. If user has not created day granularity datamap, but just created
> >>>>>>> hour
> >>>>>>>>>> granularity datamap. When query has day granularity, data will be
> >>>>>>> fetched
> >>>>>>>>>> form hour granularity datamap and aggregated ? or data is fetched
> >>>>>>> from
> >>>>>>>>> main
> >>>>>>>>>> table ?
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Ajantha
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Sep 30, 2019 at 11:46 AM Akash Nilugal <
> >>>>>>> [hidden email]>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi xuchuanyin,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the comments/Suggestions
> >>>>>>>>>>>
> >>>>>>>>>>> 1. Preaggregate is productized, but not the timeseries with
> >>>>>>>>> preaggregate,
> >>>>>>>>>>> i think you  got confused with that, if im right.
> >>>>>>>>>>> 2. Limitations like, auto sampling or rollup, which we will be
> >>>>>>>>> supporting
> >>>>>>>>>>> now. Retention policies. etc
> >>>>>>>>>>> 3. segmentTimestampMin, this i will consider in design.
> >>>>>>>>>>> 4. RP is added as a separate task, i thought instead of
> >>>>>>> maintaining two
> >>>>>>>>>>> variables better to maintabin one and parse it. But i will consider
> >>>>>>>>> your
> >>>>>>>>>>> point based on feasibility during implementation.
> >>>>>>>>>>> 5. We use an accumulator which takes list, so before writing index
> >>>>>>>>> files
> >>>>>>>>>>> we take the min max of the timestamp column and fill in
> >>>>>>> accumulator and
> >>>>>>>>>>> then we can access accumulator.value in driver after load is
> >>>>>>> finished.
> >>>>>>>>>>>
> >>>>>>>>>>> Regards,
> >>>>>>>>>>> Akash R Nilugal
> >>>>>>>>>>>
> >>>>>>>>>>> On 2019/09/28 10:46:31, xuchuanyin <[hidden email]> wrote:
> >>>>>>>>>>>> Hi akash, glad to see the feature proposed and I have some
> >>>>>>> questions
> >>>>>>>>>>> about
> >>>>>>>>>>>> this. Please notice that some of the following descriptions are
> >>>>>>>>> comments
> >>>>>>>>>>>> followed by '===' described in the design document attached in
> >>>>>>> the
> >>>>>>>>>>>> corresponding jira.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1.
> >>>>>>>>>>>> "Currently carbondata supports timeseries on preaggregate
> >>>>>>> datamap,
> >>>>>>>>> but
> >>>>>>>>>>> its
> >>>>>>>>>>>> an alpha feature"
> >>>>>>>>>>>> ===
> >>>>>>>>>>>> It has been some time since the preaggregate datamap was
> >>>>>>> introduced
> >>>>>>>>> and
> >>>>>>>>>>> it
> >>>>>>>>>>>> is still **alpha**, why it is still not product-ready? Will the
> >>>>>>> new
> >>>>>>>>>>> feature
> >>>>>>>>>>>> also come into the similar situation?
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2.
> >>>>>>>>>>>> "there are so many limitations when we compare and analyze the
> >>>>>>>>> existing
> >>>>>>>>>>>> timeseries database or projects which supports time series like
> >>>>>>>>> apache
> >>>>>>>>>>> druid
> >>>>>>>>>>>> or influxdb"
> >>>>>>>>>>>> ===
> >>>>>>>>>>>> What are the actual limitations? Besides, please give an example
> >>>>>>> of
> >>>>>>>>> this.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 3.
> >>>>>>>>>>>> "Segment_Timestamp_Min"
> >>>>>>>>>>>> ===
> >>>>>>>>>>>> Suggest using camel-case style like 'segmentTimestampMin'
> >>>>>>>>>>>>
> >>>>>>>>>>>> 4.
> >>>>>>>>>>>> "RP is way of telling the system, for how long the data should be
> >>>>>>>>> kept"
> >>>>>>>>>>>> ===
> >>>>>>>>>>>> Since the function is simple, I'd suggest using
> >>>>>>> 'retentionTime'=15
> >>>>>>>>> and
> >>>>>>>>>>>> 'timeUnit'='day' instead of 'RP'='15_days'
> >>>>>>>>>>>>
> >>>>>>>>>>>> 5.
> >>>>>>>>>>>> "When the data load is called for main table, use an spark
> >>>>>>>>> accumulator to
> >>>>>>>>>>>> get the maximum value of timestamp in that load and return to the
> >>>>>>>>> load."
> >>>>>>>>>>>> ===
> >>>>>>>>>>>> How can you get the spark accumulator? The load is launched using
> >>>>>>>>>>>> loading-by-dataframe not using global-sort-by-spark.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 6.
> >>>>>>>>>>>> For the rest of the content, still reading.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Sent from:
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>