Carbon over-use cluster resources

[Manhua Jiang]

Hi All,
Recently, I found carbon over-use cluster resources. Generally the design of carbon work flow does not act as common spark task which only do one small work in one thread, but the task has its mind/logic.

For example,
1.launch carbon with --num-executors=1 but set carbon.number.of.cores.while.loading=10;
2.no_sort table with multi-block input, N Iterator<CarbonRowBatch> for example, carbon will start N tasks in parallel. And in each task the CarbonFactDataHandlerColumnar has model.getNumberOfCores() (let's say C) in ProducerPool. Totally launch N*C threads; ==>This is the case makes me take this as serious problem. To many threads stucks the executor to send heartbeat and be killed.

So, the over-use is related to usage of threadpool.

This would affect the cluster overall resource usage and may lead to wrong performance results.

I hope this get your notice while fixing or writing new codes.

[Liang Chen]

OK, thank you feedbacked this issue, let us look into it.

Regards
Liang


Manhua Jiang wrote

> Hi All,
> Recently, I found carbon over-use cluster resources. Generally the design
> of carbon work flow does not act as common spark task which only do one
> small work in one thread, but the task has its mind/logic.
>
> For example,
> 1.launch carbon with --num-executors=1 but set
> carbon.number.of.cores.while.loading=10;
> 2.no_sort table with multi-block input, N Iterator
> <CarbonRowBatch>
>  for example, carbon will start N tasks in parallel. And in each task the
> CarbonFactDataHandlerColumnar has model.getNumberOfCores() (let's say C)
> in ProducerPool. Totally launch N*C threads; ==>This is the case makes me
> take this as serious problem. To many threads stucks the executor to send
> heartbeat and be killed.
>
> So, the over-use is related to usage of threadpool.
>
> This would affect the cluster overall resource usage and may lead to wrong
> performance results.
>
> I hope this get your notice while fixing or writing new codes.

--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[Ajantha Bhat]

Hi Manhua,

For only No sort and Local sort, we don't follow spark task launch logic.
we have our own logic of one node one task. And inside that task we can
control resource by configuration (carbon.number.of.cores.while.loading)

As you pointed in the above mail, *N * C is controlled by configuration*
and the default value of C is 2.
*I see over use cluster problem only if you configure it badly.*

Do you have any suggestion to the change design? Feel free to raise a
discussion and work on it.

Thanks,
Ajantha

On Tue, Apr 14, 2020 at 6:06 PM Liang Chen <[hidden email]> wrote:

> OK, thank you feedbacked this issue, let us look into it.
>
> Regards
> Liang
>
>
> Manhua Jiang wrote
> > Hi All,
> > Recently, I found carbon over-use cluster resources. Generally the design
> > of carbon work flow does not act as common spark task which only do one
> > small work in one thread, but the task has its mind/logic.
> >
> > For example,
> > 1.launch carbon with --num-executors=1 but set
> > carbon.number.of.cores.while.loading=10;
> > 2.no_sort table with multi-block input, N Iterator
> > <CarbonRowBatch>
> >  for example, carbon will start N tasks in parallel. And in each task the
> > CarbonFactDataHandlerColumnar has model.getNumberOfCores() (let's say C)
> > in ProducerPool. Totally launch N*C threads; ==>This is the case makes me
> > take this as serious problem. To many threads stucks the executor to send
> > heartbeat and be killed.
> >
> > So, the over-use is related to usage of threadpool.
> >
> > This would affect the cluster overall resource usage and may lead to
> wrong
> > performance results.
> >
> > I hope this get your notice while fixing or writing new codes.
>
>
>
>
>
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>

[kumarvishal09]

Hi Manhua,
In addition to what Ajantha said. All the configuration are exposed to the
user.
And by default no of threads is 2, so in 1 core launching 2 thread is okay.

-Regarda
Kumar Vishal

On Wed, 15 Apr 2020 at 9:55 PM, Ajantha Bhat <[hidden email]> wrote:

> Hi Manhua,
>
> For only No sort and Local sort, we don't follow spark task launch logic.
> we have our own logic of one node one task. And inside that task we can
> control resource by configuration (carbon.number.of.cores.while.loading)
>
> As you pointed in the above mail, *N * C is controlled by configuration*
> and the default value of C is 2.
> *I see over use cluster problem only if you configure it badly.*
>
> Do you have any suggestion to the change design? Feel free to raise a
> discussion and work on it.
>
> Thanks,
> Ajantha
>
> On Tue, Apr 14, 2020 at 6:06 PM Liang Chen <[hidden email]>
> wrote:
>
> > OK, thank you feedbacked this issue, let us look into it.
> >
> > Regards
> > Liang
> >
> >
> > Manhua Jiang wrote
> > > Hi All,
> > > Recently, I found carbon over-use cluster resources. Generally the
> design
> > > of carbon work flow does not act as common spark task which only do one
> > > small work in one thread, but the task has its mind/logic.
> > >
> > > For example,
> > > 1.launch carbon with --num-executors=1 but set
> > > carbon.number.of.cores.while.loading=10;
> > > 2.no_sort table with multi-block input, N Iterator
> > > <CarbonRowBatch>
> > >  for example, carbon will start N tasks in parallel. And in each task
> the
> > > CarbonFactDataHandlerColumnar has model.getNumberOfCores() (let's say
> C)
> > > in ProducerPool. Totally launch N*C threads; ==>This is the case makes
> me
> > > take this as serious problem. To many threads stucks the executor to
> send
> > > heartbeat and be killed.
> > >
> > > So, the over-use is related to usage of threadpool.
> > >
> > > This would affect the cluster overall resource usage and may lead to
> > wrong
> > > performance results.
> > >
> > > I hope this get your notice while fixing or writing new codes.
> >
> >
> >
> >
> >
> > --
> > Sent from:
> > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> >
>

[Manhua Jiang]

Hi Vishal,
what you said "1 core launching 2 thread" could be the view from system level, right?
In yarn mode, what application got is vCore, so carbon should not take that as a physical core.

On 2020/04/16 16:15:23, Kumar Vishal <[hidden email]> wrote:

> Hi Manhua,
> In addition to what Ajantha said. All the configuration are exposed to the
> user.
> And by default no of threads is 2, so in 1 core launching 2 thread is okay.
>
> -Regarda
> Kumar Vishal
>
> On Wed, 15 Apr 2020 at 9:55 PM, Ajantha Bhat <[hidden email]> wrote:
>
> > Hi Manhua,
> >
> > For only No sort and Local sort, we don't follow spark task launch logic.
> > we have our own logic of one node one task. And inside that task we can
> > control resource by configuration (carbon.number.of.cores.while.loading)
> >
> > As you pointed in the above mail, *N * C is controlled by configuration*
> > and the default value of C is 2.
> > *I see over use cluster problem only if you configure it badly.*
> >
> > Do you have any suggestion to the change design? Feel free to raise a
> > discussion and work on it.
> >
> > Thanks,
> > Ajantha
> >
> > On Tue, Apr 14, 2020 at 6:06 PM Liang Chen <[hidden email]>
> > wrote:
> >
> > > OK, thank you feedbacked this issue, let us look into it.
> > >
> > > Regards
> > > Liang
> > >
> > >
> > > Manhua Jiang wrote
> > > > Hi All,
> > > > Recently, I found carbon over-use cluster resources. Generally the
> > design
> > > > of carbon work flow does not act as common spark task which only do one
> > > > small work in one thread, but the task has its mind/logic.
> > > >
> > > > For example,
> > > > 1.launch carbon with --num-executors=1 but set
> > > > carbon.number.of.cores.while.loading=10;
> > > > 2.no_sort table with multi-block input, N Iterator
> > > > <CarbonRowBatch>
> > > >  for example, carbon will start N tasks in parallel. And in each task
> > the
> > > > CarbonFactDataHandlerColumnar has model.getNumberOfCores() (let's say
> > C)
> > > > in ProducerPool. Totally launch N*C threads; ==>This is the case makes
> > me
> > > > take this as serious problem. To many threads stucks the executor to
> > send
> > > > heartbeat and be killed.
> > > >
> > > > So, the over-use is related to usage of threadpool.
> > > >
> > > > This would affect the cluster overall resource usage and may lead to
> > > wrong
> > > > performance results.
> > > >
> > > > I hope this get your notice while fixing or writing new codes.
> > >
> > >
> > >
> > >
> > >
> > > --
> > > Sent from:
> > > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> > >
> >
>

[Manhua Jiang]

Hi Ajantha,
If we think of this problem in the opposite, carbon may waste resources if user do not set the properties correctly.

What about the case when concurrent loading?

So first of all, we need to figure out where and how many the executor services is used. If keeping logic of one node one task, need to keep the overall running threads in a task.

Then, a little thinking:
Is a global executor service possible? That may cause some dependencies in different steps of loading.
Is multiple executor services for each step(or others) of loading possible? Can the specific executor services change size? (like local-sort is done, then most threads work for writing and none for input reading and converting)

BTW, do you know why the cofigurtation "carbon.number.of.cores.while.loading" born ?




On 2020/04/15 13:54:50, Ajantha Bhat <[hidden email]> wrote:

> Hi Manhua,
>
> For only No sort and Local sort, we don't follow spark task launch logic.
> we have our own logic of one node one task. And inside that task we can
> control resource by configuration (carbon.number.of.cores.while.loading)
>
> As you pointed in the above mail, *N * C is controlled by configuration*
> and the default value of C is 2.
> *I see over use cluster problem only if you configure it badly.*
>
> Do you have any suggestion to the change design? Feel free to raise a
> discussion and work on it.
>
> Thanks,
> Ajantha
>
> On Tue, Apr 14, 2020 at 6:06 PM Liang Chen <[hidden email]> wrote:
>
> > OK, thank you feedbacked this issue, let us look into it.
> >
> > Regards
> > Liang
> >
> >
> > Manhua Jiang wrote
> > > Hi All,
> > > Recently, I found carbon over-use cluster resources. Generally the design
> > > of carbon work flow does not act as common spark task which only do one
> > > small work in one thread, but the task has its mind/logic.
> > >
> > > For example,
> > > 1.launch carbon with --num-executors=1 but set
> > > carbon.number.of.cores.while.loading=10;
> > > 2.no_sort table with multi-block input, N Iterator
> > > <CarbonRowBatch>
> > >  for example, carbon will start N tasks in parallel. And in each task the
> > > CarbonFactDataHandlerColumnar has model.getNumberOfCores() (let's say C)
> > > in ProducerPool. Totally launch N*C threads; ==>This is the case makes me
> > > take this as serious problem. To many threads stucks the executor to send
> > > heartbeat and be killed.
> > >
> > > So, the over-use is related to usage of threadpool.
> > >
> > > This would affect the cluster overall resource usage and may lead to
> > wrong
> > > performance results.
> > >
> > > I hope this get your notice while fixing or writing new codes.
> >
> >
> >
> >
> >
> > --
> > Sent from:
> > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> >
>

[David CaiQiang]

Hi, manhua
  Now no_sort reuse the loading flow of local_sort. It is not a good
solution and led to the situation which you have mentioned. In my opinion,
we need to adjust the loading flow of no_sort, maybe like global_sort
finally.

  In addition, the producer-consumer pattern in data encoding and
compression also can be optimized for no_sort and global_sort, maybe just
prefetch one page and process it instead of using a thread pool.



-----
Best Regards
David Cai
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[floras12]

Managing shared resources efficiently is essential in any system because small inefficiencies can quickly affect overall performance and stability. The same principle applies to everyday routines where practical tools such as the https://theingenuitybabyswing.com/ingenuity-baby-chair/ can help keep things organized and make daily tasks feel more manageable without adding extra complexity.

[clark]

When managing cluster resources efficiently, it also helps to keep practical travel gear organized for frequent business trips and work related travel. A coordinated two piece set with carry on and checked luggage, spinner wheels, TSA approved locks, and useful compartments can make airport transfers easier. The BritBag luggage range offers both hardshell and softside options, giving travelers flexibility based on their packing needs.

[michael302]

Keeping resource use efficient often comes down to having the right approach for different workloads, and cleaning pet hair can be similar when different surfaces need different tools. The https://chomchomco.com/roller-mini-bundle/ combines a full size roller for furniture and carpets with a compact version for car interiors, tight spaces, and smaller surfaces, using reusable rollers with no batteries or refill sheets. That kind of practical, reusable setup is useful for quick cleanup without adding unnecessary waste or extra supplies.

[intelligentm]

The discussion around managing demanding workloads made me think about how the right equipment can also make practical differences during long, hands on tasks. Five Ten Shoes use a Stealth C4 rubber outsole, reinforced toe cap, and breathable construction, making them useful for hiking, scrambling, and varied rocky terrain where dependable grip matters.

[ikw12]

When looking at cluster resource usage, clear visibility into how activity is prioritized can help make systems easier to manage and troubleshoot. In a similar way, Snapchat’s Friend Solar System uses interaction activity such as snaps, chats, streaks, and story replies to place friends from Mercury through Neptune, with rankings updating as activity changes. see more for a simple breakdown of how these rankings work.

[nik12]

Resource management can make a big difference when systems are handling demanding workloads, especially when monitoring usage and preventing unnecessary strain. The same idea applies to pet ownership, where understanding temperament, exercise needs, training, and health can help families make better decisions, and learn more offers useful guidance on Fox Red Labrador care and ownership.

[casee]

When managing large datasets and resource intensive workloads, keeping unnecessary network activity and file transfers under control can help reduce pressure on shared systems. For teams that also move photos, videos, apps, or documents between devices, https://thexenderapk.com/ supports fast file transfers without requiring an internet connection, which can be useful when handling files locally.

[ashtonwillanderson]

When working with large data processing jobs, keeping an eye on resource usage is important because excessive cluster consumption can affect performance and costs. The same habit of regularly checking important records applies to household utilities, and a quick mepco bill check can show the payable amount, due date, billing month, units consumed, taxes, and surcharges using a 14 digit reference number or 10 digit customer ID.

[Simon Jack]

Good point about threadpool usage and how launching too many threads can affect overall resource availability. Managing resources efficiently is important not only in distributed systems but also when planning everyday expenses. For anyone comparing the cost of their regular coffee orders, this guide on 7 brew coffee prices may also be useful.

[Johnny]

Managing resources efficiently is important in both technical and everyday environments. Just as careful resource planning can improve system performance, making informed choices when purchasing essential items can help with budgeting. For pet owners, you can check pet supplies online for useful products and options.