Topic
  • 3 replies
  • Latest Post - ‏2013-05-22T14:02:48Z by bfawcett
waldstein
waldstein
33 Posts

Pinned topic Can I balance the load dynamically when running instance on mutiple host

‏2013-05-20T07:54:32Z |

Hi all,

If the size of input tuple varies all the time, how does Streams balance the load of each host?

Is there any rules for this? Or can I control this balancing dynamically?

Thx.

  • bfawcett
    bfawcett
    14 Posts

    Re: Can I balance the load dynamically when running instance on mutiple host

    ‏2013-05-21T14:01:43Z  

    Streams balances the load on job submission, but doesn't try to balance the load automatically after that.  

    There is a feature in admin console that assists the administrator in adjusting the load across the hosts in the instance after the instance has been running for awhile.

    See the Streams Console section in the Streams Infocenter.   For Streams 3.0, this is in the following section.

    Note:  that the Operators must be specified as relocatable & restartable in the SPL program in order to take advantage of this ability.

  • waldstein
    waldstein
    33 Posts

    Re: Can I balance the load dynamically when running instance on mutiple host

    ‏2013-05-22T02:23:43Z  
    • bfawcett
    • ‏2013-05-21T14:01:43Z

    Streams balances the load on job submission, but doesn't try to balance the load automatically after that.  

    There is a feature in admin console that assists the administrator in adjusting the load across the hosts in the instance after the instance has been running for awhile.

    See the Streams Console section in the Streams Infocenter.   For Streams 3.0, this is in the following section.

    Note:  that the Operators must be specified as relocatable & restartable in the SPL program in order to take advantage of this ability.

    well, if the user needs to balance the work of hosts mannually, will Streams give user a warning that some of the hosts are overloading, say, it handles 70% or more jobs?

    In practice, the load of some jobs can not be exactly estimated......

  • bfawcett
    bfawcett
    14 Posts

    Re: Can I balance the load dynamically when running instance on mutiple host

    ‏2013-05-22T14:02:48Z  
    • waldstein
    • ‏2013-05-22T02:23:43Z

    well, if the user needs to balance the work of hosts mannually, will Streams give user a warning that some of the hosts are overloading, say, it handles 70% or more jobs?

    In practice, the load of some jobs can not be exactly estimated......

    The amount of load on a system that makes it overloaded is somewhat subjective & unique for each customer situation.

    Streams provides a instance level property (host load threshold) that allows user to define a threshold value of what should be treated as overloaded for them.   This used in conjunction with the host overload protection feature, enables Streams to detect when a host is above the threshold, and not to schedule any new application work on to that system. 

    Also, this threshold is used as basis for the Restart Recommendation feature (mentioned above).  

    This feature allows the user to request that Streams analyze the current workloads that exists on all the hosts, and the various constraints specified by the application, determining if there are better placements that are possible,  and returning a set of recommendations, along with some guidance on how strong of recommendation it is.  User can then accept the recommendations, and the re-placement performed.