Topic
5 replies Latest Post - ‏2008-09-29T18:48:12Z by lcham
astdenis
astdenis
11 Posts
ACCEPTED ANSWER

Pinned topic Loadleveler co-scheduling and max_idle.

‏2008-09-18T13:07:00Z |
In order to control queue flooding, we have set max_idle to 3 and it's been doing what we expect. Lately, our users have been puzzled by this behaviour: a job with 4+ co-scheduled steps will sit in the queue for ever...

Since co-scheduled steps must run concurrently, shouldn't they count as 1 in the idle queue?

Food for a PMR maybe?

Alain
Updated on 2008-09-29T18:48:12Z at 2008-09-29T18:48:12Z by lcham
  • lcham
    lcham
    7 Posts
    ACCEPTED ANSWER

    Re: Loadleveler co-scheduling and max_idle.

    ‏2008-09-23T15:04:43Z  in response to astdenis
    Hi.
    Coscheduled steps are counted as individual idle steps.

    In the doc the following is stated,
    "All steps within a job with the coschedule keyword set to true will be coscheduled.
    The coscheduled steps will continue to be stored as individual steps in both memory and in the job queue, but when performing certain operations, such as scheduling, the steps will be managed as a single entity."

    Regards,
    Linda
    • astdenis
      astdenis
      11 Posts
      ACCEPTED ANSWER

      Re: Loadleveler co-scheduling and max_idle.

      ‏2008-09-25T11:48:20Z  in response to lcham
      So how will co-scheduled job steps ever be managed as a single entity for scheduling if they never make it to the idle queue?

      Alain
      • lcham
        lcham
        7 Posts
        ACCEPTED ANSWER

        Re: Loadleveler co-scheduling and max_idle.

        ‏2008-09-25T14:32:54Z  in response to astdenis
        Coscheduled steps are treated as individual steps on the job queue so that limits such as max_idle can not be circumvented by simply defining multiple steps as coscheduled.

        Regards,
        Linda
        • astdenis
          astdenis
          11 Posts
          ACCEPTED ANSWER

          Re: Loadleveler co-scheduling and max_idle.

          ‏2008-09-26T20:57:44Z  in response to lcham
          Say I build a x step job with the intention of circumventing the max_idle limit. Won't the very fact I use co-scheduling defeat my purpose? My understanding of the max_idle limit is it's used to prevent queue flooding. If I fill the idle queue with x steps that will be scheduled at the same time, it means all the resources required by my job need to be available at that time, just like if it was a large single step job.

          May be I don't have enough imagination to see how one could abuse a system where co-scheduled steps are treated as one. Can you provide examples?

          Thanks.
          Alain.
          • lcham
            lcham
            7 Posts
            ACCEPTED ANSWER

            Re: Loadleveler co-scheduling and max_idle.

            ‏2008-09-29T18:48:12Z  in response to astdenis
            Hi Alain,

            A possible scenario where coscheduled steps can be abused would be a queue with 5 job steps and max idle per user set to 1 where user A has 4 job steps which each require 25% of the cluster resources and user B has a job step which requires 75% of the cluster resources. Assuming the steps all have the same priority then without coschedule, steps A1 and B1 would run first. If A's job steps are changed to coschedule then all of A's job steps may run before B's job step.

            But we do see your point, by trying to protect against abuses of coschedule steps, we force the legitimate users of coschedule steps to increase queue limits thus allowing non coschedule steps to flood the queue. We don't think we can treat coscheduled steps as a single idle step without adding some administrative control over the use of coshedule steps.

            Regards,
            Linda