当前位置: 首页>>hadoop 2.0之YARN>> 阅读正文

Hadoop YARN配置参数剖析(5)—Capacity Scheduler相关参数

Category: hadoop 2.0之YARN View: 14,653 Author: Dong
, , ,

  • 评论 (10)
  • 引用通告 (3)
发表评论 发起引用

  • 1楼tmac321 回复

    Post: 2014-04-09 03:13

    老师好,我在root队列下指定了a,b,c三个子队列,对a分配了70的容量,最大100的容量,但是在使用的时候发现使用a队列最大只能使用集群70%的容量,超出的任务都等待了,请问怎么配置可以使a能占有集群100%的容量,谢谢

    [回复]

    rainy 回复:

    我也碰到相同问题了,求回复!

    [回复]

    Dong 回复:

    把完整的配置贴出来看看

    [回复]

    tmac321 回复:

    yarn.scheduler.capacity.maximum-applications
    2000

    Maximum number of applications that can be pending and running.

    yarn.scheduler.capacity.maximum-am-resource-percent
    0.1

    Maximum percent of resources in the cluster which can be used to run
    application masters i.e. controls number of concurrent running
    applications.

    yarn.scheduler.capacity.resource-calculator
    org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator

    The ResourceCalculator implementation to be used to compare
    Resources in the scheduler.
    The default i.e. DefaultResourceCalculator only uses Memory while
    DominantResourceCalculator uses dominant-resource to compare
    multi-dimensional resources such as Memory, CPU etc.

    yarn.scheduler.capacity.root.queues
    default,hive,work

    The queues at the this level (root is the root queue).

    yarn.scheduler.capacity.root.default.capacity
    70
    Default queue target capacity.

    yarn.scheduler.capacity.root.default.user-limit-factor
    1

    Default queue user limit a percentage from 0.0 to 1.0.

    yarn.scheduler.capacity.root.default.maximum-capacity
    100

    The maximum capacity of the default queue.

    yarn.scheduler.capacity.root.default.state
    RUNNING

    The state of the default queue. State can be one of RUNNING or STOPPED.

    yarn.scheduler.capacity.root.default.acl_submit_applications
    *

    The ACL of who can submit jobs to the default queue.

    yarn.scheduler.capacity.root.default.acl_administer_queue
    *

    The ACL of who can administer jobs on the default queue.

    yarn.scheduler.capacity.root.hive.capacity
    20
    Default queue target capacity.

    yarn.scheduler.capacity.root.hive.user-limit-factor
    1

    Default queue user limit a percentage from 0.0 to 1.0.

    yarn.scheduler.capacity.root.hive.maximum-capacity
    40

    The maximum capacity of the default queue.

    yarn.scheduler.capacity.root.hive.state
    RUNNING

    The state of the default queue. State can be one of RUNNING or STOPPED.

    yarn.scheduler.capacity.root.hive.acl_submit_applications
    *

    The ACL of who can submit jobs to the default queue.

    yarn.scheduler.capacity.root.hive.acl_administer_queue
    *

    The ACL of who can administer jobs on the default queue.

    yarn.scheduler.capacity.root.work.capacity
    10
    Default queue target capacity.

    yarn.scheduler.capacity.root.work.user-limit-factor
    1

    Default queue user limit a percentage from 0.0 to 1.0.

    yarn.scheduler.capacity.root.work.maximum-capacity
    10

    The maximum capacity of the default queue.

    yarn.scheduler.capacity.root.work.state
    RUNNING

    The state of the default queue. State can be one of RUNNING or STOPPED.

    yarn.scheduler.capacity.root.work.acl_submit_applications
    *

    The ACL of who can submit jobs to the default queue.

    yarn.scheduler.capacity.root.work.acl_administer_queue
    *

    The ACL of who can administer jobs on the default queue.

    yarn.scheduler.capacity.node-locality-delay
    -1

    Number of missed scheduling opportunities after which the CapacityScheduler
    attempts to schedule rack-local containers.
    Typically this should be set to number of racks in the cluster, this
    feature is disabled by default, set to -1.

    [回复]

  • 2楼alvis 回复

    Post: 2014-07-31 08:18

    董老师。。确实有这个问题。。我也遇到了,设置最大容量没有用。。

    [回复]

  • 3楼草原 回复

    Post: 2014-10-19 06:31

    董神,请教你个问题,我现在搭建了hadoop2.x的一个集群,有一个需求:就是想创建几个新的用户,让这些用户都能在集群上运行程序,这点有什么办法吗?我在网上找了很长时间没有什么比较可行的方法,能否指点我一下

    [回复]

  • 4楼loser 回复

    Post: 2014-10-27 16:18

    yarn.scheduler.capacity..capacity Queue capacity in percentage (%) as a float (e.g. 12.5). The sum of capacities for all queues, at each level, must be equal to 100. Applications in the queue may consume more resources than the queue’s capacity if there are free resources, providing elasticity.
    官网说所有队列的容量之和应等于100,而楼主说应小于100,请问该如何理解?

    [回复]

  • 5楼loser 回复

    Post: 2014-10-27 16:32

    yarn.scheduler.capacity..capacity 总和为何小于100?

    [回复]

  • 6楼jane 回复

    Post: 2014-10-31 03:22

    其中,最小值取决于正在运行的应用程序数目,而最大值则由minimum-user-limit-percent决定。
    董老师,这句是不是有点小问题呀?是不是“而最小值由minimum-user-limit-percent决定”?

    [回复]

  • 7楼jane 回复

    Post: 2014-10-31 03:24

    其中,最小值取决于正在运行的应用程序数目,而最大值则由minimum-user-limit-percent决定。
    请问董老师,这句是不是有点小问题呀?是不是“而最大值则由minimum-user-limit-percent决定”?

    [回复]

  • 8楼zhou 回复

    Post: 2014-11-15 04:46

    董老师,您好:
    关于配置了最大容量没有起作用的问题,主要的原因是user-limit-factor参数的设置问题,比如说设置了yarn.scheduler.capacity.root.default.capacity为70,然后设置user-limit-factor为1.3,则该default队列可以使用的最大容量是0.7*1.3=0.91。当然设置最大容量是用来限制最大的资源的。

    [回复]

  • 9楼zhao 回复

    Post: 2015-05-14 06:41

    老董,这句话: user-limit-factor:每个用户最多可使用的资源量(百分比)。比如,假设该值为30,则任何时刻,每个用户使用的资源量不能超过该队列容量的30%。的解释,是不是有点问题?英文原文是:The multiple of the queue capacity which can be configured to allow a single user to acquire more resources.

    [回复]

  • 10楼大圆 回复

    Post: 2015-06-03 08:13

    你好,user-limit-factor这个参数我有些疑问。在Hadoop 1.0中我测试过,不是上面的解释,现在yarn中有了新的调整?
    我的理解是:设置queue中用户可占用queue容量的系数,默认为1表示,queue中每个用户最多只能占有queue的容量(即mapred.capacity-scheduler.queue..capacity),因此,如果queue中只有一个用户提交job,且希望用户在集群不繁忙时候可扩展到mapred.capacity-scheduler.queue..maximum-capacity指定的slots数,则必须相应调大user-limit-factor这个系数。

    [回复]

发表评论