标签： 云计算

本文采用的是CDH4+MAPREDUCE 0.20

hadoop中共有6台机作为tasktracker ，每台机配置map和reduce个2个slots

设置完资源后，设置队列的ACL，必须具有相关权限才能向指定队列中提交任务。

登录jobtracker机器

将/usr/lib/hadoop-0.20-mapreduce/contrib/capacity-scheduler/下的hadoop-capacity-scheduler-2.0.0-mr1-cdh4.0.1.jar 复制到/usr/lib/hadoop/lib/目录下

修改/etc/hadoop/conf 下的mapred-site.xml 在其中新增

<property>
<name>mapred.jobtracker.taskScheduler</name>
<value>org.apache.hadoop.mapred.CapacityTaskScheduler</value>
</property>

<property>
<name>mapred.queue.names</name>
<value>default,edp,hive</value>
</property>

<property>
<name>mapred.acls.enabled</name>
<value>true</value>
<description> Specifies whether ACLs should be checked
for authorization of users for doing various queue and job level operations.
ACLs are disabled by default. If enabled, access control checks are made by
JobTracker and TaskTracker when requests are made by users for queue
operations like submit job to a queue and kill a job in the queue and job
operations like viewing the job-details (See mapreduce.job.acl-view-job)
or for modifying the job (See mapreduce.job.acl-modify-job) using
Map/Reduce APIs, RPCs or via the console and web user interfaces.
</description>
</property>

在/etc/hadoop/conf 下新建capacity-scheduler.xml

<?xml version=”1.0″?>

<!– This is the configuration file for the resource manager in Hadoop. –>
<!– You can configure various scheduling parameters related to queues. –>
<!– The properties for a queue follow a naming convention,such as, –>
<!– mapred.capacity-scheduler.queue.<queue-name>.property-name. –>

<configuration>

<property>
<name>mapred.capacity-scheduler.maximum-system-jobs</name>
<value>6</value>
<description>Maximum number of jobs in the system which can be initialized,
concurrently, by the CapacityScheduler.
</description>
</property>

<property>
<name>mapred.capacity-scheduler.queue.default.capacity</name>
<value>10</value>
<description>Percentage of the number of slots in the cluster that are
to be available for jobs in this queue.
</description>
</property>

<property>
<name>mapred.capacity-scheduler.queue.default.maximum-capacity</name>
<value>-1</value>
<description>
maximum-capacity defines a limit beyond which a queue cannot use the capacity of the cluster.
This provides a means to limit how much excess capacity a queue can use. By default, there is no limit.
The maximum-capacity of a queue can only be greater than or equal to its minimum capacity.
Default value of -1 implies a queue can use complete capacity of the cluster.

This property could be to curtail certain jobs which are long running in nature from occupying more than a
certain percentage of the cluster, which in the absence of pre-emption, could lead to capacity guarantees of
other queues being affected.

One important thing to note is that maximum-capacity is a percentage , so based on the cluster’s capacity
the max capacity would change. So if large no of nodes or racks get added to the cluster , max Capacity in
absolute terms would increase accordingly.
</description>
</property>

<property>
<name>mapred.capacity-scheduler.queue.default.supports-priority</name>
<value>false</value>
<description>If true, priorities of jobs will be taken into
account in scheduling decisions.
</description>
</property>

<property>
<name>mapred.capacity-scheduler.queue.default.minimum-user-limit-percent</name>
<value>100</value>
<description> Each queue enforces a limit on the percentage of resources
allocated to a user at any given time, if there is competition for them.
This user limit can vary between a minimum and maximum value. The former
depends on the number of users who have submitted jobs, and the latter is
set to this property value. For example, suppose the value of this
property is 25. If two users have submitted jobs to a queue, no single
user can use more than 50% of the queue resources. If a third user submits
a job, no single user can use more than 33% of the queue resources. With 4
or more users, no user can use more than 25% of the queue’s resources. A
value of 100 implies no user limits are imposed.
</description>
</property>

<property>
<name>mapred.capacity-scheduler.queue.default.user-limit-factor</name>
<value>1</value>
<description>The multiple of the queue capacity which can be configured to
allow a single user to acquire more slots.
</description>
</property>

<property>
<name>mapred.capacity-scheduler.queue.default.maximum-initialized-active-tasks</name>
<value>200000</value>
<description>The maximum number of tasks, across all jobs in the queue,
which can be initialized concurrently. Once the queue’s jobs exceed this
limit they will be queued on disk.
</description>
</property>

<property>
<name>mapred.capacity-scheduler.queue.default.maximum-initialized-active-tasks-per-user</name>
<value>100000</value>
<description>The maximum number of tasks per-user, across all the of the
user’s jobs in the queue, which can be initialized concurrently. Once the
user’s jobs exceed this limit they will be queued on disk.
</description>
</property>

<property>
<name>mapred.capacity-scheduler.queue.default.init-accept-jobs-factor</name>
<value>10</value>
<description>The multipe of (maximum-system-jobs * queue-capacity) used to
determine the number of jobs which are accepted by the scheduler.
</description>
</property>

<!– The default configuration settings for the capacity task scheduler –>
<!– The default values would be applied to all the queues which don’t have –>
<!– the appropriate property for the particular queue –>
<property>
<name>mapred.capacity-scheduler.default-supports-priority</name>
<value>false</value>
<description>If true, priorities of jobs will be taken into
account in scheduling decisions by default in a job queue.
</description>
</property>

<property>
<name>mapred.capacity-scheduler.default-minimum-user-limit-percent</name>
<value>100</value>
<description>The percentage of the resources limited to a particular user
for the job queue at any given point of time by default.
</description>
</property>
<property>
<name>mapred.capacity-scheduler.default-user-limit-factor</name>
<value>1</value>
<description>The default multiple of queue-capacity which is used to
determine the amount of slots a single user can consume concurrently.
</description>
</property>

<property>
<name>mapred.capacity-scheduler.default-maximum-active-tasks-per-queue</name>
<value>200000</value>
<description>The default maximum number of tasks, across all jobs in the
queue, which can be initialized concurrently. Once the queue’s jobs exceed
this limit they will be queued on disk.
</description>
</property>

<property>
<name>mapred.capacity-scheduler.default-maximum-active-tasks-per-user</name>
<value>100000</value>
<description>The default maximum number of tasks per-user, across all the of
the user’s jobs in the queue, which can be initialized concurrently. Once
the user’s jobs exceed this limit they will be queued on disk.
</description>
</property>

<property>
<name>mapred.capacity-scheduler.default-init-accept-jobs-factor</name>
<value>10</value>
<description>The default multipe of (maximum-system-jobs * queue-capacity)
used to determine the number of jobs which are accepted by the scheduler.
</description>
</property>

<!– Capacity scheduler Job Initialization configuration parameters –>
<property>
<name>mapred.capacity-scheduler.init-poll-interval</name>
<value>5000</value>
<description>The amount of time in miliseconds which is used to poll
the job queues for jobs to initialize.
</description>
</property>
<property>
<name>mapred.capacity-scheduler.init-worker-threads</name>
<value>5</value>
<description>Number of worker threads which would be used by
Initialization poller to initialize jobs in a set of queue.
If number mentioned in property is equal to number of job queues
then a single thread would initialize jobs in a queue. If lesser
then a thread would get a set of queues assigned. If the number
is greater then number of threads would be equal to number of
job queues.
</description>
</property>

<property>
<name>mapred.capacity-scheduler.queue.hive.capacity</name>
<value>40</value>
</property>
<property>
<name>mapred.capacity-scheduler.queue.hive.maximum-capacity</name>
<value>80</value>
</property>
<property>
<name>mapred.capacity-scheduler.queue.hive.supports-priority</name>
<value>false</value>
</property>
<property>
<name>mapred.capacity-scheduler.queue.hive.minimum-user-limit-percent</name>
<value>20</value>
</property>
<property>
<name>mapred.capacity-scheduler.queue.hive.user-limit-factor</name>
<value>10</value>
</property>
<property>
<name>mapred.capacity-scheduler.queue.hive.maximum-initialized-active-tasks</name>
<value>200000</value>
</property>
<property>
<name>mapred.capacity-scheduler.queue.hive.maximum-initialized-active-tasks-per-user</name>
<value>100000</value>
</property>
<property>
<name>mapred.capacity-scheduler.queue.hive.init-accept-jobs-factor</name>
<value>100</value>
</property>

<!– queue: edp –>
<property>
<name>mapred.capacity-scheduler.queue.edp.capacity</name>
<value>50</value>
</property>
<property>
<name>mapred.capacity-scheduler.queue.edp.maximum-capacity</name>
<value>90</value>
</property>
<property>
<name>mapred.capacity-scheduler.queue.edp.supports-priority</name>
<value>false</value>
</property>
<property>
<name>mapred.capacity-scheduler.queue.edp.minimum-user-limit-percent</name>
<value>100</value>
</property>
<property>
<name>mapred.capacity-scheduler.queue.edp.user-limit-factor</name>
<value>1</value>
</property>
<property>
<name>mapred.capacity-scheduler.queue.edp.maximum-initialized-active-tasks</name>
<value>200000</value>
</property>
<property>
<name>mapred.capacity-scheduler.queue.edp.maximum-initialized-active-tasks-per-user</name>
<value>100000</value>
</property>
<property>
<name>mapred.capacity-scheduler.queue.edp.init-accept-jobs-factor</name>
<value>10</value>
</property>

</configuration>

在/etc/hadoop/conf 下新建mapred-queue-acls.xml

<configuration>
<property>
<name>mapred.queue.edp.acl-submit-job</name>
<value>xhdeng</value>
<description> Comma separated list of user and group names that are allowed
to submit jobs to the ‘default’ queue. The user list and the group list
are separated by a blank. For e.g. user1,user2 group1,group2.
If set to the special value ‘*’, it means all users are allowed to
submit jobs. If set to ‘ ‘(i.e. space), no user will be allowed to submit
jobs.

It is only used if authorization is enabled in Map/Reduce by setting the
configuration property mapred.acls.enabled to true.
Irrespective of this ACL configuration, the user who started the cluster and
cluster administrators configured via
mapreduce.cluster.administrators can submit jobs.
</description>
</property>
<property>
<name>mapred.queue.hive.acl-submit-job</name>
<value>hive,root</value>
<description> Comma separated list of user and group names that are allowed
to submit jobs to the ‘default’ queue. The user list and the group list
are separated by a blank. For e.g. user1,user2 group1,group2.
If set to the special value ‘*’, it means all users are allowed to
submit jobs. If set to ‘ ‘(i.e. space), no user will be allowed to submit
jobs.

It is only used if authorization is enabled in Map/Reduce by setting the
configuration property mapred.acls.enabled to true.
Irrespective of this ACL configuration, the user who started the cluster and
cluster administrators configured via
mapreduce.cluster.administrators can submit jobs.
</description>
</property>
<property>
<name>mapred.queue.default.acl-submit-job</name>
<value>*</value>
<description> Comma separated list of user and group names that are allowed
to submit jobs to the ‘default’ queue. The user list and the group list
are separated by a blank. For e.g. user1,user2 group1,group2.
If set to the special value ‘*’, it means all users are allowed to
submit jobs. If set to ‘ ‘(i.e. space), no user will be allowed to submit
jobs.

It is only used if authorization is enabled in Map/Reduce by setting the
configuration property mapred.acls.enabled to true.
Irrespective of this ACL configuration, the user who started the cluster and
cluster administrators configured via
mapreduce.cluster.administrators can submit jobs.
</description>
</property>
</configuration>

重启jobtracker，登录hadoop的map/reduce administration即可发现新的调度器生效。

Scheduling Information

如果修改了调度器的配置文件，无需重启整个jobtracker，使用以下命令刷新即可。

hadoop mradmin -refreshQueues

查看当前用户的acl可以使用以下命令查看。

mapred queue -showacls

Queue acls for user : root

Queue Operations
=====================
hive submit-job,administer-jobs
default submit-job,administer-jobs
edp administer-jobs

用root用户往edp队列中跑一个任务测试一下：

sudo  hadoop jar    hadoop-mapreduce-client-jobclient-2.0.0-cdh4.0.1-tests.jar TestDFSIO -D mapred.job.queue.name=edp  -write -nrFiles 6 -fileSize 1000

然后，必然的，报错了。

12/11/23 16:16:19 ERROR security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: User root cannot perform operation SUBMIT_JOB on queue edp.
Please run “hadoop queue -showacls” command to find the queues you have access to .

用root用户往edp用户中跑一个任务测试一下：

sudo  hadoop jar    hadoop-mapreduce-client-jobclient-2.0.0-cdh4.0.1-tests.jar TestDFSIO  -write -nrFiles 6 -fileSize 1000

然后，必然的，报错了。

跑起来了，很悲催的是只能使用1个slot。。。

那往hive队列上再调一次。。

提示：我们设置的capacity是4，而实际的则是Used capacity: 6 (150.0% of Capacity)，符合我们的预期。

使用capacity scheduler ，集群的资源得到有效的管理可控制，即不会让一个用户跑死整个集群，也不会管得过死造成资源闲置。

Hadoop的发行版除了社区的Apache hadoop外，cloudera，hortonworks，mapR，EMC，IBM，INTEL，华为等等都提供了自己的商业版本。商业版主要是提供了专业的技术支持，这对一些大型企业尤其重要。每个发行版都有自己的一些特点，本文就各发行版做简单介绍。

2008 年成立的 Cloudera 是最早将 Hadoop 商用的公司，为合作伙伴提供 Hadoop 的商用解决方案，主要是包括支持，咨询服务，培训。2009年hadoop的创始人 Doug Cutting也任职于 Cloudera 公司。Cloudera产品主要为CDH，Cloudera Manager，Cloudera Support。CDH是Cloudera的hadoop发行版，完全开源，比Apache hadoop在兼容性，安全性，稳定性上有增强。Cloudera Manager是集群的软件分发及管理监控平台，可以在几个小时内部署好一个hadoop集群，并对集群的节点及服务进行实时监控。Cloudera Support即是对hadoop的技术支持。cloudera的标价为每年每个节点4000美元。

2011年成立的Hortonworks是雅虎与硅谷风投公司Benchmark Capital合资组建的公司。公司成立之初吸纳了大约25名至30名专门研究Hadoop的雅虎工程师，上述工程师均在2005年开始协助雅虎开发Hadoop，这些工程师贡献了hadoop 80%的代码。。雅虎工程副总裁、雅虎Hadoop开发团队负责人Eric Baldeschwieler出任Hortonworks的首席执行官。Hortonworks 的主打产品是Hortonworks Data Platform (HDP)，也同样是100%开源的产品，HDP除了常见的项目外还包含了Ambari，一款开源的安装和管理系统。HCatalog，一个元数据管理系统。

HDP的Datasheet中描述的版本特点是

集成和测试封装 – HDP包括稳定版本的Apache Hadoop的所有关键组件，集成和测试封装。

安装方便– HDP包括一个现代化的，直观的用户界面的安装和配置工具。

管理和监控服务 – HDP包括直观的仪表板，为监测集群和建立警示。

数据集成服务 – HDP包括Talend大数据平台，领先的开源整合工具，轻松连接Hadoop集群，而无需编写Hadoop代码的数据系统集成工具。

元数据服务 – HDP包括的Apache HCatalog，从而简化了Hadoop的应用程序之间和Hadoop和其他数据系统之间的数据共享。

高可用性– HDP与成熟的高可用性解决方案的无缝集成。

定价以集群为基础，每10个节点每年为12500美元。

cloudera和hortonworks均是在不断的提交代码完善Apache hadoop，而2009年成立的MapR公司在Hadoop领域显得有点特立独行，它提供了一款独特的发行版 。Hadoop在性能（在当前Hadoop的设计中，所有的meta data操作都要通过集中式的Namenode来进行，Namenode有可能是性能的瓶颈；M/R 应用程序需要通过DataNode来访问HDFS, 这就涉及到格外的进程切换和网络传输开销），可靠性与扩展性（namenode，jobtracker单点问题），企业级应用上的弱点（比如完全可读写的文件系统，snapshot，mirror等等）各大厂商均知，MapR则认为，Hadoop的这些缺陷来自于其架构设计本身，小修小补不能解决问题。他们选择了一条艰难得多的路： 用新架构重写HDFS，同时在API级别，和目前的Hadoop 发行版保持兼容。这家2009年成立的创业公司，在蛰伏了两年之后，终于一鸣惊人，大放异彩。他们成功的“构建一个HDFS的私有替代品，这个替代品比当前的开源版本快三倍，自带快照功能，而且支持无Namenode单点故障（SPOF），并且在API上和兼容，所以可以考虑将其作为替代方案。” mapR版本不再需要单独的namenode机器，元数据分散在集群中，也类似数据默认存储三份。也不再需要用NAS来协助namenode做元数据备份，提供了机器使用率。还有个重要的特点的可以使用nfs直接访问hdfs，提供了与旧有应用的兼容性。镜像功能也很适合做数据备份，而且支持跨数据中心的镜像，快照功能对于数据的恢复作用明显。据报道mapR标价也为每年每个节点4000美元。

mapR有免费和商业两个版本，免费版本在功能上有所缩减。

EMC的Greenplum HD是基于mapR版本二次开发改造而成，特点同mapR。

IBM在去年5月推出了InfoSphere BigInsights软件。该软件包括Apache Hadoop发行版、面向MapReduce编程的Pig编程语言、针对IBM的DB2数据库的连接件以及IBM BigSheets，后者是一种基于浏览器的、使用电子表格隐喻（spreadsheet-metaphor）的界面，用于探究和分析Hadoop里面的数据。IBM在平台管理，安全认证，作业调度算法，与DB2及netezza的集成上做了增强。从IBM中国开发中心信息管理总经理朱辉下面这句话就可以看出IBM对于biginsights的定位：BigInsights并没有替代OLAP（Online Analytical Processing）或OLTP（Online Transaction Processing）应用程序，但它可以整合其中，用于“过滤大量原始数据并合并结果，将结果以结构化数据的形式保存在DBMS或数据仓库中”。

传统的硬件厂商，华为，Intel也提供hadoop的版本

Intel 的商业版本，主要是强调其能提供全面的软硬件解决方案设计，针对硬件具有更好的性能优化，以及提供集群管理工具和安装工具简化了 Hadoop 的安装和配置，能够提供项目规划到实施各阶段专业的咨询服务，实际中采购Intel版本貌似动力不足。

华为在硬件上具有天然的优势，在网络，虚拟化，PC机等都有很强的硬件实力。华为的hadoop版本基于自研的Hadoop HA平台，构建NameNode、JobTracker、HiveServer的HA功能，进程故障后系统自动Failover，无需人工干预，这个也是对hadoop的小修补，远不如mapR解决的彻底。华为在hadoop社区中的Contributor和committer也是国内最多的，算是国内技术实力较强的公司。

———————————————————————————————————

各发行版大事记

mapR

2011-05-25 ：宣布与EMC合作，EMC GREENPLUM HD 提供hadoop基础版本。

2012-01-18：宣布与rainstor合作，

2012-03-05：宣布与informatica合作

2012-06-13:宣布成为Amazon Elastic MapReduce的计算选项

2012-06-28：成为Google App Engine的计算引擎

HortonWorks

2011-10-12：微软宣布将于从雅虎分拆出来的Hortonworks合作开发，在Apache Hadoop上实现搭建Windows Server以及Windows Azure平台。Hortonworks作为微软的战略合作伙伴将会借助自己在此领域的专长帮助最大化将Hadoop集成到微软的产品之中。

2011-11-02：Hortonworks，Apache Hadoop项目的一个主要贡献者，将分发Informatica HParser Community Edition。为Hadoop推出Informatica HParser。作为业界首个针对Hadoop环境的数据解析转换解决方案，Informatica HParser利用MapReduce框架的并行性以有效地在Hadoop中把非结构化复杂数据变成一个结构化或半结构化的格式。

2011-03-03 在今年3月初的Strate大会上，开源数据集成软件厂商Talend宣布与Hortonworks达成协议，将合作把Talend开源数据集成工具带入Apache

2012-03-12 ：TeraData就在周二宣布将与Hortonworks合作，并为客户提供产品和服务。

2012-06-17：Hortonworks宣布将与VMware合作并推出一套运行于HDP高可靠性模式的工具。VMware的vSphere可监测Hadoop的NameNode和JobTracker服务。如果服务出现错误时，vSphere可重定向操作实时备份服务，以保持集群的运行。

Cloudera：

2011年8月5日 – 戴尔宣布与Cloudera新的合作伙伴关系

2011年10月20日 – SGI和Cloudera联合宣布，他们的公司已经签署协议，为SGI分发Cloudera的软件预装在SGI Hadoop集群中。

2012-01-26：今年1月发布的甲骨文大数据机（Oracle Big Data Appliance）将甲骨文-Sun分布式计算平台与Cloudera的Apache Hadoop发行版、Cloudera管理器管理控制台、R分析软件的开源发行版以及甲骨文NoSQL数据库结合起来。甲骨文还包括连接件，因而让数据能够在大数据机与甲骨文Exadata或传统的甲骨文数据库部署环境之间来回传送。

2012-04-26：IBM宣布将Cloudera作为Hadoop商用版本的首选大数据平台。

———————————————————————————————————————-

除了mapR以外的发行版。基本都是在Apache hadoop上做了略微改进，只有mapR与Apache hadoop有较大区别，以下表格是一些功能上的区别，EMC Greenplum HD 是基于mapR所以功能同mapR。

http://www.slideshare.net/mcsrivas/design-scale-and-performance-of-maprs-distribution-for-hadoop

http://qing.weibo.com/2294942122/88ca09aa330003wz.html

http://qing.weibo.com/2294942122/88ca09aa330003x6.html

http://qing.weibo.com/2294942122/88ca09aa330003zv.html

http://qing.weibo.com/2294942122/88ca09aa330003zz.html

https://issues.apache.org/jira/browse/HDFS-347