当前位置: 首页>>hadoop 2.0之YARN>> 阅读正文

YARN/MRv2的Client端代码分析

Category: hadoop 2.0之YARN View: 159,441 Author: Dong
, , , , ,

  • 评论 (5)
  • 引用通告 (0)
发表评论 发起引用

  • 1楼小菜 回复

    Post: 2012-06-03 14:41

    问个入门级别的问题:yarn的实现代码是不是都在hadoop-yarn-common/hadoop-yarn-node/hadoop-yarn-resource三个文件夹下面。

    感觉文件夹下面的包布局没有hadoop V1清晰。

    [回复]

    Dong 回复:

    你看的哪个版本,0.23.0还是2.0.0-alpha

    [回复]

  • 2楼李秾 回复

    Post: 2013-03-22 02:30

    你好,看了你的文章,写的细致清楚,学了很多。:)
    btw,具体细节(1)的ApplicationsMaster 应该是ApplicationsManager吧。

    [回复]

    Dong 回复:

    是的,多谢修正。

    [回复]

    李秾 回复:

    我现在要在YARN上实现MPI计算框架,不过我拿到这个题目还不到一个月,对MPI的具体JAVA实现没有一个确切的想法。之前我想应该是用JNI实现一个JAVA语言接口类,调用底层的C/C++文件。但是带我的老师说,是用JAVA去实现MPI函数(也就是起码有最基本的那6个)。

    [回复]

    Dong 回复:

    你这个任务是充满风险的,因为YARN上的MPI计算框架实现已经在开源社区的todo list上了,且已经做了2年,随时可能发布代码,你的速度不一定比他们的快。 我给你的建议是,好好阅读这个jira连接(https://issues.apache.org/jira/browse/MAPREDUCE-2911)上各路牛人的探讨,重点关注他们谈到的实现MPI的难点,另外,这个框架叫:Hamster,对应的ppt是:http://www.open-mpi.org/video/mrplus/Greenplum_RalphCastain-2up.pdf,你可通过邮件与我探讨,联系方式是:http://dongxicheng.org/about/

    [回复]

    Dong 回复:

    另外,正是由于Hamster这个项目,才导致java设计人员为JDK增加了一个C语言封装:http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.devel/70734:
    1. Why is there a Java interface in Open MPI?
    The Java interface is provided in Open MPI on a provisional basis. It is not part of the current or any proposed MPI standard. Continued inclusion of the Java interface is contingent upon active user interest and continued developer support.

    The decision to add a Java interface to Open MPI was motivated by a request from the Hadoop community. Hadoop is a Java-based environment for processing extremely large data sets. Modeled on the Google enterprise system, it has evolved into its own rapidly growing open-source community.

    Although executed in parallel, Hadoop processes are independent and thus would only possibly use MPI as a base for efficient messaging. Of greater interest, however, is the ability to add MPI-based follow-on processing to a Hadoop map-reduce operation. Since Hadoop map-reduce is typically done in Java, any subsequent MPI operations would best be done in that language.

    [回复]

  • 3楼实习生 回复

    Post: 2013-05-03 12:15

    博主,您好,又来请教你了,关于omega,mesos和yarn的具体流程与区别终于搞清楚了,现在我想在yarn中扩展使其可以支持虚拟机任务的调度,我只需要学习编写相应的client与applicationmaster?对代码应该从哪下手?

    [回复]

  • 4楼daerduochen 回复

    Post: 2014-06-25 08:32

    请问博主,appMaster和app是同时提交的吗,nodemanager启动container的时候,怎么知道appMaster在哪里?谢谢

    [回复]

  • 5楼 回复

    Post: 2014-10-29 11:44

    yarn的接口是什么呢?yarn各部分之间通信的接口,还是对上层框架的接口?

    [回复]

目前还没有任何Trackbacks和Pingbacks.
发表评论