Hadoop 2 Introduces YARN
Hadoop 2 Introduces YARN
Written by Kay Ewbank   
Tuesday, 22 October 2013

Apache Hadoop 2 is now available with support for Apache YARN, a framework for job scheduling and cluster resource management, and high availability for the HDFS filing system.

In fact, the release is 2.2.0, but it is the first stable release in the 2.x line. The biggest change in the new version of the distributed big data storage and analysis framework is the support for YARN. YARN sits on top of HDFS (Hadoop distributed file system) and serves as a large-scale, distributed operating system for big data applications, so multiple applications can run simultaneously.

This is a major change to the way Hadoop works. Previous releases relied on the MapReduce framework to handle the division of work as well as managing the resources of the servers. In the new version YARN is used as the resource manager.


Architectural View of YARN (Source: HortonWorks)

The way YARN works is by splitting the work undertaken by the MapReduce JobTracker component in two. JobTracker has until now managed both resource management and job scheduling/monitoring, but these are now run as two separate applications: a global ResourceManager and an ApplicationMaster that has a separate copy for each application running.

If you want to find out more about YARN, Arun Murthy, release manager of Apache Hadoop 2 and Founder of Hortonworks, outlines it in an informative series of posts on the Hortonworks blog.

This change opens up Hadoop so that developers will be able to build apps directly within Hadoop, instead of writing them to run externally, and represents a major opening up of Hadoop.

Alongside the introduction of YARN, the HDFS aspect of Hadoop has also been improved, with high availability for HDFS, support for HDFS snapshots so you’ll be able to use native backup and disaster recovery processes. The ability to use the NFSv3 file system to access data in HDFS also makes Hadoop more ‘mainstream’, as it can now be mounted as a standard filesystem. Native network encryption has been added to secure data in transit. The team behind Hadoop has put a lot of work into stabilizing Hadoop’s APIs. Finally, Hadoop 2.2 also adds support for running it on Microsoft Windows.



More Information
Apache Hadoop

Introducing Apache Hadoop YARN

Related Articles

Hadoop for Windows

Microsoft Open Sources Big Data REEF

Hadoop gets to 1.0

Hive on Hadoop for MongoDB


To be informed about new articles on I Programmer, install the I Programmer Toolbar, subscribe to the RSS feed, follow us on, Twitter, Facebook, Google+ or Linkedin,  or sign up for our weekly newsletter.




blog comments powered by Disqus



Atom Adds Collaborative Coding

GitHub's Atom text editor has a new feature, currently in beta, for collaborative coding. Teletype for Atom lets you write and edit code with other developers in real-time.

Programmers Against Slaughterbots

Autonomous weapons might sound like a good idea at first, why send soldiers to die, but the potential for misuse is obvious. You might not have thought of such as the slaughterbot - you need to see th [ ... ]

More News


Last Updated ( Tuesday, 22 October 2013 )

RSS feed of news items only
I Programmer News
Copyright © 2017 i-programmer.info. All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.