Hadoop 2 Introduces YARN
Written by Kay Ewbank   
Tuesday, 22 October 2013

Apache Hadoop 2 is now available with support for Apache YARN, a framework for job scheduling and cluster resource management, and high availability for the HDFS filing system.

In fact, the release is 2.2.0, but it is the first stable release in the 2.x line. The biggest change in the new version of the distributed big data storage and analysis framework is the support for YARN. YARN sits on top of HDFS (Hadoop distributed file system) and serves as a large-scale, distributed operating system for big data applications, so multiple applications can run simultaneously.

This is a major change to the way Hadoop works. Previous releases relied on the MapReduce framework to handle the division of work as well as managing the resources of the servers. In the new version YARN is used as the resource manager.


Architectural View of YARN (Source: HortonWorks)

The way YARN works is by splitting the work undertaken by the MapReduce JobTracker component in two. JobTracker has until now managed both resource management and job scheduling/monitoring, but these are now run as two separate applications: a global ResourceManager and an ApplicationMaster that has a separate copy for each application running.

If you want to find out more about YARN, Arun Murthy, release manager of Apache Hadoop 2 and Founder of Hortonworks, outlines it in an informative series of posts on the Hortonworks blog.

This change opens up Hadoop so that developers will be able to build apps directly within Hadoop, instead of writing them to run externally, and represents a major opening up of Hadoop.

Alongside the introduction of YARN, the HDFS aspect of Hadoop has also been improved, with high availability for HDFS, support for HDFS snapshots so you’ll be able to use native backup and disaster recovery processes. The ability to use the NFSv3 file system to access data in HDFS also makes Hadoop more ‘mainstream’, as it can now be mounted as a standard filesystem. Native network encryption has been added to secure data in transit. The team behind Hadoop has put a lot of work into stabilizing Hadoop’s APIs. Finally, Hadoop 2.2 also adds support for running it on Microsoft Windows.



More Information
Apache Hadoop

Introducing Apache Hadoop YARN

Related Articles

Hadoop for Windows

Microsoft Open Sources Big Data REEF

Hadoop gets to 1.0

Hive on Hadoop for MongoDB


To be informed about new articles on I Programmer, install the I Programmer Toolbar, subscribe to the RSS feed, follow us on, Twitter, Facebook, Google+ or Linkedin,  or sign up for our weekly newsletter.


blog comments powered by Disqus



Chrome Makes More Gains Firefox Continues To Decline

While Internet Explorer is still the dominant browser on the desktop Chrome now has a full quarter slice of the pie. Firefox's share has now dropped to 12%.

JavaScript 20 Years Old Today

Brendan Eich reports that he thinks it was the week of May 6th to May 15th 1995 when he worked on Mocha the language which was to be renamed JavaScript.

More News


Last Updated ( Tuesday, 22 October 2013 )

RSS feed of news items only
I Programmer News
Copyright © 2015 i-programmer.info. All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.