Naiad - Differential Dataflows
Written by Kay Ewbank   
Monday, 12 November 2012

Microsoft Reasearch is developing a new data-parallel computing platform, and there’s a new video that makes it much more understandable.

Naiad is a programming environment that takes a new approach to running programs on big data sets. The aim is to find faster ways to carry out analysis of big data. Naiad is described as an incremental, iterative, and interactive data-parallel computing platform that is being developed at Microsoft Research Silicon Valley.

The project came about because of experience among the researchers on Dryad and on DryadLINQ. Dryad and DryadLINQ are two Microsoft Research projects that can be used for large data processing in C# on clustered computers. Dryad runs sequential programs in parallel, with the parallel computation organized as a directed acyclic graph with the programs you’re running forming the graph’s vertices while the edges are the communication channels between the programs. DryadLINQ is a compiler that translates LINQ programs to distributed computations that can then be run on clustered PCs.

The experience with this pair of projects showed that many big-data algorithms contain loops, and these loops are often data-dependent, iterating until the answer doesn’t change any more. However, once the answer starts to stabilize, much of the work done in each iteration is redundant with work done in previous iterations, since much of the data are the same. If the later iterations could work on the changed data rather than the entire set, it ought to be possible to achieve significant speed increases. The result of this is Naiad.

The way Naiad works means it extends standard batch data-parallel processing models like MapReduce, Hadoop, and Dryad/DryadLINQ. With Naiad you can have efficient incremental updates to the inputs so Naiad acts as though it’s a stream processing system. Alongside this, Naiad lets you make use of nested fixed-point iteration.

The Naiad tech report "Composable Incremental and Iterative Data-Parallel Computation with Naiad" shows Naiad being used for a range of problems such as graph algorithms, but I’d recommend watching the video first. Frank McSherry introduces Naiad and discusses why it might be important, with some demos of Naiad in action.

 

 

 

The Big Data at SVC blog also has some interesting posts showing programs written in Naiad.

 

More Information

Naiad Project Page

Download Naid

Naiad tech report: Composable Incremental and Iterative Data-Parallel Computation with Naiad

Big Data at SVC blog

Related Articles

Google Supersonic Released As Open Source

New Udacity Classes On Games, Graphics, Parallel & Chips

Cloud Numerics

Halide - New Language For Image Processing

The $10 Million Computer-Assisted Programming Project

 

blog comments powered by Disqus

 

To be informed about new articles on I Programmer, install the I Programmer Toolbar, subscribe to the RSS feed, follow us on, Twitter, Facebook, Google+ or Linkedin,  or sign up for our weekly newsletter.

 

Banner


Telerik Kendo Goes Open Source
22/04/2014

Telerik has released an open source version of Kendo UI. The release consists of an open source package of UI and JavaScript tools that also includes Telerik Kendo UI Mobile.



Microsoft Releases WinJS 2.1 As Open Source
03/04/2014

WinJS is Microsoft's JavaScript library for developing WinRT or Windows Store apps. One of the lesser announcements at this year's Build is that it is being released as open source.


More News

 

Last Updated ( Monday, 12 November 2012 )
 
 

   
RSS feed of news items only
I Programmer News
Copyright © 2014 i-programmer.info. All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.