Yahoo Open Sources Vespa
Yahoo Open Sources Vespa
Written by Kay Ewbank   
Friday, 20 October 2017

The big data serving system that powers Yahoo and Flickr has been open sourced by Oath, the parent company to Yahoo.  

Most developers will be familiar with Hadoop, which Yahoo made open source in 2006. Vespa is another element in Yahoo's data management suite, the data processing and serving engine, and it too is being made available as open source on GitHub.

 

vespa

 

Vespa deals with the problem of serving results to end users where there's a need to make computations over data while a user is waiting, as is the case with applications that feature search, recommendation, and personalization. Vespa means you can create applications that compute responses to user requests, over large datasets, at real time and at internet scale.

Oath (the group of Verizon that owns Yahoo) uses Vespa across many different Oath brands – including Yahoo.com, Yahoo News, Yahoo Sports, Yahoo Finance, Yahoo Gemini and Flickr. The system handles billions of daily requests over billions of documents while responding to search queries, making recommendations, and providing personalized content and advertisements. It also serves over 3 billion native ad requests per day via Yahoo Gemini.

Vespa can be used to select content items using SQL-like queries and text search, and to organize the matches to generate data-driven pages. It can rank the matches using either hard-coded or machine-learned relevance models. The results are served with response times in the lows milliseconds, and it can write data in real-time at a rate of thousands of times per second per node.

Vespa distributes data and computation over many machines without any single master as a bottleneck. You can feed data and models of any size to the serving system, then make the final computations at request time. It can be run on premises or in the cloud, and there are both Docker images and rpm packages for Vespa.  You can read how to build and manage Vespa systems, and the Vespa blog is running a series of posts on how to make the most of Vespa.

 

vespa

More Information

Vespa Blog 

Vespa On GitHub 

Related Articles

Yahoo In Retrenchment Again

Yahoo Shuts Services

Yahoo Opens Up Flickr Deep Learning

Yahoo Releases Record Machine  

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on, Twitter, FacebookGoogle+ or Linkedin.

 

Banner


Hewlett-Packard Archive Destroyed
31/10/2017

A valuable archive of the history of Silicon Valley was among the victims of the Tubbs fire, the wildfire that has caused widespread destruction in Northern California, killing at least 23 people and  [ ... ]



Minix Inside!
11/11/2017

What is the most used operating system? You might say Windows and then you might remember all those servers and opt for Linux, but either way you would be wrong. And no, it isn't OSX or Android. It is [ ... ]


More News

 

 
 

 

blog comments powered by Disqus

Last Updated ( Friday, 20 October 2017 )
 
 

   
Banner
RSS feed of news items only
I Programmer News
Copyright © 2017 i-programmer.info. All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.