Baidu AI Team Caught Cheating - Banned For A Year From ImageNet Competition
Written by Mike James   
Thursday, 04 June 2015

The race to build the best neural network seems to have descended into tactics that are not familiar in the academic world. Chinese search engine Baidu has built a world class team that delivers first class results - but now it has been caught cheating.

 

imagenettexctimagenet

ImageNet is the benchmark standard for testing convolutional neural networks and other image recognition techniques. It was success on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), that brought Geoff Hinton's approach to convolutional neural networks to everyone's attention and recently Google's GoogLeNet did even better. The competition to produce the best neural network was intense and in the latest breakthrough Baidu's team claimed to have done even better with an error rate of just 4.58%, beating Microsoft and Google's 4.9% and 4.8% - humans have an error rate of around 5%. 

Baidu made much of the small improvement. For example, it posted an article on its technology blog headlined “Baidu Achieves Top Results on Image Recognition Challenge.”

Baidu is serious about AI and hired Google's AI expert Andrew Ng to head the project and then built a very big super computer called Minwa. This is the machine that ran the huge neural network on the ImageNet data and which produced the result that gave Baidu the crown of the best neural network on the planet. 

However the organizers of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) noticed that:

During the period of 28 November 2014 to 13 May 2015, there were at least 30 accounts used by a team from Baidu to the test server at least 200 times, far exceeding the specified limit of two submissions per week.

This is much more than the permitted two submissions per week. 

The report continues:

To make this clear, by exploiting the ability to test many slightly different solutions on the test server it is possible to

1) select the best out of a set of very similar solutions based on test performance and achieve a small but potentially significant advantage and

2) choose methods for further research and development based directly on the test data instead of using only the training and validation data for such choices.

The Baidu team has been asked not to submit to the test server for the next 12 months. In response the company has said that it was a "mistake". 

What are we to make of this?

It could be that that the extra interaction with the test data wasn't taken advantage of - but the restriction on submission rate was imposed to minimise this possibility. By creating 30 accounts to increase the submission rate, it is clear that it wasn't accidental. 

 imagenetsq

More Information

ImageNet Communication

 

Related Articles

Baidu Hires Andrew Ng 

Neural Networks Beat Humans

Google's Neural Networks See Even Better

 

To be informed about new articles on I Programmer, install the I Programmer Toolbar, subscribe to the RSS feed, follow us on, Twitter, FacebookGoogle+ or Linkedin,  or sign up for our weekly newsletter.

 

Banner


Help Curiosity Avoid Sand On Mars
21/06/2020

NASA wants help to add a new on-board capability to its Mars Rover - that of visually distinguishing between different types of terrain. AI4Mars is a Zooniverse project to label images captured by Spi [ ... ]



Chrome OS Runs Windows Apps - What's An OS Anyway?
17/06/2020

Chrome OS started out as a good idea, not a new idea, but still a good one. Since then things have gotten a little out of hand and now it has been announced that you can run Windows apps under it. Why [ ... ]


More News

 

graphics

 



 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Thursday, 04 June 2015 )