Baidu AI Team Caught Cheating - Banned For A Year From ImageNet Competition
Written by Mike James   
Thursday, 04 June 2015

The race to build the best neural network seems to have descended into tactics that are not familiar in the academic world. Chinese search engine Baidu has built a world class team that delivers first class results - but now it has been caught cheating.

 

imagenettexctimagenet

ImageNet is the benchmark standard for testing convolutional neural networks and other image recognition techniques. It was success on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), that brought Geoff Hinton's approach to convolutional neural networks to everyone's attention and recently Google's GoogLeNet did even better. The competition to produce the best neural network was intense and in the latest breakthrough Baidu's team claimed to have done even better with an error rate of just 4.58%, beating Microsoft and Google's 4.9% and 4.8% - humans have an error rate of around 5%. 

Baidu made much of the small improvement. For example, it posted an article on its technology blog headlined “Baidu Achieves Top Results on Image Recognition Challenge.”

Baidu is serious about AI and hired Google's AI expert Andrew Ng to head the project and then built a very big super computer called Minwa. This is the machine that ran the huge neural network on the ImageNet data and which produced the result that gave Baidu the crown of the best neural network on the planet. 

However the organizers of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) noticed that:

During the period of 28 November 2014 to 13 May 2015, there were at least 30 accounts used by a team from Baidu to the test server at least 200 times, far exceeding the specified limit of two submissions per week.

This is much more than the permitted two submissions per week. 

The report continues:

To make this clear, by exploiting the ability to test many slightly different solutions on the test server it is possible to

1) select the best out of a set of very similar solutions based on test performance and achieve a small but potentially significant advantage and

2) choose methods for further research and development based directly on the test data instead of using only the training and validation data for such choices.

The Baidu team has been asked not to submit to the test server for the next 12 months. In response the company has said that it was a "mistake". 

What are we to make of this?

It could be that that the extra interaction with the test data wasn't taken advantage of - but the restriction on submission rate was imposed to minimise this possibility. By creating 30 accounts to increase the submission rate, it is clear that it wasn't accidental. 

 imagenetsq

More Information

ImageNet Communication

 

Related Articles

Baidu Hires Andrew Ng 

Neural Networks Beat Humans

Google's Neural Networks See Even Better

 

To be informed about new articles on I Programmer, install the I Programmer Toolbar, subscribe to the RSS feed, follow us on, Twitter, FacebookGoogle+ or Linkedin,  or sign up for our weekly newsletter.

 

Banner


IFTTT GMail Options Removed
28/03/2019

Most of the integration of Gmail features in IFTTT are being removed on March 31. IFTTT apps will still be able to send an email, but will no longer be able to trigger actions based on emails received [ ... ]



Kotlin Enters RedMonk's Top 20
28/03/2019

RedMonk has published the latest of its twice-yearly language rankings. For the January 2019, or Q1 2019, the Top 10 remains virtually unchanged from Q3 2017 and shows only minor variation from Q3 201 [ ... ]


More News

 

Python

 



 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Thursday, 04 June 2015 )