SQLFire - Speeds and Scales
Written by Kay Ewbank   
Friday, 07 September 2012

VMware's in-memory distributed SQL database, vFabric SQLFire, can load 8 million rows in 88 seconds.

SQLFire is designed to work as an in-memory distributed SQL-based cache that can be used alongside a traditional database handling the disk-based side of the process. vFabric SQLFire has JDBC and ADO.NET interfaces that you use to query the data store using SQL.

The keys and indexes are stored in memory to provide high scalability, availability and performance. It can also use a RDBMS where data persistence is required, and there are plans to provide support for major databases including Oracle, MySQL, Sybase, DB2, SQL Server and postGres.

vfabric1

One concern of using data onto a memory based data system with large data sets is the potential time taken to actually load it. Pas Apicella of the vFabric Cloud Application Platform team has posted an interesting blog post about just how well SQLFire achieves this task, loading 8 million rows in 88 seconds. The team estimates that this means SQLFire would be able to load around 40GB of data in an hour. The data was in CSV format, and the team used a multi-threaded load approach.

The team used SQLFire 1.0.3, with two virtual machines each with 18G of memory. The virtual machines shared the same home directory as well as having separate disks for their own disk stores and log files. A separate disk store was used for persistence, and any overflow data would be removed from memory and not written to disk. The in-memory table asynchronously writes its changes to the disk store. This means it only has to do the memory insert prior to moving onto the next change. Only six threads were used, and Apicella says making the thread count too high can have an adverse effect.

Other than ensuring the process loads only for the client exists and preventing the second node from attempting to read the file as well, the rest of the setup looks remarkably straightforward. You can see the scripts and output  here.

 speed

More Information

vFabric SQLFire datasheet (pdf)

VMware vFabric blog

Related Articles

CORFU - Flash Fault Tolerant Distributed Storage

MemSQL - 80,000 queries per second

SQL Fiddle

 

blog comments powered by Disqus

 

To be informed about new articles on I Programmer, install the I Programmer Toolbar, subscribe to the RSS feed, follow us on, Twitter, Facebook, Google+ or Linkedin,  or sign up for our weekly newsletter.

 

Banner


Wire The Programmer To Prevent Buggy Code
23/07/2014

It's impossible to eliminate all coding errors, but it is certainly worth taking measures to reduce their frequency. A novel approach looks at using psycho-physiological measures.



Self Assessment For Programmers
21/07/2014

A useful proficiency matrix has been produced that could be used to track your own progress when learning to program or to advertise your skillset to prospective employers.


More News

Last Updated ( Friday, 07 September 2012 )
 
 

   
RSS feed of news items only
I Programmer News
Copyright © 2014 i-programmer.info. All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.