Facebook Open Sources Two Technologies
Written by Kay Ewbank   
Friday, 02 September 2016

Facebook has released two technologies as open source at its @Scale conference in San Jose. The Zstandard compression algorithm and the MyRocks storage engine are both being made available for open source use. 

zstandardlogo

The Zstandard compression is a lossless compression technology that Facebook believes can replace the Deflate compression algorithm used by most existing software. This is the technique used by products such as Zip, gzip, and zlib, and it has remained a market leader because rival newer technologies have either offered better compression or faster compression, but rarely both.

Zstandard 1.0 is a new compression algorithm that combines techniques such as Finite State Entropy with fast performance that is optimized for modern CPUs. Writing about the release on the Facebook Code blog, Yann Collet and Chip Turner of Facebook compared Zstandard to zlib, lz4, and xz, saying:

"Each of these algorithms offers different trade-offs: lz4 aims for speed, xz aims for higher compression ratios, and zlib aims for a good balance of speed and size."

They then showed a comparison of Zstandard compared to those algorithms per lzbench, an in-memory benchmark, in which was substantially faster than xz and zlib, and compressed as well as zlib. The developers say it is also extremely scalable.

zstandard

 

Facebook also announced that MyRocks is being made available as open source. Facebook uses MySQL and the InnoDB storage engine for data management, but the developers at Facebook say that InnoDB has inefficiencies on space and write amplification when used with flash storage.

To alleviate this, Facebook developed RocksDB, an embeddable, persistent key-value store for fast storage that is better at using space than InnoDB. However, RocksDB has no support for replication and lacks a SQL layer, so the developers at Facebook have built MyRocks, a new open source project that integrates RocksDB as a new MySQL storage engine. This enables the use of RocksDB as backend storage while supporting all the features in MySQL.

myrocks

 

 The developers are still in the process of migrating to MyRocks on the user database (UDB) tier, but say that:

"after deploying MyRocks to this database tier in one of our data center regions, we were able to use 50 percent less storage for the same amount of data compared with compressed InnoDB. Ultimately this will allow us to use half as many UDB servers." 

 

Both announcements were made at Facebook’s @Scale conference in San Jose. This is aimed at engineers working on large-scale engineering projects in the hope of increasing the use of open-source technology.

zstandardlogo

 

More Information

Facebook Code Blog

Zstandard 1.0

MyRocks

Related Articles

Zip Tests Quantum Mechanics

RocksDB on Steroids

RocksDB - Facebook's Database Now Open Source 

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter,subscribe to the RSS feed and follow us on, Twitter, FacebookGoogle+ or Linkedin

 

Banner


Swift 6 Improves Linux Support
23/09/2024

Apple has released Swift 6, with improvements to support for writing concurrent code, specifically a new, optional language mode that analyzes your code at compile-time and diagnoses possible data rac [ ... ]



David Padua Recognized With HPC Award
27/09/2024

David A. Padua is the recipient of the 2024 ACM-IEEE CS Ken Kennedy Award which recognizes achievements in parallel and high performance computing (HPC). Padua is cited for innovative and us [ ... ]


More News

 

kotlin book

 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Friday, 02 September 2016 )