.NET For Apache Spark Updated
Written by Kay Ewbank   
Thursday, 05 November 2020

The .NET bindings for Spark have been updated. The new 1.0 version adds support for .NET apps targeting .NET Standard 2.0 or later, as well as support for Apache Spark DataFrame APIs.

Spark is the general-purpose cluster computing framework that has native support for distributed SQL and enables streaming, graph processing, and machine learning. The .NET bindings for Spark are written on the Spark interop layer, and provide high performance bindings to multiple languages. Until their introduction Spark was accessible by coding using Scala, Java, Python or R but not .NET.

sparklogo

The improvements to the 1.0 release of .NET for Apache Spark start with the support for .NET applications targeting .NET Standard 2.0. In practical terms, it also now lets you use the Spark DataFrame APIs, including the ability to write Spark SQL. You can also write Spark apps using .NET user-defined functions.

netforspark

The library now comes with an API extension framework to add support for additional Spark libraries. Currently this includes Linux foundation Delta Lake, Microsoft OSS Hyperspace, ML.NET, and support for Apache Spark’s MLLib functionality.

The developers have improved the performance when using Apache Arrow to move data between the Spark runtime and .NET UDFs, and have also improved the features available for this. Overall, the team says that apps using .NET for Apache Spark that are not using UDFs show the same speed as Scala and PySpark-based non-UDF Spark applications. If the applications include UDFs, the .apps are at least as fast as PySpark programs, often faster.

Future plans include supporting deployment options, including integration with CI/CD DevOps pipelines and publishing or submitting jobs directly from Visual Studio.

sparklogo

More Information

.NET For Apache Spark On GitHub

.NET For Apache Spark On Microsoft

Related Articles

Have Your Say On .NET For Spark

Visual Spark Studio IDE For Spark Apps

.NET Is One With .NET 5

Spark BI Gets Fine Grain Security

Spark Announcements

Microsoft Asks For Help On The Future Of .NET - Where Do We Start?

Microsoft Open Sources .NET?       

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


Pi Day 2021 - Give Me Pi!
14/03/2021

On this Pi Day I give you the search for Pi. No, not its numbers but what it means to computer science, which is a little different from what it means for mathematics. It's a matter of symbols, not nu [ ... ]



Java 16 Hits General Availability
18/03/2021

Java Development Kit 16 is now generally available.  Production-ready binaries under the GPL are available from Oracle; binaries from other vendors will follow shortly. Oracle has also released t [ ... ]


More News

square

 



 

Comments




or email your comment to: comments@i-programmer.info