.NET For Apache Spark Updated
Written by Kay Ewbank   
Thursday, 05 November 2020

The .NET bindings for Spark have been updated. The new 1.0 version adds support for .NET apps targeting .NET Standard 2.0 or later, as well as support for Apache Spark DataFrame APIs.

Spark is the general-purpose cluster computing framework that has native support for distributed SQL and enables streaming, graph processing, and machine learning. The .NET bindings for Spark are written on the Spark interop layer, and provide high performance bindings to multiple languages. Until their introduction Spark was accessible by coding using Scala, Java, Python or R but not .NET.

sparklogo

The improvements to the 1.0 release of .NET for Apache Spark start with the support for .NET applications targeting .NET Standard 2.0. In practical terms, it also now lets you use the Spark DataFrame APIs, including the ability to write Spark SQL. You can also write Spark apps using .NET user-defined functions.

netforspark

The library now comes with an API extension framework to add support for additional Spark libraries. Currently this includes Linux foundation Delta Lake, Microsoft OSS Hyperspace, ML.NET, and support for Apache Spark’s MLLib functionality.

The developers have improved the performance when using Apache Arrow to move data between the Spark runtime and .NET UDFs, and have also improved the features available for this. Overall, the team says that apps using .NET for Apache Spark that are not using UDFs show the same speed as Scala and PySpark-based non-UDF Spark applications. If the applications include UDFs, the .apps are at least as fast as PySpark programs, often faster.

Future plans include supporting deployment options, including integration with CI/CD DevOps pipelines and publishing or submitting jobs directly from Visual Studio.

sparklogo

More Information

.NET For Apache Spark On GitHub

.NET For Apache Spark On Microsoft

Related Articles

Have Your Say On .NET For Spark

Visual Spark Studio IDE For Spark Apps

.NET Is One With .NET 5

Spark BI Gets Fine Grain Security

Spark Announcements

Microsoft Asks For Help On The Future Of .NET - Where Do We Start?

Microsoft Open Sources .NET?       

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


Office Development Kit For VS Code
18/09/2024

Microsoft has released a preview version of a new development tool for developing Office add-ins in Visual Studio Code.



Pulumi Envisions The Intelligent Cloud
18/09/2024

Today at PulumiUp, its 4th annual conference, Pulumi is announcing "Intelligent Cloud", its vision for AI-powered cloud infrastructure, and two new products, ESC GA and Insights 2.0.  


More News

kotlin book

 

Comments




or email your comment to: comments@i-programmer.info