Spark 3 Improves Python and SQL Support
Written by Kay Ewbank   
Monday, 22 June 2020

Spark 3 has been released with major improvements in Python and SQL support, along with changes to make it easier to explore data. It is also faster.

Spark is the general-purpose cluster computing framework that has native support for distributed SQL and enables streaming, graph processing, and machine learning.

sparklogo

Spark SQL underlies many of the actions carried out by developers using Spark, and nearly half the work in the updated release has been to the SQL engine, improving both performance and ANSI compatibility.  The SQL engine has a number of new features, including an Adaptive Query Execution (AQE) framework that improves performance and simplifies tuning by generating a better execution plan at runtime. SQL higher-level libraries have also been improved, including structured streaming and MLlib, and higher level APIs, including SQL and DataFrames.

Python is now the most widely used language on Spark and has received a lot of attention in this release, especially pandas and Koalas. The pandas API is limited to single-node processing, and work has continued on Koalas, an implementation of the pandas API on top of Apache Spark, to make data scientists more productive when working with big data in distributed environments. Koalas eliminates the need to build many functions in PySpark, to make performance across clusters more efficient.  The Koalas API coverage for pandas is now close to 80%.

Work has also continued on the PySpark APIS. There are new pandas APIs with type hints. The new pandas UDF interface uses Python type hints to make it easier to understand when more UDF types are added.  Error handling has ben improved, simplifying PySpark exceptions, hiding the unnecessary JVM stack trace, and making them more Pythonic.

Other improvements include a new UI for structured streaming, and faster calling (up to 40 times faster) for R user-defined functions.

 

sparklogo

 

More Information

Spark 3 Release Notes 

Related Articles

Have Your Say On .NET For Spark  

Visual Spark Studio IDE For Spark Apps

.NET Is One With .NET 5

Spark BI Gets Fine Grain Security

Spark Announcements

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


Nobel Prize For Chemistry For AlphaFold
11/10/2024

The Royal Swedish Academy of Sciences has awarded a half share of the 2024 Nobel Prize For Chemistry to Demis Hassabis, CEO of Google DeepMind and his colleague John Jumper for "protein stru [ ... ]



Apache Updates Wicket
03/10/2024

Apache Wicket has been updated to version 10.2, following the major release of Wicket 10 earlier this year. The open source Java web framework is now built on top of Java 17, and has a new module test [ ... ]


More News

kotlin book

 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Monday, 22 June 2020 )