Spark Performance Optimization Series: #1. Skew, by Himansu Sekhar, road to data engineering

In Spark cluster data is typically read in as 128 MB partitions which ensures even distribution of data. However, as the data is transformed (e.g. aggregated), it is possible to have significantly…

Stream Data from Kinesis to Databricks with Pyspark, by Himansu Sekhar, road to data engineering

Data engineering and intelligent computing : proceedings of IC3T 2016 978-981-10-3223-3, 9811032238, 978-981-10-3222-6

Apache Kafka With Spark Structured Streaming With Emma Liu, Nitin Saksena, Ram Dhakne, Current 2022

Spark Performance Tuning: Skewness Part 1, by Wasurat Soontronchai

PDF) Spark Performance Tuning

Stream Data from Kinesis to Databricks with Pyspark, by Himansu Sekhar, road to data engineering

Cranking the Voltage on Spark: Achieve Peak Performance with Optimization, by BlackRockEngineering

High Performance Spark, 2nd Edition [Book]

miro./v2/resize:fit:1400/1*QmlphAQ0u8_VB

Spark Performance Tuning: Skewness Part 1, by Wasurat Soontronchai

Azarudeen S on LinkedIn: #spark #apachespark #spark #optimization #interviewpreparation

Apache Spark Optimization Toolkit

Spark Tuning: Spark Resource Optimization

Performance optimization lessons from Spark+AI and Data+AI Summits on - articles about Apache Spark

Spark Performance Optimization Series: #1. Skew, by Himansu Sekhar, road to data engineering