Leanpub Header

Skip to main content
Go to Leanpub.comLeanpub

Testing Spark Applications

Writing Spark code is hard... well designed, performant Spark tests are even harder. You need a robust test suite to identify performance bottlenecks in your code and refactor with ease. This book teaches you how to write a beautiful test suite and how to run the tests whenever code is pushed to the master branch.

Matthew Powers

This book teaches you how to test Apache Spark codebases. Tests encourage your to write well designed code, help you identify bottlenecks in your code, make refactoring easier, and prevent some production deploy errors. You'll need to master testing Spark code to be a great Spark programmer.

The author is letting you choose the price you pay for this book!

Pick Your Price...
PDF
EPUB
WEB
158
Readers
92
Pages
12,049Words
About

About

About the Book

The book discusses Scala testing basics with the Scalatest framework. It uses the spark-fast-tests library to demonstrate column equality testing and DataFrame equality testing. Spark tests can run slowly so the book provides several practical workflows to keep tests running quickly. Spark code frequently reads and writes to disk and the book covers how to write tests for code with I/O. Configuring a test suite properly can make it around 70% faster and this book explains the configuration options you should have on your radar. Complex transformations (e.g. aggregations) and column types (e.g. MapType, ArrayType, StructType, BinaryType) have special testing considerations that are addressed in separate chapters.

This book has a heavy emphasis on software engineering best practices and will teach you skills that are useful for any language or framework.

Share this book

Categories

Scala

Feedback

Email the author

Price

Pick Your Price...

Minimum price

$29.99

$39.99

You pay

$39.99

Author earns

$31.99
$

All prices are in US $. You can pay in US $ or in your local currency when you check out.

EU customers: prices exclude VAT, which is added during checkout.

...Or Buy With Credits!

Number of credits (Minimum 2)

2
The author will earn $24.00 from your purchase!
You can get credits monthly with a Reader Membership

Author

About the Author

Matthew Powers

Contents

Table of Contents

Introduction

  1. Messy data
  2. Nightmare deploys
  3. Empower refactoring
  4. Tests encourage code that doesn’t have side effects
  5. Identifying code bottlenecks
  6. Test suites document behavior
  7. Technologies used

Testing Scala with Scalatest

  1. Writing a simple test
  2. Directory organization
  3. build.sbt
  4. More tests
  5. Running tests and configuring output
  6. assertThrows
  7. assertDoesNotCompile
  8. Other assertions
  9. Other test formats
  10. Test library alternatives
  11. Testing Spark applications
  12. Next steps

Column Equality Tests

  1. Custom DataFrame Transformation Refresher
  2. Spark project setup
  3. assertColumnEquality with spark-fast-tests
  4. Conclusion

Quieting Test Output

  1. Customizing test suite output

Creating DataFrames for Tests

  1. toDF
  2. createDataFrame
  3. createDF
  4. Including spark-daria in your projects
  5. Next steps

DataFrame Equality Tests

  1. Simple example
  2. assertSmallDataFrameEquality error messages
  3. Next steps

Running Tests

  1. Running from the SBT console
  2. Running a single test file
  3. Running a single test
  4. Best workflow

Approximate Equality

  1. Difference between double, float and decimal
  2. When assertColumnEquality falls short
  3. assertFloatTypeColumnEquality to the rescue
  4. assertApproximateDataFrameEquality
  5. Conclusion

Testing User Defined Functions

  1. Creating a UDF
  2. Testing a UDF
  3. Check the UDF fails with null input
  4. The billion dollar mistake
  5. Verifying test failure in the test suite
  6. Next steps

Testing Spark Column Functions

  1. Simple example
  2. How Spark functions handle null
  3. Important takeaway
  4. Why print DataFrames from the test suite?
  5. Next steps

Testing Filesystem Reads

  1. Untestable code
  2. Setting the path as a param
  3. Testing with the config pattern
  4. Elegant testing with dependency injection
  5. Abstracting custom transformation to a separate function
  6. Next steps

Testing Filesystem Writes

  1. Simple example
  2. Rude tests leave garbage behind
  3. Performance considerations
  4. Next steps

Identifying Bottlenecks

  1. Let’s find the bottleneck
  2. Benchmarking individual transformations
  3. Contrived but representative
  4. Conclusion

Organizing Tests

  1. Some pure Scala
  2. Poorly organized Spark tests
  3. Test suite rules to follow
  4. Good Spark test organization
  5. Tests should be descriptive and document behavior
  6. Quantifying performance difference
  7. Next steps

Test Suite Configuration

  1. Shuffle partitions
  2. javaOptions
  3. Conclusion

Testing Aggregations

  1. groupBy refresher
  2. groupBy with two columns
  3. groupBy with filters
  4. Conclusions

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

Now, this is technically risky for us, since you'll have the book or course files either way. But we're so confident in our products and services, and in our authors and readers, that we're happy to offer a full money back guarantee for everything we sell.

You can only find out how good something is by trying it, and because of our 100% money back guarantee there's literally no risk to do so!

So, there's no reason not to click the Add to Cart button, is there?

See full terms...

Earn $8 on a $10 Purchase, and $16 on a $20 Purchase

We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

In fact, authors have earned over $14 million writing, publishing and selling on Leanpub.

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub