Leanpub Header

Skip to main content
Go to Leanpub.comLeanpub

Data Analysis and Visualisation in Climate Science

A Programmer's Guide

Sebastian Engelstaedter

The interest of many young scientists in climate research is often cut short by the sometimes stressful experience of having to learn how to code. This book covers everything a student or young researcher needs to know covering climate data types and file formats, remote server login, the Unix command line, number crunching with CDO and Python Xarray/NumPy as well as how to create high-quality publishable plots.

The author is letting you choose the price you pay for this book!

Pick Your Price...
PDF
EPUB
WEB
180
Readers
482
Pages
92,104Words
About

About

About the Book

Master the tools well so you can focus on the science

The latest version is a substantial update of the book including 132 extra pages with many more code examples.

v1.0 includes the following updates:

  • code base update to Python 3.11
  • extensive new chapter on Xarray
  • plotting code examples updated
  • streamline plot examples added
  • introduction to pathlib library
  • introduction to Micromamba
  • broken cross-references and URLs fixed

v0.3 includes the following updates:

  • code updated to Python 3.8
  • code example for trend line plot using linear regression
  • section added on glob.glob()
  • introduction to f-strings and associated code updates
  • minor code updates
  • hi-resolution Cartopy map features
  • fixed skew-t plot monotonicity issue
  • improved solutions for creating sequences of dates
  • some more NumPy solutions
  • fixed some cross-references

v0.2 includes the following updates:

  • Hovmoeller plots updated and additional example added
  • Unnecessary np.meshgrid() commands removed from code examples
  • Improved code for contour plot time axis formatting
  • Working with Dates/Times section added (datetime module)
  • Wind vector plot section added including examples
  • Skew-T plot section added including examples
  • Section on multiple line plots updated
  • Plotting variables added to Table 8.1.2.1
  • ax.quiver() command added to Table 8.1.2.1
  • ax.subplots() commands updated (nrows and ncols)
  • Typos fixed
  • Cross-references and table/code/figure numbering fixed

Share this book

Categories

PythonData ScienceEarth and Ocean Sciences

Feedback

Contact the author

Price

Pick Your Price...

Minimum price

$7.99

$9.99

You pay

$9.99

Author earns

$7.99
$

All prices are in US $. You can pay in US $ or in your local currency when you check out.

EU customers: prices exclude VAT, which is added during checkout.

...Or Buy With Credits!

Number of credits (Minimum 1)

1
The author will earn $12.00 from your purchase!
You can get credits monthly with a Reader Membership

Author

About the Author

Sebastian Engelstaedter

I am a scientist and data analyst working at the Climate Research Lab in the School of Geography and the Environment at the University of Oxford, UK. More details about my work can be found here: https://www.geog.ox.ac.uk/staff/sengelstaedter.html

Contents

Table of Contents

Preface

  1. Acknowledgements

1Introduction

  1. 1.1Overview and Objective
  2. 1.2Concept of Local and Remote Machines
  3. 1.3Software

2Climate Data

  1. 2.1Climate Data Overview
  2. 2.2Data Use Licences
  3. 2.3Data Quality
  4. 2.4Accessing Climate Data
  5. 2.5Types of Climate Data
  6. 2.5.1Analyses and Reanalyses Products
  7. 2.5.2Climate and NWP Model Output
  8. 2.5.3Point observations
  9. 2.6Data File Formats
  10. 2.6.1Plain Text and ASCII
  11. 2.6.2Binary
  12. 2.6.3GRIB
  13. 2.6.4netCDF
  14. 2.6.5PP

3Unix

  1. 3.1Introduction to Unix
  2. 3.1.1Linux Distributions
  3. 3.1.2Desktop versus Server
  4. 3.1.3High Performance Computing on a Server
  5. 3.2Accessing a Remote Server
  6. 3.2.1Remote Server Login Details
  7. 3.2.2Virtual Private Network (VPN)
  8. 3.2.3X Window System (X11 forwarding)
  9. 3.2.4Connecting to a Remote Server
  10. 3.3First Steps on the Unix server
  11. 3.3.1The Terminal Window
  12. 3.3.2The Shell
  13. 3.3.3Linux Directory Structure and Home Directory
  14. 3.3.4Quota
  15. 3.3.5File Transfer to and from the Server
  16. 3.3.6Mapping the Linux Home Directory as a Remote Network Drive
  17. 3.4Some More Unix Server Basics
  18. 3.4.1Unix Command Syntax}
  19. 3.4.2Manual Pages
  20. 3.4.3Editing Text Files
  21. 3.4.4Full versus Relative Paths
  22. 3.4.5Special Characters
  23. 3.5Working with Files and Directories
  24. 3.5.1Creating Text Files and Directories
  25. 3.5.2Listing Files and Directories
  26. 3.5.3Moving Around in the Directory Tree
  27. 3.5.4Copying, Moving, Renaming and Deleting Files and Directories
  28. 3.6Advanced Unix Commands
  29. 3.6.1Examining Text Files
  30. 3.6.2File and Directory Properties
  31. 3.6.3File Permissions
  32. 3.6.4Changing File Permissions and Ownership
  33. 3.6.5Changing the Unix Account Password
  34. 3.6.6Redirecting Command Output
  35. 3.6.7Finding Files
  36. 3.6.8File Compression and Archives
  37. 3.6.9Download Files from the Command Line
  38. 3.7Long-running Jobs
  39. 3.7.1GNU Screen (recommended)

4Multi-dimensional Gridded Datasets

  1. 4.1The Earth’s Coordinate System and Realms
  2. 4.2The Model Grid
  3. 4.3Grid Indexing and Geographical Referencing of Data Points
  4. 4.4The Time Dimension
  5. 4.5Horizontal Resolutions and Grid Types
  6. 4.5.1Spectral Resolution
  7. 4.5.2Full and Reduced Gaussian Grid
  8. 4.5.3Regular latitude-longitude Grid
  9. 4.6Vertical Level Types
  10. 4.6.1Pressure, Potential Temperature and Potential Vorticity Levels
  11. 4.6.2Sigma (Model) Levels
  12. 4.6.3Sigma-Hybrid Levels

5The netCDF File Format

  1. 5.1Introduction to the netCDF File Format
  2. 5.2netCDF File Headers
  3. 5.2.1Exploring netCDF File Headers with ncdump
  4. 5.2.2Exploring netCDF File Headers with CDO
  5. 5.2.3Exploring netCDF File Headers with ncview
  6. 5.3Packed netCDF Files
  7. 5.4netCDF File Format Conventions

6Python - Concepts and Work Environment

  1. 6.1Python Overview
  2. 6.2Python Concepts
  3. 6.2.1Python Modules and Packages
  4. 6.2.2Package Dependencies
  5. 6.2.3Package Managers, Repositories and Channels
  6. 6.2.4Virtual Environments for Python
  7. 6.2.5Conda, Mamba or Micromamba?
  8. 6.2.6Micromamba
  9. 6.2.7Conda
  10. 6.3Python Code Development Solutions
  11. 6.3.1Python Code Editors
  12. 6.3.2Python IDEs
  13. 6.3.3Browser-based Python Code Editing
  14. 6.3.4The IPython Command Line

7Python - Programming Basics

  1. 7.1Basic Python Programming Building Blocks
  2. 7.1.1Declaring Variables
  3. 7.1.2Variable Types and Conversion Between them
  4. 7.1.3Functions
  5. 7.1.4Methods and Attributes
  6. 7.1.5Controlling the Code Flow
  7. 7.2Applying Python in Climate Data Analysis
  8. 7.2.1Error Messages when Running Code
  9. 7.2.2Looping Through Input Files
  10. 7.2.3Reading Data Files Into NumPy Variables
  11. 7.2.4Executing Unix System Commands from Within Python
  12. 7.3A Brief Introduction to Numpy
  13. 7.3.1Creating Numpy Arrays
  14. 7.3.2Indexing NumPy Arrays
  15. 7.3.3Saving and Loading NumPy Variables
  16. 7.3.4Some NumPy Solutions
  17. 7.4Working with Dates and Times
  18. 7.4.1Creating Date Objects
  19. 7.4.2Converting Datetime Objects to Strings (strftime)
  20. 7.4.3Creating a NumPy Array with Datetime Objects
  21. 7.5Tips and Solutions
  22. 7.5.1Managing Paths and Filenames Using pathlib
  23. 7.5.2String Formatting of Numbers
  24. 7.5.3Zero-padding Integer Values in Filenames
  25. 7.5.4Calculate Height from Geopotential with MetPy

8Python - Data Analysis with Xarray

  1. 8.1What is Xarray?
  2. 8.2Xarray Basics
  3. 8.2.1Xarray Terminology
  4. 8.2.2Import Xarray into Python Script
  5. 8.3Reading in netCDF Files Using Xarray
  6. 8.3.1Reading in a Single netCDF File
  7. 8.3.2Reading in Multiple netCDF Files
  8. 8.3.3Reading in Very Large (memory-intensive) netCDF Files
  9. 8.3.4Additional Considerations for Reading in netCDF Files
  10. 8.4DataSets vs DataArrays
  11. 8.5Exploring File Content
  12. 8.5.1Print DataSet
  13. 8.5.2Print DataArray
  14. 8.5.3Accessing DataArray Elements
  15. 8.6Dates and Times in Xarray
  16. 8.6.1Retrieving Date/Time Information Using Datetime Accessors
  17. 8.6.2Passing Date/Time Information to DataArray Methods
  18. 8.7Selections
  19. 8.7.1Selecting Variables
  20. 8.7.2sel() vs isel()
  21. 8.7.3Selecting Spatial Subsets (Geographical Regions)
  22. 8.7.4Selecting Vertical Levels
  23. 8.7.5Selecting Timesteps
  24. 8.8Resampling - Downsampling the Time Dimension
  25. 8.8.1Resampling to Daily Statistics
  26. 8.8.2Resampling to Monthly Statistics
  27. 8.8.3Resampling to Yearly Statistics
  28. 8.9Aggregation - Calculating Statistics Over Dimensions
  29. 8.9.1Aggregation Over the Time Domain
  30. 8.9.2Aggregation Over the Spatial Domain
  31. 8.9.3Aggregation Over the Vertical Domain
  32. 8.9.4Aggregation Over the Zonal Domain
  33. 8.9.5Aggregation Over the Meridional Domain
  34. 8.9.6Aggregation for Hovmöller Plots
  35. 8.10The Split-Apply-Combine Concept
  36. 8.10.1Group by Hour
  37. 8.10.2Group by Month
  38. 8.10.3Group by Season
  39. 8.11Interpolation
  40. 8.11.1Simple Interpolation Example
  41. 8.11.2Spatial Interpolation
  42. 8.11.3Interpolation Between two Geographical Points
  43. 8.12Xarray Computations
  44. 8.12.1Direct Application of Arithmetic Operation
  45. 8.12.2NumPy Universial Functions - ufunc()
  46. 8.12.3Xarray Universal Functions - apply_ufunc()
  47. 8.13Xarray MetPy Integration
  48. 8.14More Xarray Methods and Attributes to Explore
  49. 8.14.1Some Useful Xarray Methods
  50. 8.14.2Some Useful Xarray Attributes
  51. 8.15Xarray Plotting
  52. 8.16Applying Xarray in Climate Computations
  53. 8.16.1The 1997 Indian Ocean Dipole Event
  54. 8.16.2Rainfall Variability in the Sahel
  55. 8.16.3Winds at In Salah (Algeria)

9Python - Creating Plots

  1. 9.1Matplotlib
  2. 9.1.1Setting up a Plotting Page (Figure and Axes)
  3. 9.1.2Main Plotting Commands
  4. 9.1.3Colour Names and Colour Maps
  5. 9.2Line Plots
  6. 9.2.1Line Plot with Labels
  7. 9.2.2Line Plot with Arrows
  8. 9.2.3Multiple Lines Plot with Markers and Legend
  9. 9.2.4Multiple Lines Plot with two Scales
  10. 9.2.5Multiple Lines Plot with Standard Deviation
  11. 9.3Scatter Plots
  12. 9.3.1Scatter Plot with a Legend
  13. 9.3.2Scatter Plot with Divergent Colour Bar
  14. 9.3.3Scatter Plot on a Map with Colour Bar and Legend
  15. 9.3.4Adding Trend Line Based on Linear Regression
  16. 9.4Map Plots
  17. 9.4.1Cartopy Map Projections
  18. 9.4.2Cartopy Data Transformations
  19. 9.4.3High-Resolution Map Features
  20. 9.4.4Simple Map of SST Anomalies
  21. 9.4.5Map with Stipples for Statistical Significance
  22. 9.5Bar Graphs
  23. 9.5.1Anomalies Bar Graph
  24. 9.6Hovmöller Plots
  25. 9.6.1Hovmöller Plot with Time as a Function of Latitude
  26. 9.6.2Hovmöller Plot with Time as a Function of Pressure
  27. 9.7Vertical Cross-Section Plots
  28. 9.7.1Meridional Cross-Section
  29. 9.7.2Vertical Cross-Section Between two Points
  30. 9.8Skew-T Plots
  31. 9.8.1Brief Introduction to Skew-T Plots
  32. 9.8.2Simple Skew-T Plot
  33. 9.8.3Multiple Skew-T Plots
  34. 9.9Vector and Streamline Plots
  35. 9.9.1Black Wind Vectors on Filled Colour Contours
  36. 9.9.2Coloured Wind Vectors
  37. 9.9.3Streamline Plot
  38. 9.10Looping Through Multiple Panels
  39. 9.10.1Multiple Line Plots (axes.flat method)
  40. 9.10.2Multiple Line Plots (pop() function)
  41. 9.10.3Multiple Map Plots (axes.flat method)

10Data Analysis with CDO

  1. 10.1What is CDO?
  2. 10.2Useful CDO Resources
  3. 10.3Basic Syntax of CDO Commands
  4. 10.3.1CDO Options
  5. 10.3.2CDO Operator Categories
  6. 10.3.3Using Multiple CDO Operators
  7. 10.3.4CDO Operator Parameters
  8. 10.3.5CDO Command Input and Output Files
  9. 10.4Merging Files
  10. 10.5Selections
  11. 10.5.1Selecting Variables
  12. 10.5.2Selecting Spatial Subsets (Geographical Regions)
  13. 10.5.3Selecting Vertical Levels
  14. 10.5.4Selecting Time Subsets
  15. 10.6Basic Statistics
  16. 10.6.1Statistics over the Time Domain
  17. 10.6.2Statistics over the Spatial Domain
  18. 10.6.3Statistics over the Vertical Domain
  19. 10.6.4Statistics over the Zonal Domain
  20. 10.6.5Statistics over the Meridional Domain
  21. 10.6.6Statistics over Ensembles
  22. 10.7Interpolations
  23. 10.7.1Interpolation to a new horizontal grid (remapping)
  24. 10.7.2Interpolation in the Vertical Domain
  25. 10.7.3Interpolation in the Time Domain
  26. 10.8Basic Arithmetic
  27. 10.8.1Artithmetic Between Two Files
  28. 10.8.2Arithmetic Using a Constant Value
  29. 10.9Applying CDO in Climate Computations
  30. 10.9.1Indian Ocean Dipole Example
  31. 10.9.2Sahel Rainfall Variability Example
  32. 10.9.3Creating a Land-Sea Mask File
  33. 10.10Using CDO with Python

Appendix

  1. List of Acronyms

Get the free sample chapters

Click the buttons to get the free sample in PDF or EPUB, or read the sample online here

Download Sample PDFDownload Sample EPUB

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

Now, this is technically risky for us, since you'll have the book or course files either way. But we're so confident in our products and services, and in our authors and readers, that we're happy to offer a full money back guarantee for everything we sell.

You can only find out how good something is by trying it, and because of our 100% money back guarantee there's literally no risk to do so!

So, there's no reason not to click the Add to Cart button, is there?

See full terms...

Earn $8 on a $10 Purchase, and $16 on a $20 Purchase

We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

In fact, authors have earned over $14 million writing, publishing and selling on Leanpub.

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub