Leanpub Header

Skip to main content
Go to Leanpub.comLeanpub

Data Science Bootstrap

A practical guide to getting organized for your best data science work

Eric Ma

Learn about the best practices for getting organized on your data science projects. Tips, tricks, and all of the reasons why they are important, distilled into a short and easily readable guide.

The author is letting you choose the price you pay for this book!

Pick Your Price...
PDF
EPUB
WEB
About

About

About the Book

Hey there, data scientist! Have you found yourself struggling with your compute environment, such as package conflicts, this cryptic term called "environment variables", this weird concept called "containers", and technology terms that feel like an entire stack of things to learn? How about being lost amongst your multitude of projects, and not being able to get organized? This book, a linearized form of my freely available knowledge base, will give you a concise and practical guide to getting up-to-speed and organized, to help you do your best data science work.

The content in this book has been battle-tested since 2013 when I first switched over from bench science to computational science. You'll benefit from my distilled experience, where I learned all the hard lessons that come from not applying "best data science practices" to my projects, so you can avoid the same mistakes. You'll also benefit from the countless number of times I have guided newcomer colleagues and data practitioners to data science on getting their systems set up. You'll see the backing philosophies, the "whys" that explain why we ought to do certain things a certain way, as well as manifestations of those "whys".

End the struggle with getting your computer to do what you want. Gain full control over it instead. Come and learn how!

Share this book

Categories

Data ScienceDevOpsProgramming Cookbooks

Feedback

Email the author

Price

Pick Your Price...

Minimum price

$7.99

$14.99

You pay

$14.99

Author earns

$11.99
$

All prices are in US $. You can pay in US $ or in your local currency when you check out.

EU customers: prices exclude VAT, which is added during checkout.

...Or Buy With Credits!

Number of credits (Minimum 1)

1
The author will earn $12.00 from your purchase!
You can get credits monthly with a Reader Membership

Author

About the Author

Eric Ma

As Principal Data Scientist at Moderna Eric leads the Data Science and Artificial Intelligence (Research) team to accelerate science to the speed of thought. Prior to Moderna, he was at the Novartis Institutes for Biomedical Research conducting biomedical data science research with a focus on using Bayesian statistical methods in the service of discovering medicines for patients. Prior to Novartis, he was an Insight Health Data Fellow in the summer of 2017 and defended his doctoral thesis in the Department of Biological Engineering at MIT in the spring of 2017.

Eric is also an open-source software developer and has led the development of pyjanitor, a clean API for cleaning data in Python, and nxviz, a visualization package for NetworkX. He is also on the core developer team of NetworkX and PyMC. In addition, he gives back to the community through code contributions, blogging, teaching, and writing.

His personal life motto is found in the Gospel of Luke 12:48.

Contents

Table of Contents

The Data Science Bootstrap Notes

  1. Where I think you, the reader, are coming from
  2. Things you’ll learn
  3. Apply these ideas just-in-time
  4. Changes from the first edition
  5. Ways to support the project

Philosophies

  1. How these philosophies work together

You should know your computing stack

  1. See this philosophy in action

Automate and standardize everywhere possible

  1. See this philosophy in action

You should always know the source of truth

  1. See this philosophy in action

Categorize everything that you can

  1. See this philosophy in action

Putting it all together

  1. How you’ll see these philosophies in action

Setup your machine

  1. Why this is important
  2. What you’ll learn in this section

Configure your shell

  1. Install Starship
  2. Configure environment variables
  3. Create shell aliases
  4. Troubleshooting
  5. Quick Reference

Install and configure system-wide software

  1. Install package managers
  2. Install software
  3. Configure your PATH
  4. TL;DR: Quick installation commands
  5. Troubleshooting
  6. Quick Reference

Install and configure Git on your machine

  1. Why do we need Git
  2. How to install Git
  3. How to configure Git with basic information
  4. How to configure Git with fancy features
  5. Troubleshooting
  6. Productivity Tip: Shell Aliases
  7. Quick Reference

Install uv to manage and install Python-based command line tools

  1. Further reading

Install Homebrew on your Mac (fallback package manager)

  1. Why install Homebrew?
  2. When to use Homebrew
  3. How to install Homebrew
  4. Using Homebrew on Linux
  5. See also

Install and configure direnv for environment management

  1. Why we need direnv
  2. How to install direnv
  3. How to configure direnv
  4. Loading .env files automatically (direnv >= 2.31.0)
  5. Troubleshooting

Leverage dotfiles to get your machine configured quickly

  1. Why create a dotfiles repository
  2. How to structure a dotfiles repository
  3. Examples and resources

Configure VSCode for maximum productivity

  1. How do I access VSCode settings?
  2. What built-in settings have transformed my workflow?
  3. What extensions have actually improved my productivity?
  4. What keyboard shortcuts do I actually use?
  5. How do I handle project-specific settings?
  6. What about collaborative coding?
  7. Remember: start simple, grow gradually

Master your shell for data science productivity

  1. Why shell mastery matters for data scientists
  2. What you’ll learn in this section

Take full control of your shell environment variables

  1. Why control your environment variables
  2. How do I control my environment variables

Create shell command aliases for your commonly used commands

  1. Why create shell aliases
  2. How to create aliases
  3. Where to store these aliases
  4. Useful aliases to get started
  5. Git aliases cheat sheet
  6. Port management aliases
  7. Enhancing built-in commands with functions

Shell commands cheat sheet

  1. Basic navigation and file operations
  2. File permissions and ownership
  3. Process management
  4. Network and system info
  5. Archive and compression
  6. Git shortcuts
  7. Text processing
  8. Advanced patterns
  9. Time-saving tips

Shell-based text editors

  1. Why should I care about shell editors?
  2. What do I actually need to learn?
  3. My recommendation: start with nano
  4. What about vim and emacs?
  5. My philosophy on shell editors
  6. Getting started

Manage and configure your projects

  1. Follow the 1:1:1:1… rule
  2. When can we break this rule

Start with a sane repository structure

  1. How to structure a standard repository structure
  2. Automate the scaffolding of new projects

Use pixi for maximally ergonomic and reproducible environments

  1. The Cheat Sheet of pixi commands
  2. Long-term reproducibility through lock files
  3. Composable multi-environment projects

Structure your source code repository sanely

  1. Phase 1: Initial Exploration
  2. Phase 2: Emerging Patterns
  3. Phase 3: One-off Scripts
  4. Phase 4: Production Structure
  5. Leveraging AI Assistants
  6. Core Development Principles

Store your project documentation in your project repository

  1. Introduction to the Diataxis framework
  2. When to add documentation
  3. Code comments as documentation
  4. AI assistance in documentation
  5. Reference documentation
  6. Automating documentation with CI/CD

Use CI/CD to automate tasks

  1. Key Concepts of CI/CD
  2. Environment Considerations
  3. Configuration and Environment Variables
  4. Leveraging Pixi Environment
  5. Practical Examples with GitHub Actions

Use data catalogs to manage data

  1. What are data catalogs?
  2. Traditional data catalog examples
  3. Modern ML data storage with xarray and zarr
  4. What are the advantages of data catalogs?
  5. When should you use data catalogs?

Choose your data formats wisely

  1. The binary vs text format decision
  2. The hybrid approach: Binary source, text derivatives
  3. High-dimensional data: The xarray advantage
  4. Format-specific recommendations
  5. How to implement this in practice
  6. The bottom line

Take advantage of uv for one-off projects

  1. Using PEP723 for script dependencies
  2. Self-contained notebooks
  3. Benefits of this approach
  4. Real-world examples
  5. Conclusion

Configuration files guide

  1. Why do we even need all these config files?
  2. What are the core configuration files you should know about?
  3. What about documentation configuration?
  4. How do I handle environment variables?
  5. What’s the quick reference for which tools use which files?
  6. What are my best practices for configuration files?
  7. How do I get started with all this?

Set environment variables in a .env file

  1. Why configure environment variables per project
  2. How to configure environment variables for your project

Name things consistently

  1. What constitutes a “sane” name?

Skills for Effective Data Science

  1. Core Technical Skills
  2. Effective Ways of Working

How to write software tests

  1. Why should I bother writing tests?
  2. How do I actually write tests?
  3. What do real tests look like?
  4. What about testing data assumptions?
  5. Don’t forget to test error conditions
  6. How do I actually run these tests?
  7. What are some advanced patterns worth knowing?
  8. How do I organize my tests?
  9. How do I know if I’m testing enough?
  10. How does this fit into my development workflow?
  11. What mistakes should I avoid?
  12. How do I get started?

Refactor code

Working with AI tools

  1. The speed of thought
  2. The right kind of lazy
  3. Effective patterns for AI interaction
  4. Beyond code generation
  5. Moving forward

Collaborating on Data Science Projects

  1. The power of pair programming
  2. Leveraging AI in collaborative work
  3. The science in data science projects
  4. Effective work distribution
  5. Handling merge conflicts without the drama
  6. The art of managing unproductive patches

Use notebooks effectively

  1. Choose Marimo over Jupyter for reactivity
  2. Notebooks as prototyping tools, not production code
  3. Data access best practices
  4. Scratch pad vs. Report-style notebooks
  5. Refactor with the help of AI
  6. Publish notebooks and strip outputs before committing
  7. Jupyter hygiene (if you must use Jupyter)

Looking back, moving forward

  1. What we’ve covered together
  2. The journey ahead
  3. A personal note
  4. Where to go from here
  5. Final thoughts

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

Now, this is technically risky for us, since you'll have the book or course files either way. But we're so confident in our products and services, and in our authors and readers, that we're happy to offer a full money back guarantee for everything we sell.

You can only find out how good something is by trying it, and because of our 100% money back guarantee there's literally no risk to do so!

So, there's no reason not to click the Add to Cart button, is there?

See full terms...

Earn $8 on a $10 Purchase, and $16 on a $20 Purchase

We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

In fact, authors have earned over $14 million writing, publishing and selling on Leanpub.

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub