1. Session 1 - How To Start (Almost) Any Project
   • What makes production machine learning different from what you've learned.
   • The strategy to solve the right problem using the right solution.
   • Critical questions to ask before starting any project.
   • Problem framing, inversion, and the haystack principle for building successful applications.
   • The first rule of machine learning engineering and how to start building.
   • Data collection strategies. A technique to determine how much data you need.
   • The problem of selection bias and how to deal with it.
   • Labeling data. Human annotations, natural labels and weak supervision.
   • Active learning using the uncertainty and diversity sampling strategies.
2. Session 2 - How to Build a Model
   • The role of data cleaning and feature engineering to build better models.
   • Turning data into numbers using vectorization techniques.
   • Producing homogeneous features using normalization and standardization.
   • Handling and interpreting missing values using imputation techniques.
   • The approach to choosing the best model to solve any problem.
   • Random baselines and the zero-rule algorithm.
   • How to use overfitting to build models that don't suck.
   • Hyperparameter tuning and experiment tracking.
   • Measuring the quality of your holdout set.
   • An introduction to distributed training using data parallelism and model parallelism.
3. Session 3 - Offline Evaluation
   • The role of a baseline to contextualize the evaluation process.
   • Framing evaluation metrics in the context of business performance.
   • Evaluating models using holdout sets and cross-validation.
   • Introduction to data leakages and leaky validation strategies.
   • Invariance tests and model fairness.
   • The problems with metric summarization.
   • Testing models on specific slices of data.
   • Error analysis and measuring the impact of potential improvements.
   • Introduction to backtesting and time-based evaluation strategies.
   • Dealing with disproportionally important examples and rare cases.
4. Session 4 - Serving Predictions
   • Introduction to model versioning and the model registry.
   • The trade-offs when serving predictions. An introduction to latency and throughput.
   • Dynamic serving and static serving predictions.
   • Two-phase predictions and deploying models on the edge.
   • Enforced modularization and inference pipelines.
   • Evaluating individual pipeline models.
   • Introduction to human-in-the-loop workflows.
   • Cost-sensitive deployment architectures.
   • Using test-time augmentations to improve predictions.
   • Model compression using pruning, quantization, and knowledge distillation.
5. Session 5 - Drift and Monitoring
   • Dealing with catastrophic predictions and the problem with edge cases.
   • The problem with feedback loops and how to fix them.
   • Data distribution shifts. Understanding concept drift and data drift.
   • Using adversarial validation to detect distribution shifts.
   • An introduction to monitoring model behavior in production systems.
   • Monitoring model inputs, operational metrics, predictions, and user feedback.
   • How to keep your models working in the face of distribution shifts.
6. Session 6 - Continual Learning
   • Introduction to continual learning.
   • Techniques to determine how frequently to retrain your models.
   • Techniques to determine what data to use to retrain a model.
   • Common triggers to initiate a retraining process.
   • Understanding stateless training and stateful training.
   • Catastrophic forgetting and how to prevent it.
   • The importance of testing in production.
   • Testing models using A/B testing, shadow deployments, canary releases, and interleaving experiments.