Model Monitoring
- Data Drift: Changes in the distribution of input data.
- Concept Drift: Changes in the relationship between features and target variables.
- Monitoring System: Tracks changes and compares against the training set. Alerts are sent if performance issues are detected.
- Re-training: Regular re-training (e.g., daily, weekly) can address drift.
Amazon SageMaker Model Monitor
- Monitors models in production.
- Compares live data with training data and detects deviations.
- Integrates with Amazon CloudWatch to trigger alarms and re-train models when necessary.
MLOps (Machine Learning Operations)
- Automation: Automates tasks like model testing, deployment, and re-training.
- Version Control: Tracks configurations, data, and model changes.
- Benefits:
- Productivity: Speeds up workflows.
- Repeatability: Ensures a consistent process.
- Reliability: Increases quality and consistency.
- Auditability: Versioning all components for compliance and traceability.
- Improved Quality: Mitigates model bias and tracks data/model changes.
Amazon SageMaker Pipelines
- Orchestrates ML workflows for building, deploying, and monitoring models.
- Pipelines can be created with SageMaker SDK for Python or JSON.
- View pipelines in SageMaker Studio.
