In MLOps, it’s crucial to use various services and metrics to manage models and workflows effectively. Here’s a breakdown of key services and evaluation metrics.
MLOps Services
1. AWS CodeCommit: Repository for storing source code (like GitHub).
2. SageMaker Feature Store: Stores feature definitions for training data.
3. SageMaker Model Registry: Centralized repository for storing and tracking models.
4. SageMaker Pipelines: Orchestrates ML pipelines.
5. AWS Step Functions: Build serverless workflows with a visual interface.
6. Apache Airflow: Tool for creating and monitoring workflows; available as Amazon Managed Workflows for Apache Airflow.
Model Evaluation Metrics
1. Confusion Matrix:
- Compares actual vs. predicted results.
- Tracks True Positives (TP), True Negatives (TN), False Positives (FP), False Negatives (FN).
2. Accuracy:
- Percentage of correct predictions.
- Formula:
(TP + TN) / Total Predictions.
3. Precision:
- Correct positive predictions out of all positives identified.
- Formula:
TP / (TP + FP). - Used to minimize false positives.
4. Recall (Sensitivity):
- Correct positive predictions out of all actual positives.
- Formula:
TP / (TP + FN). - Used to minimize false negatives.
5. F1 Score:
Best for optimizing both precision and recall.
Balances Precision and Recall.
