BCG, Microsoft, MLflow detail 5 steps to choose AI models
Serge Bulaev
BCG, Microsoft, and MLflow suggest that choosing an AI model should follow five steps. First, teams may need to define needs and risks before looking at models. Next, they should filter out options that do not meet rules on data and security. Then, teams might benchmark models using standard tests to check performance. After that, it appears helpful to test models with a small part of real traffic to watch for problems. Finally, ongoing checks and controls should be set up to manage costs and keep the system safe and stable.
Selecting the right AI model requires balancing technical performance with business constraints. Drawing on frameworks from BCG, Microsoft, and MLflow, here are five essential steps with practical guidance for engineering and product teams.
How do we define requirements before evaluating any models?
Choosing an AI model involves a five-step process: first, define business requirements and risk tolerance. Next, screen candidates against governance and infrastructure constraints. Then, benchmark the shortlist on reproducible tasks, pilot the winner with real traffic, and finally, implement ongoing monitoring and cost controls in production.
Start with the business outcome, not the technology. Document what success looks like for your specific use case - whether that's response accuracy, latency thresholds, or cost per transaction. BCG's framework emphasizes matching models to your platform capabilities and cost envelope so AI can scale efficiently across the organization.
Key elements to capture:
- Task type (classification, summarization, code generation, conversational agents)
- Quality baseline - minimum acceptable performance before comparing options
- Latency requirements - real-time vs. batch processing needs
- Risk classification - high-stakes decisions demand stricter governance
Industry best practices recommend classifying workloads by volume, criticality, and domain specificity before shortlisting candidates.
What criteria should drive our model shortlist?
Screen for constraints before comparing performance. Enterprise teams increasingly use approved model lists with risk tiers and data residency requirements to narrow options quickly.
Evaluate across five dimensions:
| Dimension | Key Questions |
|---|---|
| Performance | Does it meet your quality baseline on task-specific metrics? |
| Infrastructure | Cloud, edge, or hybrid deployment? GPU requirements? |
| Commercial terms | Token pricing, fine-tuning costs, vendor lock-in risks |
| Sustainability | Energy consumption and carbon footprint |
| Governance | Model cards, audit trails, compliance certifications |
Microsoft's guidance stresses using model catalogs that align workload characteristics with cost factors rather than defaulting to the largest available model.
How do we benchmark models reproducibly?
Use benchmarks that ship with evaluation code and public task data. For current deployments, prioritize tools that provide raw artifacts, traces, and trajectories so teams can rerun scoring exactly or inspect failures.
Recommended benchmark tools by domain:
- Coding/software engineering: SWE-bench - real GitHub issues with executable tests across Python projects
- Agentic tool use: WildToolBench - releases datasets, evaluation scripts, and evaluated trajectories for reproducibility
- Web agents: WebArena and BrowserGym with standardized interactive environments
- Research automation: ResearchGym with runnable repositories and task-native graders
Document runs using standards like ReproEvalCard to specify minimal artifacts needed for reproduction.
What does a structured pilot look like before full rollout?
Deploy structured canaries to measure real user impact. MLflow's guidance recommends routing a small percentage of production traffic to new models, monitoring behavior, and expanding only if service health indicators remain stable.
Pilot structure:
- Shadow mode - run new model alongside production without affecting users
- Canary deployment - small percentage of traffic with automated rollback triggers
- Staged expansion - gradual increase with quality gates at each step
Critical metrics to track:
- Time to first token (TTFT) and time per output token (TPOT) for LLMs
- Prediction drift and input/output distribution changes
- Business KPIs tied to the original use case definition
This approach prevents broad rollout of models that perform well on benchmarks but fail in production contexts.
How do we optimize for cost and operational sustainability?
Quantization and distillation can materially reduce inference costs while preserving utility. Combine these techniques with infrastructure controls to maintain performance within budget constraints.
Operational cost controls:
| Tactic | Implementation |
|---|---|
| Right-sizing | Provision replicas using measured throughput with appropriate headroom |
| Caching | Avoid redundant inferences on repeated queries |
| Rate limiting and quotas | Prevent abuse and bound spend |
| Autoscaling triggers | Request rate, latency percentiles, queue depth, token throughput |
| Model routing | Direct traffic to smaller/cheaper models when quality requirements allow |
Monitoring essentials:
- Infrastructure health (CPU/GPU utilization, memory, restarts)
- Service performance (latency, throughput, error rates)
- Model behavior (drift detection, calibration, safety signals)
- Audit logging with model version tracking
Production ML guidance specifically recommends skew and drift detection with fine-tuned alert thresholds to catch degradation before users notice.
These five steps transform model selection from a technical preference into a data-driven, governed decision process aligned with business goals. The frameworks from BCG, Microsoft, and MLflow provide reusable structures, while tools like SWE-bench and WildToolBench ensure your evaluations hold up to scrutiny and reproduction.