AI Cost Estimation (Arlene F. Minkiewicz)

© 2025 Unison. All Rights Reserved. www.unisonglobal.com Arlene F. Minkiewicz
AI Cost Estimation NESMA Conference, October 16, 2025

© 2025 Unison. All Rights Reserved. 2 ▪Software development projects
often have a significant AI component ▪Traditional software cost estimation methods fall short with AI Software development projects ▪This paper proposes a multi-phased framework intended to address these short falls ▪On-going research focused on a new cost models aligned with this framework will be presented Introduction

© 2025 Unison. All Rights Reserved. 3 ▪Traditional Cost Estimation
– challenges for AI projects ▪Key Factors Influencing Cost and Effort in AI Projects ▪Proposed Framework for Cost Estimation in AI Projects ▪Research Review - Estimation of AI Development Projects - Estimation Implications of Generative AI ▪Wrap Up Agenda

© 2025 Unison. All Rights Reserved. 5 ▪Cost Estimation is
an important aspect of program management for SW development projects - Good estimation is critical for project success ▪Software has become ubiquitous in almost every important system ▪Consider the F-35 Joint Strike Fighter military aircraft compared to the F-16 - 177x more computer code - 300x software development effort/costs - 90% of functionality is delivered by software ▪Software cost estimation relies on traditional size and complexity measures Traditional Software Cost Estimation

© 2025 Unison. All Rights Reserved. 6 ▪Cutting edge technology
▪Unpredictable nature of data quality and availability ▪Unpredictable nature of model training and tuning– iterative process ▪Tasks such as data acquisition, cleaning, labeling and preprocessing – depend on the nature of the data not the type of project ▪High variability from project to project ▪Specialized skill requirements – in addition to SME knowledge, skills such as Data Science, Machine Learning, Big Data, NLP, etc. Where traditional software estimation methods fail…

© 2025 Unison. All Rights Reserved. 7 Where traditional software
estimation methods fail… ▪Ongoing maintenance including monitoring, retraining and model drift ▪Lack of relevant historical data ▪Iterative development experimentation in AI projects ▪Addressing ethical and regulatory requirements ▪Complexity of testing and validation ▪Computational resource demands ▪Traditional size and complexity measures are only part of the equation

© 2025 Unison. All Rights Reserved. 8 ▪Data Preparation –
acquisition, cleaning and preparation ▪Algorithm Selection – choosing the right algorithms ▪Model Training – Resource requirements and time considerations ▪Deployment and Maintenance – challenges and ongoing costs in AI systems ▪Team expertise – role of skills and experience (high priced people) ▪Tool Selection and Maturity – effect of tools on development efficiency and cost ▪Application Platforms – considerations for platform specific costs ▪Integration Challenges – issues related to legacy systems and compatibility Key factors Influencing Cost and Effort in AI Projects

© 2025 Unison. All Rights Reserved. 9 Framework for Cost
Estimation in AI Projects

© 2025 Unison. All Rights Reserved. 10 ▪ 1. Project
Scope Definition ▪ 2. Phased Development Approach - Phase 1 – Planning - Phase 2 – Data Preparation - Phase 3 – Algorithm Selection and Development - Phase 4 – Model Training and Tuning - Phase 5 – Deployment - Phase 6 – Maintenance and Monitoring ▪ 3. Cost Factors and Estimating Techniques ▪ 4. Dynamic and Iterative Estimation Process ▪ 5. Team Expertise and Resource Allocation ▪ 6. Risk Management ▪ 7. Case Study Review and Validation Framework for Cost Estimation in AI Projects

© 2025 Unison. All Rights Reserved. 11 ▪Clear identification of
project objectives - what are the project goals and desired outcomes ▪Outline the specific features and functionalities required ▪Identify clearly (or as clearly as possible) the data requirements - Types of data required - Data Volume - Quality Requirements - Potential Sources (and viability of acquisition of these sources) - Unique data considerations Framework for Cost Estimation in AI Projects – Project Scope Definition

© 2025 Unison. All Rights Reserved. 12 ▪ Phase 1
– Planning - Conduct feasibility studies - Perform preliminary risk assessments - Identify stakeholders - Create communication plans and identify owner(s) - Develop the high-level project timeline (aware that this will likely change – work that into plan) ▪ Phase 2 – Data Preparation - Estimate cost for data collection, cleaning and preparation – consider types of data, unique data characteristics, potential data sources, obstacles to data collection - Consider data acquisition methods – internal or external, data types, data collection method(s), purchased data - Consider the associated costs with acquiring data (licensing, storage, etc.) Framework for Cost Estimation in AI Projects – Phased Development Approach

– Algorithm Selection and Development - Assess skills required and evaluate resource pool, - Evaluate potential algorithms based on the requirements of the project - Estimate time and resources for model development (consult SMEs and others involved in similar projects) - Consider the impacts of prototyping and testing ▪ Phase 4 – Model Training and Tuning - Assess skills required and evaluate resource pool - Assess model complexity - Assess computational resources needed for training - consider things like GPU usage, cloud services, hardware needs, etc. - Estimate time for parameter tuning and validation (consult SME’s and others involved in similar projects) Framework for Cost Estimation in AI Projects – Phased Development Approach

– Deployment - Consider the costs related to integrating the AI model into existing systems - Consider the on-going infrastructure costs (cloud services, on-premise servers, etc.) ▪ Cloud based services such as ASW SageMaker or Google Vertex XI offer scalable solutions but can be pricey ▪ Edge computing solutions may be more cost effective but come with initial hardware investments ▪ Phase 6 – Maintenance and Monitoring - Consider ongoing costs for model maintenance, updates, retuning, performance monitoring ▪ Data drift, adversarial attacks, evolving regulatory requirements, etc. - Identify metrics for success and tracking continuous improvements - Consider the on-going infrastructure costs (cloud services, on-premise servers, etc.) Framework for Cost Estimation in AI Projects – Phased Development Approach

© 2025 Unison. All Rights Reserved. 15 ▪ Traditional estimation
methods are good but need to be tuned for the unique nature of AI development - Expert Judgement – experts are still valuable, but their opinions should be metered by experience with AI projects - Analogous Estimation – a poor choice if there are no similar projects - Parametric Estimation ▪ Beware that these models are based on historical data and project parameters ▪ Size and Complexity are still important, but estimator should also consider things outside of their current model - Data issues (acquisition, cleansing, preprocessing, availability, quality) - Iterative nature of data and model tuning should be incorporated - Considerations comparable with estimating agile projects should rule ▪ Bottoms Up Estimation - break down of costs is still valuable, but awareness of the iterative nature should contribute to estimate uncertainty – especially when organizational history is limited Framework for Cost Estimation in AI Projects – Cost Factors and Estimation Techniques

© 2025 Unison. All Rights Reserved. 16 ▪ It is
not going to be easy for organizations starting out with AI ▪ Successful estimators will embrace techniques favored for estimating agile projects - Expect things to change - Embrace this change - Communicate the risks to the stakeholders - Flexibility is key to successful projects - Probably won’t get it right the first time - Establish success metrics and track to learn from each project ▪ Regularly update estimates based on ongoing findings Framework for Cost Estimation in AI Projects – Dynamic and Iterative Estimation Process

© 2025 Unison. All Rights Reserved. 17 ▪ The resource
skill set for AI projects are currently in high demand – incorporate salary differentials ▪ Assess the skills and experience of team members, factoring in the fact that training and or new hires may be necessary to meet project requirements ▪ Assess the maturity and capability of tools and platforms beings used ▪ Consider the maturity and processing power of existing hardware ▪ Be prepared to invest in resources necessary for project success – human, technology, equipment, etc. Framework for Cost Estimation in AI Projects – Team Expertise and Resource Allocation

© 2025 Unison. All Rights Reserved. 18 Research Review –
Estimation of AI Development Projects

© 2025 Unison. All Rights Reserved. 19 ▪ AI Lifecycle
Models and Estimation Frameworks ▪ Data Preparation and Annotation ▪ Algorithm Selection and Optimization ▪ Model Training and Tuning ▪ Data Acquisition (in early stages) ▪ Deployment and Maintenance (future topic) Research Topics in Progress

© 2025 Unison. All Rights Reserved. 20 ▪ Existing AI/ML
Lifecycle Frameworks : Alignment to Cost Modeling - CDAC AI Lifecycle (Center for Data Analytics and Cognition) – Comprehensive structure; covers design, develop, deploy but light on annotation - Generic ML Lifecycle – Broad and simple; lacks governance and data lifecycle detail - ISO/IEC 8183 Data Lifecycle for AI – strong for data costs but weak on model phases - ModelOps (IBS AI Ladder) – Emphasizes operationalization and governance; weak in data prep - Hybrid Lifecycle – Combines structure, data loops, and governance coverage - CRISP-ML(Q) – ML Specific process model that extends CRISP-DM with explicit quality-assurance and risk mitigation tasks across six phases with emphasis on quality gats and risk checklists. AI Lifecycle Models and Estimation Frameworks

© 2025 Unison. All Rights Reserved. 21 ▪ Potential Hybrid
AI Development Lifecycle – A Data-Centric Cost Framework - Phases (all phases encircled with a governance layer) ▪ Problem Definition ▪ Data Lifecycle ▪ Model Development ▪ Evaluation ▪ Deployment ▪ Monitoring - Key Features ▪ Explicit mapping of cost drivers to lifecycle phases ▪ Iterative loops for annotation, retraining, and drift correction ▪ Governance ensures ethics, audit, and compliance oversight AI Lifecycle Models and Estimation Frameworks

© 2025 Unison. All Rights Reserved. 22 ▪ Toward a
Cost-Aligned AI Lifecycle Model - A lifecycle model aligned with data reality should yield more accurate, explainable, and sustainable AI cost estimation observations ▪ No single lifecycle covers full AI cost spectrum ▪ Data and human-in-the-loop phases are underrepresented ▪ Governance and retraining costs accumulate over time. - Current thoughts ▪ Base structure on CDAC or Generic ML Lifecycle Model ▪ Overlay ISO Data Lifecycle AI for upstream visibility ▪ Embed Governance as cross-cutting layer AI Lifecycle Models and Estimation Frameworks

© 2025 Unison. All Rights Reserved. 23 ▪ This activity
covers the cost needed to make raw data usable for machine learning and AI model development ▪ Key Considerations - Data Acquisition/Ingestion - Data Cleaning and Transformation - Data Integration - Feature Engineering - Data Storage and Management - Data Quality Assurance - Compliance and Governance Data Preparation Potential Cost Factors Data Volume Number of Data S ources Compute time/cycle Pipeline Runtime Complexity of Data Formats Number of Features S torage Volume S torage Costs Automated Tool Usage Tool Licenses

covers the cost of enriching raw data (images, text, audio, video, etc.) to teach models patterns and behaviors necessary for supervised learning ▪ Key Considerations - Data Collection and Preparation - Task Design and Guideline Development - Tool Configuration - Infrastructure Setup - Annotator Training and Onboarding - Manual or Semi-automated Annotation - Quality Assurance and Review - Label Validation - Version Control and Metadata Tracking Data Annotation and Labeling Potential Cost Factors Data Complexity Data Volume Annotation Type Annotator E xpertise Annotation Guidelines and Quality Control Tooling and Infrastructure Degree of Automation E thical and Legal Considerations

© 2025 Unison. All Rights Reserved. 25 ▪ This activities
covers the cost of: - Understanding the problem type, data characteristics and performance goals - Balancing tradeoffs between model complexity and performance ▪ Algorithm types include supervised, unsupervised or deep learning ▪ Key Considerations - Define the problem - Assess the data - Define constraints - Hyperparameter tuning - Quantization - Model Evaluation Tools and Metrics Algorithm Selection and Optimization Potential Cost Factors Computational Complexity Model Complexity Pre-trained or from scratch Data Requirements Overfitting Risk Feature E ngineering Hyperparameter Tuning Techinques Hyperparameter Tuning Requirements E xplainability and Fairness Requirements

covers the costs of training and AI or Machine Learning (ML) model ▪ Key Considerations - Collect and Prepare the Data - Label the Data - Choose a Model - Train the Model - Tune and Validate - Test and Evaluate - Deploy the Model - Monitor and Maintain Model Training and Tooling Potential Cost Factors Data Acquisition/Licensing Costs Cloud S torage Data Volume Annotation Platform S oftware Tools Compute Time E nergy Costs Development E ffort Infrastructure S etup

© 2025 Unison. All Rights Reserved. 27 ▪Technology and data
availability creates a situation where Artificial Intelligences applications are practical and valuable ▪Organizations intent on developing such apps need to revisit the way they estimate these projects ▪A framework has been presented to help estimators consider these unique factor as appropriate to better aid decision makers in their organizations. ▪Ongoing research intended to turn the framework and what is learned by using the framework, into a working cost estimation model for AI Development Wrap Up

© 2025 Unison. All Rights Reserved. 30 AI Lifecycle Models
and Estimation Frameworks DCAC AI LifeCycle Desing->Develop->Deploy (17 sub stages) Very granular breakdown; explicitly includes early stages (problem faming, data acquisition, benchmarking, evaluation, monitoring); good match for cost breakdown not enough emphasis on annotation.labeling cost explicitly; doesn't always emphasize iterative feedback loops or operational drift cost Good baseline to adapt or extend Generic AI/ML Lifecycle (Data S cience/MLOps style) Problem definition-> Data collection/preparation -> Model selection/training -> Validation/evaluation-> Deployment -> Monitoring and Maintenance S imple, well understood, widely used; align cleanly with our cost buckets Could be too high level - may gloss over annotation, human validation, governance costs or retraining drift Good starter model, likely need to expand sub phases AI Model Lifecycle/ModelOps / IBM's AI Ladder Collect -> Organize -> Analyze -> Infuse E mphasizes the operational side (governance, monitoring, deployment) which is critical for realistic cost modeling over time Less fine detail on annotation, human in the loop, labeling or upstrem data acquisition decisions Usefle for bounding runway costs post deployment IS O/Data Lifecycle S tandard (IS O/IE C 8183, Data Lifecycle for AI) Data ideation-> acquisition -> processing -> usage -> archival/decommisioning (applied in AI concept) S tandard focused on data so potential for rigorous accounting for data costs Does not cover algorithmic model costs or training/tuning costs Could act as a supplement to AI workflow selected to boost the data cost coverage Participatory/Co-production Lifecycle (augmented AI Lifecycle) Co-framing->Co-design->co- implementation->co-deployment->co- maintenance bringsin human/stakeholder/governance cost explicitly;good to expose non technical cost (coordination, governance) More qualitative, less engineering/metric oriented, could add a degree of complication to cost modeling usefless as a lens for capturing hidden overhead costs Hybrid or custom cost-aligned AI lifecycle S ome combination of the above tailored to the cost buckets we decide on Best aligned to the model we decide to build Not a standard, industry acceptance will need to be earned best to create a hybrid that is anchored to one or more of the industry accepted frameworks CRIS P-ML(Q) Business and Data Understanding->Data Preparation->Model Building and Tuning- >E valuation->Deployment -> Monitoring and Maintenance Quality gates (validation, documentation, risk controls) become costablework packages, bult in monitoring and maintenance phase support costing around ongoing drift/retraining annotation and lablering isnt callout at as a phase, less prescirptive about tooling/ops

© 2025 Unison. All Rights Reserved. 33 ▪ Project Goal
- Create a more versatile and efficient AI system by unifying OpenAI technologies to surpass GPT-4 through improvements to contextual understanding and creative generation ▪ Challenges - Availability of high-quality training data. OpenAI has had to employ synthetic data generation and human-created datasets - Extremely expensive training runs, with iterations approaching $500 million apiece - Disappointing performance improvements ▪ Lessons Learned - Limitations of scaling – improved performance requires more than increasing model size and data - Data quality and availability is essential to successful AI advancements - Groundbreaking improvements require innovative approaches Real World Example - OpenAI’s GPT-5 (Orion) Project

- Revolutionize healthcare by using AI for analyzing medical data, assisting doctors in making more accurate diagnoses and providing tailored treatment recommendations ▪ Challenges - Project faced significant cost issues due to the complexity integrating AI into clinical environments - Need for constant data updates - Healthcare data is messy – often inconsistent or incomplete – training is costly and time consuming - High operational and maintenance costs - Data interoperability and integration technical limitations ▪ Lessons Learned - AI projects in highly regulated fields like healthcare face substantial costs for data handling, compliance and customization to clinical workflows Real World Example - IBM Watson Health

- Tesla’s FSD software development project aimed to deliver a fully autonomous driving experience powered by advanced ML algorithms ▪ Challenges - Developing and refining autonomous driving technology – substantial costs - Heavy investment in massive computational resources to run and train AI models - Expensive sensor technology - Global test fleets to ensure safety - Software needed continuous updates and vast quantities of real-world driving data - Regulatory delays and liability issues ▪ Lessons learned - Autonomous AI requires substantial up front investment in hardware and data - High costs for maintenance, model updates and regulatory compliance Real World Example - Tesla’s Autopilot and Full Self-Driving (FSD)

- Create a fully autonomous electric vehicle that would revolutionize the automotive industry offering deep integration with Apple’s existing products and services. ▪ Challenges - Creating a vehicle with no steering wheels or pedals required unprecedented technological advancements - Lack of coherent and consistent strategy led to repeated redesigns and shifts in focus resulting in resource misallocation and escalating costs - Lack of manufacturing partner and automotive expertise resulted in difficulties with battery technology, self-driving systems and overall vehicle design. ▪ Lessons Learned - Clear vision, well-defined strategic vision and consistent leadership are crucial for success of large-scale projects - Entering a complex industry requires deep industry knowledge and experience, not just innovation - Recognizing when to pivot is vital, though the project was cancelled in 2024, Apple was able to redirect its focus toward generative AI turning to areas with greater synergy to its core competency Real World Example - Apples Project Titan

- Develop a single aircraft platform adaptable for various missions across multiple branches of the military incorporating cutting-edge stealth , sensor fusion and network centric warfare technologies depending heavily on AI technologies for situational awareness, autonomy, mission adaptability and pilot decision making. Pivotal to project success is the development of the Autonomic Logistics Information System (ALIS) ▪ Challenges - Integration challenges combining diverse data types (radar, EW, IR, comms, etc.) from multiple sources - Reliability of AI models – validating AI decisions against traditional military rules of engagement - Hardware constraints – onboard AI within limited computational and power budgets - Calibrating trust between pilot and AI system was nontrivial ▪ Lessons Learned - Human – Centric AI Design is crucial - Iterative development and testing is best strategy - AI is a force multiplier – AI does not replace human judgement - Model validation and robustness is a long-term investment - Real World Example - Lockheed Martin’s F-35 Lightening II Program

AI Cost Estimation (Arlene F. Minkiewicz)

AI Cost Estimation (Arlene F. Minkiewicz)

More Decks by Nesma

Featured

Transcript