Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Connecting Theory and Practice V

Connecting Theory and Practice V

Avatar for MIKIO KUBO

MIKIO KUBO

July 10, 2025
Tweet

More Decks by MIKIO KUBO

Other Decks in Business

Transcript

  1. Agenda How to proceed with optimization projects and their lifecycle

    Pre-processing for optimization Handling infeasibility Why is application to practice difficult? Fusion of Machine Learning and Mathematical Optimization 2
  2. "Real-world" "NP-Hard" Problems Real-world... ✓ Real-world optimization problems are diverse.

    ✓ Various additional conditions are required in practice. ✓ Often cannot be reduced to typical optimization problems as is. (If reducible, refer to "100+ Optimization Problems" for easy solution!) NP-Hard... ✓ Most real-world problems are NP-Hard. ✓ Efficient algorithms (probably) do not exist. ✓ Solving them generically can take a very long time depending on the problem instance. 4
  3. 100+ Optimization Problems https://scmopt.github.io/opt100/ Three volumes from Asakura Publishing, 3

    minutes read 100+ Optimization Problems Solvable in Practice with Python ▪ Introduces formulations and Python solutions for over 100 optimization problems useful in practice. Table of Contents Author's Page Guidelines Notes Maxims Operating Environment Install the following with Poetry. (Also use commercial solvers Gurobi, OptSeq, SCOP) 100+ Optimization Problems 5
  4. Approaches to Real-world NP-Hard Problems Give up on solving quickly

    (polynomially on average) Give up on finding optimal (exact) solutions (approximate solutions) => Trade-off between computation speed and solution quality Give up on solving all problem instances Research on special types that are easy to solve (X) => Design models and solution methods targeting typical problem instances 6
  5. Characteristics of Real-world Problem Instances Different from randomly generated benchmarks

    ✓ Customers and products often follow Zipf's law. ✓ Customer distribution is biased (e.g., concentrated in Tokyo in Japan). Different from intentionally difficult problem instances (benchmarks for research) There are parts that can be handled by heuristics (designed to be so) => Fixed by pre-processing Interviewing alone is insufficient (supplement with data) Data alone is insufficient (supplement with interviewing) 7
  6. Pitfalls of Programming Education "Desirable" programs taught in class (Diligent

    people) Model assuming arbitrary problem instances (X) Cannot solve all NP-hard instances! (Lazy people) Model assuming a few problem instances (X) Will fail if unexpected problem instances are input! 8
  7. Extracting Typical Real-world Problem Instances and Models Model design assuming

    arbitrary input data (X) Assume problem instances that may occur in practice (Not too diligent & not lazy => Art!) ✓ Collect as many real-world problem instances as possible ✓ Data analysis and visualization ✓ Interviewing ✓ Feature extraction from problem instances ✓ Pre-processing and modeling according to the characteristics of the problem instances ✓ Testing with slightly larger random data than actual problem instances 9
  8. Solving with Interviewing + Experts? Customer's explanation | Interview |

    Expert | Programmer | Sales promise Documentation | Operation | Billing Amount | Support | Real Requirement C. Alexander "The Oregon Experiment" 10
  9. Interviewing Pitfalls (Common Scenarios) People on site don't tell the

    truth in meeting rooms (in front of suits) (but will talk over drinks). Rare exceptions (bragging about how they handled it). Always adding features because they're afraid of losing their job. Minute specifications and numerical values (bragging about how difficult their job is). Conditions that cannot be expressed numerically (e.g., if possible, preferably, occasionally...) Objective function that increases costs (e.g., taking a detour for fairness?) 11
  10. Interviewing and Modeling "Diligently" modeling all interviewed conditions (X) Based

    on interviews and data, extract "truly necessary" conditions, and consider whether to obfuscate them with pre-processing, reduce them to standard problems, or properly include them in the model. Example: Delivery Planning ✓ Interviewing: Work time varies by store depending on the time of day. ✓ Ask about the number of stores, work time differences, and types of time zones (+ confirm with data). ✓ Visualize the distribution of stores and the relationship between time zones and work time. ✓ If few time zone types (day and night) & few stores => solve with two types of time slots (day and night). ✓ If large differences in work time => set time slots to one type with shorter work time (night). ✓ If many stores and various time zones => reduce to delivery optimization with time-dependent travel times. 12
  11. Optimization Project Goal Setting (Scope, Granularity, Objective) ✓ SPRINT Metrics

    Setting ✓ Response Time ✓ Evaluation Metrics Required Resources ✓ People (Skills) ✓ Budget Timeline ✓ Milestones ✓ Goals 13
  12. Data Processing • Data Collection • Data Extraction • Pre-processing

    ✓ Missing Values ✓ Outliers • Data Analysis • Simple Metrics Calculation • Data Visualization • Dashboard • Pre-processing for optimization ✓ Data Aggregation ✓ Rounding numbers 14
  13. Model Creation • Similar Research Investigation • Constraint Extraction •

    Objective Function Extraction (Solver Selection) • Solution Method Development • Hypothetical (Preliminary) Experiment • Small-scale Experiment • Global Optimization • What If Analysis • Sensitivity Analysis • Trade-off Analysis • Priority Analysis 15
  14. Deploy & Evaluation • UI Design • UI Creation •

    Deploy • Test • Evaluation • Monitoring ✓ Cloud Performance ✓ Input Data ✓ Output Metrics 16
  15. Optimization Project Flow Real-world Problem • Interviewing • Site Visit

    • Internship Model (or Algorithm) Construction • Research on similar problems • Find similar problems from 100+ optimization problems • Constru 17
  16. Test • Reproduce current state (baseline) • Small-scale problem instance

    test • Medium-scale problem instance test • Full-scale test • What If analysis • Sensitivity analysis • Robustness test Data • Data Collection • Data Pre-processing • Data Visualization 18
  17. (Meta) Heuristics and Mathematical Optimization (Meta) Heuristics Pros • Inexpensive

    • Can be fast • Effective for specific problems • No need for formalization • Can incorporate complex conditions • Can design using problem structure Cons • Not exact solutions • Cannot handle continuous variables • No model • Simple procedures (processing IT) • Solution method design takes time 19
  18. Mathematical Optimization Pros • Exact solutions (mostly) • Has a

    model (formalization) • Easy to maintain • Additional conditions (if formalizable) are easy to incorporate • Can be introduced quickly Cons • Can be expensive or slow • May take time to compute • May not be able to incorporate some additional conditions? • Computation time may increase if data changes • If data changes, solution quality may deteriorate (already mentioned in meta-heuristics cons, but more pronounced here in terms of feasibility/optimality) 20
  19. How to Handle Complex Additional Constraints with MIP MIP/CP Approach

    Constraints difficult to describe in MIP Solution Constraint-dependent cutting plane (no good cut) Constraint Violation Oracle: Checks constraints (with CP) and adds constraints to eliminate violations. Branch-and-Price (Column Generation) Method Generate feasible patterns and optimize selection. Main Problem (Linear Relaxation) Optimal Dual Variables Subproblem: Generate multiple complex constraint patterns (beam search or dynamic programming). Column (Pattern) Generation Minimize Pattern Cost s.t. Global Constraints (e.g., carry cargo) 21
  20. Learning from Failure Cases (1) Goal Setting ✓ Believed marketing

    that the solver became 1 trillion times faster, told the client it could solve anything instantly, but it failed and broke down. ✓ Told the client it could solve instantly, but in reality it was impossible and broke down. ✓ Was told delivery would be via a maximum of 2 hubs, but it became 4 hubs just before the deadline and broke down. Data Processing ✓ Failed to collect transportation costs and handling costs, leading to breakdown. ✓ Insisted on using real data with its original precision, causing the solver to break down. ✓ Fed raw, extremely large-scale data directly, leading to breakdown. 22
  21. Learning from Failure Cases (2) Modeling ✓ Optimized resource constraints

    to the very limit, but real-world data was different, leading to breakdown. ✓ Tried to solve with the only available method in-house, but ran out of time and broke down. ✓ Kept adding conditions as requested by the client, eventually leading to breakdown. ✓ Believed interviews too easily and added too many detailed conditions, leading to breakdown. ✓ Said "university's main work got busy (or the assigned student stopped coming)" and quit midway. Deployment & Evaluation ✓ Pulled data one by one from the database with SQL, ran out of time. ✓ Problem scale gradually increased in real operation, making it unsolvable and leading to breakdown. ✓ Sales slipped their tongue saying labor costs would be halved, causing a rebellion on site and breakdown. 23
  22. Project Management for Optimization System development for optimization (machine learning)

    differs from typical software development project management. ✓ Goals are ambiguous (what to achieve and to what extent is determined by trial and error). ✓ Feasibility cannot be guaranteed (whether it can be solved is unknown until tried!). ✓ Not merely process automation (processing IT), but creating intelligence (analytical IT). ✓ Time taken varies greatly depending on the developer's skill (6 months => 3 days). ✓ Skill measurement is difficult (certification exams are different from real problems). Traditional PMs lack experience and knowledge. ✓ Cannot judge problem difficulty. ✓ Tend to decide specifications at the beginning. ✓ Estimate costs based on hourly wages or lines of code. Need for PMs who understand the optimization (machine learning) process. 24
  23. Summary Solving complex real-world optimization problems is not easy. Giving

    up only on computation time and optimality is not enough. It is also necessary to give up on solving arbitrary problem instances. Extract typical problem instances based on real-world data and model them. A part of the data is simplified by pre-processing (for optimization) (Art!). Optimization project lifecycle. Precautions during real-world operation. 25
  24. Data Pre-processing Pre-processing for Machine Learning ✓ Recently automated (type

    conversion, handling missing values and outliers, separation of training/validation/test sets, etc.). ✓ Feature extraction is an art (however, deep learning can automate feature extraction). Pre-processing for Optimization ✓ Automation is difficult. ✓ Discussions with clients (+ interviewing on site) are crucial. ✓ Must be considered simultaneously with the construction of the optimization model (+ solution method) (cannot be separated). ✓ Art! 27
  25. Data Collection 1. Sign an NDA (Non-Disclosure Agreement). 2. Obtain

    multiple (as many as possible) actual real-world problem instances. 3. If possible, obtain time-series data for a suitable period (to determine if it needs to be treated as a dynamic problem). 4. If data cannot be provided => It's pointless to continue, so terminate. 5. If data is insufficient, return to step 2. 28
  26. Data Pre-processing 1. Examine the distribution of each data item

    and remove anomalies (identify the cause of anomalies such as input data units; requires discussion with client). 2. Handle missing data (how to handle it? requires discussion with client). 3. Visualize relationships between data items (scatter plots, Gantt charts). 4. Plot location information on a map (calculate latitude and longitude if necessary). Basically the same as pre-processing for machine learning or data science. 29
  27. Pre-processing for Optimization Consider simplifying the problem through pre-processing (requires

    discussion with client; especially interviewing on site). ✓ Example: Cluster analysis of locations to aggregate data. ✓ Example: ABC analysis of products to aggregate data. ✓ Example: Standardization of sizes based on product size and capacity (rounding up). ✓ Example: Fixing partial solutions that most optimal solutions have (utilizing machine learning). Bin (Capacity 10) Items (Size) 30
  28. Summary Differences between pre-processing for machine learning/data science and pre-processing

    for optimization model creation. Data collection, pre-processing, pre-processing for optimization. Pre-processing for optimization models and solution construction (Art). 31
  29. Optimization Model as a Function (Basic Framework) Diagram: Instance ->

    Optimization Model -> (Optimal) Solution. Labels for each step describing its meaning (e.g., "Problem with numerical values input", "Maximize/Minimize Objective Function", "Solution satisfying all constraints"). Problem Instance -> Optimization Model -> (Optimal) Solution Problem with numerical values input Maximize (Minimize) Objective Function subject to Constraints Optimal feasible solution with max (min) objective function value (= Solution satisfying all constraints) 33
  30. Optimization Model as a Function (Theoretician's Framework) Problem Instance ->

    Optimization Model -> (Optimal) Solution OR Infeasible OR Unbounded sup (inf) Objective Function subject to Constraints No solution exists that satisfies all constraints = Infeasible Objective function becomes infinite (small) = Unbounded 34
  31. Optimization Model as a Function (Practitioner's Hope) Problem Instance ->

    Optimization Model -> Usable Solution Maximize Objective Functions (many) subject to Constraints (No need to satisfy all) Solution considering various metrics (Allows some deviation from constraints) 35
  32. Modeling with Weighted Satisfiability Problem Problem Instance -> Optimization Model

    -> (Optimal) Solution Minimize Sum of weighted penalties for constraint violations subject to Hard Constraints (must be satisfied absolutely) Soft Constraints (penalty if violated) How are violation weights determined? 36
  33. Example of Determining Violation Weights: Shift Optimization Determine staff shifts

    and tasks within the planning period to meet the required number of personnel for each task per time slot "completely". Constraints ✓ Required personnel conditions ✓ Staff preferred shifts (day, time slot) ✓ Task change penalty ✓ Special tasks (breaks) placed in appropriate time slots ✓ Constraints on consecutive shifts (prohibition of morning shift the day after night shift) ✓ ... Approach: Constraint Optimization (Weighted Constraint Satisfaction Problem) ✓ Allows violation of all constraints. ✓ Violation penalties for constraints are determined by the user (e.g., small for task changes, large for preferred shifts) or by priority. 37
  34. Simplified Method for Priority-based Multi-objective Optimization Multi-objective Optimization Problem Diagram

    showing three objective functions (Priority 1, 2, 3) and constraints, with a flow demonstrating how higher priority objectives are optimized first, becoming constraints for lower priority ones. Minimize Priority 1 Objective Function Priority 2 Objective Function Priority 3 Objective Function Constraints Flow: Priority 1 Objective Function Constraints -> Priority 1 Objective Function z* -> Priority 2 Objective Function -> Constraints 38
  35. Example: Delivery Optimization Want to deliver all "all" cargo. Available

    vehicles are fixed. ✓ Customer time window constraints ✓ Vehicle capacity constraints ✓ Customer-vehicle compatibility (skill conditions) ✓ ... Approach 1: Allow constraint violation (penalty for violation) Approach 2: Relax "deliver all cargo" Present undeliverable cargo (also present unused vehicles) 39
  36. Approach 1 Flow 1. Solve Number of vehicles is a

    hard constraint; other constraints are soft constraints. Constraint weights are default values. 2. Infeasible solution (Return to user which constraints are violated) < Ideally visualize for clarity > 3. Adjust the weights (penalties) of violated constraints, return to 1. 40
  37. Approach 1 Problems The part of 2 (showing violated constraints

    + visualization) is difficult for theoreticians (they don't want to create GUIs). Increasing constraint violation weights does not guarantee that constraints will be met. Since it solves multiple times by changing weights, optimization needs to be sufficiently fast. 41
  38. Approach 2 Flow 1. Solve (prioritize high-priority cargo and deliver

    as much as possible) 2. Obtain undelivered cargo and unused vehicles. 3. Investigate reasons for undelivered cargo and change constraints. Examples: ✓ Cargo time window not within vehicle operating hours. ✓ Cargo volume too large to fit in vehicle capacity. ✓ Customer-requested skills are too strict, preventing vehicle assignment. 4. (Generate a problem with only the remaining cargo and vehicles) and return to 1. 42
  39. Approach 2 Problems and Advantages Problems Requires human intervention to

    change constraints, so not fully automatic. Advantages ✓ Can easily correct when a user enters incorrect data. ✓ Problem scale gradually decreases, so overall it solves faster. ✓ Low priority (company profit) cargo can be postponed to the next day or outsourced to other companies (chartering). 43
  40. Optimization is Not a Magic Wand or a Silver Bullet

    Optimization, unlike machine learning, is not easy to apply in practice. Just buying a solver is useless. Excessive expectations are not good. User cooperation is indispensable for extracting constraints and determining evaluation criteria. Obtained solutions cannot be used as is (e.g., if a company is in the red, it might suggest closing everything). Even if modeling is possible, it might not be solvable (creating a solvable model is important). Don't be swayed by consultants or professors (the former often don't understand optimization well, the latter mistakenly believe theory alone can solve anything). 45
  41. Garbage In, Garbage Out If you put in garbage, you

    get garbage out. To ensure correct input, the following are important: ✓ Pre-processing to convert input to correct data. ✓ Validation of data validity (input time check is ideal). ✓ Visualization of input data and detection of anomalies. ✓ Education and incentives for input personnel (no one answers questionnaires without a reward). ✓ Imputation of missing values. ✓ Standardization of data units (e.g., mixture of m, cm, and mm). ✓ Linkage with past data (e.g., which old products correspond to current products, linkage for customer or department name changes). 46
  42. What's Important is Not the Optimal Solution, But the Insights

    Gained From It Maxim by Arthur Geoffrion Using the optimal solution as is is dangerous. It is important to test by varying inputs. Optimization is a tool for What If analysis. A tool for understanding the current situation. A tool for adopting a data-centric approach. 47
  43. Optimization is a Decision Support Tool; Humans Make Decisions Including

    Constraints Constraints are there to be violated. Only humans who know the site well understand the impact of relaxing constraints. To create good support tools, designing an interactive interface is important. To design easy-to-use tools, system design is important. It is important to review constraints that come from past customs. 48
  44. Optimization is Not About Simplifying Manual Work An optimization system

    that merely saves the effort of creating manual work schedules will not succeed. Do not use proximity to manually created solutions as an evaluation metric for solutions. If it merely saves effort but cannot reduce costs or improve service using optimization, it's no different from processing IT (optimization is analytical IT). 49
  45. Motivation Real-world optimization problems are dynamic and uncertain. They are

    too difficult, so optimization is applied assuming they are static and deterministic problems. On the other hand, vast amounts of data related to optimization problem instances are accumulated and available. Utilize this data to overcome the following three points: i. Dynamic (optimization considering the flow of time from past to future) ii. Uncertainty (future problem instance data has uncertainty) iii. Complexity (optimization problems themselves are large-scale, leading to huge computation) 51
  46. Comparison of Fusion with (A)DP/RL/MPC/MO Dynamic Programming (DP) Approximate DP

    (ADP) Reinforcement Learning (RL) Model Predictive Control (MPC) MO Hybrid Model Yes Yes Not always needed Yes Yes Prediction No No No From past states From past problem instances and context (additional info) Value Function Yes Defined for post-action state Yes Yes Defined for post-action state and pre- action state Optimization of 52