Pre-processing for optimization Handling infeasibility Why is application to practice difficult? Fusion of Machine Learning and Mathematical Optimization 2
✓ Various additional conditions are required in practice. ✓ Often cannot be reduced to typical optimization problems as is. (If reducible, refer to "100+ Optimization Problems" for easy solution!) NP-Hard... ✓ Most real-world problems are NP-Hard. ✓ Efficient algorithms (probably) do not exist. ✓ Solving them generically can take a very long time depending on the problem instance. 4
minutes read 100+ Optimization Problems Solvable in Practice with Python ▪ Introduces formulations and Python solutions for over 100 optimization problems useful in practice. Table of Contents Author's Page Guidelines Notes Maxims Operating Environment Install the following with Poetry. (Also use commercial solvers Gurobi, OptSeq, SCOP) 100+ Optimization Problems 5
(polynomially on average) Give up on finding optimal (exact) solutions (approximate solutions) => Trade-off between computation speed and solution quality Give up on solving all problem instances Research on special types that are easy to solve (X) => Design models and solution methods targeting typical problem instances 6
✓ Customers and products often follow Zipf's law. ✓ Customer distribution is biased (e.g., concentrated in Tokyo in Japan). Different from intentionally difficult problem instances (benchmarks for research) There are parts that can be handled by heuristics (designed to be so) => Fixed by pre-processing Interviewing alone is insufficient (supplement with data) Data alone is insufficient (supplement with interviewing) 7
people) Model assuming arbitrary problem instances (X) Cannot solve all NP-hard instances! (Lazy people) Model assuming a few problem instances (X) Will fail if unexpected problem instances are input! 8
arbitrary input data (X) Assume problem instances that may occur in practice (Not too diligent & not lazy => Art!) ✓ Collect as many real-world problem instances as possible ✓ Data analysis and visualization ✓ Interviewing ✓ Feature extraction from problem instances ✓ Pre-processing and modeling according to the characteristics of the problem instances ✓ Testing with slightly larger random data than actual problem instances 9
truth in meeting rooms (in front of suits) (but will talk over drinks). Rare exceptions (bragging about how they handled it). Always adding features because they're afraid of losing their job. Minute specifications and numerical values (bragging about how difficult their job is). Conditions that cannot be expressed numerically (e.g., if possible, preferably, occasionally...) Objective function that increases costs (e.g., taking a detour for fairness?) 11
on interviews and data, extract "truly necessary" conditions, and consider whether to obfuscate them with pre-processing, reduce them to standard problems, or properly include them in the model. Example: Delivery Planning ✓ Interviewing: Work time varies by store depending on the time of day. ✓ Ask about the number of stores, work time differences, and types of time zones (+ confirm with data). ✓ Visualize the distribution of stores and the relationship between time zones and work time. ✓ If few time zone types (day and night) & few stores => solve with two types of time slots (day and night). ✓ If large differences in work time => set time slots to one type with shorter work time (night). ✓ If many stores and various time zones => reduce to delivery optimization with time-dependent travel times. 12
test • Medium-scale problem instance test • Full-scale test • What If analysis • Sensitivity analysis • Robustness test Data • Data Collection • Data Pre-processing • Data Visualization 18
• Can be fast • Effective for specific problems • No need for formalization • Can incorporate complex conditions • Can design using problem structure Cons • Not exact solutions • Cannot handle continuous variables • No model • Simple procedures (processing IT) • Solution method design takes time 19
model (formalization) • Easy to maintain • Additional conditions (if formalizable) are easy to incorporate • Can be introduced quickly Cons • Can be expensive or slow • May take time to compute • May not be able to incorporate some additional conditions? • Computation time may increase if data changes • If data changes, solution quality may deteriorate (already mentioned in meta-heuristics cons, but more pronounced here in terms of feasibility/optimality) 20
that the solver became 1 trillion times faster, told the client it could solve anything instantly, but it failed and broke down. ✓ Told the client it could solve instantly, but in reality it was impossible and broke down. ✓ Was told delivery would be via a maximum of 2 hubs, but it became 4 hubs just before the deadline and broke down. Data Processing ✓ Failed to collect transportation costs and handling costs, leading to breakdown. ✓ Insisted on using real data with its original precision, causing the solver to break down. ✓ Fed raw, extremely large-scale data directly, leading to breakdown. 22
to the very limit, but real-world data was different, leading to breakdown. ✓ Tried to solve with the only available method in-house, but ran out of time and broke down. ✓ Kept adding conditions as requested by the client, eventually leading to breakdown. ✓ Believed interviews too easily and added too many detailed conditions, leading to breakdown. ✓ Said "university's main work got busy (or the assigned student stopped coming)" and quit midway. Deployment & Evaluation ✓ Pulled data one by one from the database with SQL, ran out of time. ✓ Problem scale gradually increased in real operation, making it unsolvable and leading to breakdown. ✓ Sales slipped their tongue saying labor costs would be halved, causing a rebellion on site and breakdown. 23
differs from typical software development project management. ✓ Goals are ambiguous (what to achieve and to what extent is determined by trial and error). ✓ Feasibility cannot be guaranteed (whether it can be solved is unknown until tried!). ✓ Not merely process automation (processing IT), but creating intelligence (analytical IT). ✓ Time taken varies greatly depending on the developer's skill (6 months => 3 days). ✓ Skill measurement is difficult (certification exams are different from real problems). Traditional PMs lack experience and knowledge. ✓ Cannot judge problem difficulty. ✓ Tend to decide specifications at the beginning. ✓ Estimate costs based on hourly wages or lines of code. Need for PMs who understand the optimization (machine learning) process. 24
up only on computation time and optimality is not enough. It is also necessary to give up on solving arbitrary problem instances. Extract typical problem instances based on real-world data and model them. A part of the data is simplified by pre-processing (for optimization) (Art!). Optimization project lifecycle. Precautions during real-world operation. 25
conversion, handling missing values and outliers, separation of training/validation/test sets, etc.). ✓ Feature extraction is an art (however, deep learning can automate feature extraction). Pre-processing for Optimization ✓ Automation is difficult. ✓ Discussions with clients (+ interviewing on site) are crucial. ✓ Must be considered simultaneously with the construction of the optimization model (+ solution method) (cannot be separated). ✓ Art! 27
multiple (as many as possible) actual real-world problem instances. 3. If possible, obtain time-series data for a suitable period (to determine if it needs to be treated as a dynamic problem). 4. If data cannot be provided => It's pointless to continue, so terminate. 5. If data is insufficient, return to step 2. 28
and remove anomalies (identify the cause of anomalies such as input data units; requires discussion with client). 2. Handle missing data (how to handle it? requires discussion with client). 3. Visualize relationships between data items (scatter plots, Gantt charts). 4. Plot location information on a map (calculate latitude and longitude if necessary). Basically the same as pre-processing for machine learning or data science. 29
discussion with client; especially interviewing on site). ✓ Example: Cluster analysis of locations to aggregate data. ✓ Example: ABC analysis of products to aggregate data. ✓ Example: Standardization of sizes based on product size and capacity (rounding up). ✓ Example: Fixing partial solutions that most optimal solutions have (utilizing machine learning). Bin (Capacity 10) Items (Size) 30
for optimization model creation. Data collection, pre-processing, pre-processing for optimization. Pre-processing for optimization models and solution construction (Art). 31
Optimization Model -> (Optimal) Solution. Labels for each step describing its meaning (e.g., "Problem with numerical values input", "Maximize/Minimize Objective Function", "Solution satisfying all constraints"). Problem Instance -> Optimization Model -> (Optimal) Solution Problem with numerical values input Maximize (Minimize) Objective Function subject to Constraints Optimal feasible solution with max (min) objective function value (= Solution satisfying all constraints) 33
Optimization Model -> (Optimal) Solution OR Infeasible OR Unbounded sup (inf) Objective Function subject to Constraints No solution exists that satisfies all constraints = Infeasible Objective function becomes infinite (small) = Unbounded 34
Optimization Model -> Usable Solution Maximize Objective Functions (many) subject to Constraints (No need to satisfy all) Solution considering various metrics (Allows some deviation from constraints) 35
-> (Optimal) Solution Minimize Sum of weighted penalties for constraint violations subject to Hard Constraints (must be satisfied absolutely) Soft Constraints (penalty if violated) How are violation weights determined? 36
and tasks within the planning period to meet the required number of personnel for each task per time slot "completely". Constraints ✓ Required personnel conditions ✓ Staff preferred shifts (day, time slot) ✓ Task change penalty ✓ Special tasks (breaks) placed in appropriate time slots ✓ Constraints on consecutive shifts (prohibition of morning shift the day after night shift) ✓ ... Approach: Constraint Optimization (Weighted Constraint Satisfaction Problem) ✓ Allows violation of all constraints. ✓ Violation penalties for constraints are determined by the user (e.g., small for task changes, large for preferred shifts) or by priority. 37
showing three objective functions (Priority 1, 2, 3) and constraints, with a flow demonstrating how higher priority objectives are optimized first, becoming constraints for lower priority ones. Minimize Priority 1 Objective Function Priority 2 Objective Function Priority 3 Objective Function Constraints Flow: Priority 1 Objective Function Constraints -> Priority 1 Objective Function z* -> Priority 2 Objective Function -> Constraints 38
hard constraint; other constraints are soft constraints. Constraint weights are default values. 2. Infeasible solution (Return to user which constraints are violated) < Ideally visualize for clarity > 3. Adjust the weights (penalties) of violated constraints, return to 1. 40
+ visualization) is difficult for theoreticians (they don't want to create GUIs). Increasing constraint violation weights does not guarantee that constraints will be met. Since it solves multiple times by changing weights, optimization needs to be sufficiently fast. 41
as much as possible) 2. Obtain undelivered cargo and unused vehicles. 3. Investigate reasons for undelivered cargo and change constraints. Examples: ✓ Cargo time window not within vehicle operating hours. ✓ Cargo volume too large to fit in vehicle capacity. ✓ Customer-requested skills are too strict, preventing vehicle assignment. 4. (Generate a problem with only the remaining cargo and vehicles) and return to 1. 42
change constraints, so not fully automatic. Advantages ✓ Can easily correct when a user enters incorrect data. ✓ Problem scale gradually decreases, so overall it solves faster. ✓ Low priority (company profit) cargo can be postponed to the next day or outsourced to other companies (chartering). 43
Optimization, unlike machine learning, is not easy to apply in practice. Just buying a solver is useless. Excessive expectations are not good. User cooperation is indispensable for extracting constraints and determining evaluation criteria. Obtained solutions cannot be used as is (e.g., if a company is in the red, it might suggest closing everything). Even if modeling is possible, it might not be solvable (creating a solvable model is important). Don't be swayed by consultants or professors (the former often don't understand optimization well, the latter mistakenly believe theory alone can solve anything). 45
get garbage out. To ensure correct input, the following are important: ✓ Pre-processing to convert input to correct data. ✓ Validation of data validity (input time check is ideal). ✓ Visualization of input data and detection of anomalies. ✓ Education and incentives for input personnel (no one answers questionnaires without a reward). ✓ Imputation of missing values. ✓ Standardization of data units (e.g., mixture of m, cm, and mm). ✓ Linkage with past data (e.g., which old products correspond to current products, linkage for customer or department name changes). 46
Gained From It Maxim by Arthur Geoffrion Using the optimal solution as is is dangerous. It is important to test by varying inputs. Optimization is a tool for What If analysis. A tool for understanding the current situation. A tool for adopting a data-centric approach. 47
Constraints Constraints are there to be violated. Only humans who know the site well understand the impact of relaxing constraints. To create good support tools, designing an interactive interface is important. To design easy-to-use tools, system design is important. It is important to review constraints that come from past customs. 48
that merely saves the effort of creating manual work schedules will not succeed. Do not use proximity to manually created solutions as an evaluation metric for solutions. If it merely saves effort but cannot reduce costs or improve service using optimization, it's no different from processing IT (optimization is analytical IT). 49
too difficult, so optimization is applied assuming they are static and deterministic problems. On the other hand, vast amounts of data related to optimization problem instances are accumulated and available. Utilize this data to overcome the following three points: i. Dynamic (optimization considering the flow of time from past to future) ii. Uncertainty (future problem instance data has uncertainty) iii. Complexity (optimization problems themselves are large-scale, leading to huge computation) 51
(ADP) Reinforcement Learning (RL) Model Predictive Control (MPC) MO Hybrid Model Yes Yes Not always needed Yes Yes Prediction No No No From past states From past problem instances and context (additional info) Value Function Yes Defined for post-action state Yes Yes Defined for post-action state and pre- action state Optimization of 52
mickey_kubo
geneguchi
ienter
tanaken0515
simpleform
shungoa
kazuotanaka
smartcamp
okitsu
ballas_inc
gessy0129
inagakikay
flux
sarafernandez
jonoalderson
akashhashmi
axbom
marktimemedia
jonrohan
cromwellryan
tmiket
davidcarrasco
tammyeverts
chriscoyier
elsirapls
