The ISO Problem: Decentralized Stochastic Control via Bidding Schemes Rahul Singh, P.R. Kumar and Le Xie Texas A&M University 

p Supply Demand Price u Supply Price Demand Price The ISO problem in a simple static context ISO Generator Load u ISO has to balance supply and demand u Generator and load bid their supply and demand curves u ISO intersects to find right price 

• What if the agents do not disclose their bid functions? • The ISO needs to go through a “price discovery process”, as in a Walrasian auction 

• Start with some initial guess 0 of market-clearing price and ask agents to declare their bids at price = 0 

0 Price u Δ0 D S 

• Use the mismatch Δ0 to adjust the guess of market clearing price 

Δ1 1 Price u D S 

∞ D u Δ = 0 Price S Repeat iterations until price converges 

Uncertainties and dynamics: renewables and demand response Coal power plant Wind farm Nuclear power plant Hydropower plant ISO Commerci al load Industrial load Load serving entity Storage service • Dynamic constraints: ramping, thermal inertia • Lots of uncertainty: Wind, temperature, water flow • All choices have costs/benefits • How can ISO ensure maximum social welfare? • How much should be generated, and balance • How should generators and loads bid? Time Generation Time Consumption 

Renewable Energy: Stochastic and Dynamic Markets • But agents have stochastic and dynamic behaviour • Stochastic: renewable energy availability has temporal fluctuations that are hard to predict • Dynamic: Agents’ utilities at a time depend upon the past actions too e.g. thermal loads = 1 − + , = 1,2, … , are bid-instants 

• Thus the utilities have temporal dependencies and are “stochastic” • Such a simple one-shot iterative scheme is highly sub-optimal when the agents are stochastic and have “dynamic utilities” • The ISO has to ensure optimal operation of such a electiricity grid 

The ISO problem Generator 1 Generator 2 Storage 5 Load 4 Prosumer 6 x 1 (t +1) = f 1 (x 1 (t),u 1 (t),w 1 (t)) (x 1 (t),u 1 (t)) Î G 1 (t) Max E U 1 t=0 T å (x 1 (t),u 1 (t)) Max E U i (x i (t),u i (t)) t=0 T å i=1 N å How to assign ui (t)’s? ISO Without knowledge of: • States xi (t) • Models fi (xi ,ui ,wi ) • Utilities Ui (xi ,ui ) Balance: u i (t) i=1 N å = 0 for all t Load 3 x 3 (t +1) = f 3 (x 3 (t),u 3 (t),w 3 (t)) (x 3 (t),u 3 (t)) Î G 3 (t) Max E U 3 t=0 T å (x 3 (t),u 3 (t)) 

Coordinating multiple deterministic dynamical systems • All generators, loads, storage and prosumers are deterministic dynamical systems • Goal x i (t +1) = f i (x i (t),u i (t)) (x i (t),u i (t)) Î G i (t) Max U i (x i (t),u i (t)) t=0 T å i=1 N å s.t. u i (t) i=1 N å = 0 for all t 

The solution • Consider Lagrange Dual (() is multiplier for ∑ = 0 ) = Max ui U i (x i (t),u i (t))- p(t)u i (t) t=0 T å æ è ç ö ø ÷ i=1 N å Decomposes 

• ISO needs to announce p = (p(0), p(1), … , p(T)) • The entire sequence of prices for all future times • Agent i distributedly chooses ui = (ui (0), ui (1), … , ui (T)) to maximize its own utility • This yields value of dual function () U i (x i (t),u i (t))- p(t)u i (t) ( ) t=0 T å 

How does ISO choose price sequence p? • The ISO needs to solve dual problem: • Subgradient iteration • Now • Price iteration • Converges after weighted averaging under convexity and compactness assumptions (“ergodic” method) Min p³0 D(p) pk+1 = pk - ek ¶D(pk ) ¶pk pk+1 = pk + ek u i k (0) i=1 N å , u i k (1) i=1 N å ,..., u i k (T ) i=1 N å æ è ç ö ø ÷ ¶D ¶p = - u i (0) i=1 N å , u i (1) i=1 N å ,..., u i (T ) i=1 N å æ è ç ö ø ÷ 

Price sequences and Bid sequences Coal power plant Wind farm Nuclear power plant Hydropower plant ISO Commerci al load Industrial load Load serving entity Storage service (p1(0),….,p1(T)) (p1(0),….,p1(T)) (p1(0),….,p1(T)) (p1(0),….,p1(T)) (p1(0),….,p1(T)) (p1(0),….,p1(T)) (p1(0),….,p1(T)) (p1(0),….,p1(T)) 

Price sequences and Bid sequences Coal power plant Wind farm Nuclear power plant Hydropower plant ISO Commerci al load Industrial load Load serving entity Storage service (u1(0),….,u1(T)) (u1(0),….,u1(T)) (u1(0),….,u1(T)) (u1(0),….,u1(T)) (u1(0),….,u1(T)) (u1(0),….,u1(T)) (u1(0),….,u1(T)) (u1(0),….,u1(T)) 

Price sequences and Bid sequences Coal power plant Wind farm Nuclear power plant Hydropower plant ISO Commerci al load Industrial load Load serving entity Storage service (p2(0),….,p2(T)) (p2(0),….,p2(T)) (p2(0),….,p2(T)) (p2(0),….,p2(T)) (p2(0),….,p2(T)) (p2(0),….,p2(T)) (p2(0),….,p2(T)) (p2(0),….,p2(T)) 

Price sequences and Bid sequences Coal power plant Wind farm Nuclear power plant Hydropower plant ISO Commerci al load Industrial load Load serving entity Storage service (u2(0),….,u2(T)) (u2(0),….,u2(T)) (u2(0),….,u2(T)) (u2(0),….,u2(T)) (u2(0),….,u2(T)) (u2(0),….,u2(T)) (u2(0),….,u2(T)) (u2(0),….,u2(T)) 

Price sequences and Bid sequences Coal power plant Wind farm Nuclear power plant Hydropower plant ISO Commerci al load Industrial load Load serving entity Storage service (p3(0),….,p3(T)) (p3(0),….,p3(T)) (p3(0),….,p3(T)) (p3(0),….,p3(T)) (p3(0),….,p3(T)) (p3(0),….,p3(T)) (p3(0),….,p3(T)) (p3(0),….,p1(T)) 

Price sequences and Bid sequences Coal power plant Wind farm Nuclear power plant Hydropower plant ISO Commerci al load Industrial load Load serving entity Storage service (u3(0),….,u3(T)) (u3(0),….,u3(T)) (u3(0),….,u3(T)) (u3(0),….,u3(T)) (u3(0),….,u3(T)) (u3(0),….,u3(T)) (u3(0),….,u3(T)) (u3(0),….,u3(T)) 

…... until convergence Price Time 

Coordinating multiple stochastic dynamical systems with common uncertainty • Generators, loads, storage, prosumers are stochastic – dependent on a common uncertainty • Ex: All loads depend on common temperature of the city • Common uncertainty w() is observed by all loads/gens • Goal Max E w(×) U i (x i (t),u i (t)) t=0 T å i=1 N å æ è ç ö ø ÷ x i (t +1) = f i (x i (t),u i (t),w(t)) (x i (t),u i (t)) Î G i (t) 

Solution: Uncertainty tree • Make a tree that lists all possible realizations of the Uncertainty • = 0 , 1 , … , () w(1) w(0) w(2) w(3) v v 

• Tree policy: Choose action u(v) for each vertex v • Since probability of a node does not depend on previous actions • Optimization problem Min p(v)c(x(v,uv ),uv ) v å s.t. u i (v) = 0 for all v i=1 N å 

u Now perform bid-price iterations for each node u Very complex since number of nodes is exponentially large u What if the uncertainty is not commonly observed? 

Coordinating multiple stochastic dynamic systems with private uncertainty • Generators, loads, storage and prosumers are stochastic having private uncertainties • Goal: Max E w1 (×),w2 (×),...,wN (×) U i (x i (t),u i (t)) t=0 T å i=1 N å æ è ç ö ø ÷ x i (t +1) = f i (x i (t),u i (t),w i (t)) (x i (t),u i (t)) Î G i (t) 

Solvable case of private uncertainty • The ISO need not know the values of the uncertainties • It only needs to know and announce • Labels of remaining tree (w1 (s), w2 (s), …, wN (s)) for s ≥ t • Law of remaining labels • Labels could be hashed • Confidentiality of information can be assured • Agent i communicates the label of wi (t) to ISO at time t • Similar scheme works – but great complexity 

General intractability of distributed stochastic control problems • Witsenhausen’s counterexample • If agent i’s actions affect another agent j, but that agent i doesn’t know the action that j applied, then problem is intractable, even in LQG case • Generally intractable in stochastic case with private uncertainty • Any solution? 

Same solution as commonly observed uncertainty, but iterations for future prices at times , + 1, … , performed at each time w(1) w(0) w(2) w(3) 0 5 4 3 1 2 6 

Prices and Bids Coal power plant Wind farm Nuclear power plant Hydropower plant ISO Commerci al load Industrial load Load serving entity Storage service Price p Price p Price p Price p Price p Price p Price p Price p 

Prices and Bids Coal power plant Wind farm Nuclear power plant Hydropower plant ISO Commerci al load Industrial load Load serving entity Storage service Gen u1 Gen u2 Gen u3 Cons u7 Cons u6 Cons u5 Gen u4 Cons u8 • ISO needs to choose p • So that • Ensures least cost in simple static context u i = 0 å 

No content

Price sequences and Bid sequences for each node Coal power plant Wind farm Nuclear power plant Hydropower plant ISO Commerci al load Industrial load Load serving entity Storage service (p1(v1 ),….,p1(vM )) (p1(v1 ),….,p1(vM )) (p1(v1 ),….,p1(vM )) (p1(v1 ),….,p1(vM )) (p1(v1 ),….,p1(vM )) (p1(v1 ),….,p1(vM )) (p1(v1 ),….,p1(vM )) (p1(v1 ),….,p1(vM )) 

Price sequences and Bid sequences for each node Coal power plant Wind farm Nuclear power plant Hydropower plant ISO Commerci al load Industrial load Load serving entity Storage service (u1(v1 ),….,u1(vM )) (u1(v1 ),….,u1(vM )) (u1(v1 ),….,u1(vM )) (u1(v1 ),….,u1(vM )) (u1(v1 ),….,u1(vM )) (u1(v1 ),….,u1(vM )) (u1(v1 ),….,u1(vM )) (u1(v1 ),….,u1(vM )) 

Price sequences and Bid sequences for each node Coal power plant Wind farm Nuclear power plant Hydropower plant ISO Commerci al load Industrial load Load serving entity Storage service (p2(v1 ),….,p2(vM )) (p2(v1 ),….,p2(vM )) (p2(v1 ),….,p2(vM )) (p2(v1 ),….,p2(vM )) (p2(v1 ),….,p2(vM )) (p2(v1 ),….,p2(vM )) (p2(v1 ),….,p2(vM )) (p2(v1 ),….,p2(vM )) 

Price sequences and Bid sequences for each node Coal power plant Wind farm Nuclear power plant Hydropower plant ISO Commerci al load Industrial load Load serving entity Storage service (u2(v1 ),….,u2(vM )) (u2(v1 ),….,u2(vM )) (u2(v1 ),….,u2(vM )) (u2(v1 ),….,u2(vM )) (u2(v1 ),….,u2(vM )) (u2(v1 ),….,u2(vM )) (u2(v1 ),….,u2(vM )) (u2(v1 ),….,u2(vM ))  Very complex since number of loads is exponentially large 

Uncertainty tree • Uncertainty at each stage • Tree policy: Choose action u(v) for each vertex v • State x(v, uv) • One step cost is c(x(v, uv), u(v)) • Convexity of c(.) in uv • Optimization problem Min p(v)c(x(v,uv ),uv ) v å s.t. u i (v) = 0 for all v i=1 N å w(1) w(0) w(2) w(3) v v u Very complex since number of loads is exponentially large 

But if generators and loads can be modeled as Linear Quadratic Gaussian (LQG) systems then there is a tractable solution 

Multiple LQG Systems with private uncertainty • All generators and loads are LQG systems having private uncertainties • Goal: Max E x i (t)T Q i x i (t)+u i (t)T R i u i (t) t=0 T å i=1 N å æ è ç ö ø ÷ x i (t +1) = A i x i (t)+ b i u i (t)+ C i w i (t) y i (t) = D i x i (t)+ H i v i (t) x 0 ,w i (t),v i t ( ) ∼ N, mean 0, and independent 

Scheme for distributed LQG systems • At each time t = 0, 1, 2, … • Iterations k=1,2,3, …: • At iteration k: • ISO announces deterministic, future price sequence • Each Generator/Load i responds with optimal solution (ui k (t), ui k (t+1), …, ui k (T)) of deterministic LQ problem pk = pk-1 +e k-1 u i k-1(t) i=1 N å , u i k-1(t +1) i=1 N å ,..., u i k-1(T ) i=1 N å æ è ç ö ø ÷ x i (t +1) = A i x i (t)+ b i u i (t) Min x i (t)T Q i x i (t)+ u i (t)T R i u i (t)+ pk (t)u i (t) T å æ è ç ö ø ÷ 

Future price sequences and Future bid sequences at each time t Coal power plant Wind farm Nuclear power plant Hydropower plant ISO Commerci al load Industrial load Load serving entity Storage service (p1(t),….,p1(T)) (p1(t),….,p1(T)) (p1(t),….,p1(T)) (p1(t),….,p1(t)) (p1(t),….,p1(T)) (p1(t),….,p1(T)) (p1(t),….,p1(T)) (p1(t),….,p1(T)) 

Future price sequences and Future bid sequences at each time t Coal power plant Wind farm Nuclear power plant Hydropower plant ISO Commerci al load Industrial load Load serving entity Storage service (u1(t),….,u1(T)) (u1(t),….,u1(T)) (u1(t),….,u1(T)) (u1(t),….,u1(T)) (u1(t),….,u1(T)) (u1(t),….,u1(T)) (u1(t),….,u1(T)) (u1(t),….,u1(T)) 

No content

No content

p Supply Demand Price u • We have provided the equivalent of exchanging supply and demand curves in simple static context • With 15 min intervals, is this iterative bidding feasible? • Communication/computation infrastructure to automate this • May not be any better alternative • Can truncate the iterations after finite threshold • Investigating strategic considerations, line losses, etc Conclusion: