SocInfo22 Keynote: Modelling and Quantifying Uncertainty in Economic Indicators

© 2018 Bloomberg Finance L.P. All rights reserved. © 2022
Bloomberg Finance L.P. All rights reserved. Keynote: Modelling and Quantifying Uncertainty in Economic Indicators Social Informatics 2022 (SocInfo 2022) October 19, 2022 Edgar Meij, Ph.D. Head of AI Search and Discovery @edgarmeij | [email protected]

© 2022 Bloomberg Finance L.P. All rights reserved. Recent Macroeconomic
Headlines

© 2018 Bloomberg Finance L.P. All rights reserved. Today •
Economic indicators and macroeconomic activity • The role of central banks • The role of AI/NLP

© 2018 Bloomberg Finance L.P. All rights reserved. Bloomberg is
just finance, right? • A technology company founded in New York City in 1981 • 325,000+ subscribers in 170 countries • 20,000+ employees in 160+ locations, including 7,000+ software engineers – with 200+ engineers and data scientists working on AI and related problems • Increased use of and contributions to open source software • Increased presence in academic research

© 2022 Bloomberg Finance L.P. All rights reserved. The Bloomberg
Terminal is software that delivers a diverse array of information, news and analytics to facilitate financial decision-making.

© 2018 Bloomberg Finance L.P. All rights reserved. News moves
markets

© 2022 Bloomberg Finance L.P. All rights reserved. Sanford Bernstein’s
Toni Sacconaghi “And so, where specifically will you be in terms of capital requirements?” Real-time multi-modal data moves markets speech recognition entity recognition linking salience topic classification summarization Elon Musk “Excuse me. Next. Boring, bonehead questions are not cool. Next?”

Latency matters entity recognition linking salience sentiment

© 2022 Bloomberg Finance L.P. All rights reserved. Scale of
unstructured data at Bloomberg 80% of all pertinent data is unstructured company filings and press releases news and social media analyst reports and CRMs chats & email client feedback alternative data web sites AI >2M news articles and relevant social media posts ingested per day >1B indexed and searchable documents in multiple languages <500ms latency for monitoring and alerting Hundreds of millions of entities and relations in the Bloomberg KG

© 2018 Bloomberg Finance L.P. All rights reserved. Central banks
and economic policy

© 2018 Bloomberg Finance L.P. All rights reserved. The Fed
• The Federal Reserve conducts monetary policy "so as to promote the goals of maximum employment, stable prices, and moderate long-term interest rates” • Mandate ◦ Management and oversight of the production and distribution of the nation's currency ◦ Sharing of information and statistics ◦ Promoting economic and employment growth through changes to the interest rate

Bloomberg Finance L.P. All rights reserved. The Fed • The Federal Open Market Committee (FOMC) ◦ Eight scheduled meetings per year ▪ reviewing economic and financial conditions ▪ determining monetary policy ▪ assessing the risks to its long-run goals of price stability and sustainable economic growth ◦ Promotes economic and employment growth through changes to the interest rate

Bloomberg Finance L.P. All rights reserved. Central bankers – Of hawks and doves • A hawk is thought of as assigning a higher priority to fighting inflation ◦ >> higher interest rates • A dove supports output growth and lowering unemployment ◦ >> lower interest rates • Interest rates?

Bloomberg Finance L.P. All rights reserved. Monetary transmission • Central banks use the interest rate as its primary policy tool ◦ Overnight rate at which banks lend to each other over the very short term ◦ Interest rate at which banks can borrow from the central bank (directly or via bonds) ◦ Shifts in this rate have a knock-on effect on the economy through, e.g., bank lending/mortgage/interest rates, and thus consumer spending and borrowing • Price stability is critical to a healthy economy ◦ If interest rates go up it increases the cost of borrowing, making credit and investment more expensive – used to slow an overheated economy ◦ If rates go down it makes borrowing cheaper, encouraging credit and investment – used to stimulate a stagnant economy

© 2018 Bloomberg Finance L.P. All rights reserved. The transmission
mechanism of monetary policy

Bloomberg Finance L.P. All rights reserved. Monetary transmission • If inflation rates are rising, then there is too much money in the economy relative to the amount of goods ◦ Each dollar/euro/pound/… one has is worth less than it was a day earlier • Increasing the interest rate is a way to remove money from the economy and thus to decrease inflation ◦ Investors will pick less risky investments, people will save more and spend less, etc ◦ Decreases employment opportunities, and may be detrimental long-term • A decrease in rates allows more money to be spent, encouraging business growth – thus increasing the potential for employment

Bloomberg Finance L.P. All rights reserved. Inflation and target rates • Lag ◦ Policymakers have to anticipate future inflation trends when deciding on rate levels in the present ◦ Adherence to inflation targets can only be gauged with backward-looking statistics ◦ These can range widely amid economic shocks "In short, if making monetary policy is like driving a car, then the car is one that has an unreliable speedometer, a foggy windshield, and a tendency to respond unpredictably and with a delay to the accelerator or the brake," -- Ben Bernanke, 2004

Bloomberg Finance L.P. All rights reserved. Hawks vs. doves (and in-betweens?) • “About 20,000 articles or reports from more than 30 newspapers and business reports of Fed watchers with reference to FOMC members are read.” ◦ (Footnote: “Due to the particularity of the exercise, the process involved human reading rather than text mining/reading algorithms.”) Istrefi, Klodiana (2019). Fed Watchers’ Eyes: Hawks, Doves and Monetary Policy. Banque de France.

Bloomberg Finance L.P. All rights reserved. Economic indicators • Economic indicators provide measures of macroeconomic performance and stability • Divided into three broad groups ◦ Leading indicators, e.g., consumer confidence, jobless claims, and durable goods orders ◦ Coincident indicators, e.g., short-term interest rate, retail sales, and CPI ◦ Lagging indicators, e.g., GDP, corporate earnings, and PCE

Bloomberg Finance L.P. All rights reserved. Example: Consumer Price Index (CPI) • Measure of the average change over time in the prices paid by urban consumers for a market basket of consumer goods and services • Based on surveys of consumer purchases • Essentially compares the cost of living over time ◦ Can be used to gauge inflation levels • Market participants pay close attention to CPI for signs of inflation ◦ Rising inflation can cause higher interest rates and reduce borrowing ◦ Deflation can lower interest rates and encourage lending

Bloomberg Finance L.P. All rights reserved. Example: Gross Domestic Product (GDP) • The market value of all goods and services produced within a country during a given period • Lagging indicator and one of the most popular gauges of economic health • Can vary by political definition even if there is no difference in the economy • Commonly released every three months

Bloomberg Finance L.P. All rights reserved. Example: Personal Consumption Expenditures (PCE) • Measure of consumer spending • Include how much is spent every month on (non-)durable goods and services, so is comparable to CPI • Takes GDP into account (measured/reported quarterly!) ◦ Gaps are filled using retail sales every month • PCE is used by the Federal reserve to measure inflation ◦ Shows how people change their buying habits when prices change, providing a window into demand for products and services

© 2022 Bloomberg Finance L.P. All rights reserved. NIPA Handbook,
Chapter 5: Personal Consumption Expenditures. Bureau of Economic Analysis.

© 2018 Bloomberg Finance L.P. All rights reserved. How are
they used?

Bloomberg Finance L.P. All rights reserved. Risk analysis (shocks) • What happens if … ◦ VIX + 20% ? ◦ Oil price + x %? ◦ Pr(inflation > 4%)? ◦ If inflation is high in the US, what is the probability of inflation being low in the EU?

© 2018 Bloomberg Finance L.P. All rights reserved. How are
economic indicators created? • Surveys • Reports • Estimates • Nowcasting ◦ Combine ▪ Input indicators ▪ “Alternative data” • Weather, satellite imagery • (Semi-)structured data ◦ Transactions ◦ Web crawls • Textual data ◦ Predict ▪ Actual value ▪ Above/Below expectation

© 2018 Bloomberg Finance L.P. All rights reserved. Data is
the backbone of the financial markets • Historically mostly “structured” market data (ticks/quotes/trades) ◦ Well-understood ◦ Enabling advanced forms of automation • Increasingly non-traditional factors, based on “alternative” data, such as: ◦ Satellite images / CO2 emissions over factories ◦ Sentiment analytics on news ◦ Shopping mall footfall traffic ◦ Number of people riding the subway ◦ Credit card transactions ◦ “Pret index” ◦ etc. Bl o o mb e r g Se c o nd M e a su r e L L C © 20 21 | Th i s d o c u me nt i s st r i c t l y p r i v a t e a nd c o nf i d e nti a l a nd m a y n o t b e c o p ie d , d is t r i b u t e d o r r e p r o d u c e d i n w h o l e o r i n p a r t w i t h o u t Bl o o mb e r g Se c o nd M e a su r e ' s p r i o r w r i t t e n c o nse nt . 3 W hat is Transaction Data? P u r c h a se d a t a i s t h e p r e f e r r e d so u r c e f o r c o nsu me r t r e nd s a nd b u si ne ss p e r f o r ma nc e i nsi g h t s. T r a nsa c t i o n d a t a p r o v i d e s b e ne f i t s t h a t o t h e r so u r c e s c a nno t : • P o i nt - o f - sa l e i nsi g h t s • T i me l i ne ss • L o ng i t u d i na l a na l y se s Aw areness Interest Consideration Intent Evaluation Purchase Transaction Re c e i p t Alternative Data Sources: Se nt i me nt W e a t h e r Sa t e l l i t e G e o - l o c a t i o n W e b t r a f f i c A p p u sa g e Consumer Purchase Funnel

© 2018 Bloomberg Finance L.P. All rights reserved. Data is
the backbone of the financial markets • Increasingly non-traditional factors, based on “alternative” data, such as: ◦ Satellite images / CO2 emissions over factories ◦ Sentiment analytics on news ◦ Shopping mall footfall traffic ◦ Number of people riding the subway ◦ “Pret index” ◦ Credit card transactions ◦ etc. • But also “unstructured” data… Challenge: identify financially-relevant signal from noisy, complex tangentially-related datasets.

© 2018 Bloomberg Finance L.P. All rights reserved. But also
“unstructured” data • 80% of data exists in the form of “raw”, unstructured text, e.g., ◦ News stories ◦ Company filings, earnings call transcripts ◦ Tweets/Reddit/Facebook/… posts ◦ Research analyst reports, CRMs ◦ Economic policy, govt communications ◦ Press releases ◦ Web sites ◦ Chats & e-mail, client feedback ◦ etc. ◦ Lots of jargon and custom terminology (sometimes even firm-specific!) Our Challenge Identify financially-relevant signal from noisy, complex tangentially-related datasets

© 2018 Bloomberg Finance L.P. All rights reserved. NLP in
Practice

Bloomberg Finance L.P. All rights reserved. NLP – Ingredients • What do you need? ◦ Data (Text) ◦ Annotations ◦ Algorithms/Models ◦ Compute ◦ Human beings, to ▪ operate compute ▪ tune models ▪ interpret results ▪ annotate data ◦ Universe of “entities” (and their relationships) ▪ financial instruments, companies, people, products/brands, geopolitical, etc. ▪ supply chain, issuers, corporate structure, domicile, c-suite, board members, etc. ◦ Point-in-time

© 2020 Bloomberg Finance L.P. All rights reserved. 57 A
False Positive Entity

Bloomberg Finance L.P. All rights reserved. NLP in Practice • Generation 1: Write a bunch of rules (“templates”, “grammars”) ◦ High-precision ◦ Slow, manual, difficult to maintain or update • Generation 2: Count words, or train a statistical classifier ◦ For sequence tagging: conditional random fields ◦ For document classification: logistic regression, SVMs, decision trees/random forests ◦ Need labeled data • Generation 3: Deep learning and human in the loop ◦ Need a lot more labeled data, or distant supervision ◦ May be slower

Bloomberg Finance L.P. All rights reserved. State-of-the-art NLP for monetary policy • Circa 2000s – “counting words” ◦ Gazetteers of positive/negative words ◦ PMI over Google hits, e.g., ▪ Lucca, D. O., & Trebbi, F. (2009). Measuring central bank communication: an automated approach with application to FOMC statements. National Bureau of Economic Research.

Bloomberg Finance L.P. All rights reserved. State-of-the-art NLP for monetary policy • Circa 2000s – “counting words” ◦ Gazetteers of positive/negative words ◦ PMI over Google hits, e.g., ▪ Lucca, D. O., & Trebbi, F. (2009). Measuring central bank communication: an automated approach with application to FOMC statements. National Bureau of Economic Research. • Circa 2010s – Still counting words, LDA was on the rise ◦ E.g., ▪ Baker, S. R., Bloom, N., & Davis, S. J. (2016). Measuring economic policy uncertainty. The quarterly journal of economics, 131(4), 1593-1636. ▪ Hendry, S (2012), Central Bank Communication or the Media’s Interpretation: What Moves Markets?, Bank of Canada Working Paper 2012-9. ▪ Hansen, S, McMahon, M and Prat, A (2014), Transparency and Deliberation within the FOMC: a Computational Linguistics Approach, CEP Discussion Papers DP1276, Centre for Economic Performance, LSE.

Bloomberg Finance L.P. All rights reserved. State-of-the-art circa now • Representation learning and reusing knowledge across tasks ◦ Pretrain general language models to predict next words (BERT, GPT-k, …) ◦ Use it to initialize a task-specific model ◦ Achieve higher accuracy with fewer examples ◦ Zero or few-shot learning • Add relevant domain, task, and contextual knowledge to LMs • Scaling out deep learning architectures • Learning more from less annotations ◦ Distant supervision ◦ Transfer learning ◦ Active learning

Bloomberg Finance L.P. All rights reserved. “Augmented” intelligence • No model is static ◦ New entities, new contexts, new relationships, new data, ... • Human-experts-in-the-loop ◦ Help prevent model drift and ameliorate lack of recall, precision ◦ Provide important training data • “Automation is augmentation, not replacement” ◦ Need effective tools for humans to work with algorithms

© 2018 Bloomberg Finance L.P. All rights reserved. Measuring economic
policy uncertainty

Bloomberg Finance L.P. All rights reserved. Method • Boolean expression keyword retrieval, i.e., keyword match, and then count ◦ Match ▪ Economic terms • “economic”, “economy” ◦ AND ▪ Uncertainty terms • “uncertain” OR “uncertainty” ◦ AND ▪ Policy terms • “regulation” OR “deficit” OR “deficits” OR “legislation” OR “legislative” OR “legislature” OR “congress” OR “congressional” OR “white house” OR “federal reserve” OR “the fed” OR “regulations” OR “regulatory” OR “legislative” OR “legislature”

Bloomberg Finance L.P. All rights reserved. (Dis)entangling multiple sources of uncertainty • Real underlying uncertainty about economic outcomes • Semantic uncertainty as expressed in language • Annotator uncertainty about the labels • Modeling uncertainty inherent to NLP/AI methods • Limitations of keyword-based approaches ◦ Lacking context ◦ Missing lexical variations ◦ Inability to capture common sense / inference K Keith, C Teichmann, B O’Connor, and E Meij (2020). Uncertainty over Uncertainty: Investigating the Assumptions, Annotations, and Text Measurements of Economic Policy Uncertainty. In Proceedings of the Fourth Workshop on Natural Language Processing and Computational Social Science. Association for Computational Linguistics.

© 2018 Bloomberg Finance L.P. All rights reserved. Disentangling multiple
sources of uncertainty Demand for new clothing is uncertain because several states may implement large hikes in their sales tax rates. Some economists claim that uncertainties due to government industrial policy in the 1930s prolonged and deepened the Great Depression. The outlook for the H1B visa program remains highly uncertain. As a result, some high-tech firms fear that shortages of qualified workers will cramp their expansion plans. Uncertainty about U.S. military actions in Iraq are placing upward pressure on oil prices. It remains unclear whether the government will implement new incentives for small business hiring. The looming political fight over whether to extend the Bush-era tax cuts makes it extremely difficult to forecast federal income tax collections in 2011. Uncertainty about prospects for war in Iraq has encouraged a build-up of petroleum inventories and pushed oil prices higher. 'The budget is uncertain,' Ms. Veneman said at a hearing of the Senate Agriculture Committee. 'I can't tell you where the budget is going to go with regard to anything.' Manufacturing businesses' confidence slipped in April for the second consecutive month, partly because of uncertainty about the Clinton Administration's plans, Cahners Economics Inc. said today. Adding to the pressures on the dollar was the growing view in the currency markets that the Federal Reserve was unlikely to act soon to push up interest rates further. Policy, Economy, Uncertainty, Causal Relation

© 2018 Bloomberg Finance L.P. All rights reserved. Analysis •
Baker et al. sample documents, obtain binary labels, and construct a “human- generated” index which has “a 0.86 correlation” with their index ◦ Only 16% of documents have more than one annotator and of these, the agreement rates are moderate: 0.60 Krippendorff’s α (chance-adjusted agreement) ◦ Baker et al. did not address whether this is a result of annotator bias, error in annotations, or true ambiguity in the text

© 2018 Bloomberg Finance L.P. All rights reserved. Small improvements:
embeddings-based term expansion ◦ Match ▪ Economic terms: “economic” OR “economy” OR “economies” OR “financial” OR “economic” OR “growth” OR “recession” OR “slowdown” ◦ AND ▪ Uncertainty terms: “uncertain” OR “uncertainty” OR “unclear” OR “unsure” OR “uncertainties” OR “turmoil” OR “confusion” OR “worries” ◦ AND ▪ Policy terms: “regulation” OR “deficit” OR “deficits” OR “legislation” OR “legislative” OR “legislature” OR “congress” OR “congressional” OR “white house” OR “federal reserve” OR “the fed” OR “regulations” OR “regulatory” OR “legislative” OR “legislature”

© 2018 Bloomberg Finance L.P. All rights reserved. Method •
LogReg: multiclass logistic regression ◦ Obtain vocabulary containing most frequent words from labeled documents ◦ Maximize conditional log-likelihood in order to infer weights associated with these words, expressing how much they contribute towards the class label ◦ Aggregate and use for document-level classification

Bloomberg Finance L.P. All rights reserved. Results • Text-as-data methods require task-specific validation • Content validity: consider type of linguistic information one is measuring ◦ For instance, mapping keywords to a document-level binary label collapses all types of semantic uncertainty, many of which cannot be identified via keywords alone • Sensitivity: strengthen conclusions with multiple measurements and measurement approaches ◦ In this paper, we demonstrate that no tight correlation exist between the keywords-based approach and aggregating outputs of a document classifier

Bloomberg Finance L.P. All rights reserved. FOMC meeting minutes • 15+ pages, dense and technical prose ◦ Rich structure ▪ Numerous sections ▪ Discussions about previous meetings, future outlook, etc. ◦ Designed to influence markets • Our experiment ◦ 177 meeting documents ranging from January 2000 until August 2020 ▪ Includes ad-hoc meetings ◦ Split into ~100k sentences ◦ Manually labelled 1.8k sentences ▪ Hawkish vs dovish vs neutral Distribution of labels Negative 38.2% Neutral 27.4% Positive 34.4%

Bloomberg Finance L.P. All rights reserved. Example sentences from FOMC meeting minutes Sentence Source Comments Regarding the labor market, many participants commented that the pace of employment gains, which was quite strong in May and June, had likely slowed. July 28-29, 2020 Sentiment is governed by the final clause. One of them judged that the low level of inflation compensation could reflect increased concern on the part of investors about adverse outcomes in which low inflation was accompanied by weak economic activity, and that it was important not to dismiss this possible interpretation. March 17-18, 2015 Negation does not invert sentiment. Recent data along with anecdotal reports indicated some loss of vigor in the nation’s housing markets, though overall activity was still at a high level. December 21, 1999 Mixed sentiment.

Bloomberg Finance L.P. All rights reserved. Methods • Dictionary-based: measure the ratio of positive to negative sentiment- bearing words • Linear: multiclass logistic regression ◦ Obtain vocabulary containing all words and word pairs from labeled sentences ◦ Maximize conditional log-likelihood of all meetings in order to infer weights associated with these words and word pairs, expressing how much they contribute towards an overall positive, negative, or neutral sentiment ◦ Final classifier: one model per sentiment class using one-versus-all scheme Neutral Positive Negative committees 1.02 increased 1.12 weakness 1.23 committee 0.93 gains 0.88 weak 1.14 had remained 0.87 strong 0.88 uncertainty 1.10 mixed 0.86 rise 0.86 below 0.82 information 0.78 up 0.82 short 0.78

Bloomberg Finance L.P. All rights reserved. Methods • Dictionary-based: measure the ratio of positive to negative sentiment- bearing words • Linear: multiclass logistic regression ◦ Obtain vocabulary containing all words and word pairs from labeled sentences ◦ Maximize conditional log-likelihood of all meetings in order to infer weights associated with these words and word pairs, expressing how much they contribute towards an overall positive, negative, or neutral sentiment ◦ Final classifier: one model per sentiment class using one-versus-all scheme • Deep Learning: finetune a character-based Transformer, decomposing words into sub-word units and train using labeled sentences

Bloomberg Finance L.P. All rights reserved. Results: F1 score Negative F1 score Neutral F1 score Positive F1 score Dictionary 0.62 0.28 0.43 Linear 0.6 0.41 0.66 Deep Learning 0.72 0.55 0.66 F1 score Chance 0.28 Dictionary 0.44 Linear 0.58 Deep Learning 0.66

Bloomberg Finance L.P. All rights reserved. Results • Information contained in FOMC’s meeting minutes can help forecast the target interest rate • Strongly leads the FOMC’s decision to hike or cut rates by approximately eight meetings or one year • Manual encoding coupled with deep learning performs better than baselines

Bloomberg Finance L.P. All rights reserved. Conclusions • Economic indicators: interpret and predict macroeconomic activity ◦ Used by governments, central banks, and finance professionals ◦ Have real-life impact on everyone in society • Open challenges and the role of AI/NLP ◦ Find accurate and more real-time signals to determine effects of economic policy ◦ How to incorporate shocks? ◦ Deliver value by using NLP to add structure to unstructured data ▪ At scale with high accuracy and, in some cases, with low latency ▪ Link, connect, and identify patterns over entities, topics, and events • Compute and implementations of algorithms are commodity ◦ Finding, cleaning, labeling, and analyzing relevant data to train models is key ◦ Collaborate with experts to train, understand, and operate models and drive (continuous) annotations

Bloomberg Finance L.P. All rights reserved. https://TechAtBloomberg.com/AI https://TechAtBloomberg.com/data-science-research-grant-program/ https://www.bloomberg.com/careers @edgarmeij | [email protected] Thank you

SocInfo22 Keynote: Modelling and Quantifying Un...

SocInfo22 Keynote: Modelling and Quantifying Uncertainty in Economic Indicators

More Decks by Edgar Meij

Other Decks in Technology

Featured

Transcript