Do LLMs Know How Many Personas to Generate?​ A Benchmarking Study in Customer Segmentation​

Transcript

1. Do LLMs Know How Many Personas to Generate? Same customer data, two answers Illustrative example Clustering: 3 segments A Benchmarking Study in Customer Segmentation Danial Amin1,2, Joni Salminen1, Bernard J. Jansen 3,4 1 University of Vaasa, Finland 3 Peking University, China 2 Otherkind Tech, Finland 4 Nankai University, China LLM: 5 personas Persona matches a segment No matching segment: 2 extra
2. Introduction Method Results Discussion Personas and persona sets Persona Persona set A fictional person who represents a customer segment, with attributes drawn from the underlying data. A collection of personas representing different segments of a population. The set size, denoted by k, determines how many personas are included. Age Spending Channel k=5 One segment, one profile Cooper, The Inmates Are Running the Asylum, 1999 Set size k
3. Introduction Method Results Discussion The number of personas influences who gets represented Participants who were shown 15 personas engaged with a wider variety of groups than those shown 5 personas, although they spent less time on each persona. 5 personas Salminen et al., International Journal of Human-Computer Studies, 2022 15 personas
4. Introduction Method Results Discussion LLMs now generate personas from customer data However, little is known about how LLMs determine the number of personas to generate and whether that number reflects the underlying customer segments. k=? LLM Customer data Jung et al., IJHCS, 2025; Amin et al., CHI 2026 Personas
5. Introduction Method Results Discussion Three research questions How many personas do LLMs generate? How closely do the generated persona attributes match the underlying data? How do diversity instructions affect the generated personas?
6. Introduction Method Results Discussion Public customer segmentation data 29 datasets were selected from 90 candidates. The datasets cover retail, banking, telecommunications, insurance, and other sectors. Domains Demographic attributes Behavioral attributes Sectors in the 90 candidates Required in every dataset Required in every dataset Retail Banking Age Purchase frequency Telecom Insurance Gender Product category Automobile Fashion Income Service usage Domain, row count and column count for each dataset are in the online appendix (osf.io/dchu9) · Amin, Salminen and Jansen, AI CCSA 2026
7. Introduction Method Results Discussion Clustering provides the benchmark For each dataset, we obtained a reference set size (k*) and its corresponding cluster centroids. 7 3 ≥2 k* clustering algorithms validity indices algorithms agree on k stability check reference set size from five families two of three must agree by majority vote clusters hold when refitted on resampled data plus cluster centroids
8. Introduction Method Results Discussion Nine prompts across three models We varied the information provided to the models and the instructions for generating diverse personas. Context Example prompt (illustrative) A Dataset of 2,000 retail customers: age, gender, income and purchase frequency. B Age: mean 41, SD 12. Income: median $54K. Female: 52%. C Suggested number of personas: 4 D+ Maximise differences between personas. A Dataset description B Description with summary statistics C Description with statistics and k* Diversity instruction No diversity instruction Maximise differences between personas Generate personas representing extreme segments Generate customer personas for this population. Across 29 datasets, we conducted 783 runs. Shown: condition C with the diversity instruction
9. Introduction Method Results Discussion Three evaluation measures 1 2 Set size deviation k* Measures the difference between the number of generated personas and the reference k*. LLM Attribute fidelity Measures how closely the generated persona attributes match the nearest cluster centroid, ranked within each dataset. 3 +2 centroid persona distance Cluster coverage Measures the proportion of reference clusters represented by at least one generated persona. 3 of 4 clusters covered = 75%
10. Introduction Method Results Discussion LLMs generate more personas than clustering suggests +1.55 Across all models and experimental conditions, LLMs generated an average of 1.55 more personas than the clustering benchmark suggested. k* LLMs +1.55 Amin, Salminen and Jansen, AICCSA 2026
11. Introduction Method Results Discussion Claude Sonnet overshoots the most, while DeepSeek overshoots the least Under description-only prompts, Claude Sonnet generated the largest number of additional personas, while DeepSeek remained closest to the reference set size. None +5.66 Diverse Extreme +6.14 +3.00 +2.62 +2.69 +0.83 0.00 +0.79 −0.10 Claude Sonnet-4.6 GPT-5.4 Personas above k*, Condition A · Amin, Salminen and Jansen, AICCSA 2026 DeepSeek-v4-pro
12. Introduction Method Results Discussion Providing a reference set size aligns the counts exactly Adding summary statistics did not substantially change the number of personas generated. However, when the models were given k* as a reference, all three generated exactly that number of personas. k* +2.40 +2.24 0.00 Description only With summary statistics With k* reference Mean deviation from k* · Amin, Salminen and Jansen, AICCSA 2026
13. Introduction Method Results Discussion LLMs tend to generate persona sets within a narrow range −0.93 Personas generated LLM set sizes Across the 29 datasets, we found a strong negative correlation between k* and the amount of overgeneration (Spearman ρ = −0.93). As the reference set size increased, the models generated fewer additional personas relative to k*. overshoot Schematic Amin, Salminen and Jansen, AICCSA 2026 = k* k*, small to large
14. Introduction Method Results Discussion Additional context improves cluster coverage 88% Claude Sonnet covered 88% of the reference clusters when provided with summary statistics and an instruction to generate diverse personas. However, providing k* as an additional reference did not consistently improve attribute fidelity. Amin, Salminen and Jansen, AICCSA 2026
15. Introduction Method Results Discussion Generating extreme personas reduces attribute fidelity Diversity instructions increased the number of personas generated, but the instruction to generate extreme personas produced the lowest attribute fidelity. The general diversity instruction showed a smaller difference in fidelity compared with using no diversity instruction. Extra personas Fidelity rank +2.03 2.74 +1.91 1.55 1.70 None Diverse +0.71 None Diverse Extreme Instruction Rank 1 is closest to the data · Amin, Salminen and Jansen, AICCSA 2026 Extreme
16. Introduction Method Results Discussion Generating twenty personas requires little more effort than generating five Although LLMs make it easy to generate larger persona sets, teams still need to read, compare, and prioritise every persona. As the number of personas increases, the effort required to review and use them also increases. Generation cost 5 personas 20 personas Amin, Salminen and Jansen, AICCSA 2026; Salminen et al., IJHCS, 2022 Reading and prioritising
17. Introduction Method Results Discussion Five guidelines for generating personas 1. Provide a data-derived reference for the number of personas to generate. 2. Include summary statistics describing the underlying population. 3. Use a general diversity instruction when broader segment coverage is needed. 4. Avoid asking for extreme personas unless the task specifically requires them. 5. Test different models against your own data before selecting one. Amin, Salminen and Jansen, AICCSA 2026
18. Introduction Method Results Discussion Limitations The study used cleaned, publicly available datasets sourced from Kaggle, which may differ from real-world customer data. Our fidelity measurements depend on matching the generated attributes to the original variable names. The reference k* represents one defensible segmentation, although other clustering approaches may produce different results. We have not yet evaluated whether human users or domain experts consider the generated personas realistic or useful.
19. Thanks! Questions? The number of personas should be grounded in the data. Danial Amin University of Vaasa, Finland danial.amin@uwasa.fi academic.danialamin.com Data, prompts and outputs doi.org/10.17605/OSF.IO/DCHU9 Persona Team's research personateam.xyz PS. Stay tuned for the updated GenAI persona book in 2027!