Sophistication in GenAI Use: Field Evidence from a Large Firm

Noosaga Placements

• Artificial Intelligencesubfield90%
  The paper studies generative AI use in a workplace setting, focusing on how employees interact with LLMs, which is a core topic in artificial intelligence.
  We study how sophistication in generative AI (genAI) use varies among the back-office workforce of a large firm.Using proprietary data, we observe 713,564 employee prompts and their corresponding large language model responses.
• Neural NLPframework80%
  The study uses LLMs (neural NLP models) to analyze employee conversations and measure sophistication.
  To measure genAI sophistication, we prompt an LLM to classify each conversation by use case and assess the clarity and specificity of its instructions and the prompting techniques employed.
• Natural Language Processingsubfield85%
  The study uses LLMs to classify employee conversations and assess prompting techniques, which are central to NLP applications.
  To measure genAI sophistication, we prompt an LLM to classify each conversation by use case and assess the clarity and specificity of its instructions and the prompting techniques employed.The paper includes detailed metaprompts for LLM conversation analysis.
• Attention Mechanisms and Transformersframework50%
  The LLMs used for classification are based on transformer architectures, though the paper does not focus on the underlying model architecture.
  an enterprise-specific chat tool that provided access to commercial large language models (LLMs) such as those from Anthropic and OpenAI.
• Machine Learningsubfield60%
  The study uses LLMs for classification, which is a machine learning application, but the focus is on empirical analysis of usage rather than developing new ML methods.
  We prompt an LLM to classify each conversation by use case and assess the clarity and specificity of its instructions.

Abstract

We study how sophistication in generative AI (genAI) use varies among the back-office workforce of a large firm. Using proprietary data, we observe 713,564 employee prompts and their corresponding large language model responses from nearly 4,000 back-office employees across 15 functional areas over eight months in 2025. We document three main findings. First, senior employees exhibit more sophisticated genAI use, consistent with domain expertise complementing genAI capabilities. Second, sophistication varies considerably across functions and is highest in Strategy, Digital Innovation, and Project Management, three groups that share a focus on firmwide strategic initiatives and organizational change. Third, we observe neither improvements in sophistication over time nor lasting improvements following formal AI training, suggesting that sophisticated use can be difficult to change. Together, our study provides measures of and insights into sophisticated genAI use that managers can use to improve outcomes and that researchers can use in future research.

Paper Context