themisf.it

The case study, part two: tinkering and exploration.

Picking up the firm from the previous post. Twelve months have passed since the executive team meeting where the AI conversation produced no commitments. The Managing Partner is still asking about strategy in general terms. Marcus is still drafting responses that satisfy without committing. The practice leaders are still pressing on billable utilization. By external measures, very little has changed.

What's changed, hidden from the leadership team's view, is the volume of individual AI use across the firm. Senior associates are using ChatGPT on personal accounts to summarize tax code. Audit staff are pasting redacted snippets of workpapers into chat interfaces to ask interpretation questions. Advisory consultants are drafting first-pass client memos with AI assistance and then editing heavily before delivering. The use is widespread, individual, ad hoc, and almost entirely invisible to the firm's IT controls because almost all of it is happening on personal devices on personal accounts.

This is the substrate against which the firm is about to be forced into the next stage. The trigger, when it arrives, is not strategic. It's an incident.


A senior associate in audit, working a Saturday on a client engagement, pastes a section of the client's general ledger into a personal ChatGPT account to ask the model to help identify unusual entries. The session is innocuous in intent. The associate is trying to do good work faster. She closes the browser window when she's done and doesn't think about it again.

A client compliance officer at the same engagement, doing routine procurement diligence the following week, asks the firm whether any AI tools were used in the audit process. The standard firm answer, which Marcus has carefully crafted over the prior year, is that the firm does not use generative AI in client work and that all AI-related activity is restricted to approved tools and approved purposes. The partner on the engagement gives that answer. The compliance officer asks for confirmation in writing.

The partner forwards the request to Marcus with a note that says some version of "please confirm we're clean on this." Marcus is, in his own assessment, probably not clean on this. He has no visibility into what individual staff are doing on personal devices. He cannot confirm in writing that no AI has touched any client work product, because he doesn't actually know.

Marcus walks to Dana's office. They have the conversation neither of them has wanted to have. They estimate, between them, that probably half the staff has used AI on some piece of work in the last quarter, and that they have no way to put a number on what data has gone where. They also estimate that putting that estimate into writing to a client would be catastrophic.

The conversation that follows has two layers. The surface layer is the legal and reputational exposure. The deeper layer is the realization that the firm has been pretending to be at stage zero on AI when it has actually been at stage two all along, unsupervised. The pretense is now untenable.


The next two weeks are the firm's first real AI work, even though almost none of it is technical.

Marcus and Linda draft a temporary policy that acknowledges what's happening. It prohibits putting client data into any AI tool that the firm hasn't approved, on any device the firm hasn't approved, period. It also acknowledges, for the first time in writing, that the firm will be evaluating and approving specific AI tools for specific purposes in the coming quarter. The policy is brief. Dana signs off. The Managing Partner signs off after asking two clarifying questions, neither of which is about the technology.

Marcus then does something he should have done a year ago. He sends a survey to all staff asking what AI tools they have used in the last six months, for what purposes, and on what kind of work. The survey is anonymous and explicitly amnesty-based. The response rate surprises him: 78%. The aggregate picture is what he already suspected and what the firm's leadership had not previously imagined. Approximately 60% of respondents have used a generative AI tool for some work-related task in the prior six months. About a third have used such a tool on something that touched client data. The most common tools are ChatGPT and Claude. About 15% of respondents indicate they have used AI to draft something that was delivered to a client without disclosure of the AI's involvement.

Marcus presents the survey results to the executive team. The meeting is uncomfortable in a productive way. The Managing Partner asks whether the firm should fire the 15%. Dana asks whether the firm should fire the staff who didn't use AI, on the theory that they're going to be slower than competitors who do. Linda points out that both reactions are the wrong frame. The right frame is that the firm has been in stage two for at least a year, has had no visibility into it, has accumulated unknown exposure during that time, and now needs to decide whether to suppress the activity or formalize it.

The executive team chooses to formalize it. Not because they have a strategic conviction about AI capability, but because suppression would require the kind of enforcement infrastructure they don't have and don't want to build.


The firm enters tinkering and exploration deliberately, by recognizing what was already happening.

Marcus negotiates an enterprise license with a major AI vendor. Linda partners with him on a usage policy that distinguishes between approved tools (firm-licensed accounts, firm-controlled data flows) and unapproved tools (personal accounts, no client data). The policy is launched with a firm-wide training session that includes an honest acknowledgment from the Managing Partner that the firm has been behind on this and is now catching up. The training is well-attended. The Q&A is animated. Several senior associates ask whether they're now allowed to do things they had been quietly doing for months. The answer, in most cases, is yes, with the caveat that the firm-licensed tool is the one they're supposed to use.

Usage takes off. Within ninety days, the firm-licensed tool has roughly 70% of staff actively using it. The usage is varied: drafting, summarization, code generation in the small IT team, research assistance across all three practice areas. Most of the use is what the diagnostic post called tinkering, people getting comfortable with the tool, building a feel for what it can and can't do. Some of it is starting to become what the diagnostic post called exploration, staff identifying specific tasks where AI clearly helps and starting to use it deliberately for those tasks.

Marcus tracks usage metrics. The numbers look good: seats deployed, queries run, daily actives. He shares the metrics with the executive team monthly. The executive team interprets the metrics as evidence that the firm is making real progress. Marcus knows the metrics are largely meaningless. The firm is doing what individuals were already doing, just visibly and with better data handling. No business problem has been solved. No process has been redesigned. No outcome has been changed.


A few vantages on the firm at this stage:

A Wednesday afternoon in the audit practice. A senior associate is reviewing a quarterly close package. She drafts a memo describing the unusual items she's flagged. She uses the firm-licensed AI tool to help structure the memo and to suggest standard language for several recurring item types. The draft takes her forty minutes instead of the ninety it used to take. She delivers the memo to the engagement manager, who reviews it and asks for two minor revisions. The memo is filed. The work is better than her previous unaided drafts, in her own judgment, and meaningfully faster. None of this is captured in any firm metric. The forty-five minutes she saved is absorbed into her billable utilization without being attributed to AI. She does not think to mention the saving to her practice leader.

The advisory practice leader's experience. He has personally used the AI tool for about a month. His use is mostly exploratory: pasting in a section of a client deliverable and asking the model to critique it, generating bullet outlines for a presentation, drafting talking points. He finds the tool useful but doesn't know what to do with the finding. The leap from "this is useful in my own workflow" to "this should change how the practice operates" is one he hasn't yet considered making.

Linda's evolving frustration. The workforce conversation she tried to start a year ago has not yet happened. She continues to track external research on how AI is reshaping professional services roles. She has started having quiet one-on-one conversations with the practice leaders about what's happening with their associates' workflows. The conversations are useful but not strategic. The firm has no view on what work it should expect senior associates to be doing in three years versus today. The absence of that view is, in Linda's read, the biggest thing currently missing from the firm's AI program. She has tried to bring this to the executive team twice. Both times it has been received as an interesting concern that the firm should address eventually. Eventually has not arrived.

The Managing Partner's confidence. He has started referencing the firm's AI program in client conversations. The references are vague but plausible. He believes, based on the usage metrics Marcus shares, that the firm has made significant progress on AI in the last year. He is also genuinely impressed by what the licensed tool can do when he uses it himself. He has not connected the dots between "individuals are using the tool" and "the firm has not actually changed how it operates," because no one has surfaced the distinction in language he would have to engage with.


The firm is now legibly at stage two. The licensing is in place. The policy is in place. The usage is real and growing. The leadership team believes the firm is making good progress. The metrics being tracked tell a story of adoption.

What the firm has not done, and does not yet realize it has not done, is identify any specific business problem it intends to solve with AI. The licensing was a response to exposure, not to opportunity. The policy was a response to risk, not to strategy. The training was a response to the policy, not to a redesigned operating model. The usage is genuine but undirected. The firm has a tool. The firm does not have a plan.

This is the most common form that stage two takes in mid-market organizations. The tinkering is real. The exploration is starting. But the activity is happening at the individual level, the metrics being tracked are licensing metrics rather than outcome metrics, and the leadership team is confusing visible adoption with operational change. The firm could remain at stage two indefinitely. Many do.

What moves the firm to the next stage, when it eventually moves, is not the AI program. It's a practice leader who gets impatient with a specific bottleneck and starts asking specific questions. That's the next post.

— Chris