Fine, When I’m Using It: What Your First AI Pilot Really Tests

Legal Technology

Nicolas Charalambous gave his firm’s first AI pilot to a capable associate and asked her to see whether it was useful. Seven weeks later he could not say whether it had worked, and the reason had nothing to do with the software.

In the spring, Nicolas Charalambous came back from a conference in Athens having decided that his firm should try AI properly rather than keep talking about it. The firm is in Limassol, eleven lawyers, and a good part of its work is people buying apartments in the developments between Germasogeia and Agios Tychonas. He picked the plainest thing in the building, the engagement letter that goes out to every new purchaser, and asked Ioanna Michaelidou, four years qualified and the person most at ease with new tools, to see whether Claude could draft it from the template and the client’s first email.

That was the whole brief. “Have a play with it and tell me if it’s useful.” He meant it kindly, and on the face of it he was doing what the advice says.

1The choice was right

It was a good choice. An engagement letter for an apartment purchase is repeatable: the firm sends about eleven a month, and they differ in the names, the property, the price and the fee, and little else. It needs almost no judgement on the day, because the judgement was done when the template was written. Nothing in it is sensitive beyond what the client has already put in their own email. And it has a number attached. Ioanna timed herself before she started: thirty-five minutes to find the last letter, change what needed changing, check the fee scale and the money-laundering paragraph, and send it up for signature. Partner review took another ten.

If a pilot has to live somewhere, this is the kind of place: frequent, dull, harmless and countable, like the uses I set out earlier for Claude in a Cyprus law firm.

2Ten weeks

In week one Ioanna set it up on the account the firm had approved for the purpose: the template, the client’s email, and a short instruction of her own about what to fill in and what to leave alone. The first draft needed two corrections, both things she would have caught herself. Three letters went out that week, all three through the tool.

Week two was four of four. She showed Katerina Neophytou, the office manager, and Michalis Konstantinou, two years qualified, who sits on Stelios Kyriakou’s team. People stopped by her desk to look.

In week three Stelios said, in the Monday meeting, that he would keep doing his own letters the old way for now. His purchasers often buy two or three units, his fees are negotiated, and he wanted to be sure. Nobody argued, because it was a reasonable thing to say. Michalis, who sits opposite him, drew the obvious conclusion. One letter of three.

Week four was two of three. In week five Ioanna was in court in Nicosia most of the week on a partition dispute, and nobody else had her instruction, because nobody had been asked to cover. One of four. Week six was none of two.

In week seven Nicolas asked her, in passing, how the pilot was going. She said, “Fine, when I’m using it.” It was an honest answer that told him nothing, which is when he noticed he had no other way of finding out.

3What the numbers said

He asked Katerina to count four things any firm can pull from its own files: letters sent, letters through the tool, the signing partner’s minutes on each, and how many came back for changes.

Nineteen letters in six weeks. Eleven through the tool. Fifty-eight per cent, which sounds respectable until you see the shape of it, peaking in week two and touching zero in week six. Partner review had gone up, twelve minutes against the old ten, because partners read the new drafts more carefully, the right instinct, and a cost nobody had budgeted for. And four of the eleven drafts had come back for changes, three of them because of the template rather than the tool. There were, it turned out, three versions of the engagement letter in circulation: Nicolas’s, one of Stelios’s from 2019 with a different fee schedule, and one Katerina had tidied up that nobody had ever approved. The tool had drafted faithfully from whichever it was given.

Then there were the eight letters that bypassed it altogether.

Three were on Stelios’s team, two went out in the week Ioanna was in court, two were drafted by hand after the template mistake because the associate no longer trusted what was coming back, which was fair, and one was simply forgotten.

4Everyone was right

Stelios had good reasons. His letters genuinely differ, he has the largest book in the firm, and he is the one answerable if a purchaser of three units in Agios Tychonas is sent the wrong fee. What he could not see was that a partner staying out of a pilot is never a private decision. To Michalis it read as a verdict, because in most firms the safest thing to copy is the partner opposite you.

Ioanna did what she was asked and did it well, while carrying a full billable target with the pilot placed on top of it, unwritten, with no hour in the week that belonged to it. When court came, the pilot went, because it was the only thing in her week nobody was going to ask about.

And Nicolas had followed the delegation advice: hand it to someone capable, then step back. I have written before about partners who hand over the task and keep the authority. This was a different mistake. He had handed over a curiosity rather than a job, and a curiosity has no owner in any sense that survives a busy fortnight.

5The moment it actually turned

Ask anyone in the firm when the pilot went wrong and they will say week three, or week five. It was earlier than that. It was the afternoon in week one when Nicolas said, “Have a play with it.” A pilot with no written workflow, no owner with time to own it, no agreed measure and one partner outside it is a hobby, and hobbies stop when the diary fills. Weeks three to six had already been decided in that sentence.

6What the pilot revealed about the firm

This is the useful part. In ten weeks, for the price of a subscription and some of Ioanna’s evenings, the firm got a stress test of how it was run, and found four things a consultant would have taken a month to tell them: three templates and none of them the firm’s, a task assigned but no job, no protected time, and a partner who had exempted himself, which told everyone else what was optional. All four existed before the tool arrived, and any change would have surfaced them, a new billing system as surely as a new tool. That is why I tell firms to get delegation right before automation. If you cannot hand a piece of work to a person with a written standard and a named owner, you cannot hand it to software either. A firm capped by what one desk can personally supervise is capped in its technology by the same desk.

Most of Ioanna’s thirty-five minutes sat in the third row, the only row the tool was ever going to touch. The rows either side of it had never been settled.

7What Nicolas changed

In week eight the three partners spent ninety minutes on the engagement letter, longer than they had spent on it in the previous six years put together, and came out with one template. Stelios’s variations became a fee schedule at the back, which he accepted once he saw nothing of his had been lost. The draft became Katerina’s job at intake; Ioanna or Michalis reviews it; a partner signs. Two hours on Friday morning now belong to the pilot, in Ioanna’s diary, and Katerina puts three numbers on one page for the Monday meeting: letters sent, letters through the tool, letters sent back. Stelios put his next three through himself.

Weeks nine and ten were nine letters, nine through the tool. Partner review settled at seven minutes, because the drafts now came from a template the partners trusted. One letter came back for changes. Nobody has yet claimed a figure for hours saved, and Nicolas has asked that nobody does until the numbers have held for a quarter.

Three habits came out of it.

Write the workflow down before you point a tool at it. One page: who does what, in what order, from which template. If the page cannot be written, that is the finding, and it is worth more than the pilot.

Count use before you count savings. The share of matters where the tool was used, the review minutes, the rework, and who stopped using it and in which week. “Hours saved” is a claim. “Nine of nine” is a fact, and one you can act on when it turns into six of nine.

Partners go first, or at least go. A partner who stays out is making a statement whether he means to or not. The people sitting near him will conclude that the pilot is optional, and they will be right.

The software at Nicolas’s firm was fine. It was always going to be fine. What the ten weeks tested was whether the firm had agreed what its own letter said, who owned it, and how it would know. Your first pilot will tell you very little about AI, and a great deal about how your firm is run.

Sources & notes. Illustrative example: Nicolas Charalambous, Ioanna Michaelidou, Stelios Kyriakou, Michalis Konstantinou, Katerina Neophytou, the firm, the developments and all figures and dates are invented, and no real client matter is described. The management ideas (delegation before automation, taking a piece of legal work apart into judgement, knowledge, repeatable and administrative elements, and controlled pilots with honest adoption measures) are drawn from my own programme material for law firm leaders and from advisory practice, and are presented as a working model rather than a research finding. Claude is named because it is the tool I work with; nothing here is a claim about its features, and in the example every draft is reviewed by a lawyer before it leaves the firm. The wider pattern of firms capped by one desk is examined in The Bottleneck Partner.