Managing Risks with SMEs, Part 2

In part one of the risks discussion, we talked about managing the content risks when a subject expert does their own learning development. In this post, we’ll talk about the other risks &emdash; the costs and long-term implications. As always, I’ll include suggested mitigations.

As a reminder, the previously discussed risks were:

1. The gap between the expert and the learner
2. Time needed to generate and refine content
3. Garbage in, Garbage out

Risk #4: Costs associated with development

It’s important to look at the larger picture when considering solo SME development. How much do your SMEs cost? How much do IDs cost? What are the timelines and workloads when an ID is driving the process? What’s the cost if training is delayed or if the SMEs primary work is delayed?

Chances are you’ll save money by having IDs guide and share work with the SMEs. A single ID can work on multiple projects simultaneously and often work as project managers to keep delivery on track.

But there’s a larger cost looming: the AI charges.

Models (LLM)	Vendor	Input	Output
Gemini Pro	Google	$0.35	$1.05
GPT 4o	OpenAI	$5.00	$15.00
GPT 3.5 Turbo	OpenAI	$0.50	$1.50
Claude 3 Haiku	Anthropic	$0.25	$1.25
Claude 3 Sonnet	Anthropic	$3.00	$15.00
Claude 3 Opus	Anthropic	$15.00	$75.00

Large Language Models bill according to the number of tokens used. (A token is approximately one word.) The above prices are per million tokens.

Those prices look pretty reasonable, with more capable models commanding higher prices. Except they count tokens you’ll never see — intermediate values inside the model; content that’s generated, checked, and regenerated automatically; and costs incurred when searching the web and processing the results. They also include the cost of ingesting uploaded content and using it in prompts (“retrieval augmented generation”).

Even with the more modest models, a day’s work can run into the hundreds of dollars.

The nature of expertise can multiply these charges. AI models don’t connect concepts the way our brains do. So a SME using specialized language may not be able to generate content with more fundamental language, missing the learner’s needs entirely. Likewise, a SME may miss the mark on diagnosing the performance issue and generate generic (and ineffective) content.

Mitigation: Consider using less capable models for prototyping on a budget. Look at the trade-offs between correcting content in a session vs. fixing it by hand. Take advantage of multi-person chat with AI models to do “pair prompting” with an ID or writer (this can improve the results dramatically.)

Risk #5: Deadlines and burnout

Your subject matter experts are some of your most valuable people and need to be available to others on short notice. Training development work can interfere with this critical role and put your schedules at risk.

SMEs doing training development spend a significant proportion of the time on work outside their domain:

1. Prompting the AI,
2. Media-related tasks (scripting, recording, editing reviews, etc.),
3. Aligning content with learner needs,
4. Reviewing evaluation results, and
5. Designing updates.

These SMEs generally enjoy creation, but often get tired of the refinement. Others who care deeply about quality will want to tinker and keep refining. Both make deadlines hard to meet and multiply the costs.

To be fair, an ID can also make the process harder. If they take on content production and it’s not well-researched, the SME can drown in content reviews. You might need to step in and address the balance of the work or manage the content review process.

Burnout can become a major problem. The first course is exciting and new for a few months, but you may have a hard time getting a second one produced. The detailed work of creating enduring content (such as on video) is a long process and foreign skill set. SMEs often don’t want that kind of visibility; you don’t want to push them.

This is the right place for a professional to step in. As a training program manager and ID, I’ve helped my SMEs develop design skills, drafted their content outlines, took on managing the video team, and talked more than one SME out of giving up (by helping them see the bigger picture).

If the work goes off-track, you may be tempted to add people. But doing this mid-process generally stretches the timeline. It’s much better to start with the ID consulting with the SME and having each person work within their expertise

Mitigations: Wrap your SMEs in support. At a minimum, there should be a project manager who can solve blocking issues. Professionals who have the expertise to spot issues and fix them should do media production, such as in videos. An ID should be available to help scope the work, offload the logistics of reviews and evaluation, and help the SME see the big picture.

Risk 6: The maintenance burden

Every learning tool eventually needs revisions or becomes obsolete. Does this leave the SME responsible for content maintenance? In the short run, there are always little factual corrections to be made and re-reviewed; each edit can take a disproportionate amount of time.

There’s also the question of who keeps the LMS/LRS up to date, if one is in use.

Without the usual ID artifacts, this work will be difficult to delegate or pass to a new person when the SME leaves. (It’s not always easy even with the ID artifacts as the new developer needs to internalize the original design.)

Making the SME responsible often shortens the effective lifetime substantially, as revision now leads to a complete rework. It may look like a cost savings now, but you need to look at the long-term picture.

Mitigations: Releasing a learning intervention must include solid documentation for future developers. If an ID is involved, they could produce this package, otherwise you may need to engage a good writer. Beware doing this part with AI; accuracy and clarity are paramount.

Next time

In the next post, we’ll look at some guidelines for managing a successful engagement.