Why Generic Embeddings Fail on Maintenance Work Orders in RAG Pipelines

A technician asks the maintenance assistant about a bearing problem on pump P-2201A. The retrieval step returns notes about a different pump, a procedure for a similar-sounding fault, and a work order where "brg" means something else entirely. The language model then writes a fluent, confident answer from the wrong context.

That failure usually starts before the model writes a word. In a retrieval-augmented generation (RAG) pipeline, the answer can only be as good as the passages retrieved, and general-purpose embedding models were not built for maintenance records. This article covers why they struggle, how to structure and retrieve work orders properly, when fine-tuning is worth it, and how to measure retrieval so problems are caught before they reach a technician.

What general-purpose embeddings see in your work orders

Embedding models map text into vectors so that similar meanings land close together. They learn "similar" from the text they were trained on, which is mostly general web and document text. Maintenance records look very different:

Research on retrieval benchmarks supports caution. The BEIR benchmark, which tests retrieval models across many domains without in-domain training, found BM25 keyword retrieval to be a strong baseline, while dense and sparse retrieval models were computationally more efficient but often underperformed other approaches, showing considerable room for improvement in generalization ^[1]. Maintenance records are about as far out of domain as it gets.

Chunk by the structure of the record

Most RAG tutorials split documents into fixed token windows. That works poorly for work orders, because a fixed split can separate the asset from the symptom, or the symptom from the fix.

Chunk by logical section instead:

• One chunk per work order when the record is short, with the asset ID, failure code, symptom, cause, action, and outcome together.
• Section chunks for long records: problem description, diagnosis, corrective action, and close-out notes as separate chunks that all carry the asset ID and work order number as metadata.
• Procedures by step group, keeping safety steps with the task they belong to.
• Metadata on every chunk: asset ID, asset class, site, failure code, date, and source system, so retrieval can filter before it ranks.

Expand abbreviations into a normalized field during ingestion, using your own glossary, while keeping the original text. That gives both keyword and vector retrieval more to work with.

Combine keyword and vector retrieval

Vectors are good at meaning and bad at exact strings. Keyword search is the reverse. Asset tags, part numbers, and codes need exact matching; symptoms described in different words need semantic matching.

Run both and merge the results. Reciprocal rank fusion is a method for combining multiple result sets with different relevance indicators into a single result set ^[2], which suits merging a keyword ranking with a vector ranking. Add metadata filters on top: if the query names P-2201A, restrict or boost results for that asset before ranking.

Fine-tune embeddings when you can prove the gain

Fine-tuning teaches an embedding model your plant's notion of similarity. The Sentence Transformers documentation explains why this matters: each task requires a different notion of similarity, and training data is typically pairs or triplets, such as an anchor with a positive and a negative example ^[3].

For maintenance, useful training pairs come from your own records:

• Query to work order: a technician question paired with the work order that answered it.
• Symptom to resolution: the problem description paired with the corrective action.
• Hard negatives: a similar-looking record about a different asset or failure mode.

Do not fine-tune blind. Build the retrieval test set first (next section), measure the baseline, fine-tune, and keep the new model only if retrieval improves on held-out queries. Re-run the test whenever the model, the chunking, or the glossary changes.

Measure retrieval before it poisons agent reasoning

Retrieval quality should be measured on its own, before any language model is involved:

1. Build a test set of real technician questions, each labelled with the work orders or procedures that correctly answer it. Include questions with asset tags, abbreviations, and site-specific names. 2. Hold it out. Never train on it. Google's Rules of Machine Learning recommend sanity checks right before exporting a model, specifically making sure performance is reasonable on held-out data ^[4]. 3. Track standard metrics: whether a correct record appears in the top results (recall at k), how high the first correct result ranks (mean reciprocal rank), and how many top results are relevant (precision at k). 4. Slice the results by asset class, site, and query type, so a failure on one class is not hidden by an average. 5. Review failures by hand with a reliability engineer, and feed the fixes back into the glossary, chunking, or training pairs.

Architecture decisions that compound or collapse

The NIST AI Risk Management Framework is intended to help organizations incorporate trustworthiness considerations into the design, development, use, and evaluation of AI systems ^[5]. For a maintenance assistant, that means showing technicians the source records behind every answer, logging which records were retrieved, and giving them a fast way to flag a wrong result. For agent pipelines built on top of retrieval, see our article on multi-agent orchestration in production.

Frequently asked questions

Can I skip fine-tuning and just normalize abbreviations?

Often that is the right first step. Normalizing abbreviations and adding hybrid keyword and vector retrieval can fix many problems. Fine-tune when your test set shows a remaining gap.

How many work orders do I need before fine-tuning is worth it?

Enough to build a held-out test set and a separate training set of query-answer pairs with hard negatives. Measure the baseline first and let the test set decide.

Should I use a maintenance-specific language model instead?

The language model cannot fix bad retrieval. Improve chunking, hybrid search, and embeddings first, then evaluate whether a different generator helps.

How do we keep retrieval accurate as records change?

Re-index new and closed work orders on a schedule, update the glossary when technicians introduce new shorthand, and re-run the held-out test set after every change to chunking, embeddings, or ranking. Add the questions technicians flag as wrong to the test set, so the same failure cannot quietly return. Treat the test set as a living asset owned by the reliability team, not a one-time project artifact.

Your 30-day retrieval quality improvement plan

Week 1: collect real technician questions and label the correct records to build a held-out test set. Measure the current pipeline.

Week 2: rechunk work orders by logical section, attach asset and code metadata, and add an abbreviation glossary. Re-measure.

Week 3: add keyword retrieval and merge it with vector results using reciprocal rank fusion, with asset filters. Re-measure.

Week 4: if a gap remains, fine-tune embeddings on query-record pairs with hard negatives, and keep the new model only if the held-out metrics improve. For the data foundation behind all of this, see our CMMS integration guide.

References

[1] Thakur et al., "BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models," arXiv:2104.08663. https://arxiv.org/abs/2104.08663

[2] Elastic, "Reciprocal rank fusion." https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion

[3] Sentence Transformers, "Training Overview." https://sbert.net/docs/sentence_transformer/training_overview.html

[4] Google for Developers, "Rules of Machine Learning." https://developers.google.com/machine-learning/guides/rules-of-ml?hl=en

[5] NIST, "AI Risk Management Framework." https://www.nist.gov/itl/ai-risk-management-framework