What is Named Entity Recognition (NER)? How NER Models Work

A practitioner’s explanation of named entity recognition: what counts as an entity, how NER models move from tokens to labeled spans, how BIO tagging works, how NER is scored, and why annotation guidelines and inter-annotator agreement decide how well a model performs in production.

Named entity recognition is the NLP task of finding spans of text that refer to named things, such as people, organizations, locations, dates and monetary amounts, and assigning each span a category. It is how a sentence becomes a set of structured records that a database can hold and an application can query.

That definition fits on a slide. Building a NER system that works on your documents does not, and the part that decides whether it works is rarely the model.

Key takeaways

• A NER model does not predict entities. It predicts a label for every token, then reassembles those labels into spans.
• Two decisions dominate outcomes: the entity ontology, and where annotators agree an entity starts and ends.
• Published F1 scores are only comparable when the span matching rule is stated. Strict and partial matching produce different numbers from the same predictions.
• Even the reference datasets contain labeling errors. Researchers found mistakes in roughly 5.4% of CoNLL-2003 test sentences.
• Prompting an LLM is not yet a substitute for a fine-tuned model on production extraction, though it is a good pre-labeling tool.
• Inter-annotator agreement predicts model quality better than any other project metric available before training starts.

What is named entity recognition?

Named entity recognition, also called entity extraction or entity chunking, is a sub-task of information extraction. It locates mentions of named things in unstructured text and classifies each one into a predefined category.

Take a sentence:

Maria Alvarez joined Siemens in Munich on 3 March 2024 to lead a €40 million program.

A NER model with a standard entity set returns:

Those five rows can go into a table. The original sentence could not. That transformation is the whole point of the task, and it is why NER sits underneath search ranking, document routing, compliance screening, knowledge graph construction and most document AI pipelines.

The task is not new. The Message Understanding Conference introduced it as a formal evaluation in 1995, with a seven-type schema over Wall Street Journal text. Everything since has changed the method rather than the objective.

What counts as a named entity

This is where most projects have their first real argument.

A named entity is a rigid designator: a reference that picks out one specific thing regardless of context. Proper names qualify. So do certain fixed references such as dates and monetary amounts, which is why most schemas include them even though they are not strictly names.

What does not qualify:

• Pronouns. “She” refers to a person but does not name one. Resolving it is coreference resolution, a separate task.
• Descriptive references. “The CEO” picks out someone by a property, not a name.
• Category nouns. “Bank” names a kind of institution, not an institution.

Those exclusions sound tidy until you hit real documents. “The Fed” is a nickname, not a proper name, and every financial NLP team includes it anyway. “CEO Tim Cook” raises the question of whether the title belongs inside the PERSON span. “Bank of America Corporation” contains “America”, which is a location on its own and part of an organization name here.

None of these have a correct answer in the abstract. They have a correct answer in your guideline document, and that is the only place the answer exists. Two skilled annotators disagreeing on a boundary is a guideline failure, not an annotator failure.

Entity types and where they come from

An entity schema, or ontology, is the list of categories your model can output. It is usually inherited from a benchmark or designed for a domain.

Two things are worth noticing in that table.

The MISC class in CoNLL-2003 is a catch-all for entities that do not fit the other three. It is the class models score worst on, for the obvious reason that it is defined by exclusion rather than by anything the model can learn a pattern for.

And OntoNotes has four and a half times as many types as CoNLL-2003, which means a far larger label space, more ambiguity between neighboring types, and correspondingly more annotation effort per document. Type count is a cost driver, not a free upgrade.

For domain work, design the ontology before anything else and design it slowly. A badly structured ontology cannot be patched halfway through a project. It has to be redone, and redoing it means re-annotating everything labeled so far. In our experience that is the single most expensive failure mode in annotation work.

How NER models work, step by step

A modern NER model runs five stages between raw text and labeled output.

• Tokenization. The text is split into tokens. Transformer models use subword tokenization, so “Alvarez” might become two or three pieces. Entity boundaries have to be tracked back to the original character offsets, which is a routine source of off-by-one bugs in production pipelines.
• Contextual encoding. Each token is converted into a vector that depends on the tokens around it. This is what makes modern NER work. The word “Jordan” in “Jordan scored 40 points” and “flights to Jordan” produces different vectors, so the model can type them differently without any explicit rule.
• Token classification. A classification layer assigns each token a label from the tag set. This is the step that gets misdescribed everywhere else. The model is not predicting entities. It is predicting one label per token, and it has no direct concept of a span.
• Sequence decoding. The per-token labels are combined into a valid sequence. A CRF layer, or constrained decoding, prevents illegal transitions such as an inside-tag appearing without a beginning-tag before it. Skip this and models emit fragmentary spans that break downstream code.
• Span reconstruction and post-processing. Consecutive labels are collapsed back into spans with character offsets. Post-processing handles normalization, gazetteer overrides for known entity lists, and any business rules the application needs.

The stages that matter for accuracy are two and three. The stages that generate support tickets are one and five.

How tagging schemes turn tokens into spans

Because the model works token by token, the tag set has to encode position as well as type. Otherwise “Bank of America” and “Bank” followed by “America” produce identical labels.

BIO tagging, also called IOB2, solves this with three prefixes. B- marks the first token of an entity, I- marks subsequent tokens of the same entity, and O marks everything outside an entity.

Applied to a sentence:

Two adjacent entities of the same type stay separate because the second one starts with B- rather than continuing with I-.

BIOES adds two more prefixes: E- for the last token of a multi-token entity and S- for single-token entities.

BIOES tends to help when entity boundaries are hard and you have enough labeled data. BIO is the safer default on smaller datasets. Pick one before annotation starts, because the choice propagates into export formats and evaluation scripts.

Four approaches to building a NER model

The neural row is where nearly all production systems sit. Fine-tuning a pretrained transformer on a few thousand well-labeled domain examples typically beats every other option, and the pretrained weights do most of the heavy lifting.

Can LLMs do NER?

They can. They are just not very good at it yet, relative to a fine-tuned model, and the gap is wider than the demos suggest.

A published evaluation of ChatGPT and GPT-3.5 on the CoNLL-2003 test set reported an overall zero-shot F1 of 53.2 and 53.5 respectively, against 93.0 for Flair, 93.9 for LUKE and 94.6 for ACE. The per-class breakdown is more informative than the headline. On PERSON, zero-shot ChatGPT reached 87.2, which is respectable. On MISC it reached 4.1, because MISC is defined by an annotation guideline rather than by anything inferable from the text, and the model has never seen the guideline.

Few-shot prompting narrows the gap. A recent method reported 83.15 F1 on CoNLL-2003 and 70.37 on OntoNotes 5.0 under strict matching, which is roughly 88% and 76% of supervised performance on those datasets. A separate French-language experiment found GPT-4o at 0.70 token-level F1 against 0.93 for a BERT model fine-tuned on 1,800 documents.

There is a pattern in those failures worth internalizing. LLMs are good at recognizing entity types that correspond to real-world categories they learned during pretraining. They are poor at entity types that exist only because your guideline says so. The more your ontology diverges from common sense, the worse prompting performs, and domain ontologies diverge from common sense by design.

Where LLMs do earn their place is pre-labeling. Running a model over unlabeled text to produce draft annotations, then having trained annotators correct them, is faster than labeling from scratch and avoids inheriting the model’s errors wholesale. On a recent financial NER engagement, a human-in-the-loop workflow of this shape took entity F1 from 0.78 with the model alone to 0.94 with human review, at 38% of the cost of full manual annotation.

The judgment call is what to review. Reviewing everything wastes the speed advantage. Reviewing a random sample misses the cases that matter. Routing the model’s least confident predictions to humans is the approach that holds up.

How NER accuracy is measured

NER is scored with precision, recall and F1 computed over entity spans, not over tokens. Token-level scores look better and mean less, because the O label dominates and getting it right is trivial.

The matching rule changes everything:

• Strict match. Both boundaries and the type must be exactly right. “Bank of America” predicted as “America” is a miss and a false positive at once.
• Relaxed or partial match. An overlapping span with the correct type counts as correct.

Strict scores are always lower. A paper reporting relaxed-match F1 and one reporting strict-match F1 are not comparable, and the difference can run to several points. When a vendor quotes an F1, ask which rule produced it.

The annotation layer that decides the score

Everything above assumes labeled data exists. Producing it is where NER projects succeed or quietly fail, and it gets almost no coverage in the material that ranks for this topic.

We have watched NER models lose 12 to 18 points of F1 because two annotators could not agree on whether a job title should be tagged as PERSON or ROLE. The architecture was fine. The label set was not.

The workflow that avoids this looks like the following. It follows the same sequence described in this practitioner guide to text annotation for NLP, applied specifically to entity work, and it keeps getting revised.

Design the ontology first. Define every type, its boundary rules and its relationship to neighboring types before anyone labels anything. Write down what is explicitly excluded, not just what is included.

Write guidelines with an edge-case bank. Positive examples, negative examples, and a living list of the hard cases found during the pilot. Nested entities, enumerations such as “John, Mary, and the team”, and titles attached to names each need an explicit rule. Expect three to five rounds of revision before the guideline stabilizes. Locking it earlier just means unlocking it in week three.

Calibrate annotators against gold examples. A calibration set of 100 to 200 labeled examples, with a pass threshold around 90% agreement, catches most quality problems before they scale.

Run a pilot. Two thousand to five thousand records, roughly 5 to 10% of total volume. Measure inter-annotator agreement, surface the ambiguous cases, update the guideline. Skipping the pilot is the single biggest waste of project hours in this work.

Track inter-annotator agreement weekly. Cohen’s kappa above 0.80 is the working floor for entity tasks. Our own standard for NER work is 0.92 or higher. Below 0.80, adding annotators makes the dataset noisier rather than larger. Annotation drift is gradual, and weekly measurement catches it before 50,000 mislabeled records exist.

Sample QA by difficulty, not at random. Stratify the QA sample so a disproportionate share comes from the hardest decile: long sentences, high entity density, unusual register. Drift shows up there first.

None of this requires an external team. It does require someone to own it full time, which is the usual reason it ends up with a text annotation services provider rather than with the ML engineers who need the data.

If you take one operational point from this article, take this one. Inter-annotator agreement measured during the pilot is the best predictor of final model quality available before training starts, and it costs almost nothing to measure.

How much training data a NER model needs

There is no single number, but there are useful reference points.

Those are research corpora built for general-purpose models. A domain project with a narrow ontology usually needs far less, because a pretrained transformer arrives already knowing what a person name looks like. What it does not know is your definition of a CLAUSE_TYPE or an ADVERSE_EVENT.

A reasonable starting point for domain fine-tuning is a few hundred to a couple of thousand clean examples per entity type, with rare types needing proportionally more. Class imbalance is usually the binding constraint rather than total volume. A type appearing 40 times in 10,000 documents will not be learned no matter how large the corpus is, which is a sign the type either needs targeted sampling or does not belong in the ontology.

On cost, published ranges put multi-layer NER with QA at roughly $0.15 to $0.60 per record, with clinical and legal domains higher. A pilot of 2,000 to 5,000 records typically runs one to two weeks; a full project of 100,000 records runs four to eight weeks with a calibrated team. Ontology complexity, language coverage and QA depth each move those figures, and the breakdown of data annotation pricing models and rates shows by how much.

Whether that work happens internally or with a partner depends on data sensitivity, volume and whether your team is bottle-necked on modeling or on labeling. A side-by-side look at in-house versus outsourced data annotation covers the loaded-cost arithmetic that internal forecasts usually leave out.

Where NER models break in production

A model at 0.95 F1 on its test set can perform substantially worse on live documents. This surprises people more than it should.

Domain shift. The test set comes from the same distribution as the training data. It shares vocabulary, document structure and entity mix with it. Production documents do not. A model trained on newswire will under-perform on clinical notes, contracts or support tickets, and the drop is not gradual.

Entity drift. New companies, drugs, products and people appear continuously. A model trained on a 2023 corpus has no representation for an organization founded in 2025. Gazetteer overrides and scheduled retraining are the usual mitigations.

Long-tail entities. Frequent entity types get learned well. Rare types stay weak even when overall F1 looks healthy. Per-class metrics reveal this immediately, which is why aggregate F1 alone is a poor project health indicator.

Boundary disagreement between model and consumer. The model reproduces the annotation guideline. If a downstream team expects titles excluded from PERSON spans and the guideline included them, the model is correct and the integration is still broken. This is a specification problem, and it gets found in production far more often than it should.

Format and encoding drift. OCR output, PDFs with unusual layouts, and inconsistent Unicode normalization break offset tracking in ways that look like model failures and are not.

NER in practice, by industry

Healthcare. Clinical NER extracts conditions, medications, dosages and adverse events from unstructured notes. De-identification annotation removes the 18 PHI categories defined under HIPAA, where a missed entity is a compliance exposure rather than an accuracy metric.

Legal. Contract intelligence models extract parties, jurisdictions, governing law, dates and obligations. Nested entities are common here, since an organization name frequently contains a location and a defined term frequently contains a party name.

Financial services. Earnings transcripts and filings are annotated for companies, tickers, amounts and forward-looking statements. KYC pipelines pair NER with entity linking so a name in a news article resolves to the right customer record.

E-commerce. Product attribute extraction pulls brand, material, dimensions and model numbers out of titles and descriptions to populate structured catalog fields.

Customer support. Entity and intent annotation together drive ticket routing and self-service deflection.

Across all five, the same pattern holds: the model architecture is broadly interchangeable, and the annotation ontology is what makes the system specific to the business. Most of these programs also need image or document labeling alongside the text work, which is why entity annotation is normally scoped as one stream within wider data annotation services rather than as a standalone project.

Conclusion

The modeling side of named entity recognition is close to a solved problem for common entity types in well-resourced languages. Fine-tune a pretrained transformer, use BIO or BIOES tagging, decode with constraints, score with strict matching, and you will get a competent model.

The part that is not solved, and never will be by a model release, is deciding what your entities are and getting a team to label them the same way. That decision is made in a guideline document by people arguing about whether a job title belongs inside a PERSON span. The model reproduces whatever they decide, faithfully and at scale, including the parts they got wrong.

Budget accordingly. Ontology design and guideline iteration look like overhead until the alternative arrives, which is re-annotating 60,000 records in month four.

The practical next step, if you are scoping one of these, is a pilot on 2,000 to 5,000 records measured against your own gold set. That produces an inter-annotator agreement number, which tells you whether the ontology holds before you spend the annotation budget.

If you want to run that pilot with our team, get in touch.