Natural Language Processing

01Purpose and scope

• This manual supports the course Natural Language Processing. It describes how practitioners move a language task from a business question to a working text system and how they keep that system reliable after it goes live.
• The scope covers text classification, information extraction, search and retrieval, summarization, and text generation with pretrained models. It touches speech only where a transcript becomes the text input. It does not teach the mathematics of model training in depth, since that belongs to a separate study of machine learning.
• Practice changes with the context. A team extracting invoice numbers from a fixed template may do well with rules and patterns, while a team classifying customer messages across many dialects and topics usually needs trained statistical or neural models and far more labeled data. Each approach trades setup effort, transparency, cost per prediction and tolerance for error against the others. Rules are easy to audit but brittle. Large pretrained models adapt quickly but can be expensive to run and hard to explain.
• Health, legal, financial, tax and safety-related text raises the stakes. The guidance here does not replace qualified professional judgment, approved procedures or required authorization. Where a system will influence a decision about a person, confirm the requirements with the responsible authority and the current authoritative source for your jurisdiction.

02Core principles

• Define the task precisely before collecting anything. A request such as find the important parts of support emails is not yet a task. A usable definition names the output, the label set, the unit of analysis such as a message or a sentence, and what counts as a correct answer in borderline cases.
• Train and test on data that looks like production input. Clean benchmark text often hides the misspellings, abbreviations, mixed languages and copied signatures that real users send. Sample from the actual source and keep a held-out set that the team does not tune against.
• Start with the simplest method that could meet the target. A keyword rule or a logistic regression model on word features gives a baseline that is cheap to build and easy to compare against. A larger model needs to earn its cost by beating that baseline on the metrics that matter.
• Errors do not cost the same. Missing a safety report can be far worse than flagging a harmless message for review. Decide the cost of each error type with the people who will live with the consequences, and set thresholds to match.
• Language varies by region, dialect, register, domain and time. Terms that are common in one product line may be unknown in another. Plan to revisit vocabulary and examples as the subject matter changes.

03Planning and preparation

• Identify the stakeholders before writing code. Domain experts know what the labels mean in practice. Data owners control access and retention rules. Engineers build and deploy the system. Reviewers check outputs and handle escalations. A privacy or legal contact should see the plan early if personal data or regulated content is involved.
• Prepare a short set of deliverables before modeling starts. These usually include a task definition, a labeling guide with examples of each class and several hard cases, a data inventory listing sources, volumes, owners and sensitive fields, and a baseline result. The labeling guide is the most neglected item and often the most valuable.
• Choose tools that match the team's skills and the data environment. Common open source choices include spaCy for pipelines and entity recognition, scikit-learn for classical classifiers, and the Hugging Face transformers library for pretrained neural models. Annotation tools such as Label Studio or a well-structured spreadsheet can handle small projects. Confirm that the chosen tools are permitted for the data you hold.
• Estimate the labeling budget early. A reasonable first pass for a multi-class classifier may need a few hundred to a few thousand examples per class, depending on how hard the classes are to tell apart. Treat any figure here as a starting assumption to test, not a rule.

04Recommended workflow

1. 1
   : Write the task definition in one page. Include the input unit, the output format, the label set, the users of the output, and the decision the output will support. Get sign-off from the person who owns that decision.
2. 2
   : Build a data inventory and draw a sample. Record where text comes from, who may use it, whether it contains personal or confidential information, and how long it can be kept. Draw a sample that reflects the mix of sources in production, not only the cleanest source.
3. 3
   : Remove or mask sensitive content before annotation. Use pattern-based redaction for obvious identifiers such as account numbers, then have a person check a sample for misses. Keep the raw data in its approved location and give annotators only the text they need.
4. 4
   : Write the labeling guide and run a calibration round. Have two or three annotators label the same fifty to one hundred items independently, then discuss every disagreement. Update the guide with each rule that emerges from those discussions.
5. 5
   : Label the main set and hold out a test set. Use the guide and record the annotator for each item. Lock the test set before any model tuning begins, and do not look at its errors until the final evaluation.
6. 6
   : Build a simple baseline and record its results. A keyword or rules baseline and a classical model give a reference point. Report precision, recall and F1 for each class, not only overall accuracy, because a majority class can hide poor performance on the rare class that matters.
7. 7
   : Try a stronger model and compare on the same test set. Fine-tune a pretrained model or test a prompted large language model if the task and budget allow. Include latency, cost per thousand items and the hardware or service required in the comparison, since a small accuracy gain may not justify a large operating cost.
8. 8
   : Prepare for deployment with a review path and a fallback. Decide which outputs are applied automatically, which go to a human queue, and what happens when the service is down. Run the system in shadow mode, where it predicts but humans make the decisions, before allowing it to act.

05Quality checks, risk controls and compliance

• Measure agreement between annotators before trusting any label. For two annotators, Cohen's kappa is a common statistic. For more than two, Krippendorff's alpha handles missing labels and multiple raters. Low agreement usually means the guide is vague, not that the annotators are careless, so revise the guide before relabeling.
• Evaluate by slice as well as overall. Break results down by source, dialect, length, language, customer segment and any group where unfair error rates could cause harm. A system that scores well on average but fails for one dialect or one product line is not ready.
• Check for leakage between training and test data. Duplicate messages, quoted replies and templated signatures can make results look better than they are. Deduplicate before splitting and document how the split was made.
• Set risk controls based on the error costs agreed in planning. Define confidence thresholds, route low-confidence items to people, and log every automated decision that affects a customer or employee. Keep a rollback path that the operations team can execute without the original developers.
• Handle personal and regulated data under the rules that apply to your organization and location. Data protection laws such as the EU General Data Protection Regulation apply in some settings, and sector rules may impose more. Confirm retention periods, access logs and data transfer terms with the responsible authority and the current authoritative source before moving data to a new service.

06Collaboration and documentation

• Keep one shared record of decisions. Write down why a label was merged, why a threshold was chosen, and why a model was rejected. Teams that skip this step repeat the same debates each time staff change.
• Produce documents that match the audience. A short model card or system note should describe the intended use, the training and test data sources, the evaluation results by slice, known failure modes and the person responsible for the system. A datasheet for the labeled data should record collection methods, labeling procedures, exclusions and known gaps. These are common formats in the field, and an existing template in your organization may serve the same purpose.
• Hold regular review meetings with the domain experts. Bring a sample of recent errors, not only aggregate scores, and ask the experts whether each error is a model problem, a label problem or a policy question. Many apparent model failures turn out to be unclear definitions.
• Maintain a change log for data, labels, code and model versions. Each deployed version should be traceable to the data and configuration that produced it, so that a complaint about last month's output can be investigated.

07Measures and continuous improvement

• Track quality indicators in production. Useful measures include the share of items routed to human review, the agreement rate between the system and reviewers on a sampled basis, precision and recall on newly labeled samples, latency, and cost per thousand predictions. Pick a small set and report them on the same schedule each time.
• Watch for drift. Vocabulary, product names, customer complaints and regulations all change. Compare the distribution of incoming text with the training data each period, and sample recent predictions for review. A rise in low-confidence items or a sudden shift in predicted class mix is often the first visible sign.
• Set retraining triggers in advance. Decide what level of drop in recall, what rise in review volume or what new category would prompt relabeling and retraining. Pre-set triggers prevent both panic retraining and long periods of quiet decline.
• Close the loop with reviewers. When a person corrects a prediction, store the correction with a reason code. These corrections become the most useful new training examples because they come from the cases the system actually handles.
• Review the whole system periodically. Ask whether the task definition still matches the decision it supports, whether the labels still reflect policy, and whether a simpler method would now be sufficient. Retiring a model is a valid outcome.

08Common mistakes and how to prevent them

• Starting with a model before defining the task. Teams often choose a model family first and then discover that the labels do not match the decision. Prevent this by completing the task definition and labeling guide before any training run.
• Evaluating on the wrong data. A test set drawn from a curated corpus can produce strong numbers that collapse in production. Prevent this by sampling from production sources and keeping the test set locked.
• Reporting only overall accuracy. A classifier that predicts the common class for almost everything can look accurate while missing the rare, important cases. Prevent this with per-class precision, recall and slice results.
• Ignoring annotator disagreement. Teams that skip agreement checks inherit noisy labels and then blame the model. Prevent this with calibration rounds and regular agreement measurement.
• Skipping the privacy review until late. Personal data in training sets creates rework and risk. Prevent this by completing the data inventory and redaction step in the first week.
• Treating a demo as a deployment. A system that works on a few examples in a meeting may fail under real volume, latency limits or unusual inputs. Prevent this with shadow mode, load tests and a documented fallback.
• Letting the system run without an owner. Models decay and nobody notices when no one is assigned to review them. Prevent this by naming an owner, a review schedule and the triggers that prompt action.

09Practical checklist

• Write a one-page task definition with input unit, output format, label set and the decision it supports.
• Confirm the data inventory, permitted uses, retention limits and any personal or regulated content.
• Redact obvious identifiers and have a person spot check a sample of the redacted text.
• Write a labeling guide with examples and hard cases, then run a calibration round with agreement measurement.
• Lock a test set sampled from production sources and deduplicated against training data.
• Report precision, recall and F1 by class and by slice, alongside latency and cost per thousand items.
• Run the system in shadow mode with a human review path and a documented rollback procedure.
• Record model versions, data versions and decisions in a shared log, and set retraining triggers before launch.

10Frequently asked questions

★Learning sources

• Micro Degree: Natural Language Processing
• Micro Degree: Data Analytics Applications
• Micro Degree: Azure Cloud Fundamentals
• Micro Degree: Google Cloud Platform Basics