Machine Learning and Artificial Intelligence Engineering

01Purpose and scope

• This manual supports the course Machine Learning and Artificial Intelligence Engineering.
• It describes how engineers build, evaluate, deploy and maintain systems that learn patterns from data and produce predictions, rankings, classifications or generated outputs. It is written for practitioners who already write code and want a dependable way to move a model from a notebook into a production service. It covers the activities, decisions, deliverables and roles that appear on most projects, along with the trade-offs that change between a small internal forecasting tool and a customer facing recommendation service.
• The scope stops short of the mathematics behind particular algorithms, which is better learned from textbooks and structured coursework. It also does not replace qualified professional judgment when a system touches medicine, lending, hiring, safety or legal decisions. In those settings, approved procedures, formal model risk reviews and any required authorization govern what may be built, tested and released.

02Core principles

• The quality of the data sets the upper limit on what any model can do. Most failed projects trace back to labels that were inconsistent, samples that did not match the population the model would see in use, or features that quietly included information unavailable at prediction time. Spend time on the data before spending time on the architecture.
• A simple baseline comes first. A logistic regression, a gradient boosted tree or even a rules based heuristic gives a reference point. If a complex model cannot beat the baseline by a margin that matters to the business, the extra complexity adds maintenance cost without adding value.
• Evaluation must reflect the real decision. Offline accuracy on a random holdout set is rarely enough. Use splits that respect time, users or geography when the deployed system will face those shifts. Report metrics that match the cost of errors, such as precision and recall for a fraud queue or calibration for a risk score.
• Every model is a component of a larger system. The surrounding code for data ingestion, feature computation, serving, logging, fallbacks and human review often causes more incidents than the model itself. Design those parts with the same care.
• Reproducibility is a working requirement, not an academic ideal. A result that cannot be regenerated from versioned data, code and configuration cannot be audited, compared or safely rolled back.

03Planning and preparation

• Start by writing down the decision the system will support, who will act on its output, and what happens when it is wrong. A model that ranks support tickets for a human agent has a very different risk profile from one that automatically denies a refund. Record these answers in a short problem statement that the product owner signs off on before modeling begins.
• Confirm that the data is available, permitted for this use and representative. Ask data owners for the lineage of each source, the refresh schedule and any restrictions on personal or sensitive attributes. Check whether the labels exist, how they were generated and whether the labeling process introduced bias. If labels must be created by hand, budget time for labeling guidelines, a pilot round and measured agreement between annotators.
• Choose success criteria before looking at results. Define a primary metric, one or two guardrail metrics such as latency or subgroup performance, and the minimum improvement over the baseline that justifies deployment. Agree on the acceptable operating point, for example the false positive rate the review team can absorb each day.
• Set up the workspace with version control for code, a data versioning approach or immutable snapshots, an experiment tracker and a shared environment definition. Pin library versions. These steps look like overhead at the start but prevent the most common forms of confusion later.
• Identify the roles early. A typical team includes a product owner who defines the decision, a data engineer who owns pipelines, an applied scientist or engineer who trains and evaluates models, a platform engineer who runs serving infrastructure, and a reviewer from risk, legal or privacy functions when the use case requires it. In small teams one person may hold several roles, but the responsibilities should still be named.

04Recommended workflow

1. 1
   : Frame the problem by converting the business question into a prediction target, the unit of prediction, the time horizon and the action the output will trigger. Write the target definition in plain language and in code so that both readers reach the same meaning.
2. 2
   : Build the data set by joining sources, removing duplicates, and applying the time boundary that matches prediction time. Write a script that generates the data set from raw inputs rather than editing a file by hand. Log row counts, class balance and missingness at each stage.
3. 3
   : Create features and check for leakage. Compute each feature using only information that would exist at the moment of prediction. Review every feature with a skeptical eye, especially aggregates, identifiers and fields updated after the outcome. Leakage is the most frequent reason a model looks excellent in testing and fails in production.
4. 4
   : Split the data and train a baseline. Use a training, validation and test split that matches the deployment pattern. Train the simple baseline first, record its metrics, and only then move to more flexible models. Keep the test set untouched until the final comparison.
5. 5
   : Tune and compare candidate models. Run a bounded hyperparameter search with a fixed budget. Compare candidates on the agreed primary metric, the guardrail metrics and calibration. Inspect errors by segment, looking for groups where performance drops sharply.
6. 6
   : Evaluate for the deployment context. Run the model on a recent slice of data, simulate the decision threshold, and estimate the workload created for reviewers or downstream systems. If a large language model or other generative component is involved, add evaluation sets of representative prompts with graded reference answers, and test for refusals, hallucinated facts and unsafe outputs.
7. 7
   : Package and deploy. Wrap the model in a versioned artifact with its preprocessing code, schema, dependencies and a model card that states intended use, training data, evaluation results and known limits. Deploy first to a shadow mode or a small percentage of traffic. Compare live predictions against the offline expectations before widening exposure.
8. 8
   : Monitor and retrain on a schedule or on trigger. Track input distribution drift, prediction distribution, outcome based performance once labels arrive, latency and error rates. Define the conditions that trigger investigation, rollback or retraining, and rehearse the rollback path before it is needed.

05Quality checks, risk controls and compliance

• Quality checks should run automatically wherever possible. Unit tests cover feature functions and preprocessing. Data tests check schema, ranges, null rates and referential integrity. Model tests confirm that the artifact loads, produces outputs of the expected shape, and meets minimum metric thresholds on a fixed evaluation set. Fail the pipeline when a check fails rather than logging a warning that nobody reads.
• Risk controls depend on how much harm a wrong output can cause. For a movie recommendation, a poor ranking is a minor annoyance. For a model that flags patients for follow up or prioritizes loan applications, errors can affect health, housing and livelihoods. In higher risk uses, add subgroup performance analysis, documented human override paths, appeal mechanisms where applicable, and a formal sign off from the accountable owner.
• Privacy and security controls apply throughout. Minimize personal data in training sets, restrict access to raw data, and check whether models can memorize or expose sensitive records. Review whether training data use is permitted under the contracts and consent terms that apply to it.
• Compliance requirements vary by jurisdiction and sector. Some regions regulate automated decision making, require explanations to affected people, or classify certain AI systems as high risk with specific obligations. Frameworks such as the NIST AI Risk Management Framework and ISO/IEC 42001 offer structured ways to organize governance, but they do not establish legal requirements by themselves. Confirm the current obligations with your legal or compliance team and the authoritative source for your location.
• This guidance does not replace qualified professional judgment, approved procedures or required authorization. For regulated fields such as medicine, finance, tax and law, a model may only be deployed after the organization's model risk, clinical, legal or safety processes have approved it.

06Collaboration and documentation

• Good documentation is mostly short, specific and kept close to the code. A project folder should contain the problem statement, the data dictionary, the label definition, the experiment log, the evaluation report, the model card and the runbook for operations. Each document should name an owner and a last reviewed state rather than a date.
• Hold a brief review at each major gate: problem framing, data readiness, offline evaluation, and pre-deployment. Invite the product owner, the engineers and the risk reviewer. Walk through the evidence and record decisions, including the options that were rejected and why. Those notes become essential when someone asks later why a threshold was set where it was.
• Set expectations for handoffs between data scientists and platform engineers. The scientist should deliver a model artifact with a clear interface, expected input schema, preprocessing that matches training exactly, and load tests run at realistic volume. The platform engineer should own the serving environment, scaling, alerting and rollback. Disagreements usually come from unclear interface definitions, so write them down.
• Share results in language that fits the audience. Executives need the expected business effect, its uncertainty and the conditions under which the system should be paused. Operations staff need the meaning of each score, the threshold and what to do with edge cases. Engineers need the full technical detail.

07Measures and continuous improvement

• Measure outcomes at three levels. Model level measures include the primary metric, calibration, subgroup gaps and stability across time. System level measures include latency, throughput, error rates and the share of predictions that fall back to rules. Business level measures include the downstream result the model was meant to improve, such as reduced manual review hours, fewer chargebacks or faster resolution of incidents.
• Compare these measures against the baseline and against the agreed guardrails. A gain in the primary metric that increases complaints or harms a subgroup is not a success. Track the gap between offline and online performance, since a persistent gap usually points to data pipeline differences or a changed population.
• Use a fixed review cadence for improvement. Monthly or quarterly reviews are common, with the exact interval set by how quickly the input distribution changes. Each review should decide whether to keep, retrain, redesign or retire the model. Retirement is a legitimate outcome and should have a documented process.
• Recognize the limits of measurement. Some outcomes take a long time to appear, and labels may arrive late or be biased by earlier decisions. A model that shapes its own training data can create feedback loops. Where possible, reserve a small randomized holdout or exploration slice to estimate what would have happened without the model.

08Common mistakes and how to prevent them

• Leakage appears when a feature carries information from after the outcome, such as a status field updated during resolution. Prevent it by defining prediction time explicitly, building features with timestamp filters, and reviewing feature importance for suspicious spikes.
• Overfitting to the validation set happens after many rounds of tuning and comparison. Keep a final test set sealed, limit the number of experiments, and confirm the result on a fresh slice of data before deployment.
• Ignoring class imbalance leads to models that look accurate while missing the rare events that matter. Choose metrics such as precision recall curves, apply weighting or resampling deliberately, and review the operating threshold with the people who will use it.
• Deploying without a rollback plan turns a small defect into a long incident. Keep the previous model version runnable, automate traffic switching, and rehearse the rollback steps.
• Treating the model as finished after launch causes slow decay. Assign an owner for monitoring, set alert thresholds, and schedule evaluation reviews from the start.
• Skipping stakeholder definitions produces systems that technically work but do not match the decision. Write the decision definition, the review workload and the escalation path before training begins.

09Practical checklist

• Written problem statement with the decision, the affected people and the action taken on each output.
• Data lineage, permissions and label definitions recorded and signed off by the data owner.
• Prediction time defined, and every feature checked against it for leakage.
• Baseline model trained and recorded before any complex model is considered.
• Primary metric, guardrail metrics and deployment threshold agreed before evaluation on the test set.
• Subgroup performance and calibration reviewed, with gaps documented.
• Model card, data dictionary, experiment log and runbook stored with the code.
• Shadow or limited rollout completed, with live predictions compared to offline expectations.
• Monitoring for drift, latency, errors and outcome performance active, with alert owners named.
• Rollback path tested and the prior model version still deployable.
• Risk, privacy and legal review completed where the jurisdiction or use case requires it, with approvals on record.

★Learning sources

• Micro Degree: Machine Learning and Artificial Intelligence Engineering
• Micro Degree: Data Analytics Applications
• Micro Degree: Azure Cloud Fundamentals
• Micro Degree: Google Cloud Platform Basics