AI Research Methods Flashcards

Question 1

What is a testable hypothesis in AI research?

Accepted Answer

A testable hypothesis is a specific, falsifiable statement predicting how one factor will affect a measurable outcome. For example, a researcher might predict that adding dropout to a neural network reduces overfitting on a held-out validation set. A good hypothesis names its variables and states the expected direction of effect.

Question 2

What distinguishes an independent variable from a dependent variable?

Accepted Answer

The independent variable is the factor the researcher deliberately changes or manipulates, while the dependent variable is the outcome measured in response. For instance, learning rate might be the independent variable and test accuracy the dependent one. Controls are held constant so changes in the outcome can be attributed to the manipulated factor.

Question 3

Why do researchers use a control group in an experiment?

Accepted Answer

A control group receives the baseline condition, such as no new treatment or a standard model, so researchers can compare results against it. Without this comparison, improvements might come from chance or other unrelated factors. Random assignment of participants or examples to groups helps ensure the groups differ mainly by the treatment being tested.

Question 4

What is a confound in experimental design, and how is it reduced?

Accepted Answer

A confound is an outside variable that changes along with the factor under study and could explain the observed effect. For example, if a new model is tested only on a newer, cleaner dataset, the improvement may reflect the data rather than the model. Researchers reduce confounds through randomization, matching, and holding conditions constant.

Question 5

What is the difference between correlation and causation in study results?

Accepted Answer

Correlation means two variables move together, while causation means changes in one variable actually produce changes in the other. A strong correlation between GPU usage and model quality does not prove that GPU usage causes better quality, since a third factor such as training budget may drive both.

Question 6

What is an observational study, and how does it differ from an experiment?

Accepted Answer

An observational study records data about subjects or systems without assigning conditions or intervening. The researcher watches what happens naturally, such as analyzing logs of user interactions. An experiment, by contrast, deliberately assigns conditions to units, which makes causal conclusions more defensible but is not always practical or ethical.

Question 7

Why should a study state its unit of analysis before collecting data?

Accepted Answer

The unit of analysis is the entity about which conclusions are drawn, such as an individual image, a user, a model run, or a company. Stating it early prevents mixing levels, for example reporting per-image accuracy as if it described per-user behavior. Clear units also guide how samples are drawn.

Question 8

What is a research question, and why is it useful before running experiments?

Accepted Answer

A research question is a focused question that defines what the study will investigate and what counts as an answer. It guides decisions about data, methods, and metrics, and it keeps experiments from drifting into unfocused comparisons. It also clarifies what a negative result would mean.

Question 9

What is the purpose of splitting data into training, validation, and test sets?

Accepted Answer

Splitting data into training, validation, and test sets lets a model learn from one portion, tune choices on another, and be judged on data it has never seen. The training set fits parameters, the validation set guides decisions such as hyperparameters, and the test set gives a final estimate of performance.

Question 10

Why is it important that the test set be kept separate during development?

Accepted Answer

Keeping the test set separate prevents information from leaking into design decisions, which would make measured performance look better than it truly is on new data. Once test results influence model choices, the estimate no longer reflects generalization. Researchers usually consult the test set once, after tuning is finished.

Question 11

What is data leakage, and what forms can it take?

Accepted Answer

Data leakage occurs when information that would not be available at prediction time enters the training process, inflating apparent performance. Common forms include preprocessing such as scaling computed over the full dataset, duplicate examples split across train and test sets, and features that encode the target. Leakage often goes unnoticed because the resulting metrics look unusually strong.

Question 12

How does sampling bias affect what a dataset can support?

Accepted Answer

Sampling bias occurs when the data collected systematically differs from the population the researcher intends to describe. A model trained on images from one region, camera, or demographic group may perform poorly elsewhere, and reported accuracy will overstate real-world reliability. Good practice includes documenting how data was gathered and testing performance across relevant subgroups.

Question 13

What is labeling noise, and why does it matter?

Accepted Answer

Labeling noise refers to errors or inconsistencies in the labels assigned to training or evaluation examples, such as annotators disagreeing about a category. It matters because models learn from these labels, so noise can limit achievable accuracy and distort comparisons between methods. Researchers often measure annotator agreement and adjudicate disputed cases to reduce it.

Question 14

Why do researchers document datasets with datasheets or data cards?

Accepted Answer

Dataset documentation records a dataset's origin, collection process, intended uses, known limitations, and licensing so others can judge whether it fits their work. Such documents make hidden assumptions visible, which helps prevent misuse and supports reproducibility. Documentation is most useful when written during collection, not reconstructed afterward.

Question 15

What is the difference between a random sample and a convenience sample?

Accepted Answer

A random sample gives every member of the population a known, nonzero chance of selection, which supports generalization to that population. A convenience sample uses whatever data is easiest to obtain, such as volunteers or logs from one platform, and its results may not transfer. Convenience samples are sometimes necessary, but their limits should be stated plainly.

Question 16

What is class imbalance, and why can it mislead a study?

Accepted Answer

Class imbalance means some categories have far fewer examples than others, so a model can appear accurate by favoring the majority class. In a dataset where ninety-five percent of examples are negative, always predicting negative yields high accuracy but finds no positives. Common remedies include resampling, class weights, and metrics like recall.

Question 17

What is a confusion matrix, and what does it show?

Accepted Answer

A confusion matrix is a table that counts how often each actual class was predicted as each possible class. For binary classification, its four cells are true positives, false positives, true negatives, and false negatives. It reveals error patterns that a single summary number such as accuracy can hide.

Question 18

How are precision and recall defined, and how do they differ?

Accepted Answer

Precision is the fraction of predicted positives that are actually positive, while recall is the fraction of actual positives that the model correctly identifies. Precision asks how trustworthy a positive prediction is, and recall asks how many true cases were found. The right balance depends on the costs of each error type.

Question 19

What does the F1 score combine, and when is it useful?

Accepted Answer

The F1 score is the harmonic mean of precision and recall, producing a single value that is high only when both are high. It is useful when a task needs a balance between false alarms and misses, and when classes are imbalanced enough that accuracy alone is misleading. Because the harmonic mean penalizes large gaps, a model with extreme values in one measure scores poorly.

Question 20

How does balanced accuracy differ from ordinary accuracy?

Accepted Answer

Ordinary accuracy is the share of all predictions that are correct, which can be dominated by the largest class. Balanced accuracy averages the recall obtained for each class, giving minority classes equal weight. For a dataset with many more negatives than positives, balanced accuracy reveals poor positive detection that ordinary accuracy would hide.

Question 21

What is cross-validation, and why do researchers use it?

Accepted Answer

Cross-validation splits data into several folds, trains on all but one fold, and tests on the held-out fold, rotating until every fold has served as the test set. Averaging the scores gives a more stable estimate than a single split, especially with limited data.

Question 22

Why should researchers report uncertainty, such as confidence intervals or standard deviations across runs?

Accepted Answer

Reporting uncertainty shows how much a result might vary under different random seeds, data samples, or conditions. A confidence interval gives a plausible range for the true value, while a standard deviation across repeated runs shows run-to-run variability. Small differences between methods may fall within this variability and should not be read as improvements.

Question 23

What is the difference between a metric and a benchmark?

Accepted Answer

A metric is a quantitative measure that scores performance on some aspect of a task, such as error rate or F1 score. A benchmark is a standardized evaluation setup, usually including a fixed dataset, a defined task, and agreed metrics, that allows fair comparison across methods.

Question 24

What is the area under the ROC curve, and what does it measure?

Accepted Answer

The area under the ROC curve summarizes how well a classifier ranks positive examples above negative ones across all possible decision thresholds. A value of 0.5 corresponds to random ranking, while 1.0 indicates perfect separation. Precision-recall curves are often more informative on heavily imbalanced data.

Question 25

What is a baseline, and why must one be included in a comparison?

Accepted Answer

A baseline is a simple, established, or previously published method used as a reference point for judging a new approach. Without an appropriate baseline, a reported gain has no context, and a complex model may look impressive while a simple heuristic performs nearly as well. Baselines deserve equal tuning effort.

Question 26

What is an ablation study, and what does it reveal about a model?

Accepted Answer

An ablation study removes or disables individual components of a system one at a time and measures the effect on performance. It helps show which parts actually contribute to results and which are unnecessary. For example, a researcher might remove an attention layer or a data augmentation step to see how much accuracy drops.

Question 27

Why does controlling random seeds matter for reproducibility?

Accepted Answer

Fixing random seeds makes operations such as weight initialization, data shuffling, and dropout produce the same sequence on each run, so results can be repeated. Without fixed seeds, differences between runs may reflect chance. Even with seeds set, some GPU operations may be nondeterministic, so researchers should also report software and hardware details.

Question 28

What is statistical significance, and what does a p-value indicate?

Accepted Answer

Statistical significance indicates that an observed difference would be unlikely if no true difference existed. A p-value is the probability of seeing a result at least as extreme as the observed one, assuming the null hypothesis is true. A small p-value does not measure how large or important an effect is.

Question 29

What is a preregistered study, and why do some researchers use one?

Accepted Answer

A preregistered study records its hypotheses, methods, and analysis plan in a public or time-stamped document before data is collected or examined. Committing in advance limits flexible analysis choices that can inflate false positive findings. Preregistration does not prevent exploratory work, but it helps readers distinguish planned tests from analyses added after seeing the results.

Question 30

What is a hyperparameter, and why should tuning effort be reported?

Accepted Answer

A hyperparameter is a setting chosen before training, such as learning rate, batch size, or regularization strength, rather than a value learned from data. Reporting tuning effort matters because extra trials can make a method look better. Stating the search space, number of trials, and selection criteria lets readers judge whether comparisons are fair.

Question 31

Why is it important to report the compute budget and training cost of a method?

Accepted Answer

Reporting compute budget, including hardware, training time, and number of training runs, allows readers to judge whether a result is practical and whether comparisons are fair. Two methods with similar accuracy may differ greatly in cost, and results obtained with far more compute may not be replicable by smaller labs.

Question 32

What is the role of a replication study?

Accepted Answer

A replication study repeats an earlier study's procedures, sometimes with new data or settings, to check whether its findings hold. Direct replications use the same methods, while conceptual replications test the same idea with different approaches. Successful replication strengthens confidence in a finding, and failed replication prompts scrutiny of the original methods, data, or conditions.

Question 33

What is error analysis, and how does it guide model improvement?

Accepted Answer

Error analysis is the systematic review of a model's mistakes to find patterns, such as particular input types, subgroups, or edge cases where performance drops. Grouping errors by cause reveals whether problems stem from data quality, labeling, model capacity, or distribution shift. Targeted fixes based on these patterns usually produce more reliable gains than adjusting settings at random.

Question 34

What is distribution shift, and why does it threaten deployed models?

Accepted Answer

Distribution shift occurs when the data a model sees after deployment differs statistically from the data it was trained and evaluated on. Such differences can come from changes in users, sensors, language, or time, and they often cause performance to degrade without obvious warning. Monitoring inputs and outputs in production helps detect shifts early.

Question 35

What should a responsible research report disclose about its limitations?

Accepted Answer

A responsible report describes where the results may not apply, including the populations, languages, settings, and time frames covered by the data. It also discloses known failure modes, unresolved biases, and assumptions that could change conclusions. Stating limitations plainly helps readers avoid overgeneralizing and gives future researchers a clear starting point.

Question 36

What does informed consent require when research uses data about people?

Accepted Answer

Informed consent means people understand what data is collected, how it will be used, and the risks, and they agree voluntarily. In AI research, consent questions arise when scraping public text, collecting faces, or reusing records originally gathered for another purpose. When consent is impractical, ethics board review and data minimization are common safeguards.

Question 37

What is a model card, and what does it typically contain?

Accepted Answer

A model card is a short document describing a trained model's intended uses, performance across relevant conditions, training data summary, and known limitations. It helps users decide whether a model fits their task and warns against uses the developers did not evaluate.

Question 38

What should a methods section include so that others can replicate a study?

Accepted Answer

A methods section should describe the data sources and preprocessing, model architecture, training procedure, hyperparameters, evaluation metrics, and the exact splits used. Enough detail should be given that a capable researcher could reproduce the work without contacting the authors. Code and data availability, where permitted, strengthen this goal.

Question 39

How can a researcher communicate AI results to nonspecialist decision makers responsibly?

Accepted Answer

Researchers should state the main finding, its uncertainty, and its practical meaning in plain language, avoiding jargon and inflated claims. Visualizations that show ranges and error rates often communicate more honestly than single percentages. Stakeholders should also hear what the model cannot do, which helps prevent misplaced trust and misuse.

Question 40

What is a human-in-the-loop check, and when is it useful?

Accepted Answer

A human-in-the-loop check means a person reviews, confirms, or can override model outputs before they affect decisions. It is useful when errors are costly, when the model's confidence is unreliable, or when legal or ethical accountability requires human judgment. Reviewers who cannot see the model's inputs cannot judge its outputs.

AI Research Methods Flashcards

1Research Design Fundamentals

2Data Collection and Dataset Splits

3Evaluation Metrics and Measurement

4Experimental Rigor and Reproducibility

5Applied Research and Reporting

Learning Resources