Big Data Technologies Flashcards

Question 1

What does the term big data generally describe?

Accepted Answer

Big data generally describes datasets so large, fast, or varied that traditional tools and single servers struggle to capture, store, and analyze them efficiently. The term is not defined by a fixed size threshold. Instead, it refers to a set of challenges that call for distributed systems, specialized storage, and parallel processing methods.

Question 2

What are the three Vs commonly used to describe big data?

Accepted Answer

The three Vs are volume, velocity, and variety. Volume refers to the sheer amount of data, velocity describes how quickly data arrives and must be processed, and variety captures the range of formats, including structured tables, semi-structured logs, and unstructured text, images, or audio. Some authors add veracity and value as further dimensions.

Question 3

Why does a single conventional database server often fail to handle big data workloads?

Accepted Answer

A single machine has limited storage capacity, memory, and processing power, and its disk and network speeds cannot keep pace with growing data. Scaling up with a larger machine eventually reaches hardware and cost limits. Big data systems instead scale out, spreading storage and computation across many commodity machines that work together.

Question 4

What is the difference between scaling up and scaling out?

Accepted Answer

Scaling up means adding more power, such as more memory or faster processors, to one machine. Scaling out means adding more machines to a cluster and dividing the work among them. Scaling out is generally preferred for big data because it relies on inexpensive hardware, though it adds coordination and fault tolerance challenges.

Question 5

What is a cluster in the context of big data systems?

Accepted Answer

A cluster is a group of interconnected computers, called nodes, that work together as a single logical system. Each node contributes storage and processing capacity, and a coordinating layer assigns tasks and tracks their status. Clusters let organizations handle workloads that no single machine could manage on its own.

Question 6

What is data locality and why does it matter for performance?

Accepted Answer

Data locality means running computation on or near the machine that already stores the data. It matters because moving large volumes of data across a network is slow compared with reading from local disks. Keeping code close to data reduces network traffic and usually shortens job completion times.

Question 7

What is fault tolerance and why is it essential in distributed systems?

Accepted Answer

Fault tolerance is the ability of a system to keep operating correctly when components fail. In distributed systems, failures of individual machines, disks, or network links are routine when hundreds of nodes run together. Techniques such as replicating data and restarting failed tasks let the system recover without losing work or data.

Question 8

What is the difference between structured, semi-structured, and unstructured data?

Accepted Answer

Structured data follows a fixed schema, such as rows and columns in a relational table. Semi-structured data has tags or markers that organize it without a rigid table layout, as with JSON or XML documents. Unstructured data, including free text, images, audio, and video, has no predefined model and needs interpretation before analysis.

Question 9

What is the Hadoop Distributed File System and what problem does it solve?

Accepted Answer

HDFS, the Hadoop Distributed File System, stores very large files across many machines. It solves the problem of storing more data than any single disk can hold. Each block is copied to several nodes so data survives hardware failures, and the design favors large sequential reads over small random updates.

Question 10

What is block replication and what default replication factor is commonly used in HDFS?

Accepted Answer

Block replication is the practice of storing several identical copies of each data block on different nodes or racks. If one node fails, the system reads from another copy and rebuilds the missing replica elsewhere. The default replication factor in HDFS is three, which balances durability and storage cost.

Question 11

What is an object store and how does it differ from a file system?

Accepted Answer

An object store keeps data as discrete objects, each with its content, a unique identifier, and metadata, usually in flat buckets rather than nested directories. A traditional file system organizes data in folders and supports in-place modification. Object stores trade fine-grained updates for massive scalability and low-cost storage of large unstructured content.

Question 12

What is a data lake and how does it differ from a data warehouse?

Accepted Answer

A data lake is a central repository that stores raw data in its native format until it is needed. A data warehouse holds curated, structured data that has been cleaned and modeled for reporting before it is loaded. Lakes favor flexibility and schema on read, while warehouses favor consistency and schema on write.

Question 13

What is replication lag in a distributed storage system?

Accepted Answer

Replication lag is the delay between a write being accepted by one copy of the data and that change appearing on other replicas. During this window, readers hitting a lagging replica may see stale values. Systems can reduce this risk by directing sensitive reads to up-to-date copies.

Question 14

What is a distributed hash table used for in large storage systems?

Accepted Answer

A distributed hash table maps keys to nodes so that any node can find which machine holds a given key without a central directory. Nodes join and leave with only a small portion of keys needing to move. This approach underlies many key-value stores and peer-to-peer systems.

Question 15

Why is partitioning data across nodes important for scalability?

Accepted Answer

Partitioning, or sharding, splits a large dataset into smaller pieces that are stored on different nodes. Each node handles only a fraction of total reads and writes, so throughput can grow as machines are added. Choosing a good partition key matters, because a poor choice can overload one node while others sit idle.

Question 16

What is consistency in distributed storage and what trade-off does it involve?

Accepted Answer

Consistency describes whether all readers see the same data at the same time after a write. Strong consistency guarantees up-to-date reads but can increase latency and reduce availability during network partitions. Eventual consistency allows faster responses and higher availability, at the cost of temporarily stale results. The CAP theorem frames this choice.

Question 17

What is MapReduce and what are its two main phases?

Accepted Answer

MapReduce is a programming model for processing large datasets in parallel across a cluster. Its two main phases are map, which transforms each input record into intermediate key-value pairs, and reduce, which aggregates all values sharing the same key. A shuffle step between them groups intermediate data by key before the reduce phase runs.

Question 18

How does Apache Spark differ from classic MapReduce in how it handles intermediate data?

Accepted Answer

Spark keeps intermediate results in memory across operations where possible, instead of writing them to disk after every step as classic MapReduce does. This reduces disk input and output, which makes iterative algorithms and interactive queries much faster. Spark still spills to disk when memory is insufficient.

Question 19

What is the difference between batch processing and stream processing?

Accepted Answer

Batch processing collects a bounded set of data and processes it together as a job, usually on a schedule, with results available after the job finishes. Stream processing handles data continuously as individual events arrive, producing results incrementally with low latency. Batch favors throughput and completeness, while streams favor timeliness.

Question 20

Why do distributed frameworks often break a job into stages with dependencies between them?

Accepted Answer

Many computations depend on earlier results, like a join that needs grouped data. Splitting a job into stages lets the framework run independent tasks in parallel within each stage and wait only where a dependency requires it. A scheduler tracks which stages are complete, retries failed tasks, and moves data between stages.

Question 21

What is a directed acyclic graph in the context of data processing engines?

Accepted Answer

A directed acyclic graph, or DAG, represents a processing job as nodes for operations and directed edges for data dependencies, with no cycles. Because the graph is known in advance, an engine can optimize the whole plan, combine steps, and schedule independent branches in parallel. Spark, Flink, and similar engines use this structure.

Question 22

What does a combiner do in a MapReduce job and why is it useful?

Accepted Answer

A combiner is an optional local aggregation step that runs on each mapper's output before data is shuffled across the network. For operations such as sums or counts, it can merge many intermediate pairs with the same key into fewer pairs. This cuts network traffic and reducer workload, but only for associative operations.

Question 23

What is data skew and why does it slow distributed jobs?

Accepted Answer

Data skew occurs when a few keys or partitions hold a disproportionate share of the data. Because a distributed job finishes only when its slowest task finishes, one overloaded task can delay the entire stage while other workers sit idle. Common remedies include salting hot keys, repartitioning, and filtering or pre-aggregating heavy values.

Question 24

How does Spark recover lost partitions when a worker node fails?

Accepted Answer

Spark tracks the lineage of each resilient distributed dataset, which is the sequence of transformations that produced it. If a partition is lost, Spark recomputes only that partition from its source data and earlier lineage rather than restarting the whole job. Caching or checkpointing can shorten recovery when lineage chains grow long.

Question 25

What is a message queue or event log in a data pipeline?

Accepted Answer

A message queue or event log is a buffer that stores records from producers until consumers read them. It decouples the systems that generate data from those that process it, so each side can run at its own pace. Durable logs retain records for a period, allowing consumers to replay or recover from failures.

Question 26

What is Apache Kafka and what role does it typically play in big data architectures?

Accepted Answer

Apache Kafka is a distributed event streaming platform that stores streams of records in partitioned, replicated logs. In big data architectures it typically serves as the central backbone that collects events from many sources and delivers them to processing engines, storage systems, and monitoring tools. Its partitions let many consumers read in parallel.

Question 27

What is the difference between ETL and ELT in data pipelines?

Accepted Answer

ETL means extracting data from sources, transforming it in a separate processing stage, and then loading the cleaned result into a target system. ELT loads raw data into the target first and transforms it there using the target's compute. ELT suits cloud warehouses with cheap, scalable compute, while ETL suits strict cleansing needs.

Question 28

Why is exactly-once processing difficult to guarantee in distributed streams?

Accepted Answer

Machines and networks can fail at any point, so a processor may crash after writing output but before recording its progress. On restart, the record might be processed again, producing duplicates. Achieving exactly-once effects requires coordinating checkpoints, offsets, and idempotent or transactional writes to the output.

Question 29

What is a stream windowing operation and why is it needed?

Accepted Answer

A window groups an unbounded stream into finite slices, such as each five-minute interval. Windows make aggregation possible, because operations like averages or counts need a bounded set of values to produce a result. Tumbling, sliding, and session windows differ in how their boundaries are defined.

Question 30

What is backpressure in a streaming system and why does it matter?

Accepted Answer

Backpressure is a feedback mechanism by which a slow consumer signals upstream producers to slow their rate. Without it, buffers grow without bound, leading to memory exhaustion or dropped events. With it, the system adapts its flow to the throughput each stage can sustain.

Question 31

What does idempotent mean when applied to a data write?

Accepted Answer

An idempotent write produces the same final result no matter how many times it is applied. For example, upserting a record by its unique key can be repeated safely, while appending a row to a table repeats the insertion. Idempotent writes let pipelines retry after failures without creating duplicate or inconsistent data.

Question 32

How does a dead letter queue help a data pipeline handle bad records?

Accepted Answer

A dead letter queue is a separate destination where records that repeatedly fail processing are sent instead of blocking the pipeline. Engineers can inspect these records later, fix the underlying problem such as a malformed field or schema mismatch, and replay them. This keeps the main flow moving and preserves evidence of failures.

Question 33

What is a NoSQL database and how does it generally differ from a relational database?

Accepted Answer

NoSQL is a broad family of databases that store data in models other than fixed-schema relational tables, such as key-value, document, column-family, and graph models. They generally trade some relational features, such as multi-table joins and rigid schemas, for horizontal scalability and flexible data models. The exact trade-offs vary widely between products.

Question 34

What is a key-value store and when is it a good fit?

Accepted Answer

A key-value store maps each unique key to a value, usually opaque to the database, and supports fast lookups by key. It fits workloads such as session storage, caching, shopping carts, and user profiles, where access is mainly by primary key. It is less suited to complex queries that filter or join across attributes.

Question 35

What is a column-family store and how does it organize data?

Accepted Answer

A column-family store groups related columns together into families and stores each family's data contiguously on disk, with rows identified by a row key. Sparse rows can omit many columns without wasting space. This layout suits wide tables with many attributes and time-series or event data read by row range.

Question 36

What is columnar storage and why does it speed up analytical queries?

Accepted Answer

Columnar storage keeps the values of each column together on disk rather than storing whole rows contiguously. Analytical queries often read only a few columns across millions of rows, so the engine can skip irrelevant data entirely. Similar values also compress well, which further reduces input and output and speeds up scans.

Question 37

What is an MPP query engine and how does it execute a query?

Accepted Answer

A massively parallel processing engine splits a query into fragments and runs them simultaneously on many nodes, each working on its local slice of the data. Results from the nodes are combined at a coordinator, which returns the final answer. Such engines keep data partitioned across nodes and avoid moving volumes over the network.

Question 38

What is a data lakehouse and what problem is it meant to address?

Accepted Answer

A data lakehouse is an architecture that adds warehouse-style management, like transactions and schema enforcement, on top of low-cost object storage used by data lakes. It aims to address the split where teams maintain both a lake for raw data and a separate warehouse for reliable analytics. Keeping one copy of data for both purposes reduces duplication and complexity.

Question 39

What is a materialized view and why do analytics systems use it?

Accepted Answer

A materialized view stores the precomputed result of a query so that later reads can return the answer without recomputing it from base tables. Analytics systems use these views for expensive aggregations that many dashboards request repeatedly. The trade-off is that stored results must be refreshed as underlying data changes.

Question 40

How would you choose between a NoSQL store and a relational database for a new application?

Accepted Answer

Start from the access patterns, consistency needs, and expected scale rather than from a preferred technology. Relational databases suit data with complex relationships, transactions spanning several tables, and ad hoc queries. NoSQL stores suit high write volumes, simple lookups by key, or flexible schemas, but may require application logic for joins and integrity.

Big Data Technologies Flashcards

1Core Big Data Concepts

2Distributed Storage Foundations

3Batch and Distributed Processing

4Ingestion and Streaming Pipelines

5NoSQL and Analytics at Scale

Learning Resources