Database Administration and Data Engineering
By vaayaa Editorial Team · Subject panel: Hiroshi Tanaka (PhD, Computer Science), Carlos García (Master's, Data Science)
Database administrators keep business data available, secure and recoverable, while data engineers build the pipelines that move it into reports and models. The two roles overlap heavily in SQL, cloud platforms and governance practice.
Key takeaways
- Administration keeps database systems reliable and recoverable, while engineering moves data dependably between systems.
- SQL, PostgreSQL or MySQL, Python and infrastructure-as-code appear again and again in job postings [1][2][3].
- Governance and audit pressure now shape daily database work, especially in finance [4][5].
- Cloud migrations and platform consolidation are large projects with measurable cost, downtime and staffing stakes [6][7][8].
- A learner builds competence by completing small systems in stages: model, secure, automate, monitor and document.
Almost every organization keeps its operational data in databases: orders, subscriber records, account balances and sensor readings. The people who keep that data correct, fast and recoverable work in database administration or data engineering. Their work sits behind dashboards, fraud checks, billing runs and machine learning models, so when it fails, the damage shows up quickly across the business.
The two roles are distinct but often blur in practice. Some postings ask for a database administrator who also writes Python automation and manages infrastructure code [3]. Others describe a data engineer who must tune SQL and understand production databases as deeply as a DBA [9][10]. This discipline therefore rewards people who understand both how systems stay up and how data moves through them.
This article explains what the work involves, the skills and tools it uses, real projects that show the stakes, typical roles and pay signals from job postings, and the governance risks that now dominate planning. It closes with a staged learning path and a checklist a reader can apply without enrolling in any particular course.
Where these two disciplines begin and end
Database administration is the work of keeping database systems running, secure and recoverable. A database administrator, usually called a DBA, handles installation, upgrades, backups, access control, capacity and performance problems in production. Job descriptions for these roles typically list monitoring, capacity planning, backup and recovery strategy, and high-availability configuration across several database engines [11].
Data engineering is the work of moving data from where it is created to where people and models can use it. A data engineer designs extraction, transformation and loading steps, assembles large data sets for analysis, and keeps those pipelines dependable as sources change [10]. The discipline grew because analysts were spending much of their time preparing data rather than analyzing it, which is why data engineering is now treated as a separate role [12].
The need for both roles grows with scale. The LUX-ZEPLIN experiment, which describes itself as the world's most sensitive dark matter direct-detection experiment, acquires over a petabyte of data and keeps metadata in six heterogeneous databases [13]. A telecom company serving over 15 million subscribers faced more than 30,000 fragmented reporting flows, many running outdated scripts that frequently failed [14].
Fact In the telecom case, executives were making commercial decisions based on lagging data because the on-premises Oracle database beneath the reports was under heavy load [14].
Ownership confusion causes similar problems at smaller scale. Copenhagen Airport described a data setup built on SSIS packages and extensive SQL reporting, with no clear ownership of data domains, so managers could receive different answers to the same question depending on who they asked [15]. Clear ownership and well-built pipelines solve that problem together.
Core skills: data modeling, reliability and security
Data modeling is the starting point. A practitioner must translate a logical data model into a physical database and then manage it over its life [16]. Postings in this area ask for schema design from first principles, not only maintenance, along with partitioning strategies and the ability to lead a migration between database types [3].
Reliability is the second core area. Backup and recovery procedures are described in job postings as the insurance policy of the DBA [16]. Practitioners also need to plan disaster recovery, incident response and replication. Commentary on this topic notes that backups often exist but are rarely tested end-to-end, and that disaster recovery plans are defined but not executed under pressure [17].
Performance and security
Performance work means monitoring production, identifying the cause of slow queries, and changing data structures, SQL or configuration to fix them [16]. Security work covers encryption, column-level access control and secure user management. A capstone project on relational administration lists these skills alongside automated restore procedures and scheduled jobs [18]. Postings also ask for row-level security and least-privilege access in multi-tenant systems [3].
- Data modeling and schema design, including normalization and partitioning
- Backup, restore and disaster recovery testing
- Query tuning, indexing and resource monitoring
- Access control, encryption and audit trails
- Python or shell scripting for repetitive operational work
Opinion In this editor's view, the skill most often underrated by beginners is recovery testing. A backup that has never been restored has not yet been proven to work [17].
Source: vaayaa analysis of cited data
View chart data
| AI use, earlier survey | 15% |
| AI use, later survey | 44% |
| Compliance anxiety | 40% |
| Security and privacy concerns | 64% |
| Formal governance framework | 23% |
Tools, platforms and automation methods
The tool list varies by employer, but several patterns recur. Job postings name Oracle, SQL Server, SAP HANA, Linux and Windows Server for enterprise administration [11]. Others combine MySQL and MongoDB with BigQuery and cloud services such as RDS and Aurora [2]. A Python and Azure SQL stack might include App Service, Azure Functions, Data Factory and API Management, depending on whether the workload is an API, a scheduled pipeline or analytics [19].
Pipelines usually depend on orchestration tools. A capstone project built ETL workflows with Apache Airflow, while another moved data from transactional MySQL and NoSQL MongoDB stores into a staging warehouse on PostgreSQL before loading a production warehouse [20][18]. Spark appears in the same project for analytics connections [20].
Automation and change control
Infrastructure-as-code tools such as Terraform and Ansible appear in many postings [2][3]. Change data capture, which streams row changes from a source database, is often built with Debezium and Kafka [2][3]. Automated schema change tools such as Liquibase and Flyway are listed as bonus skills, and a governance guide from the vendor Liquibase argues that manual database changes remain common in banks [21][3].
Guidance for pipeline design also gives concrete limits. One documentation set for a data pipeline tool recommends 10 to 30 blocks per pipeline, a maximum of 5 to 7 levels of dependency, and 5 to 10 parallel branches, and it suggests splitting larger work into multiple pipelines [22]. Treat such figures as vendor guidance for one tool, not universal rules.
| Aspect | Database administration | Data engineering | Database migration | Basis |
|---|---|---|---|---|
| Main purpose | Keep production databases available, secure and recoverable [11][17] | Build pipelines that extract, transform and load data [10] | Move systems to a new platform with controlled cutovers [6][8] | Fact |
| Typical tools | Oracle, SQL Server, SAP HANA on Linux and Windows [11] | SQL, Apache Airflow, Spark, Python [3][20] | AWS DMS and SCT, RDS for Oracle [1][6] | Fact |
| Experience often asked | 7 to 10 years for a senior SQL Server DBA [9] | 6 to 10 years for a senior data engineer [3] | 8 to 12 years for a migration DBA [1] | Fact |
| Typical risk | Backups that are rarely tested end-to-end [17] | Fragmented pipelines that fail silently [13][14] | Downtime and rollback failures during cutover [8][6] | Fact |
| Where to start | Relational engine, backups, access control [18] | SQL queries and scheduled ETL jobs [20] | One source and one target with a test cutover [8] | Opinion |
Real-world projects: migration, reporting and governed change
Large migrations show the work at full scale. U.S. Bank planned to move four Oracle Exadata databases totaling approximately 119 TB to Amazon RDS for Oracle over a shared AWS Direct Connect link, under a strict deadline to leave its on-premises data centers [6]. The migration used discrete transfer tasks that retried after transient network problems, with shared storage at the center of the pipeline [6].
A second migration involved smaller teams. Plaid moved 234 databases across 100 services from Amazon Aurora MySQL to self-hosted TiDB, and it chose controlled cutovers with 60 to 120 seconds of writer downtime instead of zero-downtime complexity [8]. Its team categorized services by business criticality and frontloaded its riskiest services to reduce program risk [8].
Fact Plaid's program reported that each of three TiDB upgrades cost four engineer weeks, compared with 12 engineer weeks for the same kind of upgrade on Amazon Aurora, according to the company's webinar description [8].
Cost work is another common project. bigbasket, an online grocery platform, worked with a managed services provider to stabilize and optimize its RDS MySQL environment, reporting 33% in AWS cost savings through instance rightsizing, storage optimization and database tuning [7]. Its case also shows that instability and cost problems often arrive together.
Regulated industries add a further layer. A finance team may need to prove exactly who changed a schema, when a rollback plan existed and what policy checks ran [21]. Copenhagen Airport reported that it built a new scalable data platform and eliminated the technical debt and legacy data technology that had caused conflicting reports [15].
Source: vaayaa analysis of cited data
View chart data
| Compliance audit | 81% |
| Security as strategic concern | 79% |
| Database DevOps tooling | 55% |
| Use AI for database management | 47% |
| Permanent hybrid model | 42% |
| Four or more database platforms | 36% |
| Governance or quality frameworks | 14% |
Roles, job requirements and pay signals
The job titles vary: database administrator, database reliability engineer, data platform engineer, senior database engineer and data engineering consultant all appear in current postings [2][9][10][3]. Titles matter less than the mix of duties. A platform engineer may own schema design, CI/CD pipelines and disaster recovery together, while a consultant may focus on migrating SQL Server workloads to cloud services [9][3].
Experience requirements differ. One migration-focused role asks for 8 to 12 years of experience [1]. A senior SQL Server role asks for 7 to 10 years [9]. A data platform engineering role asks for 4 to 9 years of data, database or platform work in production [3]. Employers often combine seniority levels with overlapping skills, so entry paths usually start with a narrower role and grow.
- Database administrator: operations, backups, performance and access control [11]
- Data engineer: pipelines, transformation and data delivery [10][20]
- Migration specialist: moving workloads between platforms with minimal downtime [1][6]
- Platform or reliability engineer: infrastructure-as-code, monitoring and incident response [2][3]
Posted pay ranges are one signal among many. One migration posting lists a range of ₹20L to ₹40L, and a data platform posting lists ₹15L to ₹21L, both as yearly pay [1][3]. These are single job postings, so they show what employers offered in those listings, not typical pay across the field.
To build competence step by step, a learner can start with SQL and one relational engine, add Python for automation, then learn backups and access control on that same engine. Capstone-style projects that combine an operational database, a warehouse and a scheduled pipeline give a learner concrete evidence to show an employer [20][18].
Governance, compliance and the risk of fast change
Governance is now a central part of the work. Redgate's AI edition survey reported that 99% of respondents using AI see at least one measurable benefit for database work. The same dataset reported security and privacy concerns at 64% and regulatory compliance anxiety at 40%, and only 23% said their organization had a formal data governance framework [23].
Fact In Redgate's finance edition survey of 2,150 IT professionals worldwide, 81% of finance organizations reported undergoing a compliance audit, and 79% of respondents described database security as a strategic concern [4][24].
Regulation shapes these patterns. Frameworks named in one governance guide include SOX, PCI DSS, SOC 2 and DORA, and the guide argues that manual database changes leave gaps in policy checks, rollback planning and audit records [21]. A DORA-focused guide notes that 19 critical ICT third-party providers have been designated for direct oversight [5].
Commentators also warn about speed. One article argues that AI is accelerating changes across environments that already had inconsistent foundations, which amplifies existing gaps rather than creating new ones [23]. Another notes that AI can take on repetitive tuning and documentation, while people still need to judge when to trust a recommendation [25].
Opinion Managing risk does not mean slowing all change. The safer path is to automate routine changes with review, logging and rollback, so that speed and control rise together rather than trade off.
The learner who understands this area can speak about data protection rules in concrete terms. Frameworks such as GDPR, CCPA and HIPAA are named in governance guidance as difficult to manage without repeatable database processes [26]. This article describes the work only and does not offer legal advice on how any rule applies to a specific organization.
Original analysis
vaayaa analysis: gaps between AI adoption, security and governance in the published survey figures
- Take the AI usage figures from the AI edition survey: 15% at the earlier measurement and 44% at the later one [23].
- Subtract the earlier figure from the later figure: 44 minus 15 gives a rise of 29 percentage points.
- Divide the later figure by the earlier one: 44 divided by 15 gives about 2.9, so usage grew by nearly three times.
- Take the security and privacy concern figure of 64% and the formal governance framework figure of 23% from the same dataset [23].
- Subtract the smaller from the larger: 64 minus 23 gives a gap of 41 percentage points between reported concern and formal controls.
- Take the Plaid engineer week figures of 4 and 12 and divide 12 by 4 to get 3, meaning the Aurora upgrade cost three times the effort [8].
The sources show that adoption grew faster than formal controls, with a 41 point gap between concern and governance. The Plaid figures suggest that automation and disciplined upgrades can cut effort sharply. These are calculations from published figures, not new surveys or experiments.
A staged learning path for this discipline
- 1
Learn SQL and one engine
Write queries, joins and indexes on a single relational database until you can explain each query plan.
- 2
Model a real domain
Design a schema for a simple business, including keys, constraints and partitioning decisions for growth [16][3].
- 3
Automate operations
Write Python scripts for backups, restore tests and row-count checks, then schedule them [3][18].
- 4
Build a pipeline
Create an extraction, transformation and loading workflow with an orchestration tool and a warehouse target [20][22].
- 5
Govern and document
Add access control, audit logs and a written rollback procedure, then test the rollback before you call the work finished [21][17].
Worked example
Planning a database migration from published figures
- Start with the U.S. Bank volume of approximately 119 TB spread across four Oracle Exadata databases [6].
- Divide 119 TB by 4 databases to get an average of 29.75 TB per database.
- Assume each database will be moved in its own transfer task, as the case describes discrete tasks for each tablespace [6].
- Assume a retry allowance of one extra cycle per failed transfer, following the case's description of automatic retries [6].
- List the checks to run before cutover: row counts, checksums and a restore test on a separate instance [17].
- Record the cutover window for each database and the rollback steps, then review them with the team [8].
A planner who divides the total by the number of databases gets a per-database size of 29.75 TB. That figure, plus a defined retry rule and a tested rollback, turns a large migration into a set of smaller, checkable steps.
Practical checklist before you start learning or changing a production database
- Write down which databases you would own, and who owns each data domain [15].
- Confirm that backups exist and that you have restored one to a test system [17].
- Document the data model, keys and partitioning choices for each core table [16][3].
- Automate one repetitive task with a script and store its output in a log [3][18].
- Check which access controls, encryption settings and audit records apply to sensitive data [3][18].
- Plan a rollback for every schema change and keep the change under version control [21].
- Record a baseline of query performance before tuning, then measure again afterward [11].
Frequently asked questions
Do I need a degree before working in this field?
Job postings vary. Many list years of production experience rather than a specific credential, and some ask for a mix of work and education [16]. A learner can build evidence through projects, documented capstones and steady practice with SQL, Python and one database engine [20][18].
Which should I learn first, database administration or data engineering?
Most people start with SQL and a relational engine, since both roles depend on it. After that, choose by interest: operations and reliability lead toward administration, while transformation and pipelines lead toward engineering [11][10]. Many postings combine both areas, so early work in both is useful.
Is cloud experience required?
Cloud experience appears often in postings, including managed RDS and Aurora services, Azure SQL and cloud warehouses [1][2][9][19]. Learn the basics of one cloud platform and the reasons teams use managed services, because many migrations move workloads from on-premises systems.
How important is automation?
Automation is repeatedly listed as a core skill. Postings ask for Python-based automation, infrastructure-as-code with Terraform and CI/CD pipelines [2][3]. Start with one small script that replaces a manual task, then add scheduling, logging and tests.
What is the biggest risk for newcomers?
A common risk is changing production systems without a tested recovery plan. Commentary notes that backups are rarely tested end-to-end [17]. Practice restores on a separate system and write rollback steps before making any change.
How do I show progress without paid credentials?
Publish documented projects that include a data model, a pipeline, automation scripts, a governance checklist and a short write-up of decisions. Capstone-style repositories that describe architecture and tradeoffs show skill to a reader without any certificate [20][18].
Conclusion
Database administration and data engineering both exist to make data reliable. Administrators protect systems from failure, security problems and slow queries, while engineers make sure data arrives in usable form. Job postings, migration case studies and governance surveys all show that the two roles increasingly share tools and responsibilities [11][3][8][4].
A learner should start with SQL, a single relational engine and simple automation, then add pipelines, governance and recovery testing. Keep records of each project. Clear documentation, tested restores and disciplined change control are the habits that employers and auditors look for first [21][17][18].
Learning sources
References
- cutshort.io. Suzuki Digital is hiring Database Administrator (PostgreSQL) job in Gurugram | Cutshort - - AWS Certified Database - Specialty.
- lenskartcareers.recruitee.com. Lenskart - Database Administrator (MySQL).
- cutshort.io. Wissen Technology is hiring Database Administrator job in Bengaluru (Bangalore) | Cutshort - - Build Python-based automation to eliminate repetitive DB work.
- tmcnet.com. 81% of Finance Organizations Hit by Audits in 12 Months as Hybrid Database Complexity Reaches a Breaking Point - | [May 13, ].
- liquibase.com. DORA Compliance in : What Article 9 Requires at the Database Layer.
- aws.amazon.com. How U.S. Bank migrated 119 TB from Oracle Exadata to Amazon RDS for Oracle | Amazon Web Services.
- percona.com. bigbasket Cuts AWS Database Costs by 33% with Percona ExpertOps.
- pingcap.com. How Plaid Migrated 234 Databases to TiDB | Webinar.
- careers.unitedhealthgroup.com. Data Engineering Consultant at UnitedHealth Group.
- careers.tieto.com. Senior DataBase Engineer - Tietoevry Create (m/f/d).
- bayt.com. Sr. Database Engineer US Shift at TD SYNNEX - India - Bayt.com - Submitting more applications increases your chances of landing a job.
- learning.dell.com. D&LL Technologies.
- zenodo-rdm.web.cern.ch. Cloud-Native Graphical and Programmatic Cross-Database Access to Experiment Metadata with Trino, Superset and Ibis.
- itrexgroup.com. Data warehouse modernization for telecom giant.
- ing.dk. How Copenhagen Airport built a new AI framework in 18 months and eliminated all technical debt | Ingeniøren.
- jobs.pnptc.com. Plug and Play Job Board.
- dev.sirma.com. The Cloud Didn't Replace the DBA - It Elevated the Role | Sirma.
- github.com. Relational-Database-Administration-Capstone-Project/README.md at 80bded2a836d782bff89b02bd4337a135e104bc7 · Willie-Conway/Relational-Database-Administration-Capstone-Project - Skip to content.
- learn.microsoft.com. best practices working with Azure - Microsoft Q&A.
- github.com. GitHub - Sidkian/SoftCart-Data-Engineering-Project: IBM Data Engineering Professional Certificate Capstone · GitHub - GitHub - Sidkian/SoftCart-Data-Engineering-Project: IBM Data Engineering Professional Certificate Capstone · GitHub.
- cfotech.co.nz. Liquibase urges banks to fix manual database changes.
- docs.mage.ai. Best Practices - Mage AI.
- red-gate.com. 99% of database professionals are seeing AI benefits. So why are the security challenges increasing? - 99% of database professionals are seeing AI benefits. So why are the security challenges increasing?.
- itbrief.in. Finance firms face rising audit pressure over databases - IT Brief India - Technology news for CIOs & IT decision-makers.
- red-gate.com. Progress Without Control in the Age of AI and Compliance - Progress Without Control in the Age of AI and Compliance.
- liquibase.com. Data & database governance challenges that risk pipeline productivity & potential.