Senior Data Engineer – Plano, TX | Full-Time / Hybrid
Position Overview
We are seeking a highly skilled and experiencedSenior Data Engineerto help design, develop, and optimize a modern cloud-based data platform.
This role will focus heavily onAzure Databricks, PySpark, Spark SQL, scalable ETL/ELT pipelines, and data lakehouse architecture. The ideal candidate will have experience working in high-volume data environments and collaborating with data architects, data scientists, business stakeholders, and cloud infrastructure teams to deliver reliable, secure, and high-performance data solutions.
Key Responsibilities
Data Architecture & Engineering
- Design and implement scalable ETL/ELT pipelines using Azure Databricks, PySpark, and Spark SQL.
- Build robust data ingestion workflows from diverse sources, including APIs, event-streaming platforms, relational databases, and cloud storage.
- Develop and maintain Bronze, Silver, and Gold data layers using a modern medallion architecture.
- Establish best practices for Delta Lake, schema evolution, data versioning, and governance.
- Optimize large-scale data pipelines for performance, reliability, and cost efficiency through partitioning, caching, clustering, and Spark configuration tuning.
Cloud Data Platform
- Integrate Databricks with Azure cloud services, including cloud data lakes, data integration services, event-streaming platforms, data warehouses, secrets management, and CI/CD tooling.
- Implement secure and governed access to enterprise data using managed identities, role-based access controls, and data protection techniques.
- Support the development and evolution of a scalable cloud-native data platform.
Collaboration & Technical Leadership
- Partner with data analysts, data scientists, business stakeholders, and other technical teams to understand data requirements.
- Build trusted datasets, reporting structures, and business-critical KPIs.
- Provide technical leadership and mentorship to junior and mid-level engineers.
- Conduct code reviews and contribute to architectural and engineering standards.
- Collaborate with architecture, security, and infrastructure teams to ensure data solutions meet organizational security and compliance requirements.
Data Quality & Monitoring
- Implement data quality checks, validation frameworks, alerts, and lineage tracking.
- Develop monitoring and observability solutions for mission-critical data pipelines.
- Establish automated failure detection, alerting, and recovery processes.
- Continuously improve data reliability, scalability, and operational performance.
Required Qualifications
- Bachelor's or Master's degree in Computer Science, Engineering, Information Systems, or a related technical field.
- 5+ years of professional data engineering experience, including3+ years working extensively with Azure Databricks and Apache Spark.
- Strong expertise with Delta Lake, Unity Catalog, Databricks Notebooks, and SQL-based data warehousing.
- Hands-on experience with Azure data services and cloud infrastructure.
- Strong programming skills inPython and SQL; familiarity with Scala is a plus.
- Solid understanding of data modeling, including OLTP/OLAP environments and star/snowflake schemas.
- Experience with performance optimization and tuning of large-scale data workloads.
- Experience implementing CI/CD for data engineering environments using tools such as Azure DevOps, Terraform, or Databricks CLI.
- Strong understanding of data security, governance, and access-control practices.
Preferred Qualifications
- Databricks certification or comparable professional certification.
- Experience building real-time or streaming data pipelines using technologies such as Structured Streaming, Event Hubs, or Kafka.
- Experience supporting machine learning workflows or ML platforms.
- Familiarity with business intelligence and visualization platforms.
- Experience with data cataloging, lineage, and enterprise governance tools.
- Knowledge of data privacy, security, and compliance frameworks.
- Experience working in high-volume, production data environments.
Work Arrangement
- Location:Plano, TX
- Schedule:Full-Time
- Work Arrangement:Hybrid 4 days per week on-site
- Remote:1 day per week
- Relocation Assistance:Not Provided
- Work Authorization:Candidates must be authorized to work in the United States. Sponsorship is not available for this position.
- Local Requirement:Candidates must be able to commute to the Plano/Dallas–Fort Worth area and work on-site four days per week.
Compensation & Benefits
The organization offers competitive compensation and a comprehensive benefits package. Additional details regarding compensation and benefits will be discussed during the interview process.
