Senior Data Engineer

Team: Data

Location: Remote

Commitment: Full time

Workplace Type: remote

Job Summary

Essential Functions:

In addition to working in accordance with appropriate conduct and behavioral standards, specific responsibilities of this role include:
Design and build robust data pipelines and lakehouse solutions using Databricks, Microsoft Fabric, PySpark, and Python. 
Design AI-ready schemas, semantic layers, and metadata structures that allow AI agents to query data accurately and reliably. 
Build automated validation processes for agent-generated queries. 
Pilot agentic automation for data quality and reconciliation workflows while measuring accuracy and maintaining clear rollback plans. 
Curate and maintain the semantic layer that grounds AI agents and reduces incorrect responses caused by ambiguous or poorly defined data. 
Catalog and steward enterprise data assets to support internal analytics, financial planning, and product engineering needs. 
Partner proactively with Product Engineering, FP&A, and other stakeholders to identify and understand data requirements. 
Troubleshoot performance issues and support SQL Server and Azure SQL environments while continuing to develop operational expertise. 
Mentor junior engineers on data engineering best practices, architecture patterns, and technical standards. 
Contribute to data protection, security, and compliance reviews to ensure that tools and processes meet applicable standards, including HIPAA requirements where relevant. 
Carry out additional responsibilities assigned based on business needs. 

Minimum Requirements:

• 7+ years of production experience in data engineering roles, including hands-on experience building and maintaining data pipelines at scale. 
• Advanced proficiency with Python and PySpark or Apache Spark in production data engineering environments. 
• Hands-on production experience with Databricks, Microsoft Fabric, or comparable modern data platforms. 
• Demonstrated experience designing, implementing, and operating lakehouse architectures using Delta Lake, Parquet, or equivalent technologies. 
• Experience designing AI-ready schemas, semantic layers, or metadata structures with downstream AI or machine learning query patterns in mind, or experience building automated validation for programmatically generated queries. 
• Experience with agentic or automation-driven data quality workflows, such as AI-assisted reconciliation, agent-grounded semantic layers, or comparable applied solutions. 
• Demonstrated ability to partner directly with product, analytics, finance, or business stakeholders to understand and shape data requirements proactively. 
• Familiarity with Microsoft Azure cloud concepts, including platform as a service, infrastructure as a service, Azure SQL, and hybrid cloud models. 
• Willingness and ability to develop deeper operational expertise within Azure SQL and SQL Server environments. 
• Meaningful experience mentoring junior or mid-level engineers on technical standards, architecture patterns, and data engineering best practices.

Preferred Qualifications:

Hands-on production experience with Azure SQL, SQL Server, or Azure SQL Managed Instance. 
Proficiency with T-SQL and PowerShell for automation and operational support. 
Experience with infrastructure-as-code tools such as Terraform, Bicep, or Azure Resource Manager templates. 
Familiarity with streaming data architectures and technologies such as Azure Event Hubs or Apache Kafka. 
Understanding of data mesh principles and distributed data ownership. 
Experience working within healthcare or another regulated industry. 
Familiarity with HIPAA requirements or healthcare data standards such as FHIR and HL7.

Working Environment/Physical Demands:

• 100% Remote in the United States
• Activities require a significant amount of work in front of a computer monitor
• Long-distance or air travel as needed – not to exceed 10% travel