Spark / Databricks Platform Engineer
Hybrid - US
Spark / Databricks Platform Engineer
Full Time • Hybrid - US
Benefits:
- Competitive salary
Location
Austin, TX
Austin, TX
Experience Level
Senior Level (8 or more years of experience)
Senior Level (8 or more years of experience)
Role Overview
The Databricks Engineer designs, develops, and optimizes scalable data solutions on the Databricks platform, leveraging PySpark or Scala for large-scale data processing. Operating with a high degree of creativity and latitude, this role involves constructing ingestion pipelines, implementing medallion architecture (Bronze, Silver, Gold layers), and designing robust dimensional data models. The engineer collaborates with cross-functional stakeholders to gather requirements, perform cost/benefit analyses of alternative solutions, and implement data governance, security, and quality frameworks to support enterprise public health and business intelligence initiatives.
The Databricks Engineer designs, develops, and optimizes scalable data solutions on the Databricks platform, leveraging PySpark or Scala for large-scale data processing. Operating with a high degree of creativity and latitude, this role involves constructing ingestion pipelines, implementing medallion architecture (Bronze, Silver, Gold layers), and designing robust dimensional data models. The engineer collaborates with cross-functional stakeholders to gather requirements, perform cost/benefit analyses of alternative solutions, and implement data governance, security, and quality frameworks to support enterprise public health and business intelligence initiatives.
Key Responsibilities
Data Pipeline & Platform Engineering
• Design, develop, and optimize highly scalable data solutions on Databricks using Apache Spark, PySpark, or Scala for large-scale data processing.
• Build, configure, and maintain robust ETL/ELT ingestion pipelines utilizing Lakeflow Declarative Pipelines (formerly Delta Live Tables / DLT).
• Implement and maintain Delta Lake architectures and medallion database structures across Bronze, Silver, and Gold data layers.
• Orchestrate, schedule, and monitor offline production jobs utilizing Lakeflow Jobs (formerly Databricks Workflows) or equivalent tools such as Apache Airflow.
• Design, develop, and optimize highly scalable data solutions on Databricks using Apache Spark, PySpark, or Scala for large-scale data processing.
• Build, configure, and maintain robust ETL/ELT ingestion pipelines utilizing Lakeflow Declarative Pipelines (formerly Delta Live Tables / DLT).
• Implement and maintain Delta Lake architectures and medallion database structures across Bronze, Silver, and Gold data layers.
• Orchestrate, schedule, and monitor offline production jobs utilizing Lakeflow Jobs (formerly Databricks Workflows) or equivalent tools such as Apache Airflow.
Data Modeling & Quality Governance
• Design, develop, and support comprehensive enterprise data warehouses utilizing dimensional data modeling, including Star and Snowflake schemas.
• Implement rigorous data quality validation, data security controls, platform access models, and data governance frameworks.
• Analyze system specifications, evaluate operational limitations, and perform data audits to ensure scalability, security, and cost efficiency.
• Design, develop, and support comprehensive enterprise data warehouses utilizing dimensional data modeling, including Star and Snowflake schemas.
• Implement rigorous data quality validation, data security controls, platform access models, and data governance frameworks.
• Analyze system specifications, evaluate operational limitations, and perform data audits to ensure scalability, security, and cost efficiency.
Business Analysis & Operational Insights
• Collaborate directly with business stakeholders, program managers, and technical peers to understand operational objectives, identify problems, and analyze current procedures.
• Translate high-level business goals into formal technical requirements, system design documentation, and cost/benefit analyses.
• Develop dynamic analytical dashboards and reporting solutions natively within Databricks, including Databricks SQL dashboards and Databricks Apps, to deliver actionable operational insights.
• Collaborate directly with business stakeholders, program managers, and technical peers to understand operational objectives, identify problems, and analyze current procedures.
• Translate high-level business goals into formal technical requirements, system design documentation, and cost/benefit analyses.
• Develop dynamic analytical dashboards and reporting solutions natively within Databricks, including Databricks SQL dashboards and Databricks Apps, to deliver actionable operational insights.
Required Qualifications
• 8 or more years of experience in IT, supporting the design, development, deployment, or delivery of technology solutions.
• 8 or more years of experience with Databricks, including building and optimizing ETL/ELT data pipelines using Apache Spark.
• 8 or more years of experience in data warehousing and dimensional data modeling, including Star and Snowflake schemas.
• 8 or more years of professional proficiency utilizing SQL and Python (or Scala) for large-scale data processing.
• 8 or more years of experience designing and developing dashboards and applications natively within Databricks, such as Databricks SQL dashboards or Databricks Apps.
• 8 or more years of experience implementing data governance, data quality metrics, and data security practices.
• 8 or more years of experience implementing Lakeflow Declarative Pipelines (formerly Delta Live Tables / DLT) to build and manage production pipelines.
• 8 or more years of experience with Delta Lake, medallion architecture (Bronze, Silver, Gold layers), data lakehouse design, and scheduling offline jobs using Lakeflow Jobs (formerly Databricks Workflows) or similar orchestration tools.
• 8 or more years of experience communicating technical specifications and presenting data-driven insights to technical and non-technical stakeholders.
• 8 or more years of experience in IT, supporting the design, development, deployment, or delivery of technology solutions.
• 8 or more years of experience with Databricks, including building and optimizing ETL/ELT data pipelines using Apache Spark.
• 8 or more years of experience in data warehousing and dimensional data modeling, including Star and Snowflake schemas.
• 8 or more years of professional proficiency utilizing SQL and Python (or Scala) for large-scale data processing.
• 8 or more years of experience designing and developing dashboards and applications natively within Databricks, such as Databricks SQL dashboards or Databricks Apps.
• 8 or more years of experience implementing data governance, data quality metrics, and data security practices.
• 8 or more years of experience implementing Lakeflow Declarative Pipelines (formerly Delta Live Tables / DLT) to build and manage production pipelines.
• 8 or more years of experience with Delta Lake, medallion architecture (Bronze, Silver, Gold layers), data lakehouse design, and scheduling offline jobs using Lakeflow Jobs (formerly Databricks Workflows) or similar orchestration tools.
• 8 or more years of experience communicating technical specifications and presenting data-driven insights to technical and non-technical stakeholders.
Preferred Qualifications
• 1 or more years of experience working within public sector or state government environments.
• 1 or more years of experience implementing CI/CD practices for data pipelines, including DevOps and Git-based version control workflows.
• Professional Databricks certification, such as Databricks Certified Data Engineer Associate or Professional.
• 1 or more years of experience working within public sector or state government environments.
• 1 or more years of experience implementing CI/CD practices for data pipelines, including DevOps and Git-based version control workflows.
• Professional Databricks certification, such as Databricks Certified Data Engineer Associate or Professional.
Core Skills & Attributes
• Strong analytical, critical-thinking, and problem-solving skills with a high degree of creativity and latitude.
• Excellent written and verbal communication skills to write detailed descriptions of user needs and document system capabilities.
• Proactive, self-motivated approach to managing complex, cross-functional datasets and resolving workload issues.
• Strong collaborative capability to work effectively with data engineers, database administrators, and business partners.
• Strong analytical, critical-thinking, and problem-solving skills with a high degree of creativity and latitude.
• Excellent written and verbal communication skills to write detailed descriptions of user needs and document system capabilities.
• Proactive, self-motivated approach to managing complex, cross-functional datasets and resolving workload issues.
• Strong collaborative capability to work effectively with data engineers, database administrators, and business partners.
Flexible work from home options available.
Established in 2004, Smart Tech Skills is a top technology and professional services firm specializing in innovative technologies.
Headquartered in Marlborough, MA, the company effectively addresses clients’ technology needs nationwide, making advanced technology management easier.
Headquartered in Marlborough, MA, the company effectively addresses clients’ technology needs nationwide, making advanced technology management easier.
(if you already have a resume on Indeed)