Sr Data Engineer

  • Anywhere

Role – Senior Data Engineer (Databricks)

Experience – 8+ years

Location – Remote

 

Required Skills:-

 

·       8+ years in Data Engineering project development/design, handling large volumes of data.

·       At least 5+ years using cloud data engineering services (preferably on AWS).

·       5+ years as a Big Data architect/solution architect on Big Data platforms.

·       3+ years of proven experience as an architect/solution architect on the

Databricks platform.

·       Designed and implemented at least 2-3 end-to-end projects in Databricks.

·       Expertise in E2E architecture of unified data platforms covering data lifecycle aspects:

·        Data Ingestion, Transformation, Serving, and Consumption.

·       Experience in composable architecture to fully leverage Databricks

capabilities.

·       Implemented and configured Databricks environments (clusters, notebooks, libraries) for optimal performance and resource utilization.

·       Experience in integrating data from various sources (structured, semi-structured, and (unstructured) into Databricks for processing and analysis

 

Technical Skills:

 

·       Designed and developed scalable batch and streaming data pipelines, data lake architectures, and data warehousing solutions on the Databricks platform using Spark and Delta Lake.

·       Knowledgeable in the Databricks Lakehouse concept and its enterprise implementation.

·       Strong understanding of data warehousing, governance, and security standards related to Databricks.

·       Skilled in cluster optimization, integration with various cloud services, and performance optimization to improve efficiency and reduce costs.

·       Proficient in writing unit and integration tests, and setting best practices for Databricks CI/CD.

 

Qualifications:

 

·       Proven experience as an Architect/Solution Architect on Databricks.

·       Hands-on experience with AWS Databricks platform for data processing, warehousing, and analytics solutions.

·       Strong background in cloud data engineering, ETL, data integration, and data governance.

 

Responsibilities:

 

·       Enforce adherence to architectural standards, global product-specific guidelines, and usability design standards.

·       Proactively guide engineering methodologies, standards, and leading practices.
Identify, communicate, and mitigate risks, assumptions, issues, and decisions throughout the full lifecycle.

·       Provide guidelines for best practices and repeatable methodologies in Cloud Data Engineering, Data Storage, ETL (Extract, Transform, Load), Data Integration & Migration, Data Warehousing, and Data Governance.