Sr Data Engineer
Role – Senior Data Engineer (Databricks)
Experience – 8+ years
Location – Remote
Required Skills:-
· 8+ years in Data Engineering project development/design, handling large volumes of data.
· At least 5+ years using cloud data engineering services (preferably on AWS).
· 5+ years as a Big Data architect/solution architect on Big Data platforms.
· 3+ years of proven experience as an architect/solution architect on the
Databricks platform.
· Designed and implemented at least 2-3 end-to-end projects in Databricks.
· Expertise in E2E architecture of unified data platforms covering data lifecycle aspects:
· Data Ingestion, Transformation, Serving, and Consumption.
· Experience in composable architecture to fully leverage Databricks
capabilities.
· Implemented and configured Databricks environments (clusters, notebooks, libraries) for optimal performance and resource utilization.
· Experience in integrating data from various sources (structured, semi-structured, and (unstructured) into Databricks for processing and analysis
Technical Skills:
· Designed and developed scalable batch and streaming data pipelines, data lake architectures, and data warehousing solutions on the Databricks platform using Spark and Delta Lake.
· Knowledgeable in the Databricks Lakehouse concept and its enterprise implementation.
· Strong understanding of data warehousing, governance, and security standards related to Databricks.
· Skilled in cluster optimization, integration with various cloud services, and performance optimization to improve efficiency and reduce costs.
· Proficient in writing unit and integration tests, and setting best practices for Databricks CI/CD.
Qualifications:
· Proven experience as an Architect/Solution Architect on Databricks.
· Hands-on experience with AWS Databricks platform for data processing, warehousing, and analytics solutions.
· Strong background in cloud data engineering, ETL, data integration, and data governance.
Responsibilities:
· Enforce adherence to architectural standards, global product-specific guidelines, and usability design standards.
· Proactively guide engineering methodologies, standards, and leading practices.
Identify, communicate, and mitigate risks, assumptions, issues, and decisions throughout the full lifecycle.
· Provide guidelines for best practices and repeatable methodologies in Cloud Data Engineering, Data Storage, ETL (Extract, Transform, Load), Data Integration & Migration, Data Warehousing, and Data Governance.