Job Description: –
* Design, build, and optimize data pipelines and ETL/ELT workflows using Databricks and Apache Spark (PySpark).
* Develop scalable, high performance data solutions using Spark distributed processing.
* Lead engineering initiatives focused on automation, performance tuning, and platform modernization.
* Implement and manage CI/CD pipelines using Git-based workflows and tools such as GitHub Actions or Jenkins.
* Collaborate with cross-functional teams to translate business needs into technical solutions.
* Ensure data quality, governance, and security across all processes.
* Troubleshoot and optimize Spark jobs, Databricks clusters, and workflows.
* Participate in code reviews and develop reusable engineering frameworks.
* Should have knowledge of utilizing AI tools to improve productivity and support daily
engineering activities.
* Strong knowledge and hands-on experience in Databricks Genie, including prompt engineering, workspace usage, and automation.
Required Skills & Experience:
* 5+ years of experience in Data Engineering or related fields.
* Strong hands-on expertise in Databricks (notebooks, Delta Lake, job orchestration).
* Deep knowledge of Apache Spark (PySpark, Spark SQL, optimization techniques).
* Strong proficiency in Python for data processing, automation, and framework
development.
* Strong proficiency in SQL, including complex queries, performance tuning, and analytical
functions.
* Strong knowledge of Databricks Genie and leveraging it for engineering workflows.
* Strong experience with CI/CD and Git-based development workflows.
* Proficiency in data modeling and ETL/ELT pipeline design.
* Experience with automation frameworks and scheduling tools.
* Solid understanding of distributed systems and big data concepts