Data Engineer

Job Description

We are looking for a skilled Databricks Engineer to design, build, and optimize analytics pipelines while ensuring seamless integration of machine learning models into production. The ideal candidate will have strong expertise in Databricks, data engineering, and MLOps, with a deep understanding of cloud-based architectures.

• Key Responsibilities:
• • Design and develop scalable data pipelines on Databricks using Apache Spark and Delta Lake.
  • Implement ETL/ELT processes for structured and unstructured data from various sources.
  • Develop and maintain end-to-end data solutions, including ingestion, transformation, and serving layers.
  • Collaborate with data scientists to operationalize machine learning models in Databricks, ensuring performance, monitoring, and governance.
  • Optimize data workflows and queries for cost and performance efficiency.
  • Ensure best practices in data security, governance, and compliance.
  • Troubleshoot issues and optimize Databricks infrastructure and performance
• Required Skills & Qualifications:
• • 5+ years of experience in data engineering, analytics, or ML operations.
  • Proficiency in Databricks, Apache Spark, and Delta Lake.
  • Process large datasets with Apache Spark, Delta Lake, and PySpark to ensure high-performance model training.
  • Strong programming skills in Python, PySpark, or SQL.
  • Experience with cloud platforms (AWS, Azure, or GCP) and data lake architectures.
  • Implement automated ML pipelines, model versioning, and A/B testing using Databricks and cloud services.
  • Experience with machine learning models, linear and logistic regression, decision trees, random forests, and gradient boosting, clustering
  • Build, train, and optimize ML models using Databricks MLflow, AutoML, and Feature Store.
  • Experience with workflow orchestration tools like Apache Airflow or Azure Data Factory.
  • Familiarity with CI/CD practices
  • Strong problem-solving and analytical skills.
• Preferred Qualifications:
• • Databricks certification (e.g., Databricks Certified Data Engineer Professional, Certified Machine Learning Professional).
  • Prior experience in BFSI domain