4Bell Technology

Job Description

Key Responsibilities:

●      Modernize legacy, non-scalable architectures and define a scalable target-state platform

●      Design and implement Medallion Architecture (Bronze, Silver, Gold) using Delta Lake and Databricks components such as Delta Live Tables (DLT), Delta Sharing and Workflows (LakeFlow, LakeBase)

●      Design and build scalable, reliable and production-ready ETL/ELT pipelines using PySpark, SQL and Databricks notebooks to ingest and transform data from diverse sources.

●      Create and manage workflows using Databricks Workflows (Jobs) or orchestration tools to automate pipelines and dependencies

●      Optimize Spark jobs for performance, scalability and cost efficiency (partitioning, caching, query tuning, cluster optimization)

●      Implement data quality checks (e.g., DLT Expectations) and enforce governance via Unity Catalog (access control, PII masking, lineage)

●      Design and implement event-driven and streaming pipelines (Kafka or equivalent)

●      Ensure high data reliability through monitoring, observability and alerting

Requirements:

●      BE/MS in Computer Science or a related field with 8+ years of experience in data engineering

●      Strong experience in ETL/ELT pipelines, data modeling, and distributed data systems, with hands-on expertise in Databricks

●      Deep proficiency in PySpark, including performance optimization, job orchestration, and large-scale data processing

●      Good understanding of event streaming systems such as Kafka or equivalent technologies

●      Experience working with Azure data ecosystem (ADLS, Azure services, AHDS or similar data platforms)

●      Strong foundation in event-driven architecture and scalable distributed systems

●      Proven ability to design, review, and clearly articulate system architecture

●      Experience leveraging AI-assisted development tools (e.g., Claude, Antigravity, etc.) to improve productivity

●      Solid experience in Agile delivery, estimation, and program execution

●      Experience working with global customers (US/EU) in a client-facing role

●      Excellent written and verbal communication skills across engineering, business, and customer stakeholders

●      Strong analytical thinking and structured problem-solving ability

●      High ownership, reliability, and execution focus; consistently delivers despite constraints

●      Strong attention to detail while maintaining a clear big-picture perspective