Job Description

Note :- Above 13+ with last 3-4 years as Technical Lead (No Relocation)

Custom screening - sub directly to job, previous scores don't transfer

Top 3 Skills: Databricks Certified Technical Lead, Python, Data Domain experience

Project: Supporting Kroger's transition from MS Azure to Databricks; early-stage setup of data domains

Soft Skills: Strong leadership, clear communication across agile on/offshore teams, transparency, and adherence to timelines

Requirements

• 10+ years of experience in data engineering, big data, or API development, with 3+ years in a leadership role
• Proven experience leading product-centric data engineering initiatives in an agile delivery environment
• Expertise in Azure Databricks, Apache Spark, Azure SQL, and other Microsoft Azure services
• Strong programming skills in Python, Scala, and SQL for data processing and API development
• Experience building and managing APIs (REST, GraphQL, gRPC) and microservices
• Hands-on experience with Azure Data Factory (ADF), Azure Synapse Analytics, and Delta Lake
• Proficiency in CI/CD pipelines, Infrastructure-as-Code tools like Terraform or Bicep
• Experience implementing data security and compliance for sensitive HR/employee data (encryption, access control, auditing)
• Strong problem-solving skills with a focus on performance tuning, security, and cost optimization
• Experience with containerization (Docker, Kubernetes) and event-driven architecture (preferred)
• Exposure to Informatica for ETL/ELT and data integration
• Excellent communication and leadership skills in a fast-paced environment
• Willingness to transition from contract to full-time employment

Preferred Qualifications

• Microsoft Certified: Azure Solutions Architect Expert or Databricks Certified Data Engineer/Architect
• Experience with agile methodologies such as Scrum or SAFe
• Familiarity with machine learning workflows in Azure Databricks
• Knowledge of Azure API Management and Event Hub for API integration
• Experience with Informatica PowerCenter or IICS
• Hands-on experience with Oracle HCM database models and APIs
• Background in HR analytics and workforce data integration

Core Responsibilities:

• Develop and maintain the HR data domain in a secure, compliant, and efficient manner aligned with industry best practices
• Lead the development of a data engineering team responsible for building scalable, high-performance data solutions, APIs, and microservices in Azure Databricks, Azure SQL, and Informatica
• Ensure the highest levels of security and privacy for sensitive HR data across all systems and integrations
• Bring deep expertise in big data processing, data architecture, cloud migrations, API development, data security, and agile methodologies within the Azure ecosystem

Key Responsibilities:

Azure Databricks & Big Data Architecture:

• Design and implement scalable data pipelines and architectures using Azure Databricks
• Optimize ETL/ELT workflows to ensure efficient data processing, storage, and retrieval
• Leverage Apache Spark, Delta Lake, and Azure-native services to build high-performance data solutions
• Apply best practices in data governance, security, and compliance across Azure environments
• Troubleshoot and fine-tune Spark jobs to improve performance and reduce costs

Azure SQL & Cloud Migration:

• Lead migration of workloads from Azure SQL to Azure Databricks, ensuring a seamless transition
• Design and implement scalable ETL/ELT pipelines from Azure SQL to Databricks Delta Lake
• Optimize Azure SQL queries and indexing strategies pre-migration to enhance post-migration performance
• Maintain data governance, security, and compliance throughout the migration lifecycle
• Use Azure Data Factory (ADF), Informatica, and Databricks to orchestrate automated migration workflows
• Enable integration of migrated data with APIs, ML models, and BI tools
• Establish post-migration performance monitoring and cost optimization strategies

API & Services Development:

• Design and build RESTful APIs and microservices for efficient data access and integration
• Implement secure and scalable API frameworks to expose data processing capabilities
• Utilize GraphQL, gRPC, or streaming APIs for real-time data consumption needs
• Integrate APIs with Azure data lakes, warehouses, Oracle HCM, and other enterprise systems
• Apply API performance tuning, monitoring, and security practices including OAuth, JWT, and Azure API Management

HR Data Domain & Security:

• Build and manage a scalable, secure, and well-governed HR data architecture
• Implement RBAC, encryption, and data masking to protect sensitive employee data
• Ensure compliance with data privacy regulations such as GDPR, CCPA, and HIPAA
• Design and implement audit logging and monitoring for data access and modifications
• Collaborate with HR and security teams to define access permissions, retention policies, and data anonymization strategies
• Enable secure APIs and data-sharing mechanisms for HR analytics while maintaining employee privacy
• Integrate Oracle HCM data structures within the Azure Databricks ecosystem

Product-Centric & Agile Delivery:

• Drive a product-focused approach to data engineering aligned with user needs and business outcomes
• Operate within an agile delivery framework using Scrum or Kanban methodologies
• Collaborate with product managers and stakeholders to define use cases and prioritize backlog items
• Foster a continuous improvement culture using feedback loops and data-driven decision-making
• Implement DevOps and CI/CD pipelines for rapid and iterative delivery of data solutions

Leadership & Collaboration:

• Provide technical leadership and mentorship to data engineers and developers
• Collaborate with business stakeholders, HR teams, product managers, and architects to translate requirements into data solutions
• Advocate for automation and Infrastructure-as-Code practices using tools like Terraform and Bicep
• Cultivate a culture of innovation, collaboration, and continuous learning within the team
• Stay current with advancements in Azure Databricks, Azure SQL, Informatica, Oracle HCM, and other cloud technologies

Job Tags

Similar Jobs