Senior Data Architect - Databricks / Data Lakehouse / GenAI

Senior Data Lakehouse Architect Client: Bank Sector Client About the Role D L Resources is supporting a leading banking-sector client in hiring an experienced Senior Data Lakehouse Architect to lead the end-to-end architecture, design, and evolution of an enterprise Lakehouse platform. The role will be responsible for defining the technical vision, target architecture, and engineering standards for modern data platforms, including data products, data marketplace, knowledge layers, real-time data processing, Generative AI, RAG, vector search, graph technologies, and agentic workloads. The successful candidate should bring deep experience in large-scale enterprise data architecture, distributed computing, cloud and hybrid platforms, performance engineering, data governance, and modern DevSecOps practices. Key Responsibilities

• Own the end-to-end architecture and technical roadmap for the enterprise Data Lakehouse platform.

• Design and evolve platform capabilities supporting:

• Data products and data marketplace

• Knowledge and semantic layers

• Structured, semi-structured, and unstructured data

• Real-time and streaming workloads

• RAG and Generative AI use cases

• Vector and graph-based data services

• Agentic AI and autonomous workflow patterns

• Define target architectures for applications and platform services with a focus on reusability, scalability, resilience, security, and operational efficiency.

• Develop reusable architecture patterns, frameworks, and technical accelerators for:

• Unstructured and multimodal content extraction

• Batch and streaming architectures

• Lambda and event-driven architectures

• Retrieval-Augmented Generation (RAG)

• Agentic workloads and AI-driven data processing

• Partner with business and technology stakeholders to define data contracts, SLAs, data quality standards, and governance requirements for enterprise data products.

• Provide architecture oversight and quality assurance to ensure solutions comply with the client’s software engineering, security, and delivery standards.

• Review solution designs, technical specifications, non-functional requirements, and implementation approaches produced by engineering teams.

• Participate in technology and product evaluations, proof-of-concepts, and RFP processes.

• Guide installation, customization, integration, and operationalization of enterprise software platforms and technologies.

• Lead performance engineering, capacity planning, scalability reviews, and optimization of distributed data workloads.

• Partner with infrastructure, security, application, cloud, AI/ML, and operations teams to deliver integrated technology solutions.

• Drive continuous service improvement, engineering automation, platform standardization, and operational excellence.

• Produce architecture documentation, solution designs, implementation guidelines, operational standards, and technical runbooks.

Required Experience

• 10–15 years of experience in enterprise Data Engineering, Big Data, Data Architecture, Data Lake, or Lakehouse implementations.

• Strong experience designing and delivering large-scale Data Lakehouse platforms, preferably within banking, financial services, or another highly regulated industry.

• Proven experience across one or more leading data and cloud platforms such as: Databricks, Snowflake, Cloudera, Azure, AWS, Google Cloud Platform, Huawei Cloud, or Alibaba Cloud.

• Strong experience designing distributed compute and MPP workloads across on-premise, hybrid, and cloud environments.

• Deep understanding of enterprise data architecture, scalability, resilience, security, governance, and performance optimization.

Core Lakehouse & Data Architecture Skills Strong experience in several of the following areas:

• Open Table Formats: Apache Iceberg, Apache Hudi, Delta Lake

• Object Storage: Cloud and enterprise object storage, including hot/warm/cold tiering strategies

• Data Federation: Trino, Denodo, Dremio

• Distributed Query Technologies: Hive, Impala, Apache Kudu and similar platforms

• Data Processing: Spark, PySpark, SQL, Java, Python, Scala

• Real-Time & Streaming: Apache Kafka, Confluent, Azure Event Hubs, Amazon Kinesis, Apache Flink, Spark Streaming, Structured Streaming, Apache NiFi

• Workflow & Scheduling: Airflow, Control-M

• Data Modelling & Governance: Enterprise data modelling, metadata, lineage, data contracts, data quality, and governance frameworks

Generative AI, RAG & Agentic Architecture Experience designing or supporting modern AI-enabled data architectures, including:

• Retrieval-Augmented Generation (RAG)

• Embedding strategies and vectorization

• Vector databases and vector search

• Graph databases and knowledge graphs

• Prompt and context management

• Agentic workflow orchestration

• Knowledge and semantic layers

• AI-driven analytics and Generative BI

Relevant technologies may include: Vector Search / Vector Databases

• Databricks Vector Search

• Azure AI Search

• Pinecone

• ChromaDB

• Weaviate

• Snowflake Cortex

Graph Databases

• Neo4j

• JanusGraph

• TigerGraph

• Microsoft Fabric / Cosmos DB

• Amazon Neptune

• Stardog

Agentic & AI Orchestration Frameworks

• LangGraph

• OpenAI Agents SDK

• Microsoft Agent Framework

• LlamaIndex Workflows

• Google Agent Development Kit (ADK)

Data Products & Data Marketplace

• Experience designing and delivering foundation and business data products.

• Experience defining and implementing data contracts, service levels, governance, and quality controls.

• Ability to expose data products through:

• APIs

• Publish/subscribe and event-driven architectures

• Real-time dashboards

• BI and Generative BI platforms

• Data marketplace capabilities

• Experience designing data products for enterprise consumption, reuse, discoverability, and governance.

Cloud & Hybrid Architecture Strong understanding of cloud and hybrid architecture patterns, including:

• Workload placement and cloud optimization strategies

• Private and dedicated cloud connectivity such as AWS Direct Connect and Azure ExpressRoute

• Data egress and network cost optimization

• Infrastructure-as-Code

• Hybrid and multi-cloud data architecture

• Security and network integration

• High availability and disaster recovery

DevOps, Platform Engineering & Automation Experience with modern DevOps and software delivery practices, including:

• CI/CD: Jenkins, Azure Pipelines, AWS CodePipeline, Google Cloud Build / Deploy

• Source Control: Git, Bitbucket

• Code Quality: SonarQube

• Artifact Repositories: JFrog Artifactory, AWS CodeArtifact, Amazon ECR, Azure Artifacts, Google Artifact Registry

• Infrastructure-as-Code: Terraform, AWS CloudFormation, Azure ARM

• Containerization: Docker, Kubernetes, OpenShift

• Deployment: Helm, Kustomize

• Monitoring: AWS CloudWatch, Azure Monitor, Google Cloud Monitoring

• Incident / Service Management: Remedy or equivalent platforms

• Testing / Defect Management: JIRA, QuerySurge or similar tools

Programming & Automation Strong knowledge of one or more of the following:

• Python

• Scala

• Java

• SQL

• JavaScript / Node.js

• Shell scripting

• Groovy

Experience automating engineering and operational processes is strongly preferred. Migration & Modernization Experience Experience with migration and modernization programs involving legacy or MPP data platforms will be advantageous, including:

• Teradata

• Greenplum

• Netezza

• Other enterprise MPP platforms

Experience with bulk migration, workload modernization, automated migration tooling, and AI-assisted migration accelerators is a plus. Education

• Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related discipline.

• Equivalent relevant professional experience may also be considered.

Preferred Certifications Candidates with relevant architecture, data, and cloud certifications will have an advantage. Examples include:

• Databricks Certified Data Engineer / Data Architect

• Microsoft Azure certifications

• AWS Cloud / Data certifications

• Google Cloud certifications

• DAMA Certified Data Management Professional (CDMP)

• Data modelling certifications such as Erwin

• Relevant Kubernetes, DevOps, data engineering, or architecture certifications

What Will Help You Succeed

• Strong architectural thinking with the ability to balance business outcomes, engineering quality, cost, scalability, security, and operational requirements.

• Ability to understand enterprise-wide technology landscapes and translate them into practical technical roadmaps.

• Strong analytical, troubleshooting, and decision-making capabilities.

• Ability to resolve complex architecture and integration challenges.

• Strong focus on engineering quality and continuous improvement.

• Excellent communication skills, including the ability to explain complex technical concepts to non-technical stakeholders.

• Strong stakeholder management and collaboration skills across business, technology, vendors, and distributed engineering teams.

• Experience working in Agile and modern software delivery environments.

• Ability to manage multiple initiatives and priorities in a fast-paced enterprise environment.

Key Technology Stack Lakehouse & Data Platforms: Databricks, Snowflake, Cloudera, Iceberg, Hudi, Delta Lake, Trino, Denodo, Dremio, Hive, Impala, Kudu Cloud: Azure, AWS, GCP, Huawei Cloud, Alibaba Cloud Data Processing & Streaming: Spark, PySpark, Python, Scala, Java, SQL, Kafka, Confluent, Flink, Spark Streaming, Structured Streaming, NiFi AI / GenAI: RAG, Vector Search, Embeddings, Graph Databases, Knowledge Graphs, LangGraph, OpenAI Agents SDK, LlamaIndex, Microsoft Agent Framework, Google ADK DevOps & Platform Engineering: Kubernetes, OpenShift, Docker, Terraform, Helm, Kustomize, Jenkins, Git, SonarQube, CI/CD Data Architecture & Governance: Data Products, Data Marketplace, Data Contracts, Data Quality, Metadata, Lineage, Data Modelling, Governance