EY - GDS Consulting - AI And DATA - Databricks Architect - Senior Manager

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. 

EY-Consulting - Data and Analytics – Senior Manager - Databricks Architect

EY's Consulting Services is a unique, industry-focused business unit that provides a broad range of integrated services that leverage deep industry experience with strong functional and technical capabilities and product knowledge. EY’s financial services practice provides integrated Consulting services to financial institutions and other capital markets participants, including commercial banks, retail banks, investment banks, broker-dealers & asset management firms, and insurance firms from leading Fortune 500 Companies. Within EY’s Consulting Practice, Data and Analytics team solves big, complex issues and capitalize on opportunities to deliver better working outcomes that help expand and safeguard the businesses, now and in the future. This way we help create a compelling business case for embedding the right analytical practice at the heart of client’s decision-making.

The opportunity

Senior Data & AI Architect with 13+ years of experience in data engineering and data architecture, strong hands-on expertise in Databricks, Hadoop, Spark/PySpark, and cloud data platforms.

This role combines deep technical ownership of enterprise Lakehouse and data platform architecture with hands-on experience building Agentic AI and Retrieval-Augmented Generation (RAG) solutions on Databricks. The ideal candidate is equally comfortable designing large-scale, production-grade data pipelines, architecting modern AI-driven solutions, and engaging directly with clients on strategic proposals and solution design.

Your key responsibilities

• Architect, design, and deliver scalable Data & AI solutions on the Databricks Lakehouse platform, covering ingestion, transformation, storage, governance, and consumption layers.
• Design and optimize large-scale distributed data processing pipelines using Hadoop, Apache Spark, and PySpark.
• Build reusable, production-grade Python frameworks and pipelines for batch and streaming data engineering workloads.
• Define and implement cloud data architecture on AWS and/or Azure, covering storage, IAM, networking, security, and monitoring, with native Databricks integration.
• Own end-to-end ETL/ELT design, data ingestion, transformation, orchestration, and data quality frameworks across the platform.
• Design dimensional data models (star/snowflake schemas, fact/dimension tables, SCD) to support analytical and reporting use cases.
• Design and build Agentic AI solutions on Databricks, including LLM integration, tool/function calling, agent workflows, guardrails, and enterprise data integration.
• Architect and implement RAG (Retrieval-Augmented Generation) solutions using Databricks — document ingestion, chunking, embeddings, vector search, retrieval, grounding, and evaluation.
• Establish and enforce engineering best practices for Git-based version control, code reviews, and CI/CD across development, test, and production environments.
• Lead performance optimization and production deployment of Databricks jobs, workflows, and pipelines at scale.
• Participate in strategic RFP/RFI responses, technical proposals, solution shaping, effort estimation, and architecture presentations.
• Act as a trusted technical advisor in customer workshops, translating business requirements into robust technical architectures.
• Mentor engineering teams on Databricks, Spark, cloud architecture, and modern AI/RAG engineering practices.

Skills and attributes for success

13+ years of experience in Data Engineering / Data Architecture, with strong hands-on experience across Databricks, Hadoop, Spark/PySpark, and cloud data platforms.

• Databricks
  • Strong expertise in Databricks Lakehouse, Delta Lake, Databricks SQL, and Unity Catalog.
  • Hands-on experience with Jobs/Workflows, performance optimization, and production deployments.
• Hadoop & Spark
  • Strong knowledge of the Hadoop ecosystem, HDFS, and Hive.
  • Strong hands-on experience with Apache Spark and PySpark for large-scale distributed data processi
• Python
  • Strong proficiency in Python for building scalable, reusable, production-grade data pipelines and frameworks.
  • Cloud
  • Strong experience with AWS and/or Azure, including cloud storage, IAM, networking, security, and monitoring.
  • Experience integrating Databricks with cloud-native services.
• Data Engineering
  • Expertise in ETL/ELT, batch and streaming processing, data ingestion, transformation, orchestration, and data quality.
  • Proven ability to design scalable, production-ready pipeline architectures.
• SQL & Data Modelling
  • Strong SQL skills, with experience in dimensional modelling, star/snowflake schemas, fact/dimension models, SCD, and analytical data models.
• Agentic AI
  • Hands-on experience designing and building Agentic AI solutions with Databricks, including LLMs, tool/function calling, agent workflows, guardrails, and enterprise data integration.
• RAG (Retrieval-Augmented Generation)
  • Strong experience building RAG solutions using Databricks, including document ingestion, chunking, embeddings, vector search, retrieval, grounding, and evaluation.
• Git & CI/CD
  • Strong knowledge of Git, branching strategies, pull requests, and code reviews.
  • Experience with CI/CD pipelines and deployment across multiple environments.
  • Experience with Databricks Asset Bundles and/or Terraform is a plus.
• Strategic RFPs & Solutioning
  • Ability to participate in strategic RFP/RFI responses, solution shaping, technical proposals, and effort estimations.
  • Experience delivering architecture presentations and leading customer workshops.
• Client Communication
• Strong client-facing and communication skills, with the ability to understand business requirements, translate them into scalable technical solutions, and clearly communicate architectural decisions to both technical and non-technical stakeholders.

Ideally, you’ll also have

• Prior experience in a consulting, systems integrator, or client-facing delivery role.
• Relevant cloud or Databricks certifications (e.g., Databricks Certified Data Engineer / Solutions Architect, AWS/Azure certifications).
• Exposure to enterprise data governance, MLOps, or LLMOps practices.

What working at EY offers

At EY, we’re dedicated to helping our clients, from start–ups to Fortune 500 companies — and the work we do with them is as varied as they are.

You get to work with inspiring and meaningful projects. Our focus is education and coaching alongside practical experience to ensure your personal development. We value our employees and you will be able to control your own development with an individual progression plan. You will quickly grow into a responsible role with challenging and stimulating assignments. Moreover, you will be part of an interdisciplinary environment that emphasizes high quality and knowledge exchange. Plus, we offer:

• Support, coaching and feedback from some of the most engaging colleagues around
• Opportunities to develop new skills and progress your career
• The freedom and flexibility to handle your role in a way that’s right for you

EY | Building a better working world 

 
EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets.  

 
Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate.  

 
Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.