Manager, Senior Databricks Technical Lead & Solution Architect, Engineering, AI & Data, Technology & Transformation

Connect to your Industry

Do you want to lead the engineering delivery and shape the solution architect of some of the UK's largest and most ambitious Data & AI programmes?

Within Technology & Transformation organisations are increasingly relying on modern data platforms and AI to accelerate innovation, improve decision-making, and unlock new business value. As a Senior Databricks Data & AI Engineer / Technical Lead, you'll provide the technical leadership that enables these transformations—leading engineering teams, shaping solution architecture, and delivering enterprise-scale data, machine learning, and AI solutions on the Databricks Data Intelligence Platform.

Working alongside some of the industry's leading architects, engineers, and AI specialists, you'll help clients build secure, scalable, and high-performing data and AI platforms that power the next generation of intelligent applications.

Connect to your career at Deloitte     

Deloitte drives progress. Using our vast range of expertise, we help our clients' become leaders wherever they choose to compete. To do this, we invest in outstanding people. We build teams of future thinkers, with diverse talents and backgrounds, and empower them all to reach for and achieve more.  

What brings us all together at Deloitte? It’s how we approach the thousands of decisions we make every day. How we behave, our beliefs and our attitudes. In other words: our values. Whatever we do, wherever we are in the world, we lead the way, serve with integrity, take care of each other, foster inclusion, and collaborate for measurable impact. These five shared values lead every decision we make and action we take, guiding us to deliver impact how and where it matters most.

Connect to your opportunity

We're expanding our Databricks Engineering practice and are looking for experienced technical leaders who combine deep engineering expertise, solution architecture capability with strong delivery leadership. You'll work alongside exceptional data, AI, and cloud specialists to design, build, and deliver complex enterprise solutions while leading distributed engineering teams and shaping technical direction across multiple workstreams.

Depending on your strengths and client needs, you may focus more on hands-on engineering leadership and delivery, end-to-end solution and platform architecture, or a combination of both across enterprise Databricks transformation programmes. The ideal candidate will possess extensive hands-on expertise across the Databricks Lakehouse Platform, modern data engineering, machine learning, and Generative AI, while providing technical leadership, design authority, and engineering governance throughout the delivery lifecycle.

As a key member of our team, you will:

• Lead the technical delivery and/or solution architecture of enterprise-scale Databricks Data & AI implementations from solution design through to production deployment.
• Act as the technical lead and design authority across multiple workstreams, ensuring solution quality, scalability, security, and alignment with engineering best practices.
• Lead and mentor onshore and offshore engineering teams, fostering technical excellence, collaboration, and continuous improvement.
• Drive engineering delivery by establishing technical standards, reviewing solution designs, managing technical risks, and ensuring successful execution against project objectives.
• Conduct code reviews and engineering quality assurance, promoting clean, maintainable, high-quality code and adherence to software engineering best practices.
• Optimise platform performance across Spark workloads, Delta Lake, data pipelines, storage, and compute to maximise scalability, reliability, and cost efficiency.
• Design and deliver enterprise-grade Machine Learning, Generative AI and Agentic AI solutions leveraging the Databricks Data Intelligence Platform, MLflow, Mosaic AI, Vector Search, and modern LLM capabilities where appropriate.
• Partner with solution architects, product owners, client stakeholders, and engineering teams to translate business requirements into scalable technical solutions.
• Coach and develop engineers through technical mentoring, knowledge sharing, and establishing reusable engineering frameworks and best practices.
• Own technical governance by maintaining engineering standards, architectural integrity, documentation, and design decisions throughout project delivery.
• Define enterprise solution and platform architectures on Databricks — producing reference architectures, blueprints, and target-state designs that span data engineering, ML, and Generative AI.
• delivery, and advising client leadership on architecture strategy, platform roadmaps, and target operating models.

Connect to your skills and professional experience

Overall, you're an inspiring technical leader, an experienced engineer, and a delivery-focused problem solver. You understand that successful programmes are built on strong technical leadership, engineering excellence, and collaborative teams.

You're passionate about mentoring others, making sound architectural decisions, and delivering innovative Data & AI solutions that create measurable business value.

You'll have strong technical skills including some or all of the following:

• Extensive experience delivering enterprise-scale Data Engineering and AI solutions using the Databricks Data Intelligence Platform.
• Proven experience as a solution or platform architect, defining reference architectures, integration patterns, and target-state designs across enterprise Databricks environments.
• Deep expertise across Databricks including Delta Lake, Unity Catalog, Spark, PySpark, Databricks SQL, Databricks Workflows, Lakeflow, Lakehouse architecture, MLflow, AI/BI Genie, Mosaic AI, and Agent Bricks.
• Proven experience leading engineering teams across both onshore and offshore delivery models.
• Strong experience establishing engineering standards, technical governance, and design authority across large programmes.
• Expertise in software engineering best practices including Git, CI/CD, automated testing, Infrastructure as Code, and peer code reviews.
• Strong experience designing, implementing and optimising large-scale batch and streaming data pipelines.
• Advanced Spark performance tuning and optimisation experience across storage, compute, and workload execution.
• Experience delivering production Machine Learning solutions using MLflow, feature engineering, model lifecycle management, and MLOps principles.
• Experience designing and delivering enterprise AI solutions, with deep expertise in Agentic AI architectures, AI Agents, Retrieval-Augmented Generation (RAG), Vector Search, LLM orchestration, prompt engineering, AI governance and responsible AI practices.
• Excellent communication and stakeholder management skills, with the ability to influence technical and non-technical audiences alike.
• Experience working across Azure, AWS, or GCP cloud ecosystems.

Essential:

• Proven experience leading enterprise-scale Databricks implementation programmes.
• Extensive experience acting as a Technical Lead, Engineering Lead, Solution/Platform Architect, or Technical Design Authority, with responsibility for architecture, solution governance, and technical decision making.
• Demonstrable experience leading offshore engineering teams.
• Strong hands-on expertise in Python, PySpark, and SQL.
• Experience designing & delivering enterprise Generative AI & Agentic AI solutions including AI Agents, RAG, Vector Search, LLM Orchestration and AI governance.
• Experience designing & delivering production-grade Machine Learning solutions.
• Proven experience conducting solution design reviews, architecture governance, and code reviews.
• Strong understanding of modern software engineering and DevOps practices.
• Experience working within Agile delivery methodologies.
• Deep expertise in the Databricks Data Intelligence Platform, including Apache Spark, PySpark, Delta Lake, Unity Catalog, Databricks SQL, Lakeflow, Mosaic AI ,AI/BI Genie and Agent Bricks.
• Strong experience designing and delivering production-grade batch and streaming data pipelines using modern Databricks engineering patterns.
• Advanced knowledge of Spark performance tuning, workload optimisation and Databricks platform cost optimisation.
• Strong understanding of Databricks governance, security, enterprise platform architecture, and multi-workspace operating models including Unity Catalog.
• Experience implementing CI/CD, automated testing and deployment practices for Databricks, including Infrastructure as Code and Databricks Asset Bundles.
• Experience enabling ML and Generative AI workloads using MLflow, Mosaic AI and associated Databricks capabilities.

Preferred:

• Databricks Machine Learning Engineer or Generative AI certifications. Databricks Certified Data Engineer Professional (or equivalent advanced Databricks certification).
• Experience with Azure DevOps, GitHub Actions, or equivalent CI/CD tooling.
• Experience designing and delivering scalable, production-grade batch and streaming data pipelines on Databricks, including orchestration and integration with enterprise data sources.
• Experience optimising and operationalising data pipelines for performance, reliability and cost, including CI/CD, automated testing and monitoring.
• Experience with orchestration technologies such as Azure Data Factory, Airflow, or Databricks Workflows.
• Experience with Docker, Kubernetes, and containerised deployments.
• Consulting experience delivering complex Data & AI engagements across multiple industries.

Connect to your business - Technology and Transformation

Distinctive thinking, deep expertise, innovation and collaborative working. That’s what connects us. That’s what makes us Deloitte. If you want to help solve some of the biggest tech and transformational challenges around, join us. Together, we’ll make an impact that matters.

Engineering, AI and Data

We lead transformation at the heart of our clients; re-engineering the core of our clients’ organisations, helping to unleash growth, and creating better futures for their customers, citizens and employees.

Personal independence

Regulation and controls are standard practice in our industry and Deloitte is no exception. These controls provide important legal protection for both you and the firm. We are subject to a number of audit regulations, one of which requires that certain colleagues abide by specific personal independence constraints (e.g., in relation to any financial interests and employment relationships). This can mean that you and your "Immediate Family Members" are not permitted to hold certain financial interests (shares, funds, bonds etc.) with audit clients of the firm, and also prohibitions on certain employment relationships (e.g., you are not permitted to hold a secondary employment role with SEC audit clients of the firm whilst being employed by the firm). The recruitment team will provide further detail as you progress through the recruitment process or you can contact the Independence team upon request.

Connect with your colleagues

“The great culture, flexibility, and range of experiences that Deloitte offers is what sets this firm apart. In Technology & Transformation, Deloitte is far ahead of any of our immediate competitors.” - Sophia, Technology & Transformation

“What attracted me to Deloitte were the endless opportunities and the collective experience of other like-minded individuals. Deloitte’s clients include many of the world’s largest organisations; I wanted to be part of a team that made a difference that I could be proud of.” - Dan, Technology & Transformation

Our hybrid working policy

You’ll be based in London with hybrid working.

At Deloitte we understand the importance of balancing your career alongside your home life. That’s why we’ll support you to work flexibly through our hybrid working policy. Depending on the requirements of your role, you’ll have the opportunity to work in your local office, virtual collaboration spaces, client sites and remotely. You’ll get the chance to meet face to face when needed, while you collaborate and learn from colleagues, share your experiences, and build the relationships that will fuel your career and prioritise your wellbeing. Please check with your recruiter for the specific working requirements that may apply for your role.

Our commitment to you

Making an impact is more than just what we do: it’s why we’re here. So we work hard to create an environment where you can experience a purpose you believe in, the freedom to be you, and the capacity to go further than ever before.

We want you. The true you. Your own strengths, perspective and personality. So we’re nurturing a culture where everyone belongs, feels supported and heard, and is empowered to make a valuable, personal contribution. You can be sure we’ll take your wellbeing seriously, too. Because it’s only when you’re comfortable and at your best that you can make the kind of impact you, and we, live for.

Your expertise is our capability, so we’ll make sure it never stops growing. Whether it’s from the complex work you do, or the people you collaborate with, you’ll learn every day. Through world-class development, you’ll gain invaluable technical and personal skills. Whatever your level, you’ll learn how to lead.

Connect to your next step

A career at Deloitte is an opportunity to develop in any direction you choose. Join us and you’ll experience a purpose you can believe in and an impact you can see. You’ll be free to bring your true self to work every day. And you’ll never stop growing, whatever your level.

Discover more reasons to connect with us, our people and purpose-driven culture at deloitte.co.uk/careers

WPFULL SLTTECH BAENGAI BAAIDAT