Senior QA Automation Engineer (Python, Selenium & AI Testing) - Assistant Vice President

The IT Quality Sr Analyst is a seasoned professional role. Applies in-depth disciplinary knowledge, contributing to the development of new techniques and the improvement of processes and work-flow for the area or function. Integrates subject matter and industry expertise within a defined area. Requires in-depth understanding of how areas collectively integrate within the sub-function as well as coordinate and contribute to the objectives of the function and overall business. Evaluates moderately complex and variable issues with substantial potential impact, where development of an approach/taking of an action involves weighing various alternatives and balancing potentially conflicting situations using multiple sources of information. Requires good analytical skills in order to filter, prioritize and validate potentially complex and dynamic material from multiple sources. Strong communication and diplomacy skills are required. Regularly assumes informal/formal leadership role within teams. Involved in coaching and training of new recruits Significant impact in terms of project size, geography, etc. by influencing decisions through advice, counsel and/or facilitating services to others in area of specialization. Work and performance of all teams in the area are directly affected by the performance of the individual.

Responsibilities:

• Responsible for initiatives related to User Acceptance Testing (UAT) process and product rollout into production, making recommendations of solutions and improvements. Responsibilities may also include software quality assurance testing.
• Conducts complex process monitoring, product evaluation and audit assignments; reports issues and makes recommendations for solutions.
• Ensures project standards and procedures exist, are fully / clearly documented and followed throughout the software development life cycle; makes recommendations for enhanced documentation.
• Monitors products for conformance to standards; monitors processes for conformance to procedures; identifies risks and recommends solutions; ensures annual organizational reviews of standards and procedures are conducted.
• Documents findings and performs root cause analyses to provide recommended improvements. Independently recommends solutions and improvements to existing standards and procedures; collaborates with other areas in technology to develop new standards and procedures.
• Ensures all corrective actions are assigned, documented and completed. Verifies traceability in project work products. Gathers, maintains and creates reports on quality metrics.
• Exhibits sound understanding of concepts and principles in own technical area and a basic knowledge of these elements in other areas.
• Requires in-depth understanding of how own area integrates within IT Quality and has basic commercial awareness.
• Acts as advisor and mentor for junior members of the team.
• Makes evaluative judgments based on analysis of factual information in complicated and novel situations and provides solutions to problems.
• Has direct impact on the team and closely related teams by ensuring the quality of the tasks/services/information provided by self and others.
• Requires sound and comprehensive communication and diplomacy skills to exchange complex information.
• Performs other duties and functions as assigned.
• Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency.
• Design, develop, and execute comprehensive test strategies, plans, and test cases for AI/ML and Generative AI applications.
• Build and maintain robust, scalable test automation frameworks using Python and Selenium.
• Perform rigorous data validation and integrity testing using advanced SQL queries to ensure data quality.
• Evaluate AI model performance, accuracy, fairness, and robustness to ensure they meet the highest quality standards.
• Collaborate closely with data scientists, software engineers, and product managers to integrate testing seamlessly into the development lifecycle.
• Identify, document, and track defects and performance issues from discovery through to resolution.
• Champion quality engineering best practices and contribute to the continuous improvement of our testing processes and methodologies. 

Recommended Qualifications:

• 7+ years of professional experience in Quality Engineering or Software/Data Quality Assurance.
• Strong proficiency in Python scripting for test automation and framework development.
• Hands-on experience with web automation tools, particularly Selenium.
• Solid expertise in SQL for writing complex queries and performing backend data validation.
• Proven experience in testing AI/Machine Learning systems; familiarity with Generative AI concepts is highly desirable.
• Deep understanding of QE fundamentals, test methodologies, and the software development lifecycle (SDLC). 

• Design and execute comprehensive evaluation frameworks for Generative AI, Large Language Model (LLM), and Retrieval-Augmented Generation (RAG) applications.

• Assess and benchmark AI models (e.g., GPT, Gemini, and other foundation models) against defined business and technical requirements, providing recommendations on model selection and suitability.

• Implement and manage AI observability and monitoring solutions using platforms such as Langfuse, Arize AI, and Fiddler AI.

• Implement automated AI quality evaluation frameworks such as RAGAS, Promptfoo, DeepEval, and TruLens to measure key quality metrics including accuracy, relevance, precision, faithfulness, context recall, and response quality.

• Establish AI guardrails and governance controls to detect and mitigate risks related to bias, toxicity, hallucinations, prompt injection, jailbreak attempts, and unsafe outputs.

• Excellent analytical and problem-solving abilities with a meticulous eye for detail. 

• Strong communication and interpersonal skills, with a proven ability to work effectively in a collaborative team environment.

Education:

• Bachelor’s/University degree or equivalent experience

------------------------------------------------------

Job Family Group:

------------------------------------------------------

Job Family:

------------------------------------------------------

Time Type:

------------------------------------------------------

Most Relevant Skills

------------------------------------------------------

Other Relevant Skills

------------------------------------------------------

Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.

If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi.

View Citi’s EEO Policy Statement and the Know Your Rights poster.