What You'll Be Doing
- Champion a shift-left testing culture by embedding quality across every phase of our software development lifecycle.
- Design, build, and maintain robust test automation frameworks for functional, integration, performance, security, and accessibility testing.
- Collaborate closely with product managers, developers, and AI engineers to test and evaluate LLM-integrated features using both traditional and GenAI-specific testing methods.
- Implement tools and techniques to test LLM-based functionality, including prompt evaluation, RAG-enhanced responses, and API behavior.
- Drive the use of modern CI/CD pipelines, ensuring testing is efficient, scalable, and automated wherever possible.
- Analyze, interpret, and share meaningful insights from test results – especially when comparing LLM model responses, prompt outcomes, and system behavior.
- Mentor team members in test strategies, automation practices, and GenAI testing methodologies.
- Continuously assess and improve the quality landscape, identifying opportunities to increase speed, accuracy, and innovation in how we test.
What We're Looking For
- Core QA Expertise
- Strong experience with end-to-end testing of distributed systems and SaaS applications.
- Deep understanding of testing methodologies including exploratory, regression, risk-based, performance, and security testing.
- Skilled in writing automated tests using tools likeCypress,Pact, or similar frameworks, with coding experience inJava/JavaScript.
- Solid understanding of theTest Pyramid, CI/CD pipelines, and agile methodologies (Scrum, Kanban).
- Experience working with cloud platforms likeAWS,GCP, orAzure, plus comfort withSQLand database testing.
- A team player with a QA-first mindset and a drive to elevate engineering standards across the board.
- GenAI/LLM Knowledge (Essential)
- Hands-on experience working withLLM APIslikeOpenAI,Anthropic, orHuggingFace.
- Familiarity withprompt engineering, fine-tuning prompts for desired outcomes.
- Practical experience usingRetrieval-Augmented Generation (RAG)to enrich LLM responses.
- Understanding of key GenAI concepts likesystem/user prompts,tokens,embeddings,temperature,top-p,context windows, andstop sequences.
- Nice-to-Have
- Experience withfine-tuning LLMsor training smaller models for specific tasks.
- Familiarity withprompt compressiontechniques and advanced GenAI patterns likeprompt chaining,agentic workflows, orLLM routing.
- Exposure to abstraction libraries such asSpring AIor similar.
- Experience withLLM eval frameworksto benchmark model performance and prompt effectiveness.