What you’ll be doing:
- Ensure AI agents perform consistently before and after going live;
- Monitor performance across different healthcare environments and languages;
- Liaise with the clinical team to create protocols and understand what clinically relevant parameters should be monitored;
- Establish quality standards that work for AI systems, not traditional software;
- When the AI Agent does not follow the defined protocol, trace why it happened;
- Fix the underlying prompt logic and reasoning chains;
- Create new testing methods to prevent similar issues;
- Figure out how to test systems designed to be intelligent and adaptive;
- Build frameworks for testing across infinite healthcare scenarios;
- Define what "good enough" means for AI in healthcare;
- Manage compliance with MDR (Medical Device Regulation) and AI Act requirements for AI systems together with the compliance team;
- Document quality processes and verification and validation procedures for regulatory submissions;
- Help ensure the algorithms perform with the expected accuracy and do not lead to bias;
- Work with compliance teams to achieve and maintain certification of the relevant algorithms;
- Help establish industry standards for AI quality assurance;
- Become one of the few experts in AI healthcare testing methodologies.
What you need to have:
- Comfortable speaking and understanding English (doesn't need to be perfect);
- 2-6 years of experience in QA, testing, or analytical roles;
- Strong analytical and problem-solving skills;
- Comfortable learning new technologies and methodologies;
- Curious about the evolving AI landscape.
What we would love to see:
- Familiarity with secure handling of health data (GDPR, HIPAA);
- Experience with medical device regulations (MDR, 21 CFR 820, ISO 13485, IEC62304);
- Comfortable speaking and understanding Portuguese;
- Experience with AI tools or prompt engineering;
- Background in healthcare, enterprise software, or regulated industries;
- Familiarity with quality metrics and monitoring systems;
- Basic programming or scripting experience.