Conversational AI Engineer Jobs

JOB TITLE: Multilingual Conversational AI UX and Safety Evaluator

LOCATION: Remote

JOB TYPE: Contract

PAY: $75.00 – $100.00 per hour (USD)

SCHEDULE: 10–20 hours per week (Flexible)

HIRES NEEDED: 1

HIRING TIMELINE: ASAP

About the Product

We are building an AI-powered emotional wellness app designed primarily for South Asian, Middle Eastern, and immigrant communities. The app supports people navigating stress, burnout, relationships, marriage, cultural expectations, family pressure, emotional regulation, identity, grief, and difficult life decisions.

Many of our intended users may not feel comfortable beginning traditional therapy, but still want private, thoughtful, culturally informed emotional support. Our goal is not to create another generic chatbot. We are building an experience that feels deeply understanding, practical, emotionally intelligent, culturally aware, multilingual, and safe.

We are looking for an experienced evaluator to rigorously test the product before a wider release.

The Role

You will test and evaluate the app from the perspective of real users across multiple languages, cultures, ages, and emotional situations. This is not limited to testing whether buttons and technical features work.

You will evaluate whether the app:

• Understands what users are truly trying to communicate
• Responds appropriately in different languages
• Maintains the intended meaning when users switch languages
• Understands cultural expressions, idioms, indirect communication, and family terminology
• Responds with warmth without sounding robotic or overly therapeutic
• Asks thoughtful and relevant follow-up questions
• Provides specific, practical guidance rather than generic advice
• Remembers and appropriately uses previous context
• Understands South Asian and immigrant family dynamics without stereotyping
• Responds safely to emotionally sensitive and high-risk situations
• Creates enough value that users would choose to return

You should be willing to challenge the product, confuse it, contradict it, reject its suggestions, switch languages unexpectedly, and deliberately attempt to expose weaknesses.

Required Language Testing

The app must be tested in: English, Urdu, Hindi, Punjabi, Arabic, Bengali, Gujarati, Tamil, Telugu, and additional South Asian languages when relevant.

Testing must include both native-script and Romanized communication, such as Roman Urdu, Roman Hindi, and Roman Punjabi.

Candidates may apply individually or as part of a small multilingual testing team. A strong individual candidate should be fluent in at least two of the listed languages and demonstrate the ability to coordinate testing with qualified native speakers for the remaining languages. Machine translation alone is not sufficient. Native or near-native review is required.

Primary Responsibilities

• Test Complete User Journeys — Onboarding, account creation, initial conversations, recurring sessions, saved information, recommendations, and return-user experiences.
• Evaluate Conversation Quality — Realistic conversations involving burnout, family pressure, marriage conflict, grief, boundaries, guilt, dating, spirituality, bicultural identity, and more.
• Conduct Multilingual Quality Testing — Compare performance across languages. Determine if meaning is preserved, response feels natural, cultural nuance is respected, and safety protections remain equally strong.
• Conduct Adversarial and Edge-Case Testing — Attempt to make the app fail through contradictions, sudden topic shifts, code-switching, angry inputs, high-risk indirect language, and more.
• Evaluate Safety — Test scenarios involving self-harm, domestic violence, child abuse, medical emergencies, panic, religious guilt, and more. Ensure the app remains compassionate while recognizing limitations and directing users to appropriate support.
• Evaluate Cultural Intelligence — Assess understanding of izzat, sharam, family reputation, marriage pressure, gender roles, immigration identity, in-law dynamics, and cultural nuance without stereotyping.

Required Deliverables

• Documented test conversations across all required languages
• A language-by-language quality comparison
• A structured scoring system for response quality
• A prioritized list of product failures (Critical / High / Medium / Low / Cosmetic)
• Screen recordings or screenshots of significant issues
• User-journey and abandonment analysis
• Multilingual safety-testing report
• Cultural-intelligence assessment
• Recommended rewrites for weak conversations
• Reusable regression-testing library for future product updates
• Weekly summary of highest-priority findings
• Final presentation outlining highest-impact improvements

Each report should include: exact user input, language used, English translation when necessary, app's response, what worked, what failed, possible user impact, recommended correction, and priority level.

Ideal Background

You may be a strong candidate if you have experience in one or more of the following:

• Conversational AI evaluation
• Generative AI or chatbot testing
• Multilingual AI evaluation
• AI localization or linguistic quality assurance
• UX research or usability testing
• AI safety, trust and safety, or red teaming
• Digital health or mental-health technology
• Psychology, behavioral science, or human-computer interaction
• Trauma-informed communication
• Product quality assurance
• South Asian or Middle Eastern cultural research
• Translation, interpretation, or linguistic evaluation

Direct experience in every category is not required. We care most about your ability to recognize emotional nuance, test rigorously, understand cultural context, identify safety concerns, document problems clearly, and recommend practical improvements.

Required Qualifications

• Fluency in English
• Fluency in at least two additional required languages
• Ability to coordinate qualified native-language testers for languages not personally covered
• Experience evaluating conversational AI, digital products, or complex user experiences
• Strong written documentation skills
• Comfort testing emotionally sensitive scenarios
• Ability to distinguish translation accuracy from emotional and cultural accuracy
• Respect for confidentiality and sensitive user information

Preferred Qualifications

• Experience with mental-health, wellness, medical, or digital-health products
• Experience conducting AI red-team or trust-and-safety testing
• Familiarity with South Asian family and cultural dynamics
• Experience testing Romanized South Asian languages
• Experience creating AI evaluation rubrics or regression-test suites
• Background in linguistics, psychology, behavioral science, UX research, or human-computer interaction

Important Boundaries

This role does not involve providing therapy to users or diagnosing mental-health conditions. Testing must use fictional, synthetic, or properly de-identified scenarios. Real client information must not be entered into the product unless the company has explicitly approved a compliant testing procedure. Confidentiality is essential. The selected contractor will be required to sign confidentiality, data-protection, and intellectual-property agreements.

Compensation and Schedule

• Compensation: $75–$100 per hour (USD), depending on experience
• Initial commitment: 10–20 hours per week
• Work arrangement: Remote
• Schedule: Flexible, with agreed-upon weekly deliverables
• Contract length: Initial testing period with the possibility of ongoing work
• Hours must be documented by testing activity and deliverable

Finalists may be invited to complete a paid evaluation exercise (2–3 hours, compensated at the role's stated hourly rate) before selection.

How to Apply

Please submit:

• A brief introduction describing your relevant experience
• The languages you speak and your level of proficiency in each
• Your experience with conversational AI, multilingual testing, UX research, safety evaluation, or digital health
• An example of a chatbot, AI product, or digital product you have evaluated
• A sample testing report or anonymized work product, when available
• Your proposed approach to evaluating an emotionally sensitive multilingual AI product
• Whether you are applying individually or with a multilingual testing team
• Your hourly rate and weekly availability
• A response to the application scenario below

Application Scenario

A user tells the app:
"My family says I am selfish because I want to move out, but I have spent my whole life doing everything for them."

In no more than 400 words:

• Explain what a strong response from the app should accomplish
• Identify what a weak or harmful response might do
• Explain how you would score the response
• Identify the cultural considerations that should be evaluated
• Translate the user's message into one South Asian or Middle Eastern language you speak
• Explain whether the emotional meaning changes when expressed in that language

Pay: $75.00 - $100.00 per hour

Benefits:

• Flexible schedule

Work Location: Remote