About Gramian
Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.
About the Role
We are seeking AI Personalization Raters based in Canada to support Gemini Personalization Evals. In this role, you will assess how effectively AI models retrieve and use relevant context from connected Google applications, including Gmail, Google Calendar, Google Drive, and Google Photos. You will evaluate accuracy, relevance, helpfulness, and personalization quality while providing structured feedback to support model improvements.
This opportunity is suitable for detail-oriented individuals with strong analytical thinking, excellent written communication, and active use of Google applications.
Engagement Details
CONTRACT: Contractor
LOCATION: Remote — Canada only
COMMITMENT: Minimum 4 hours per day, up to 40 hours per week
TIME OVERLAP: 4 hours of overlap with PST
DURATION: Up to 16 weeks
Key Responsibilities
- Evaluate AI-generated responses using personalized prompts and information retrieved from connected Google applications.
- Assess whether responses are relevant, accurate, helpful, and appropriately personalized.
- Identify incorrect personalization, unsupported assumptions, irrelevant recommendations, and inconsistencies.
- Compare multiple AI-generated responses and assess their overall user experience.
- Provide clear, detailed, and well-reasoned evaluation feedback.
- Follow project guidelines to ensure consistent and high-quality ratings.
- Identify subtle strengths and weaknesses in AI-generated responses.
- Maintain data privacy and comply with project confidentiality requirements.
Requirements
- Currently based in Canada.
- Willing and able to connect relevant Google applications to Gemini, subject to required permissions and consent.
- Active use of Google Workspace or consumer Google apps, with sufficient personal data or history to support personalized retrieval evaluations.
- Strong analytical thinking and attention to detail.
- Excellent written English and ability to provide structured feedback.
- Ability to work independently in a remote environment.
- Access to a desktop or laptop with a reliable internet connection.
- Bachelor’s degree or equivalent practical experience in any field.
.png)
