Remote Data Annotation Specialist

Overview

Rex.zone is seeking full-time remote professionals to join as Data Annotation Specialists aiding the development of AI training datasets for large language models and computer vision applications. The role involves labeling data, performing Reinforcement Learning from Human Feedback (RLHF) evaluations, prompt assessments, and quality assurance checks to enhance the accuracy and effectiveness of training datasets within live LLM training workflows.

Key Responsibilities

• Label multi-modal data including text, images, and structured data adhering to comprehensive annotation guidelines and scoring rubrics.
• Execute diverse annotation tasks such as Named Entity Recognition (NER), classification, data extraction, creating bounding boxes and polygons, and identifying content safety parameters.
• Conduct RLHF preference evaluations, pairwise comparisons, and rubric-based scoring while documenting clear justification.
• Perform quality assurance evaluations, handle exceptional and ambiguous cases, and flag uncertain items for further review.
• Work asynchronously with project leaders, reviewers, and calibration processes to maintain and improve annotation consistency.

Required Qualifications

• Experience at mid to senior level in data annotation or labeling with demonstrated high-quality outputs.
• Aptitude for consistently applying detailed rubrics and articulating clear quality assurance rationales.
• Understanding of RLHF methodologies, prompt evaluation strategies, and signals used in large language model assessments.
• Proficient comprehension of English and effective written communication skills for interpreting guidelines and reporting issues.

Quality and Measurement

• Performance is evaluated through adherence to guidelines, audit success rates, calibration precision, and agreement between annotators.
• Constructive feedback is delivered via reviewer comments, audits, and regular calibration sessions.