Work Management Overview

Annotation Workflow in the Dataloop allows you annotating data, reviewing annotations, labeling images, performing quality assurance checks, or any other data-related task that requires human input is performed using tasks. Tasks are created by managers, who define the requirements for each task, such as:

1. General Settings: Task priority, due date, and other parameters.
2. Data Source: The scope of data to be annotated or reviewed.
3. Instructions: A recipe that includes the labels/attributes to use (ontology/taxonomy), the labeling tools to use (for example, classification, bounding-box, polygon, etc.), and work instructions (a PDF attachment).
4. Status:Status of the task. By default, completed is displayed. You can add a new status, if needed.
5. Assignments: Selected annotators, groups of annotators, reviewers, or domain experts.
6. Allocation method: Items can be Distributed in advance, providing a known workload for team members, or Pulled on-demand according to individual progress.
7. Quality: Configuring the task with quality features, such as consensus tasks.

Upon creation, tasks are fragmented into assignments. Each assignment has subtasks that are assigned to individual annotators. Work progress is then monitored through analytic metrics, from high-level task progress and quality to individual assignment performances.

Annotation Workflow (Task and Assignment) Features

The Annotation Workflow in the Dataloop allows you to perform the following functions:

Note: By default, a new UI version of the Annotation Workflow is displayed. Click Switch to Classic Annotation Workflow to view the previous version.

1. Creating any number of tasks.
2. Status assignment to items per task, with predefined and custom statuses alike.
3. Task editing, such as adding items to a task, adding team members, changing configuration, etc.
4. The distribution data allocation method enables planning individuals' workload within a task.
5. The pulling data allocation method provides agility and scalability by adding the required workforce to meet deadlines and quality goals.
6. Assign assignments to ensure work continuity for inactive team members.
7. Redistribute work to balance workloads.
8. Consensus tasks for a majority vote and high-quality data.

Annotation and QA Tasks

Annotation Workflows on the Dataloop platform are created using any combination of two different task types:

Annotation Task

The task of creating annotations of any type over data items. Task managers distribute work, monitor progress and performance, and allocate resources to meet deadlines and budget requirements.

QA Task

The task of reviewing annotation work is done as an annotation task. It has the option to flag annotations as having an 'issue' and send them for correction by the original annotator.

Actions and Statuses

Work on items is declared done by setting a status on the item. Statuses are defined when creating a task. There are default statuses, but the task creator can define custom ones.

• Annotation tasks
  • Complete: The default status to declare that annotation work is completed.
  • Discard: Disqualified item (cannot be annotated).
• QA tasks
  • Approve - the default status to declare that QA review work is done.
  • Discard - Disqualified item (cannot be reviewed).

An item will consequently have several statuses, one for every task it was added to. The status of an item in a task can be changed. The original annotator/reviewer, or other project users with privileges, such as the task owner, can open the item again and set a different status.

Users and Roles

These are the main users that typically take part in a task, correlated with their role in a project:

• Task Owner: A user with the role of a project manager is responsible for arranging the data, preparing and delivering it to tasks, and creating them.
• Task Manager: A user with the role of an annotation manager in the project who is responsible for task execution, workforce management, and accuracy assurance.
• Annotator: A user with the role of an annotator in the project is responsible for actual labeling or quality assurance but is limited to labeling/QA work.

Task and Assignment Status

Tasks and assignments have a status set by the system that cannot be changed by the users. 

To-Do:  A task or assignment that is not started yet.
In-Progress: An assignment is being worked on by assignees. In a task, at least one of its assignments is being worked on.
Completed: All items in the task or assignment are completed.
Completed with issues: All items had statuses, but one or more items have an issue on at least one of the annotations, therefore the task or assignment is not completed.

QA Process

The purpose of the QA task is to increase the quality of annotations by reviewing annotation work and triggering problematic ones for correction by their original creator.

The QA process is based on the following steps:

1. During the QA of an item, a reviewer can use two tools to indicate quality problems.
   1. Raise an Issue on an annotation. It is used when there is a quality problem with an annotation, such as a wrong position, label, attribute, etc.
   2. Create a Note annotation. It is used to indicate that an annotation is missing. The note annotation is created with an issue on it.
2. Having an issue with an item in a QA task removes its status from the original task where the annotation was created. For note annotations, since they are created in the QA task, the issue is assigned by default to the last person who set a status on the item. But the reviewer can manually adjust that for other project users.
3. Annotators will see open issues on assignments, and the task owner or manager can see them on the task level. Annotators can then go and correct the annotations.
4. After correcting an annotation, the annotator flags it for review and sets the status to Complete. This will make the item appear again in the QA task.
5. The reviewer can see how many for-review annotations are pending on their assignments, and open them for review.
6. If corrections are accepted, the reviewer can flag them as Approved. If all corrections are approved, the reviewer can set the item's status to Approve.

Status Overlay On Completed Items

Setting a status as Completed on items in annotation or QA tasks flags the item as done and often triggers further automation steps in Pipelines and FaaS. It is therefore important to ensure that no changes to items/annotations are made after setting the status, otherwise, those changes won't be in effect for the automation steps. For example, setting a status may trigger a simple step for downloading the annotations JSON. If changes are made to the annotations afterward, there won't be a trigger for a new download process.

Now Dataloop platform by forcing all users can view an overlay of the status and must remove the status from the item before making any changes, even as part of fixing issues opened during a QA task.

To enable or disable this feature, refer to Project Settings.