Task Allocation Method

Tasks can be set to work in one of the two allocation methods, which determine how items are distributed to the assignments.

• Distribution: Dataset items are distributed into assignments when tasks are created. Changes to assignments, such as adding/removing assignees, or rebalancing items, require a recalculation of the assignments.
• Pulling: Dataset items are queued in a hidden assignment and distributed to assignees in small batches, according to their progress. Changes to the team are very easy to make, as no assignee has a large number of items in their assignment.

Distributing method

Items in a task are distributed among all contributors. The automatic option will assign an equal portion of items per contributor, while the manual option gives you more control.

While the task is actively worked on by annotators, you can choose to redistribute and balance the remaining items again between annotators.

Pulling Method

When creating an annotation or QA task and selecting the Pulling allocation method, the Dataloop system will allocate to each assignee only X number of items at a time (batch size).
As assignees are working on their assignments, the system will allocate new batches (of the same batch size) whenever their pool of items requires it. An assignee cannot have more than Y items pending in their assignment ("Max items" setting).
Items not yet allocated are maintained in a queue, ready for allocation. The queue cannot be interacted with or seen.

Managing Pulling Tasks

Pulling based tasks are easier to control while in progress, and allows for better adjustment to changing needs.

Changing team size

Use the Edit task option from a task's ellipsis icon (3-dots) to add or remove contributors.

• New contributors show as new assignments.
• Removed contributors show as zeroed and locked assignments.

It allows you to track past activity and analytics.

Clear an Assignment

With Pulling-based tasks, you can clear a specific assignment from its items and put them back into the general queue.
For example, if an assignee is not working on a specific day, you can clear the respective assignment from items without removing the contributor from the task.

1. Open Workflows > Tasks page from the left portal menu.
2. Double-click on the task and click on Ellipsis icon.
3. Select Clear from the list. A confirmation message is displayed.

Load an Assignment

When the assignee is available, you can just load their assignment with a new batch, and they continue working with the new items they receive to the assignment.

1. Open Workflows > Tasks page from the left portal menu.
2. Double-click on the task and click on Ellipsis icon.
3. Select Load Batch from the list. A confirmation message is displayed.

QA Considerations

Items completed by a contributor might go through a QA task later on, and might have an issue opened on one of the annotations. In such case, the status will be removed from the item, and the item is returned back to the queue of annotation task.
Such items with issues will be directed back to their original contributor who created the annotations, with the intention of allowing contributors to account for their work. The item with issues will get a priority in the queue, and will be the first to be assigned again with the next batch pulled by the assignee.

Working in Annotation Studio

When first entering the annotation studio, wether its as part of a labeling or a QA task, the studio loads the first batch of items and the counter will show that we're on the first item of 5.

After setting a status for 4 items (e.g. Complete/Approve), the counter will show we're on the 5th item of 5 in total.

Competing the 5th item will cause the studio to pull the next batch of 5 items, and we will be back as the start point - item 1 of 5.

However if at the point of being at 5/5 items we will not complete the 5th item, and just try to move to the next one, we will find ourselves on the 1st item our of 6. That happens because we have 1 item from the previous batch, and the studio was able to load 5 more items, becasue 6 is still less than the "Max items" settings (which is 7 in our example).

Similarly, if we skip 2 items from a batch, the studio will be able to load 5 more, and we will back at 1 out of 7 items.

If for example we have 5 items in our batch, and we skip 3 of them, we will be left with these 3 items, and won't be able to load more, becasue in our example loading 5 more items will take us over the "Max-Items" limit of 7. We will be forced to complete at least 1 more item from the 3 we have, before we can pull the next batch of 5.

As we browse through our batch of items and complete/approve them, we can use the back button to review items we already passed through. However, after the studio reloads a new batch of items, we cannot go back to items we already completed/approved by using the back button - we can still get to them by going to the assignment browser, finding the item and double-clicking it.