Pipeline Overview

Overview

Create automated models that weave together humans and machines to process data in a pipeline architecture – a series of nodes, where each node’s output is the input of the next one.

The Dataloop pipeline process allows transitioning data between

• Labeling tasks
• Quality assurance tasks
• Functions installed in the Dataloop system
• Code snippets
• Machine learning (ML) models

Your data can be filtered by any criteria, split, merged, and change status in the process.

Altogether, Dataloop’s pipeline can:
🗸  Facilitate any production pipeline
🗸  Preprocess data and label it
🗸  Automate operations using applications and models
🗸  Postprocess data and train models of any type or scale at the highest performance and availability standards

The following example shows a pipeline where data is preprocessed by code (e.g., a video is cut into frames) and then directed to three different tasks that run in parallel. The items marked as completed are sent to a separate task (e.g., QA task), whereas the items that are of status discard are sent to a separate dataset.

The Dataloop’s Project Pipelines page is where you manage and monitor your pipelines, start new pipelines from scratch or from a template, and access each pipeline’s Pipeline Details page.

Pipelines Page

The metrics bar is where you find information about your existing pipelines: How many pipelines currently exist and how many are in status running, failed, pending, or standby.

Pipeline List

The Project Pipeline Table contains important data about your existing pipelines.

• Customize the table to include different columns by clicking the + icon on the right.
• Sort the table to present meaningful data by clicking the header of each column.
• Search the table to easily find the pipeline you are looking for by entering any part of the pipeline’s name.

Each row in the table allows you to perform actions specific to the pipeline listed on that row.

• Start or pause a pipeline by clicking the play/pause icon.
• View or edit a pipeline by clicking the view mode or edit mode icons.
• Click the three-dot icon for additional options, such as deleting a pipeline, copying the pipeline ID (e.g., you may need to include the pipeline ID in SDK scripts), or managing the pipeline’s secrets.

Managing the pipeline’s secrets allows you to create environment variables that will be saved in the vault. This is recommended when using sensitive information, like username and password, in your code.