Create Dataset & Upload Data

Dataset Overview

In the Dataloop system, a dataset is a collection of items (files), their metadata and annotations. A dataset can have a file-system-like structure, with folders and subfolders at any level.

Dataset Types

There are different types of Datasets:

• Master - Original dataset, managing the actual binaries
• Clone - Contains pointers to original files, enabling management of virtual items that do not replicate the binaries of the underlying storage once cloned or copied. When cloning a dataset, users can decide if the new copy will contain metadata and annotations created over the original.
• Merge - Multiple datasets can be merged into one, enabling multiple annotations to be merged onto the same item.

Storage Types

The Dataloop platform has a flexible storage engine, which enables to attach different binary storage devices such as:

• Cloud storage devices like GCS, S3, Elastifile etc.- connect your data without copying them to Dataloop storage system, to have a single point of truth for your files and comply with various regulations.
• File system storage
• Network drives
• Databases, such as: mongo GridFS

Each storage medium is supported through its drivers and additional drivers are continuously being added.

Creating a Dataset

1. Click “NEW Dataset” from the Project-Overview page, or the Datasets main page of your project (Data-Management->Datasets on the left-side navigation menu).
2. Type in the dataset’s name in the popup box, Select your Storage type (Dataloop file system or external storage) and then click "OK".
   
   
   
   

3. If you’re connecting your Dataset to external/cloud storage:
   a. Select the storage type (GCS/Azure/S3) from the drop-down menu
   b. Select your storage driver from the list. To learn how to set up your cloud storage driver/integration, read here.
   c. Select the ‘Sync’ option to have changes made in the Dataloop system reflected in your cloud storage.

Upload Files

The dataset browser enables you to organize your files in file-system-like folders and subfolders structure. From the dataset-browser, click on the Upload icon and select if you would like to upload individual files, or entire folders. This selection is required, since the browsers' pop-up windows for selecting files or folders are different.
Another option is to drag-and-drop your files and folder onto the thumbnails area, and the upload process will initiate.