Data is messy. It’s unclear. It’s noisy. Therefore, it makes sense that when we think of “real data,” we’re not referring to data that is organized in a zip file, with all the correct annotations. We’re talking about multiple sources with different parameters, configurations, noise, standards, etc. The challenge here is to align all these resources and make this data usable for training and evaluating machine and deep learning models.
When organizing data that machines can understand, we first need to answer these basic questions:
- Can we tell what’s important and relevant?
- Is the data sufficient? How much of it is redundant?
- Can we find the data’s secret to better understand it?
- How can we visualize and get insights from mass amounts of data?
Here are some tips to get you started and to help you clear the noise so even your clueless machines will understand.
3 Sure-Fire Tips to Organizing Your Data
- Identify Relevant Data
- Achieve Transparency
- Start Small
I’ve come up with 3 sure-fire tips to guarantee you’ll be able to teach your machines to understand the data you feed them. Let’s explore each tip, and why it works.
Identify Relevant Data
The first thing you need to ask yourself is do I have too much, or too little data? You need to be able to identify which samples are missing, and where your data is duplicated. It’s important that you extract the essence of the data and not waste time on the rest (the noise).
Let’s break it down further, between human and machine:
Human:
For best results, I recommend you first extract the essence of your data sources and only annotate the “core” images. You can use unsupervised learning techniques, or get initial annotated patches, to cluster the data and help visualize the world you’re trying to represent (ontologies). Try to identify the classes that overlap and where you may need to merge or split classes to get better results.
Machine:
Keep in mind that for machines, more data does not equal better answers. Thousands of samples of the same image might not provide you with more information and probably won’t contribute anything to the question you’re trying to answer. In reality, more data means higher costs, both in annotation (human) and training (machine).
Let’s take an example – suppose you have lots of images and you want to identify the “similar” ones. This could mean a lot of things. Pixel-wise similarity, MSE for example, can be high for two images that won’t add more information than a simple augmentation.
Let’s take a look at the 2 images below:
A simple rotation gets a high mean square error (~32K), but our brain can easily see that this is the same image and does not need to be labeled twice. Instead, using a feature-based similarity will be better and will give a low distance scoring for the above images. Basically, we want the minimum amount of images with the highest variance in the model embedding plane. This will help us save money on the annotation, and for the model, we’ll use augmentations to make the most out of the core data, also helping us to avoid overfitting the model.
As Dipanjan Sarkar, Data Science Lead at Applied Materials clearly states, “There are no shortcuts or direct mathematical formulae to say if we have enough data. The only way would be to actually get out there and build relevant ML models on the data and validate based on performance metrics […] to see if we are getting a satisfactory performance.” Therefore, if your data is relevant then your performance will prove so, too.
Achieve Data Transparency
Training a neural network requires many examples and you don’t want to ‘get lost’ while iterating through each batch. Make sure you stay on top of the data every step of the way and don’t be afraid to get your hands dirty with data; don’t assume the model understands you. Don’t assume people understand you.
Human:
It’s not easy to express your labeling requirements, it’s even harder in writing, in different languages, across many hands, and all over the world.
You have to keep track of the annotation process and keep in touch with the individual annotator to get the best of their work.
Examples are better than explanations – it’s better to annotate a few examples and it will help you explain what exactly needs to be done. Be sure to have an open communication channel over the data to answer questions and review edge cases. Same as the model, the annotation requirements never stop evolving…
Machine:
If by any chance you don’t get perfect training on the first attempt, one of the first things to review is what the model is seeing. You can do this by saving input tensors with the annotation drawn on top. Reverse the augmentation and any pre-processes manipulation and plot.
In my experience, we’ve traced many “weird” training issues to one simple error in the data and annotation loading process. Bottom line: don’t give the model data you wouldn’t eat. Here at Dataloop, we recommend using the SDK tools to achieve that flow easily.
The ability to fully understand the machine will, in turn, help you determine the reason and will help guide data scientists in building an optimal model.
Start Small With Your Model Training
Don’t go running days of training without getting some results on small subsamples. It’s not always true that “the more data, the better.” Instead, get quick and fast results and catch bugs and errors before going over the entire dataset. Keeping it small can help you keep on track. You can ascertain whether you’re making progress, whether your model needs improvement, and if you find bugs or errors you can quickly make the fix and move on.
Let’s apply this to the human/machine scenario:
Human:
Don’t rush to annotate every single image you have. Making annotators work for nothing is one of the best ways to waste your budget. Start with small loops of labelers, thus allowing you to improve and better define your project’s recipe (instructions). Throughout this loop, identify where your labeling requirements “pass” and where more labeling work is required. Dataloop provides a very intuitive recipe-building option, giving you extensive support to build complex (and simple) instructions that will guarantee you’re set up for success.
Machine:
The same goes for the training phase; a lot of time and money can be wasted on annoying coding errors and wrong data processing (this is where the model transparency kicks in). The best practice is to always start small. It’s better to test the entire training flow on a very small sample of images rather than a large sample that will lead to many more errors and issues. See how your model converges and try loading back the trained weights for a quick inference before you waste days of training just to see if an error pops up after the last epoch (“[Errno 2] No such file or directory”).
Summed Up
Machine learning is based predominantly on data and is what enables the training of algorithms. The important thing to keep in mind is that if you have loads of data but most of it is useless – then you won’t be able to make the machine learn successfully. Datasets require proper preparation to render successful results. Furthermore, if a machine can’t make sense of the data it can in fact cause harm. Following these tips will help you stay on track and will help make your datasets more suited for machine learning. Ready to make your data “stupid” enough for machines with Dataloop? Find out how. Let’s talk.
