What is a classification?
Labelbox separates its annotations into two general categories: objects and classifications. An easy way to remember the difference between the two is that object-type annotations are visible on the editor stage and classifications are not.
Structurally, objects cannot be nested within other objects. However, classifications can be nested within objects. Classifications can also be at the global level (can apply to the entire asset), and can be marked as required or optional.
User goal
We were finding that users relying on classification and not visual-based labeling (seg masks, bounding boxes, etc.) wanted a way to quickly apply these classifications to hundreds if not thousands of data rows at a time.
The question became: How can we allow our users to create these classifications in bulk directly from their data Catalog without forcing them into the labeling editor?
The project container problem
The first problem I needed to solve for was that all labeling - even for these bulk classification jobs - needs to technically happen with the container of a project.
Secondarily, the classifications have to belong to an ontology which has to be connected to that project.
User journeys
So, in order for a classification to be applied in bulk, users have to select a project with the ontology containing the classification they want to use.
Solving for other required classifications
For the cases where a user is using an existing project for these bulk classification jobs, in some instances there will be other required classifications that need to be completed.
How can we make this a clear and fast experience? After many sketches and comprehension tests, I finally landed on an elegant solution that adequately solved the problem.
Project task destinations
There was one more wrench thrown into this flow. A project technically contains at least four tasks by default: initial labeling, review, rework, and done. These tasks correspond to different steps in the AI labeling workflow.
And again, since these bulk classification jobs technically have to take place within a chosen project, we also needed to give the user a control to designate which task these data rows should be sent to in order for the classification to be applied.
Overcomplicating the problem
Adding a simple dropdown control that allowed users to designate which task they wanted to send these data rows to within the project was simple.However, I started running into a maze of problems for two scenarios:
1. What do we do with data rows that were included in the bulk classification job but already exist in the project and are queued for labeling (initial labeling task)? Should the classifications be applied as ground truth or prelabels?
2. What about those that were included in the bulk classification job but already exist in the project and have already been labeled (any other task)? Should the values be retained or overwritten?
Making things simple
After numerous comprehension tests, it became clear that adding more controls made the experience more confusing. I dug deeper and uncovered the three outcomes users wanted:
1. The admin didn’t realize these data rows already existed in the given project and want them to be excluded from the bulk classification job.
2. For data rows that already exist in the “initial labeling task”, admin wants these classifications to be applied, but as prelabels so as not to interfere with the previously queued labeling job.
3. For data rows that have already been labeled, admin wants these classifications to overwrite the previous value.
Moving to full screen experience
Originally, I was orienting all of the designs in a side-sheet layout because I didn't anticipate all of the extra controls that came out of my expanded understanding of the scope of the problem. Once I had all the pieces in place, I decided that a more luxurious, full screen experience would better suit the user.
Introducing "jobs" UX
It was important to make it easy for users to track these bulk classification jobs as they were running. In some cases, they could take multiple hours. I didn’t want to reinvent the wheel here. After doing some competitive analysis on other products, I decided to adopt the Google drive “upload” pattern to give users visibility on these jobs.
Solving for new users and speed
Using an existing classification within an existing project is mostly solved. But what we’re really trying to solve for are inexperienced first time users that aren’t familiar with any of these concepts.
There’s no way around the technical constraint of needing a project and an ontology, but how can we coach users through getting their bulk classification job started from scratch?
Let's not reinvent the wheel
Once I had a clear vision of all the steps a user needed to take, I started to see how I could re-use our existing components to create a fast flow for first time users trying to bulk classify.
Time is money
This was one of the most impactful features that I designed for our customers when it came to directly delivering value. We immediately received countless examples of positive feedback from our users - mainly around them being able to work far quicker than before. We even helped the Texas Rangers win the 2023 World Series with this workflow.
“This is amazing. We labeled 3k images in 2 days. This process used to take a full quarter for us to complete...but now I can see and label so many images at the same time, it’s great!"
- Liberty Mutual ML engineer on using bulk classification to deploy their model to production a month and a half earlier than their deadline.
