Machine Learning Development: Custom Features for Media Content
Automated metadata collection for media content using AI and ML
An LA company is developing a new way of creating and extracting metadata across the enterprise. Its main clients are entertainment multinationals such as Disney, Warner, BBC, FOX News, and HBO; huge players in the market that rely on this platform to search across multiple sources of media content faster and better by using Machine Learning. This platform is used to process raw footage material and then search, identify and organize data such as people, symbols, sounds, brands, and anything that can be automated thanks to ML.
Looking for custom development in the Machine Learning field
Historically, this company was one of Amazon AI services’ main consumers. However, at some point, it had some specific requirements that Amazon could not meet, so the team decided to build its own in order to have a more custom solution and to better utilize its resources.
They soon found out that it isn’t easy to find data engineers and machine learning experts in the USA though. This is mainly due to the fact that these roles command very high salaries and there’s great competition for them between the giants in AI such as Microsoft, Facebook, Google, and Amazon itself.
Nearshore outsourcing: the right skills at a reasonable cost
Looking for options abroad, its executives reached out to us and we promptly assembled a nearshore team composed of experts in different fields such as mathematics, data manipulation, machine learning, computer vision and Python development.
It was decided that the US team would remain in charge of the platform’s general development and the nearshore team would focus on ML custom development, following the company’s strategy of outsourcing this very specific (and traditionally costly) part of the process. For this reason, the UruIT nearshore team has come to be known as this client’s ML team. While its in-house staff works on the product's database and UI/Front End aspects, our devs are dedicated to developing the APIs it uses as Machine Learning services. When the client identifies a new opportunity for creating a custom ML service, this is what the development process looks like:
- Research: During this discovery phase, the developers focus on finding references for solving the challenge presented by the client, and based on that, they propose a solution. In this phase, the team identifies the state-of-the-art options in the field, open source products, existing datasets, and other tools they could re-use.
- Validation: In order to endorse the theories and estimate the effort, the team runs different experiments, called Proofs of Concept. In this step, the team trains several models in order to obtain results, gather metrics and document them. Based on the outcomes, they can evaluate and compare the different options. As a result, a general documentation is generated, detailing the different approaches for tackling the API development, plus benefits, costs and risks. We share this document with the client so we can discuss the alternatives and select one of them. Part of the decision is to define sources of data to use, the main features to extract, the ML algorithms, training and evaluation methods to apply, etc.
- Development: After the decision is made, our team gets started with the API development. Some of the good practices we put in place are unit testing, continuous integration, and code reviews. One of the developers from the in-house team is key during this process because he's responsible for connecting the API with the client's platform. Also, from the very beginning of the process until the API is ready to go live, we work very closely with another in-house dev who is focused on DevOps so we can discuss the feature implementation and the required resources (hardware, software, etc) to do so.
- Final demo: When the API is up and running, we often have another demo with the sales team. The goal is to present the solution, answer the sales reps’ questions, and support them in understanding it. Afterwards, the sales team presents this information to customers, so it’s key for them to have a clear vision of what the solution can do.
Delivering various features powered by Machine Learning
Since the beginning of our partnership in February 2018, the team has grown, as well as the list of features we’ve worked on:
- Visual recognition, such as face recognition, object detection, image captioning (natural language description from images), OCR (optical character recognition), etc.
- Custom object detection
- Audio classification & speech recognition, with 300+ general sounds detected
- Detection of events in sports such as a goal completion or a red/yellow card
- Celebrity recognition
- Language detection, both written and spoken from audio
All of these developments were proposed by the client mainly to meet its customers’ demands and others were thought of by the client to boost sales. Likewise, our nearshore team is proactive in suggesting new implementations and approaches for making the product even more robust.
In 2019, the CTO visited our headquarters in Uruguay and a few months later, the VP of Product Development also came. We’re planning to receive more of its executives in the near future in order to continue to strengthen our work relationship and set new goals for the machine learning development process. On the technical side, we’ll now be focusing on areas such as football event recognition, custom logo detection, animated character recognition, custom audio detection, and the improvement of text recognition services so the platform can cover these demands and offer clients all the tools they need.