Case Study Frame Slider

The Nielsen Company is responsible for the marketing-oriented evaluation of recorded sports events. The video analysis verifies the performance that is to be delivered.

Nielsen's outdated video analysis tool was to be replaced by a cloud-based app. The goal was to read out the videos that were to be provided in the cloud with frame accuracy and to navigate within videos more easily. In its basic analysis functionality, we oriented ourselves to the existing tool. A central change was made in the configuration of the user interface. The read-out frames were to be integrated into the user interface in order to replace the previous solution of a pop-up video player.

For the duration of the project, a team of frontend and backend developers as well as testers were made available to our client. The videos are first loaded from the server into an S3 bucket. In order to be able to handle the videos on the server in a way that is optimized for the browser, we created a frame worker that extracts frames from the videos and stores them in the S3 bucket. An SQS Queue informs the frame worker that new videos are uploaded to the bucket. The SQS Queue groups the videos according to the “first in, first out” principle.

Independently from our development team, Nielsen developed a GraphQL API that provided a central interface for our work. To communicate with the GraphQL API, we created a Backend For Frontend (BFF) that prepares the extracted data for the GUI. The BFF is also connected to a frame server, which in turn connects to the bucket where the videos are stored. The BFF reads out and transmits the frames needed for the ones in the frontend.

To avoid having to wait for cloud services or deployments while developing the tool, we set up a local minIO server in addition to the S3 bucket. This allows developers to switch between cloud-based services and the local server as needed via configuration.

For a user-friendly video analysis option, we provided Nielsen with a new video playing applicationn named frame slider. The framer slider displays the video in individual frames in the frontend, meaning only a small part of the video file actually has to be read out to ensure functionality. We designed the tool in Canvas before integration into the user interface. This was also enhanced with numerous features, including user interface configuration.

User authorization for the video analytics tool takes place through Okta via single sign on. In addition, we created a Grafana dashboard through which data and logs can be visualized and monitored.

The user-friendly video analysis is done via the video playing application called frame slider. The analyzed video is streamed via websocket. In the user interface, solely the frames that need to be displayed are read out which minimizes data traffic. The user interface of the previously used tool was adapted and supplemented with functions that alleviate the daily analysis routine. New users can easily familiarize themselves with the functionality of the frame slider via an interactive walk-through.