That’s when a remarkable team of innovators in video technology, including Actas, a Milestone technology partner, and Aalborg University (AAU) in northern Denmark, came together.
With sponsorship from the Danish Tryg Fonden, they embarked on a ground-breaking project to combine artificial intelligence (AI) with video technology to enhance safety significantly in the Port of Aalborg – one of Denmark’s busiest ports.
Transforming Tragedy into Progress
At the heart of the project was the development of the AAU/Milestone thermal dataset, comprising the largest published collection of annotated thermal images. By training AI models on this dataset, the goal is to create an advanced video analytics solution that will be able to promptly detect and alert rescue teams when a person falls into the harbour. This article explores the ground-breaking collaboration that has the potential to redefine safety standards in ports, ultimately saving countless lives in years to come.
The primary challenge faced by the team was training the AI system to identify instances when a person falls into the harbour. Training an AI system is akin to teaching a child to recognise objects or comprehend language. We show the child numerous pictures of dogs, enabling them to learn the defining characteristics of a dog.
Similarly, when training AI, we provide it with abundant data, in this case videos of people falling, enabling it to identify patterns associated with falls. The more data we provide, the more accurate the AI becomes in recognising falls. Unlike a child, an AI system necessitates thousands or even millions of data points and images to learn effectively.
Volunteers Take a Dip into Danger
To train the AI in detecting individuals falling into the harbour, volunteers initially simulated falls for analysis. However, due to safety concerns, they could not ask volunteers to actually fall. Instead, they were asked to jump, which is a different pattern of motion. It is important to note that genuine falls involve sudden and involuntary movements that were not captured in this data, rendering it less effective for AI training.
An AI model requires a diverse dataset to detect different types of falls consistently across various conditions and seasons.
For example, a fall in winter may present unique challenges due to the individual wearing bulky clothing. However, the potential risks associated with volunteers jumping into the freezing water of winter were too great, so an alternative approach was needed.
Test Dummies, a Safer Way to Train AI
Using a test dummy proved to be a safer and more effective method of collecting data. Instead of putting human lives at risk, researchers used a test dummy filled with warm water, making it detectable by thermal cameras, to simulate multiple falls into the harbour. This allowed the team to simulate many different types of falls, resulting in a more comprehensive dataset.
However, this approach also had its limitations. The dummy could only imitate human movements to a certain extent and lacked the complexity of a real person falling into the harbour. Additionally, gathering a large amount of data was unfeasible as simulating a person falling into the water many times with slight variations was not a viable option.
Scaling up with Synthetic Data
Initially, the team used thermal cameras to capture real images over eight months to create a thermal dataset. The resulting dataset is the largest collection of annotated thermal images ever published, with approximately 8 million bounding box annotations.
It covers four classes: person, bicycle, motorcycle and vehicle; however, luckily it does not include many examples of abnormal behaviours – including falls. Using synthetic data, the goal was to expand the thermal dataset by including many abnormal behaviours and incidents.
Synthetic data is a technique to provide additional data for an AI’s comprehension and identification of real-world events, such as when an individual falls into a harbour.
At AAU, it originated as a student project aimed at generating an artificial depiction of a person falling. However, the initial simulation was simplistic, and it soon became evident that these examples were inadequate.
Innovative Partnership to Save Lives
Recognising the need for an advanced dataset, Milestone Systems’ research department joined forces with AAU to extend the synthetic dataset. The drive for better and more precise synthetic data was not solely focused on improving an AI algorithm; it was about saving lives.
The training data was enhanced to simulate scenarios with more intricate behaviours, such as falls from wheelchairs, skating and biking. Each enhancement to the data had the potential to increase the system’s ability to detect real-life falls swiftly and accurately.
The innovative partnership between Milestone Systems and AAU has harnessed the power of synthetic data and AI to tackle the significant issue of drowning in Danish harbours. The team’s objective was to develop a comprehensive dataset capable of training an AI system to detect a wide range of hazardous situations.
The development of the AAU/Milestone thermal dataset represents a ground-breaking step forward in video technology. Ultimately, the goal is to help rescue teams respond faster and potentially reduce future drowning incidents.
