From care assistant to lifesaver: raising a drone with added social value

Foto: Bram Saeys TU/e Studententeam Blue Jay

“I may be more experienced, but these young people come with fresh ideas. We learn from each other.”

Take a team of smart students from the Eindhoven University of Technology (TU/e) with a vision to develop an autonomous drone that adds real value in healthcare and related fields, make them aware of the ever-present threat of children drowning in swimming pools and you’ve got a challenge to get passionate about.

Despite the best efforts of thousands of well-trained lifesaving professionals and volunteers, on average each year 60 people will suffer injuries as a result of almost drowning in Dutch swimming pools, and 2 will actually drown. By far the most common group of victims is children.

According to Anja de Brouwer, a long-serving volunteer at the ‘Eindhovense Reddingsbrigade’ (the life rescue volunteers service in Eindhoven), who is trained to realistically simulate accident victims, “When a child’s drowning they usually don’t shout or wave their arms as you’d expect. It just goes quiet and, before you know it, you’ve lost them.”

Serious roles

Anja is one of the many volunteers who recently took part in a large-scale event at the Pieter van den Hoogenband Swimming Stadium organized by the rescue service to help the Blue Jay Eindhoven team develop a drone that can assist lifeguards by performing three key tasks: monitoring activity in a busy pool, detecting a swimmer in distress and responding appropriately.

During the evening, eight cameras continuously film the volunteers as they simulate drowning people. “To learn how to detect which movements mean someone may be in trouble rather than playing, the drone must be shown lots of different videos of drowning people,” explains Liza Boormans, the Blue Jay team manager. “Once it has enough data, it can recognize and detect drowning behavior itself.” Thankfully, film footage of people really drowning is rare. Hence the need to film the LOTUS volunteers role-playing victims.

Not waving but drowning

This process of detecting someone drowning is one of the real innovations of the lifeguard assistant project, and one of its biggest challenges. “There are already detection methods on the market that spot if someone has drowned, but the Blue Jay team is developing a drone to detect if someone is about to drown — in time to help them. That’s a much more complex challenge,” says Arjen van de Wetering, an Internet of Things (IoT) consultant at IBM who has been supporting the Blue Jay team since IBM became a main partner to the project a few months back.

“Distinguishing between play and distress is difficult using traditional analytics methods” explains Arjen. “So the team is using artificial intelligence (AI), employing IBM’s Watson technology. The team will use the same drowning simulation videos to train the system and to test whether it can make the drowning-playing distinction, and later whether it can make that judgement effectively in real-life situations.”

This detection process isn’t simple. Having singled out individuals from its aerial view of the whole pool, the system must follow people using approximately 2 seconds of consecutive video frames to determine how someone is moving. The oldest frame from those 2 seconds is then dropped and replaced by a new one. The system is thus continuously analyzing how people move within a 2-second timeframe. The calculations to determine whether a swimmer’s movements constitute playing or drowning take a huge amount of computer power. In our first pilot setup, all these videos are streamed live from the cameras to the cloud, where the calculations are then done on IBM’s Watson AI mainframes. The ‘decision’ about the swimmer’s movements is then made by Watson and sent back in a matter of milliseconds to the drone, which acts accordingly — a classic IoT solution. For the final system, the team probably don’t want to depend on an unreliable internet connection, so it might consist of a trained Watson-based application on-premise. But that decision has yet to be taken.

International and multi-disciplined

Blue Jay Eindhoven is an on-going project with the ultimate goal of developing a drone that can assist professionals in healthcare settings. The team is made up of students from the Technical University of Eindhoven (TU/e) who take time out from their studies to join Blue Jay for a year.  “Because you get no elective points or financial support for working on Blue Jay,” explains Liza, “only people who are very committed to our goals and vision join the team.” Liza’s own role as team manager ends in September when she returns to her full-time studies in Medical Sciences & Technology.

The current team is an international group of 18 students from 9 different academic disciplines. As the Blue Jay personnel change pretty well entirely each academic year, continuity is assured through an advisory board that includes some of the previous year’s students who support the next year’s team. “The board ensures the team follow the long-term Blue Jay vision,” says Liza, “but gives them sufficient freedom to add their own insights and innovations.”

Last year’s team focused on developing a drone to assist in the Máxima Medical Center near Eindhoven. Much progress was made. It could, for example, play games with child patients. But a stumbling block, which has not yet been overcome, is noise: a drone is still pretty noisy and that obviously isn’t handy in a hospital environment.

The perfect setting

Ironic, then, that it was during a brainstorming session on solving the noise problem that the team heard about the case of a 9-year-old refugee girl who drowned in a swimming pool in Rhenen. They immediately saw the potential for a Blue Jay drone to help prevent such tragedies. “Swimming pools are a perfect setting,” explains Liza, “Noise isn’t an issue, but we can still demonstrate the huge potential value of the drone: the ideal proof of concept.”

But as team project manager, Mike van Sighem, points out, the swimming pool drone has its own challenges. “As well as drowning detection, it must be able to autonomously navigate buildings, notify personnel, and bring bulky and heavy objects to an accident scene.”

Pim van Dommelen, Blue Jay’s mechanical engineer, sees this last as probably the biggest mechanical challenge of the lifeguard drone. “It needs to be light, otherwise you have to switch batteries every few minutes, and a drone that’s out of the air too often isn’t much use as a lifeguard assistant. But at the same time, it must be powerful enough to fetch heavy objects such a lifebuoy, defibrillator or first aid kit.”

Partners make perfect

NXP, the second main project partner, is helping the team with making the drone both robust and responsive enough to do its job. “Last year we helped the Blue Jay team develop a robust communication channel between ground station and drone,” explains Gino Knubben, Principal System Engineer at NXP and closely involved with the company’s support to Blue Jay. “This year they’re using a slightly newer version of the same technology, but our role is even more drone-focused, looking mainly at two major systems. First, the flight control system, which deals with the drone’s movement, orientation, balance and remote control – the team chose this year to switch to our flight controller, NXPhlite. And secondly, the application processor, which deals with the display and the interaction.”

For the project, functionality is critical and one big benefit of working with world-class companies like NXP, IBM and Fourtress, the third and final main Blue Jay sponsor, is the Blue Jay team’s access to advanced technology not found in consumer products. “Compared to regular Wi-Fi on your phone,” explains Gino, “our automotive-qualified connectivity solutions are superior in robustness, RF reflections, power consumption and the number of ad-hoc connections you can do per second.” The robust connections come from 802.11p technology used in the automotive industry, and can handle the drone’s rapid movement and ad-hoc communications. Gino laughs, “Two years ago, everybody had to remember to switch off the Wi-Fi on their phones before testing or demonstrating. Now with 802.11p we’ve resolved that particular interference issue.”

One-stop shop

Another benefit of having NXP as a partner is that, unlike many semiconductor manufacturers, they can provide a wide range of components. “We’re providing a large part of the electronics they need for drone development,” says Gino. “Not just the wireless link but also high-performance microcontrollers; proven reference boards with all the components assembled; and even complete sub-systems, including supported Operating Systems. The team then adds their peripherals, displays, sensors, applications, graphical user interfaces, etc.”

But Gino is first to say the collaboration is not one-way traffic. “The Blue Jay team are basically beta-testing some of this technology for us and their feedback is invaluable. For example, on development of the FMU (Flight Management Unit): along with all the other partners involved, they provide feedback, they test… It’s outside the students’ comfort zone and the remit of the project. So that wider commitment is great. They also visit us at NXP on a regular basis, which is a lot of fun.”

And there are areas where old hands learn from the students. “They’re very knowledgeable, for example, on the ‘total drone’, such as how to calibrate and configure it. And have some seriously skilled drone pilots! Another example is hardware assembly experience. We’d been assembling drones at NXP ourselves, which took a lot of time. Then Jari van Ewijk, electrical engineer from Blue Jay turned up and built one in under 90 minutes. I asked how he could do it so quickly. He said ‘how many prototypes you think I’ve had to build? We’ve crashed quite of few of them…’ He could probably have done it blindfold! The next day we received pictures and movie clips of their first flight with that NXP-drone kit!”

Another spin-off from Blue Jay is that the technology involved can have applications beyond the narrow (albeit very important) fields of lifesaving and hospitals. “For example, there’s huge potential for FMU technology and drones in areas like warehouse logistics,” says Gino. “So the beta-testing also contributes to making this a proven solution NXP can offer in other contexts and sectors where the same parameters apply.”

Priceless know-how

Fourtress provide software solutions for technical systems. “We’ve been involved since the first Blue Jay team,” says field manager, Douwe Gerritsen. “Initially, we simply provided technical knowledge. But as we have engineers with pretty strong drone-related experience, for the past two years instead of money we have loaned the team an engineer for 3 months.”

It’s win-win as the engineer also gets interesting experience. He has worked on the low-level systems, Lidar, obstacle detection, the drone’s inner workings, the ultra-wide band and data communication systems. Fourtress also provided a student on placement with them to work on the Blue Jay mobile app development and drone-API as their thesis project.

“Probably our biggest challenge in developing the app is that Blue Jay team projects are very dynamic,” says Douwe. “They set ambitious goals. That’s inspiring, of course, but means implementation tends to change along the way as they encounter problems. You try to work with that flow, so you don’t become an obstacle yourself. But sometimes you need to help them set boundaries in terms of what’s achievable within the timescale.

“There’s also the technical challenge of trying to fly a drone in populated environments, including the paradox of wanting a drone that both works totally autonomously, but is also controlled by people.” This means a person must be able to use the app to give the drone commands, after which the drone figures out for itself how to execute the task. Sometimes deciding autonomously, for example, which route to take or task to prioritize. But the interface needs to let people easily set new tasks and get a clear overview that updates them on what each drone is doing at that moment.

“The app is basically the drone’s manager. It gives the drone tasks, but the drone decides how to execute them. The prioritization aspect is interesting. It’s important the user can schedule tasks to be done later or immediately. But also, that they get feedback on how quickly the drone will actually do the task. Because when an emergency happens it will override everything else. This variety of tasks makes things a lot more complex.”

Another challenge is notifications. “The tricky part is that people will use their phones not just to control the drone, but also to surf the internet, take calls, etc. Technically, this means the app must be able to run a service in the background even when it’s not active, so it can still give users notifications, for example of an emergency or a completed task, when their drone app isn’t active.

Douwe sees the Blue Jay partnership as an invaluable one for Fourtress. “How we operate with Blue Jay is a bit of a model for how we try to operate with all our customers. In that, if you hire one Fourtress person, you hire all of Fourtress: all our knowledge is at your disposal.”

Professional fans

Blue Jay gets students out of their ivory tower and working with real-world organisations to address real-world needs. So it’s reassuring when society responds so positively. “The Reddingsbrigade were very enthusiastic from the start,” recalls Mike. “It’s cool how they’re so open to adopting new technology. Ultimately, we share the same goals. By working together, we can help each other achieve those goals better and quicker.”

At IBM, Arjen has quickly become a huge fan of the students and what they’re doing. “Like much of my own work, this project is at the forefront of technology. I bring my experience to the project, but these young people have a different perspective and come with fresh ideas, so we definitely learn from each other.” Douwe echoes this view on behalf of Fourtress. “Because with Blue Jay you’re working with such enthusiastic and dedicated people, it makes it so much fun.”

At NXP, Gino is particularly impressed by how the Blue Jay team leverage maximum value from partners’ expertise and function so maturely as a unit. “They take technology from us for the drone, from IBM for in the cloud, use other partners for the mechanical chassis… operating just like an efficient little company, each with their own areas of responsibility. Of 18 team members, I only deal with 2 or 3 hardware engineering, electronics and software people. For young, inexperienced students it’s pretty impressive.”

With her hands-on experience, LOTUS’ Anja is equally excited by the drone’s potential. “It’s basically an extra pair of eyes. Even the best lifeguard gets tired, but not the drone. It stays alert, and that means once it’s fully operational it will help ensure people are safer.”

Those days are still a way off. For now, it’s all about proof of concept. Even so, the team will continue to work on a drone that can provide first aid in emergency situations. Because, as Arjen says, “if one day the drone saves just one life, the whole project will have been worth it.” And you don’t need AI to detect the truth in that statement.

Learn more about Blue Jay Eindhoven: https://www.bluejayeindhoven.nl/

Or get in touch with the team if you’re interested in supporting their work, getting involved or becoming a partner. Send an email to: info@bluejayeindhoven.nl

Processing personal data is according to the GDPR only allowed with informed consent of the user (person involved). To get that informed consent it is required to be transparent about what personal data is processed and why it is processed. The user must be able to access this data, edit, delete or transfer it when he/she demands that.

Decision tree (GDPR tool)

In cooperation with the TU/e we have developed a decision tree that provides insights in what needs to be done to get informed consent regarding the GDPR law. The tool can be used from two perspectives:

User perspective:
The user perspective will give you insights if the organisation processes your personal data regarding the GDPR law.

Company perspective:
The company perspective will give you insights of what an organisation needs to do to properly process personal data of its users regarding the GDPR law.

Starting with informed consent is mandatory but does not say anything about technical implementations or security. Fourtress is also a member of Brainport Techlaw to work together with other technical companies and lawyers to stay informed of new developments in the field of tech and law and to take the necessary actions.

Do you ask for informed consent the way the GDPR prescribes? Or do you want to know as a user if an organisation meets the requirements for informed consent according to the GDPR?

Test it with our tool!

How do you translate an artistic vision into a technical implementation? That was our main challenge to tackle throughout the E.E.G. kiss project. What started as a nice idea ‘programming a kiss’ related to the 15th anniversary of Fourtress, turned into a full-scale project.

In the beginning we were tackling the very basics, such as getting a proper readout of the raw E.E.G. data from the headset. Less than one year later, we have implemented a complete platform in Python code with various filters and adjustable for multiple sensor types. On top of that, we built an engine in C++ code that runs the visual and audio feedback of the EEG kiss.

Apart from the technical challenges in the project, we got a lot of satisfaction from the interaction with professional artists. We are very proud knowing that the result of our efforts will travel around the world in the form of an artistic installation.

“Domestic Drone Experience”

A drone as your domestic help: the Blue Jay student team from TU Eindhoven (Eindhoven University of Technology) developed a ‘domestic drone’ that is ‘safe’, ‘social’, ‘helpful’ and ‘autonomous’. A household help that can understand and support you. The question is no longer whether this is possible, but how it will affect the distribution of household chores. Blue Jay matched the technology of the future to the way we live today: exactly what Fourtress is all about.

When the Blue Jay team asked us to support them in developing (Low-Level) Embedded Software, we didn’t hesitate. During a three-month period, we started work on solving a height issue.

Any change in altitude resulted in the drone correcting itself, a problem we managed to solve, after thorough research, by combining a Lidar-sensor with the accelerometer-sensor. This allows the drone to determine when a (valid) altitude difference has been detected. We ended up working on several aspects of the drone, including the wireless connection and gripper-software, the ‘eyes’ and ‘ears’ of the drone.

Fourtress’s solutions helped complete the project with a successful demo at the Maxima Medical Centre. Under the watchful eye of the BBC and RTL and other broadcasters, the drone showed what it was capable of, playing noughts and crosses with patients.

 

Our client wanted to equip one of the products from its Robot Vacuum Cleaner (RVC) range with connectivity for indoor use. For this, the RVC had to be controllable using an app (iOS and Android) indoors and outdoors. The RVC has been expanded with a WiFi module, and both apps needed to be compliant with the new standards that the client applies to all its connected products.

For this purpose, our client developed so-called common components. These standard components are to be used as the basis for the company’s connected products. An RVC is one of the most complex connected products. There was no reference project, which meant that Fourtress had to develop several common components in close collaboration with the client. All communication between the app and the RVC runs through a cloud service.

Fourtress developed both apps (native iOS and Android) in close collaboration with several stakeholders. We set up two App teams (Eindhoven and Minsk) to develop the apps simultaneously. Deployment of an App team in Minsk made it possible to reduce costs significantly. The RVC and its apps have now been launched in parts of Asia and are available in various Asian app stores.

Following the development of both apps, Fourtress also took care of their maintenance and release management, and continues to do so. These developments provide a foundation for future variations on the RVC Series: watch this space!

Pentair is a ‘multinational diversified industrial company’ with over 27,000 employees worldwide. As part of an acquisition, the company took over Hoffman’s, renowned for its ‘content meter’, a CO2 measuring solution for breweries. In fact, every brewery in the world uses Haffmans’ products. This segment of the company focuses on breweries, whilst other parts provide components for mobile water filter installations. These may be used to provide a small city with clean drinking water in case of natural disaster, for example.

Pentair develops and manufactures a variety of products for the soft drink and beer industries in-house. Design, calculation models, mechanical engineering and production support and software development come together in the company’s R&D department combines. Fourtress is working on several products, such as the Keg Monitor, a control system for the cleaning of beer barrels. When a barrel returns to the brewery after being used in catering, it is cleaned and checks are carried out to determine whether the closure is still working properly. For this purpose, a barrel is equipped with various sensors (temperature, pressure, etc.) as well as an embedded system for storing data. This ‘measurement barrel’ is placed on the cleaning conveyor belt to collect data.

Another project carried out with Fourtress is the inline alcohol meter. ‘Inline’ indicated that the sensor is placed in a beer tube, unlike lab equipment that sits on a worktop. The sensor measures the alcohol percentage as a meter of beer runs through the line every second. This measurement is based on a laser shining through the beer onto a spectrometer. The spectrum is influenced by the different components in the beer, and the presence of alcohol causes a slight deviation in the signal.

Finally, Fourtress and Pentair jointly developed the turbidity meter: a checking system that can calculate the turbidity of liquids. Although these projects involve different instruments, they have a great deal in common: they are all embedded systems with a so-called ‘ARM Cortex core’. Each runs on freeRTOS with its own library of hardware drivers and application modules, all written in C. One of the things Fourtress was involved in is the generation of generic blocks for a hardware-specific driver with a common interface. As a result, we have a greater choice of hardware, testing is easier, and we promote code re-use.