#interactiveexperience
#creativecoding
Immersea
Immersea
Immersea
Immersea
Immersea
An interactive experience for pediatric patients in hospital waiting spaces. This project was created as Deirdre's Master's Thesis at NYU Tandon School of Engineering.
An interactive experience for pediatric patients in hospital waiting spaces. This project was created as Deirdre's Master's Thesis at NYU Tandon School of Engineering.
An interactive experience for pediatric patients in hospital waiting spaces. This project was created as Deirdre's Master's Thesis at NYU Tandon School of Engineering.
An interactive experience for pediatric patients in hospital waiting spaces. This project was created as Deirdre's Master's Thesis at NYU Tandon School of Engineering.
An interactive experience for pediatric patients in hospital waiting spaces. This project was created as Deirdre's Master's Thesis at NYU Tandon School of Engineering.
January 2019-January 2020
Timeline
Design, Research, & Development
Design, Research, Development
Role
January 2019-January 2020
Timeline
Design, Research, & Development
Role
Background
Background
As part of my Master's Thesis at NYU Tandon School of Engineering, I developed an interactive aquarium experience called "Immersea" to enhance hospital waiting areas for pediatric patients and their families.
Using creative coding and AI, the project provides a dynamic and engaging distraction to patients. It was successfully tested with former patients at NYU Hassenfeld Children’s Hospital, and their feedback was used to improve the overall design and functionality. Future iterations aim to add more personalization features to cater to an even wider range of children's needs and preferences.
As part of my Master's Thesis at NYU Tandon School of Engineering, I developed an interactive aquarium experience called "Immersea" to enhance hospital waiting areas for pediatric patients and their families.
Using creative coding and AI, the project provides a dynamic and engaging distraction to patients. It was successfully tested with former patients at NYU Hassenfeld Children’s Hospital, and their feedback was used to improve the overall design and functionality. Future iterations aim to add more personalization features to cater to an even wider range of children's needs and preferences.
The goal of Immersea is to relax, distract, and entertain pediatric patients waiting in the hospital.
Problem Statement
Problem Statement
Waiting in a hospital can be anxiety provoking for pediatric patients and their families.
1
Waiting in a hospital can be anxiety provoking for pediatric patients and their families.
There are a lack of interventions that target a patient’s well being at various points of their hospital stay. There is also an expectation of personalized experiences in healthcare and the need to cater to children who are going up digitally.
2
There are a lack of interventions that target a patient’s well being at various points of their hospital stay. There is also an expectation of personalized experiences in healthcare and the need to cater to children who are going up digitally.
These findings led me to the question... How might we improve the experience of pediatric patients in hospital waiting spaces?
3
These findings led me to the question... How might we improve the experience of pediatric patients in hospital waiting spaces?
Users and Audience
Users and Audience
The primary user population is pediatric patients, typically aged 0 to 18.
It was important that I considered other users like family members and clinicians as well. I specifically chose waiting spaces because I found it was important to prevent anticipatory anxiety prior to treatment as early as possible and the waiting room is one of the first places a patient comes in contact with.
The primary user population is pediatric patients, typically aged 0 to 18.
It was important that I considered other users like family members and clinicians as well. I specifically chose waiting spaces because I found it was important to prevent anticipatory anxiety prior to treatment as early as possible and the waiting room is one of the first places a patient comes in contact with.
Roles and Responsibilities
Roles and Responsibilities
At NYU Tandon School of Engineering, completing a Master’s Thesis is an individual endeavor. Nonetheless, throughout my journey, I engaged with over 200 industry experts to brainstorm, iterate, and refine my ideas. Participating in numerous ideation and feedback sessions with my classmates and professors was also integral to my project's development.
I was lucky to have the backing of NYU Langone's Digital Design Lab, which provided valuable support.
I conceived this project from the ground up, taking charge of every phase: from the initial brainstorming and research, to conducting interviews, creating prototypes, developing the final product, learning necessary programming languages, crafting the visual design, and conducting extensive testing.
At NYU Tandon School of Engineering, completing a Master’s Thesis is an individual endeavor. Nonetheless, throughout my journey, I engaged with over 200 industry experts to brainstorm, iterate, and refine my ideas. Participating in numerous ideation and feedback sessions with my classmates and professors was also integral to my project's development.
I was lucky to have the backing of NYU Langone's Digital Design Lab, which provided valuable support.
I conceived this project from the ground up, taking charge of every phase: from the initial brainstorming and research, to conducting interviews, creating prototypes, developing the final product, learning necessary programming languages, crafting the visual design, and conducting extensive testing.
Scope & Constraints
Scope & Constraints
The project began as part of a Research Methods course to help us come up with a Thesis topic. The project had two primary deliverables: the actual product - the interactive aquarium, and a comprehensive 70-page thesis paper detailing the research and development process.
Tackling this solo, the 11-month journey was intense, covering extensive research, design, and programming. It was a deep dive into new UX, technological, and healthcare territories for me, packed with learning and hands-on experience.
From the beginning, I made it a priority to get feedback from a diverse group of people. This included designers from leading healthcare companies, medical professionals from local hospitals, academic experts in the industry, and most importantly, the pediatric patients themselves. Their insights were invaluable in shaping the project's direction and outcomes. Their insights were key to shaping and refining Immersea's direction and success.
The project began as part of a Research Methods course to help us come up with a Thesis topic. The project had two primary deliverables: the actual product - the interactive aquarium, and a comprehensive 70-page thesis paper detailing the research and development process.
Tackling this solo, the 11-month journey was intense, covering extensive research, design, and programming. It was a deep dive into new UX, technological, and healthcare territories for me, packed with learning and hands-on experience.
From the beginning, I made it a priority to get feedback from a diverse group of people. This included designers from leading healthcare companies, medical professionals from local hospitals, academic experts in the industry, and most importantly, the pediatric patients themselves. Their insights were invaluable in shaping the project's direction and outcomes. Their insights were key to shaping and refining Immersea's direction and success.
Being a student, testing my ideas within active healthcare settings and with the right user groups was tough due to patient confidentiality and general healthcare regulations. This meant I had to constantly think on my feet and get creative with how I'd test my project. But teaming up with NYU Langone's Digital Design Lab made it possible and connected me with the appropriate groups.
Being a student, testing my ideas within active healthcare settings and with the right user groups was tough due to patient confidentiality and general healthcare regulations. This meant I had to constantly think on my feet and get creative with how I'd test my project. But teaming up with NYU Langone's Digital Design Lab made it possible and connected me with the appropriate groups.
Tools
Tools
Figma
p5.js
TensorFlow
HTML5
Adobe Photoshop
Adobe Illustrator
Design Process
Design Process
Initially, I delved into understanding the emotional impact of hospital stays on pediatric patients and their families, focusing on anxious and negative feelings. I engaged multiple medical experts from different fields like physical therapy, pediatric neurology, and nursing, among others, to guide my extensive research. Small-scale experiments were conducted to extract user insights to formulate a product idea, pictured below.
Several themes arose from these explorations, such as patient well-being and mental health, the influence of hospital settings, holistic healthcare, distraction tactics, non-medical treatments, societal healthcare expectations, digital media's role, and the changing landscape of children's play. My project, aimed at transforming waiting areas into interactive environments for pediatric patients and their families, evolved from these themes.
The Human Centered Design process facilitated iterative testing of my hypotheses and designs. This interdisciplinary approach was instrumental in formulating solutions within the healthcare space.
Initially, I delved into understanding the emotional impact of hospital stays on pediatric patients and their families, focusing on anxious and negative feelings. I engaged multiple medical experts from different fields like physical therapy, pediatric neurology, and nursing, among others, to guide my extensive research. Small-scale experiments were conducted to extract user insights to formulate a product idea, pictured below.
Several themes arose from these explorations, such as patient well-being and mental health, the influence of hospital settings, holistic healthcare, distraction tactics, non-medical treatments, societal healthcare expectations, digital media's role, and the changing landscape of children's play. My project, aimed at transforming waiting areas into interactive environments for pediatric patients and their families, evolved from these themes.
The Human Centered Design process facilitated iterative testing of my hypotheses and designs. This interdisciplinary approach was instrumental in formulating solutions within the healthcare space.
First Prototype
First Prototype
Progress towards the first prototype involved small-group interviews, expert consultations, mock-ups, and feedback sessions. I decided to use the Xbox 360 Kinect to allow users to engage with the interactive aquarium, reducing physical contact in light of high infection rates in hospitals. A basic yet fully functional prototype was created for testing and subsequent enhancement for the pre-thesis presentation.
Progress towards the first prototype involved small-group interviews, expert consultations, mock-ups, and feedback sessions. I decided to use the Xbox 360 Kinect to allow users to engage with the interactive aquarium, reducing physical contact in light of high infection rates in hospitals. A basic yet fully functional prototype was created for testing and subsequent enhancement for the pre-thesis presentation.
Second Prototype
Second Prototype
Initial testing with young children revealed that the Kinect sensor failed to accurately capture body proportions of children under 40 inches, including those in wheelchairs, beds, or in other seated positions.
Initial testing with young children revealed that the Kinect sensor failed to accurately capture body proportions of children under 40 inches, including those in wheelchairs, beds, or in other seated positions.
Project Evolution
& Technology Integration
Project Evolution
& Technology Integration
To ensure broad accessibility, I decided that the Kinect concept wasn't going to work in a hospital setting. So, I spent weeks exploring other technological possibilities.
Considering the vast age range among pediatric patients, I investigated behavioral and developmental milestones relevant to different age groups through parent interviews and research. These insights were translated into empathy maps, journey maps, and user scenarios and personas.
After much experimentation, I opted to use p5.js (a JavaScript creative coding language), ml5.js (Machine Learning), and PoseNet to enhance accessibility for pediatric patients. PoseNet, which detects human poses through 17 body-mapped keypoints, was more adaptable to varying body types and proportions than the Kinect was.
This creative coding library allowed me to design interactions with certain keypoints, such as using left and right wrist keypoints for popping bubbles and mapping the snorkel to the users' face using the eyes and nose keypoints. The iterative learning of this technology's capabilities and limitations was challenging but so exciting.
To ensure broad accessibility, I decided that the Kinect concept wasn't going to work in a hospital setting. So, I spent weeks exploring other technological possibilities.
Considering the vast age range among pediatric patients, I investigated behavioral and developmental milestones relevant to different age groups through parent interviews and research. These insights were translated into empathy maps, journey maps, and user scenarios and personas.
After much experimentation, I opted to use p5.js (a JavaScript creative coding language), ml5.js (Machine Learning), and PoseNet to enhance accessibility for pediatric patients. PoseNet, which detects human poses through 17 body-mapped keypoints, was more adaptable to varying body types and proportions than the Kinect was.
This creative coding library allowed me to design interactions with certain keypoints, such as using left and right wrist keypoints for popping bubbles and mapping the snorkel to the users' face using the eyes and nose keypoints. The iterative learning of this technology's capabilities and limitations was challenging but so exciting.
Frequent user testing of individual program elements informed my final decisions. I created a fully functional prototype and revised it based on feedback from users, classmates, professors, and my wider research network. This technology offered me the freedom to implement user and expert feedback through testing.
Frequent user testing of individual program elements informed my final decisions. I created a fully functional prototype and revised it based on feedback from users, classmates, professors, and my wider research network. This technology offered me the freedom to implement user and expert feedback through testing.
A notable feedback group was NYU Hassenfeld Children’s Hospital's Youth and Family Advisory Council, composed of ~40 former patients (ages 12-21), parents, and clinicians. Their invaluable insights significantly shaped my project, leading to improvements in interactivity, visual design, functionality, and accessibility. I must note, however, that due to patient privacy protection, no documentation was made during these sessions. Instead, I created mock-ups using the actual program and final product to illustrate the testing for my portfolio and presentations.
A notable feedback group was NYU Hassenfeld Children’s Hospital's Youth and Family Advisory Council, composed of ~40 former patients (ages 12-21), parents, and clinicians. Their invaluable insights significantly shaped my project, leading to improvements in interactivity, visual design, functionality, and accessibility. I must note, however, that due to patient privacy protection, no documentation was made during these sessions. Instead, I created mock-ups using the actual program and final product to illustrate the testing for my portfolio and presentations.
"I always find that watching animals interact with each other and listening to sounds of nature is a distraction from anythign negative I might be feeling."
Youth Advisory Member
NYU Hassenfeld Children's Hospital
"I always find that watching animals interact with each other and listening to sounds of nature is a distraction from anythign negative I might be feeling."
Youth Advisory Member
NYU Hassenfeld Children's Hospital
"When I'm sick, my moods are always changing. Sometimes I don't want to be on my phone and could see myself listening to relaxing nature sounds, playing a game, or just watching the scene."
Youth Advisory Member
NYU Hassenfeld Children's Hospital
"When I'm sick, my moods are always changing. Sometimes I don't want to be on my phone and could see myself listening to relaxing nature sounds, playing a game, or just watching the scene."
Youth Advisory Member
NYU Hassenfeld Children's Hospital
The Final Product & Summary
The Final Product & Summary
I developed an AI-powered interactive aquarium experience using creative coding, Machine Learning, and PoseNet. This single or multi-player experience assigns each user a snorkel and doesn't require touching a screen or holding a controller. While it starts as a tranquil underwater scene, user interaction can turn it into a game. However, users can also choose to just enjoy its calming elements.
The project works via a webcam and is designed to be projected on a wall or displayed on a television but is also functional on personal devices like laptops, phones, and tablets. Due to IRB restrictions and the project's nature, I mostly relied on qualitative data from pre-planned user testing sessions within the hospital. While ideally, I would have liked to use parent rating scales and patient heart rates as metrics, my actual success criteria were users' emotional reactions and verbal feedback.
I explored healthcare personalization by simulating a patient adding their own drawn fish into the aquarium from their device. This was done using NFC stickers, an Arduino, physical toy fish, and a breakout board sensor. While this feature won't be used in real-life hospital scenarios due to infection risk, the otherwise positive feedback encourages further exploration for future versions.
I developed an AI-powered interactive aquarium experience using creative coding, Machine Learning, and PoseNet. This single or multi-player experience assigns each user a snorkel and doesn't require touching a screen or holding a controller. While it starts as a tranquil underwater scene, user interaction can turn it into a game. However, users can also choose to just enjoy its calming elements.
The project works via a webcam and is designed to be projected on a wall or displayed on a television but is also functional on personal devices like laptops, phones, and tablets. Due to IRB restrictions and the project's nature, I mostly relied on qualitative data from pre-planned user testing sessions within the hospital. While ideally, I would have liked to use parent rating scales and patient heart rates as metrics, my actual success criteria were users' emotional reactions and verbal feedback.
I explored healthcare personalization by simulating a patient adding their own drawn fish into the aquarium from their device. This was done using NFC stickers, an Arduino, physical toy fish, and a breakout board sensor. While this feature won't be used in real-life hospital scenarios due to infection risk, the otherwise positive feedback encourages further exploration for future versions.
Conclusion &
Next Steps
This project was completed prior to the start of the COVID-19 Pandemic, however, I did have plans in early 2020 to test with patients at Advent Health Children's Hospital in Orlando, Florida. That hospital system frequently works with Disney's Imagineers, so It would have been cool to get feedback from patients. Maybe I'll have to revisit that idea in the future!
I had the privilege to test this with past pediatric patients at NYU Hassenfeld Children’s Hospital. Despite considering accessibility throughout the process, not all features could be implemented due to time constraints. The project's future enhancements will focus on personalizing features for individuals with low vision or blindness, cognitive impairments, and physical disabilities.
