Socially-Assistive Robots for Dementia Care on Embodied Voice Assistant (Eva)
Overview
Eva platform is an open-source social robotics platform that provides you with the resources to build a social robot and its interactions cheaply, fast, and easily.
Keywords
EVA-SIM, multimodal data, dementia, HCI, HRI, Embodied Voice Assistant (EVA)
Getting started
Welcome to EvaPlatform! We're thrilled that you've decided to join our community. Our goal is to provide you with an exceptional experience every time you use our platform. Don't hesitate to reach out if you have any questions or feedback.
Build your IoT prototypes faster with JavaScript and no servers here. You can find some interesting and useful notebooks on GitHub or full experiments on EvaStore. Test your amazing ideas with Evasim. Ready? Build your own Evabot following the instructions on Eva official repository.
Would you like to contribute to the project(s)?
https://github.com/sanchezcarlosjr/evasim/
https://github.com/sanchezcarlosjr/javascript-notebook
Installation
Issues.
Code mobility
Naming
Database
Communication
Goals
Our main goal is to create an EVA platform to aid patients with dementia. To achieve this, we are currently focused on developing an EVA simulator using research from papers [5] and [6], as well as utilizing tools such as Blender, 3D printing, WebGL, and other relevant technologies. Because researchers strive for smart improvements, we bring out EvaSim [7], a simulator that enhances fast prototyping.
The development of the simulator will be followed by rigorous testing and evaluation of our system to ensure its efficacy in aiding dementia patients.
Methodology
The software development process of this project is user-centered and RUP-based, this involves understanding the human beings who use our systems and their needs before writing a single line of code.
We’re going to find user needs by understanding the discourse of the domain, observing the user behavior in the current workflow, and prototyping insofar as we have uncertainty or doubts.
So, our goals are to understand the language from the available literature focused on HCI, Ambient Intelligence, and Social Assistance; print and build a robot model to identify improvement opportunities using the 3D printer and Blender; document the available software; write down the system requirements; develop the Eva-SIM [7] [8] —which has been built in Brazil; and work with other interested groups.
We propose the following work plan, with activities starting on 01-18-2023 and ending on 07-30-2023.
Work plan
Theory
HCI
Computer simulations
Robots as therapy
We use robots to assist people, we call such robots Socially Assistive Robots (SARs). Eva is a low-cost hardware SAR that assists dementia and geriatric cognitive disorder patients with non-pharmacological interventions. Nowadays, Evabot has become an Eva platform referring to the open-source social robotics platform that builds around it —EvaSim, Evabot, Eva Language, Evaweb, and so on.
Some of the multimodal interaction Evabot capabilities are Text-To-Speech (e.g. you choose vendors such as Google Cloud or IBM Watson), facial expression recognition, controlling light sensory effects, Wizard and Oz approach, and connecting IoT devices.
Serious games
The imitation game
Game of Colors
The game of emotions
Eva Language
Eva Language is a visual scripting language (VPL) built using Google Bloocky whose purpose is that non-technical people generate new interactions in the Eva language.
The language should include what is shown such that Eva Sim can show custom interfaces.
function NumericComparison({ A, B, OP }) {
let value1 = parseInt(A);
let value2 = parseInt(B);
if (OP === 'EQ') {
return value1 == value2;
} else if (OP === 'NEQ') {
return value1 != value2;
} else if (OP === 'LT') {
return value1 < value2;
} else if (OP === 'LTE') {
return value1 <= value2;
} else if (OP === 'GT') {
return value1 > value2;
} else if (OP === 'GTE') {
return value1 >= value2;
}
}
const ProcessCounterNode = function ({ ops, count, value, first, second }) {
let temp = getCounter();
if (ops === 'ADD') {
temp[count] = getValue(first, temp) + getValue(second, temp);
} else if (ops === 'MINUS') {
temp[count] = getValue(first, temp) - getValue(second, temp);
} else if (ops === 'MULTIPLY') {
temp[count] = getValue(first, temp) * getValue(second, temp);
} else if (ops === 'DIVIDE') {
temp[count] = getValue(first, temp) / getValue;
} else if (ops === 'DIVIDE') {
temp[count] = getValue(first, temp) / getValue(second, temp);
} else if (ops === 'POWER') {
temp[count] = Math.pow(getValue(first, temp), getValue(second, temp));
} else if (ops === 'assign'){
temp[count] = value;
} else if (ops === 'random'){
temp[count] = generarNumeroRandom(parseInt(first), parseInt(second));
}
setCounter(temp);
}
Interaction consists of code, script, and assets.
name.json
script/name.json
assets/image1.pngAn example is “Pasalabra”
Pasapalabra.json
script/Rosco-Pasapalabra 20220404.json
sonidos/acierto.wavWe check out the Pasapalabra.json
https://codebeautify.org/jsonviewer/y2313d691
https://blockly-demo.appspot.com/static/demos/code/index.html
VPL Blocks
https://rhasspy.readthedocs.io/en/latest/
https://github.com/matrix-io/matrixio-models/blob/master/matrix-voice/matrix-voice.stl
Eva Sim
Due to the risk of damaging the assembled robot and a slow feedback loop, Marcelo Marques da Rocha proposed a simulator that interprets the Eva language and emulates the capabilities of Eva [7] [11]. This simulator is intended for use by developers, HRI researchers, and enthusiasts who need to safely and quickly test their real-world scenarios, enabling better training.
To summarize, the goal of the Eva Sim project, and therefore our goal, is to develop a simulator that allows Eva developers to prototype, learn, and iterate more quickly.
graph LR
JSON["JSON (Eva Language)"] --> EvaSimParser["Eva Sim x Parser"]
EvaSimParser["Eva Sim Parser"] --> XML
XML --> EvaSimModule["EvaSim Execution Module"]
EvaSimModule --> UserInterfaceFor example, on the simulator, you can run Serious games such as The Imitation Game (not the movie), Game of Colors, and Game of Emotions easily and with the same behavior as in the Evabot.
You can only visualize what is going on, but you cannot program on EvaSim.
TODO: Eva Sim user experience evaluation.
EvaML (EVA Markup Language)
EvaML is an XML vocabulary in the Eva language domain that is obtained from another system by JSON. EvaML is deprecated.
Manuals
https://github.com/midiacom/eva-robot
Mockups
Project vision
We would like to enhance EVA researchers in training, programming, and testing new interactions.
Observations
What is the current workflow for researchers to develop new interactions?
Without a server or broker, the current performance of the system is slow, leading to user frustration. To improve the user experience, faster prototyping is essential. Additionally, users interact more frequently with computer vision, computer voice, and gestures, making it necessary to optimize the system for these interactions.
Although the system could support multiple programming languages, we’ve chosen VPL due to its ease of use for developers.
Prototypes
v1
https://www.figma.com/file/w63waPDDeIInLPiUpxQtM9/EVA?node-id=0%3A1&t=4USALfnKimt3aYEr-1
v3
Construction. Team versions.
Ecosystem
WBS
@startwbs * Eva Platform ** User-Centered Design Process *** RUP *** Prototypes ** Teams *** CICESE *** Universidade Federal Fluminense (UFF) *** Universidad de Castilla-La Mancha (mamilab) *** USCD ** EvaNotebook *** No servers. Please don't confuse it with serverless. *** Angular *** EditorJS *** Capabilities **** Send emails **** Send SMS **** Call people **** Control EVA and environment **** Connect to ChatGPT **** Connect to APIs **** Web Workers **** Collaborative notebook **** WebAssembly **** Blockchain? **** Decentralized database **** Observable Plot *** Protocols **** MQTT **** WebSockets **** HTTP **** WebRTC ** EvaSim *** Unity *** 3D *** WebRTC ** EvaLanguage *** Visual Programming Language *** Google Blockly *** JSON Domain-Specific Language ** Evabot *** 3D impression **** Blender *** Software components **** New version ***** MQTT ***** Rhasspy **** MEAN stack. ***** MongoDB ***** Express ***** AngularJS ***** NodeJS ***** PM2 (i.e. Process manager) ***** Interactions by VPL **** Computer vision ***** Mediapipe **** Voice and natural language processing ***** IBM Watson ***** Dialogflow by Google ***** Telegram API ***** The Wizard of Oz *** Hardware components **** Amoled Touch Display **** Raspberry PI 4 **** ArbotiX-M **** Two servomotors **** Bluetooth speaker **** Matrix voice **** Webcam **** Smart Bulb @endwbs
CICESE
Capabilities
- EvaNotebook. Build your scenarios faster with no servers.
- EvaSim 3D. Prototype your scenarios on a platform that works right out of the box.
- EvaLanguage. Control everything with custom scenarios.
- EvaStore. Share your ideas and experiments.
- EvaBot. Build a Smart and Affective Assistance with no pain.
EvaNotebookFluminense Federal University (UFF)
[11]
Capabilities
Universidad de Castilla-La Mancha
[12]
UCSD
Requirements
We are migrating the current version of EvaSim to the Unity platform on the Web. This will eliminate the need for researchers to go through an installation process, allowing for immediate use. Faster feedback from patients is crucial for research purposes.
Researchers require the ability to visualize the 3D robot and execute interactions in their preferred programming language. Additionally, they need to load customized environments, including keys for services such as Dialogflow, IBM Watson, ChatGPT, and others. EvaSim must support all features available on Evabot.
Researchers aim to connect to EvaSim through automation tasks such as "The Game of Emotions" and execute interactions automatically. However, due to cost and performance considerations, external servers are not allowed, and therefore EvaSim must run entirely on the client side. To meet these constraints, we propose using a Terminal with WebRTC and WebSockets, which enables EvaSim to operate without user interaction and run entirely on the client side.
Continuous integration
https://game.ci/docs/github/activation
Testing and evaluations
Conclusions
References
[1] Dix, Alan J., Janet E. Finlay, et al. Human-Computer Interaction. 2nd ed. Prentice Hall, 1998. ISBN: 9780132398640.
[2] Olsen, Dan R. Developing User Interfaces (Interactive Technologies). Morgan Kaufmann, 1998. ISBN: 9781558604186.
[3] Cook, Albert M., and Jan Miller Polgar. Cook & Hussey’s Assistive Technologies: Principles and Practice. 3rd ed. Mosby Elsevier, 2007. ISBN: 9780323039079.
[4] Item 1007/3283 | Repositorio CICESE. (2020, August 06). Retrieved from https://cicese.repositorioinstitucional.mx/jspui/handle/1007/3283
[5] Raskin, Jef. The Humane Interface: New Directions for Designing Interactive Systems. Addison-Wesley Professional, 2000. ISBN: 9780201379372.
[6] D. Cruz-Sandoval and J. Favela, "A Conversational Robot to Conduct Therapeutic Interventions for Dementia," in IEEE Pervasive Computing, vol. 18, no. 2, pp. 10-19, 1 April-June 2019, doi: 10.1109/MPRV.2019.2907020.
[7] M. Marques da Rocha, D. Cruz-Sandoval, J. Favela and D. C. Muchaluat-Saade, "EvaSIM: a Software Simulator for the EVA Open-source Robotics Platform," 2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), Napoli, Italy, 2022, pp. 714-721, doi: 10.1109/RO-MAN53752.2022.9900561.
[8] EVA - An open-source social robotics platform. (2022, December 20). Retrieved from https://eva-social-robot.github.io
[9] google. "mediapipe." GitHub, 23 Jan. 2023, github.com/google/mediapipe.
[10] Technologies, Unity. "Unity - Manual: Building your WebGL application." 20 Jan. 2023, docs.unity3d.com/Manual/webgl-building.html.
[11] midiacom. "eva-robot." GitHub, 29 Jan. 2023, github.com/midiacom/eva-robot.
[12] Villa, L., Hervás, R., Dobrescu, C.C., Cruz-Sandoval, D., Favela, J. (2022). Incorporating Affective Proactive Behavior to a Social Companion Robot for Community Dwelling Older Adults. In: Stephanidis, C., Antona, M., Ntoa, S., Salvendy, G. (eds) HCI International 2022 – Late Breaking Posters. HCII 2022. Communications in Computer and Information Science, vol 1655. Springer, Cham. https://doi.org/10.1007/978-3-031-19682-9_72