WordFinder app: Harnessing generative AI on AWS for aphasia communication

On this put up, we showcase how Dr. Kori Ramajoo, Dr. Sonia Brownsett, Prof. David Copland, from QARC, and Scott Harding, an individual residing with aphasia, used AWS providers to develop WordFinder, a cellular, cloud-based answer that helps people with aphasia enhance their independence via using AWS generative AI know-how.

Within the spirit of giving again to the group and harnessing the artwork of the doable for constructive change, AWS hosted the Hack For Objective occasion in 2023. This hackathon introduced collectively groups from AWS prospects throughout Queensland, Australia, to sort out urgent challenges confronted by social good organizations.

The College of Queensland’s Queensland Aphasia Analysis Centre (QARC)’s mission is to enhance entry to know-how for individuals residing with aphasia, a communication incapacity that may influence a person’s skill to specific and perceive spoken and written language.

The problem: Overcoming communication limitations

In 2023, it was estimated that greater than 140,000 individuals in Australia have been residing with aphasia. This quantity is predicted to develop to over 300,000 by 2050. Aphasia could make on a regular basis duties like on-line banking, utilizing social media, and attempting new units difficult. The purpose was to create a cellular app that might help individuals with aphasia by producing a glossary of the objects which are in a user-selected picture and lengthen the listing with associated phrases, enabling them to discover different communication strategies.

Overview of the answer

The next screenshot reveals an instance of navigating the WordFinder app, together with sign up, picture choice, object definition, and associated phrases.

Within the previous diagram, the next state of affairs unfolds: 

1. Register: The primary display reveals a easy sign-in web page the place customers enter their e-mail and password. It contains choices to create an account or get well a forgotten password.
2. Picture choice: After signing in, customers are prompted to Choose a picture to look. This display is initially clean.
3. Picture entry: The following display reveals a popup requesting non-public entry to the consumer’s photographs, with a grid of pattern pictures seen within the background.
4. Picture chosen: After a picture is chosen (on this case, an image of a koala), the app shows the picture together with some preliminary tags or classifications corresponding to Animal, Bear, Mammal, Wildlife, and Koala.
5. Associated phrases: The ultimate display reveals an inventory of associated phrases based mostly on the number of Associated Phrases subsequent to Koala from the earlier display. This step is essential for individuals with aphasia who typically have difficulties with word-finding and verbal expression. By exploring associated phrases (corresponding to habitat phrases like tree and eucalyptus, or descriptive phrases like fur and marsupial), customers can bridge communication gaps when the precise phrase they need isn’t instantly accessible. This semantic community method aligns with frequent aphasia remedy strategies, serving to customers discover alternative routes to specific their ideas when particular phrases are tough to recall.

This circulation demonstrates how customers can use the app to seek for phrases and ideas by beginning with a picture, then drilling down into associated terminology—a visible method to increasing vocabulary or discovering related phrases.

The next diagram illustrates the answer structure on AWS.

Within the following sections, we focus on the circulation and key elements of the answer in additional element.

1. Safe entry utilizing Route 53 and Amplify 
   1. The journey begins with the consumer accessing the WordFinder app via a website managed by Amazon Route 53, a extremely out there and scalable cloud DNS internet service. AWS Amplify hosts the React Native frontend, offering a seamless cross-environment expertise. 
2. Safe authentication with Amazon Cognito 
   1. Earlier than accessing the core options, the consumer should securely authenticate via Amazon Cognito. Cognito gives sturdy consumer identification administration and entry management, ensuring that solely authenticated customers can work together with the app’s providers and sources. 
3. Picture seize and storage with Amplify and Amazon S3 
   1. After being authenticated, the consumer can seize a picture of a scene, merchandise, or state of affairs they want to recall phrases from. AWS Amplify streamlines the method by robotically storing the captured picture in an Amazon Easy Storage Service (Amazon S3) bucket, a extremely out there, cost-effective, and scalable object storage service. 
4. Object recognition with Amazon Rekognition 
   1. As quickly because the picture is saved within the S3 bucket, Amazon Rekognition, a robust laptop imaginative and prescient and machine studying service, is triggered. Amazon Rekognition analyzes the picture, figuring out objects current and returning labels with confidence scores. These labels type the preliminary phrase immediate listing inside the WordFinder app, kickstarting the word-finding journey. 
5. Semantic phrase associations with API Gateway and Lambda 
   1. Whereas the preliminary glossary generated by Amazon Rekognition gives a strong place to begin, the consumer is likely to be searching for a extra particular or associated phrase. To handle this problem, the WordFinder app sends the preliminary glossary to an AWS Lambda perform via Amazon API Gateway, a completely managed service that securely handles API requests. 
6. Lambda with Amazon Bedrock, and generative AI and immediate engineering utilizing Amazon Bedrock
   1. The Lambda perform, performing as an middleman, crafts a fastidiously designed immediate and submits it to Amazon Bedrock, a completely managed service that provides entry to high-performing basis fashions (FMs) from main AI firms, together with Anthropic’s Claude mannequin.
   2. Amazon Bedrock generative AI capabilities, powered by Anthropic’s Claude mannequin, use superior language understanding and technology to supply semantically associated phrases and ideas based mostly on the preliminary glossary. This course of is pushed by immediate engineering, the place fastidiously crafted prompts information the generative AI mannequin to offer related and contextually applicable phrase associations.

WordFinder app part particulars

On this part, we take a better have a look at the elements of the WordFinder app.

React Native and Expo

WordFinder was constructed utilizing React Native, a preferred framework for constructing cross-environment cellular apps. To streamline the event course of, Expo was used, which permits for write-once, run-anywhere capabilities throughout Android and iOS working programs.

Amplify

Amplify performed an important position in accelerating the app’s improvement and provisioning the mandatory backend infrastructure. Amplify is a set of instruments and providers that allow builders to construct and deploy safe, scalable, and full stack apps. On this structure, the frontend of the phrase discovering app is hosted on Amplify. The answer makes use of a number of Amplify elements:

• Authentication and entry management: Amazon Cognito is used for consumer authentication, enabling customers to enroll and sign up to the app. Amazon Cognito gives consumer identification administration and entry management with entry to an Amazon S3 bucket and an API gateway requiring authenticated consumer classes.
• Storage: Amplify was used to create and deploy an S3 bucket for storage. A key part of this app is the power for a consumer to take an image of a scene, merchandise, or state of affairs that they’re searching for to recall phrases from. The answer must quickly retailer this picture for processing and evaluation. When a consumer uploads a picture, it’s saved in an S3 bucket for processing with Amazon Rekognition. Amazon S3 gives extremely out there, cost-effective, and scalable object storage.
• Picture recognition: Amazon Rekognition makes use of laptop imaginative and prescient and machine studying to determine objects current within the picture and return labels with confidence scores. These labels are used because the preliminary phrase immediate listing inside the WordFinder app.

Associated phrases

The generated preliminary glossary is step one towards discovering the specified phrase, however the labels returned by Amazon Rekognition may not be the precise phrase that somebody is in search of. The venture crew then thought-about how one can implement a thesaurus-style lookup functionality. Though the venture crew initially explored totally different programming libraries, they discovered this method to be considerably inflexible and restricted, typically returning solely synonyms and never entities which are associated to the supply phrase. The libraries additionally added overhead related to packaging and sustaining the library and dataset shifting ahead.

To handle these challenges and enhance responses for associated entities, the venture crew turned to the capabilities of generative AI. By utilizing the generative AI basis fashions (FMs), the venture crew was in a position to offload the continuing overhead of managing this answer whereas growing the pliability and curation of associated phrases and entities which are returned to customers. The venture crew built-in this functionality utilizing the next providers:

• Amazon Bedrock: Amazon Bedrock is a completely managed service that provides a selection of high-performing FMs from main AI firms like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon via a single API, together with a broad set of capabilities to construct generative AI apps with safety, privateness, and accountable AI. The venture crew was in a position to rapidly combine with, check, and consider totally different FMs, lastly settling upon Anthropic’s Claude mannequin.
• API Gateway: The venture crew prolonged the Amplify venture and deployed API Gateway to simply accept safe, encrypted, and authenticated requests from the WordFinder cellular app and cross them to a Lambda perform dealing with Amazon Bedrock entry. 
• Lambda: A Lambda perform was deployed behind the API gateway to deal with incoming internet requests from the cellular app. This perform was accountable for taking the provided enter, constructing the immediate, and submitting it to Amazon Bedrock. This meant that integration and immediate logic may very well be encapsulated in a single Lambda perform.

Advantages of API Gateway and Lambda

The venture crew briefly thought-about utilizing the AWS SDK for JavaScript v3 and credentials sourced from Amazon Cognito to instantly interface with Amazon Bedrock. Though this might work, there have been a number of advantages related to implementing API Gateway and a Lambda perform:

• Safety: To allow the cellular shopper to combine instantly with Amazon Bedrock, authenticated customers and their related AWS Identification and Entry Administration (IAM) position would must be granted permissions to invoke the FMs in Amazon Bedrock. This may very well be achieved utilizing Amazon Cognito and short-term permissions granted via roles. Consideration was given to the potential of uncontrolled entry to those fashions if the cellular app was compromised. By shifting the IAM permissions and invocation dealing with to a central perform, the crew was in a position to enhance visibility and management over how and when the FMs have been invoked.
• Change administration: Over time, the underlying FM or immediate may want to alter. If both was laborious coded into the cellular app, any change would require a brand new launch and each consumer must obtain the brand new app model. By finding this inside the Lambda perform, the specifics round mannequin utilization and immediate creation are decoupled and might be tailored with out impacting customers. 
• Monitoring: By routing requests via API Gateway and Lambda, the crew can log and monitor metrics related to utilization. This allows higher decision-making and reporting on how the app is performing. 
• Knowledge optimization: By implementing the REST API and encapsulating the immediate and integration logic inside the Lambda perform, the crew to can ship the supply phrase from the cellular app to the API. This implies much less information is shipped over the mobile community to the backend providers. 
• Caching layer: Though a caching layer wasn’t applied inside the system throughout the hackathon, the crew thought-about the power to implement a caching mechanism for supply and associated phrases that over time would scale back requests that must be routed to Amazon Bedrock. This may be readily queried within the Lambda perform as a preliminary step earlier than submitting a immediate to an FM.

Immediate engineering

One of many core options of WordFinder is its skill to generate associated phrases and ideas based mostly on a user-provided supply phrase. This supply phrase (obtained from the cellular app via an API request) is embedded into the next immediate by the Lambda perform, changing {phrase}:

immediate = "I've Aphasia. Give me the highest 10 commonest phrases which are associated phrases to the phrase provided within the immediate context. Your response must be a sound JSON array of simply the phrases. No surrounding context. {phrase}"

The crew examined a number of totally different prompts and approaches throughout the hackathon, however this primary guiding immediate was discovered to present dependable, correct, and repeatable outcomes, whatever the phrase provided by the consumer.

After the mannequin responds, the Lambda perform bundles the associated phrases and returns them to the cellular app. Upon receipt of this information, the WordFinder app updates and shows the brand new listing of phrases for the consumer who has aphasia. The consumer may then discover their phrase, or drill deeper into different associated phrases.

To keep up environment friendly useful resource utilization and value optimization, the structure incorporates a number of useful resource cleanup mechanisms:

• Lambda automated scaling: The Lambda perform accountable for interacting with Amazon Bedrock is configured to robotically scale all the way down to zero cases when not in use, minimizing idle useful resource consumption.
• Amazon S3 lifecycle insurance policies: The S3 bucket storing the user-uploaded pictures is configured with lifecycle insurance policies to robotically expire and delete objects after a specified retention interval, releasing up cupboard space. 
• API Gateway throttling and caching: API Gateway is configured with throttling limits to assist forestall extreme requests, and caching mechanisms are applied to cut back the load on downstream providers corresponding to Lambda and Amazon Bedrock.

Conclusion

The QARC crew and Scott Harding labored carefully with AWS to develop WordFinder, a cellular app that addresses communication challenges confronted by people residing with aphasia. Their profitable entry on the 2023 AWS Queensland Hackathon showcased the ability of involving these with lived experiences within the improvement course of. Harding’s insights helped the tech crew perceive the nuances and influence of aphasia, resulting in an answer that empowers customers to search out their phrases and keep related.

References

In regards to the Authors

Kori Ramijoo is a analysis speech pathologist at QARC. She has in depth expertise in aphasia rehabilitation, know-how, and neuroscience. Kori leads the Aphasia Tech Hub at QARC, enabling individuals with aphasia to entry know-how. She gives consultations to clinicians and gives recommendation and help to assist individuals with aphasia achieve and preserve independence. Kori can also be researching design issues for know-how improvement and use by individuals with aphasia.

Scott Harding lives with aphasia after a stroke. He has a background in Engineering and Laptop Science. Scott is likely one of the Administrators of the Australian Aphasia Affiliation and is a client consultant and advisor on numerous state authorities well being committees and nationally funded analysis tasks. He has pursuits in using AI in creating predictive fashions of aphasia restoration.

Sonia Brownsett is a speech pathologist with in depth expertise in neuroscience and know-how. She has been a postdoctoral researcher at QARC and led the aphasia tech hub in addition to a analysis program on the mind mechanisms underpinning aphasia restoration after stroke and in different populations together with adults with mind tumours and epilepsy.

David Copland is a speech pathologist and Director of QARC. He has labored for over 20 years within the area of aphasia rehabilitation. His work seeks to develop new methods to grasp, assess and deal with aphasia together with using mind imaging and know-how. He has led the creation of complete aphasia remedy applications which are being applied into well being providers.

Mark Promnitz is a Senior Options Architect at Amazon Net Companies, based mostly in Australia. Along with serving to his enterprise prospects leverage the capabilities of AWS, he can typically be discovered speaking about Software program as a Service (SaaS), information and cloud-native architectures on AWS.

Kurt Sterzl is a Senior Options Architect at Amazon Net Companies, based mostly in Australia.  He enjoys working with public sector prospects like UQ QARC to help their analysis breakthroughs.