Who doesn’t love the challenge of solving a puzzle? Jigsaw puzzles are a popular hobby for all ages and have been an important tool in a child’s physical, cognitive and emotional development. Children can develop physical skills such as better hand/eye coordination; cognitive skills such as shape recognition, memory, and problem solving; and emotional skills such as setting goals and patience by completing puzzles. But what if you are a visually impaired or a blind parent? How can you help your child to accomplish this important developmental task?

Puzzle Solving Toolkit

At IBM Research – Ireland our team built a 3D computer vision driven task completion prototype called the Puzzle Solving Toolkit which interacts with visually impaired users to guide them in solving a jigsaw puzzle in a natural and intuitive way. We achieved this by providing a combination of computer vision algorithms and Watson services.

Our toolkit recognizes the puzzle pieces via a camera input and computes the best strategy for solving the puzzle. The system interacts with the user by leveraging Watson Speech to Text and Text to Speech services to communicate the necessary instructions about which piece to pick up next and also adapt to user’s feedback.

Watch our video where we explain how our Puzzle Solving Toolkit prototype works. In the video we also share additional research projects being developed by our team on Indoor Positioning Systems and Visual Recognition Systems. 


Components of the toolkit

We utilize a head-mounted Intel 3D camera as video input and microphone/speaker to achieve natural human-computer interaction. The video input is fed to a deep learning pipeline, which identifies the puzzle pieces in a given scene and matches them to the original image of the puzzle. Upon completion of the pipeline, the core puzzle solving algorithm computes the solution of the puzzle and coordinates the instructions that need to be communicated to the user to successfully complete the puzzle.

The core puzzle solving algorithm and the deep learning pipeline are executed in a Nvidia TK1 board and leverage the computational speed of its GPUs. This demo unveils the capabilities we can provide in assisting visually impaired people to execute simple everyday tasks by bringing sophisticated AI algorithms together with advanced hardware.

Real-world applications

With our toolkit we could envision helping, for example, visually impaired parents to guide their children in completing collaborative learning tasks. Our research can bring real-time, context-specific cognitive analytics to the edge, in tandem with deep device specialization and sensory augmentation, to transform the human experience.

Performing an everyday task such as cooking, navigating an indoor/outdoor environment, or attending a meeting at work is quite simple for most people. However, approximately 285 million visually impaired and 39 million blind people face an enormous challenge in accomplishing these daily tasks. [1] With the rapid convergence of hardware equipped with powerful sensors and 3D computer vision algorithms, we are now able to repurpose IoT devices and create AI-powered assistive technologies that naturally interact, using cognitive Watson services, to provide blind and visually impaired people with more independence. The spectrum of possible applications is endless, and we’re looking to enable people to accomplish more complex and challenging everyday tasks at a rapid pace.

The goal of our research is to help people maximise their full potential, accomplish more, and have a better quality of life. We will continue to enhance our prototypes, combined with the cognitive capabilities of Watson, to bring visually impaired people closer to complete autonomy.

[1]                  World Health Organization, Fact Sheet No 282, Aug 2014

Was this article helpful?

More from Business transformation

Business process management (BPM) examples

7 min read - Business Process Management (BPM) is a systematic approach to managing and streamlining business processes. BPM is intended to help improve the efficiency of existing processes, with the goal of increasing productivity and overall business performance. BPM is often confused with other seemingly similar initiatives. For example, BPM is smaller in scale than business process reengineering (BPR), which radically overhauls or replaces processes. Conversely, it has a larger scope than task management, which deals with individual tasks, and project management, which…

Using generative AI to accelerate product innovation

3 min read - Generative artificial intelligence (GenAI) can be a powerful tool for driving product innovation, if used in the right ways. We’ll discuss select high-impact product use cases that demonstrate the potential of AI to revolutionize the way we develop, market and deliver products to customers. Stacking strong data management, predictive analytics and GenAI is foundational to taking your product organization to the next level.   1. Addressing customer inquiries with an AI-driven chatbot  ChatGPT distinguished itself as the first publicly accessible GenAI-powered…

Integrating AI into Asset Performance Management: It’s all about the data

3 min read - Imagine a future where artificial intelligence (AI) seamlessly collaborates with existing supply chain solutions, redefining how organizations manage their assets. If you’re currently using traditional AI, advanced analytics, and intelligent automation, aren’t you already getting deep insights into asset performance? Undoubtedly. But what if you could optimize even further? That’s the transformative promise of generative AI, which is beginning to revolutionize business operations in game-changing ways. It may be the solution that finally breaks through dysfunctional silos of business units,…

IBM Newsletters

Get our newsletters and topic updates that deliver the latest thought leadership and insights on emerging trends.
Subscribe now More newsletters