Use crewAI to build multimodal multi-agent systems for retail shelf optimization with watsonx.ai

Imagine transforming your retail store’s performance with just a smartphone and the power of artificial intelligence (AI). This isn’t a far-off future scenario; it’s a reality that small and mid-size retail stores can implement today! In this tutorial, we’ll explore an exciting real-world use case where store owners and managers can use AI to optimize their shelf arrangements, boost sales and enhance the customer experience. We will leverage both the latest IBM Granite model in this project as well as Meta's Llama 3.2 vision model.

With the advent of agent-based generative AI, small and mid-size retail stores now have access to expert-level analysis and recommendations that were once the domain of large corporations with dedicated data science teams. This democratization of AI technology can be groundbreaking for your local grocery store, boutique shop or regional chain.

Here’s what makes this approach so revolutionary:

• Simplicity: All you need to start is a simple picture of your store aisle.
• Expertise on demand: The AI agents act as your personal team of retail experts, analyzing your space and current market trends.
• Actionable insights: You’ll receive a detailed, practical plan for rearranging your shelves to maximize sales and customer satisfaction.
• Cost-effective: This approach eliminates the need for expensive consultants or complex software systems.
• Adaptability: As market trends change, you can quickly re-analyze and adjust your store layout to stay ahead of the curve.

Let’s dive into the technical details and see how this AI-powered retail optimization works, step by step. By the end of this tutorial, you’ll have a clear understanding of how to implement this system in your own store, potentially revolutionizing your retail space with the power of AI.

Sarah is a local grocery store owner who was struggling to compete with larger chains. Despite her best efforts, she noticed that certain products weren't selling as well as they should, while others were constantly out of stock. One day, while reorganizing her fresh produce aisle for the third time that month, she wondered if there was a better way.

That's where our AI-powered solution comes in. Using just her smartphone and our intelligent system, Sarah was able to transform her store's performance. Let's dive into how to build such a system.

We can use crewAI, an open-source agent framework that orchestrates agent interactions in crews. The term "crew" refers to multi-agent systems. Our crew is a team of agents role-playing retail experts that are available 24 x 7, each with their own specialty. Tasks can either be directly assigned to an agent or handled through crewAI’s hierarchical process that assesses specific roles and availability.

For crewAI beginners, check out the crewAI explainer along with the official docs. On the official crewAI GitHub repository, you can also find examples of crews performing stock analysis, data analysis, RAG, LangGraph integration and much more.

Let's have a look at the team of agentic retail experts we will use in this tutorial.

The task workflow is as follows.

You can find this project on Github.

We first need to set up our environment. You can find these steps in the Markdown file on GitHub or by following along here.

• Ensure you have Python >=3.10 <=3.13 installed on your system. You can check your Python version using the python3 --version command.
• Set up a virtual environment to avoid Python package dependency conflicts.

• Clone the repository using https://github.com/IBM/ibmdotcom-tutorials.git as the HTTPS URL. For detailed steps on how to clone a repository, refer to the GitHub documentation.

The project structure of thecrew-ai-projects directory should resemble the following:

src/my_retail_advisor/

├── config/
│ ├── agents.yaml # Agent configurations
│ └── tasks.yaml # Task definitions
├── tool/
│ ├── custom_tool.py # Custom crewAI tool implementations
│ └── tool_helper.py # Vision helper functions
├── crew.py # Crew orchestration
└── main.py # Application entry point

1.  Log in to watsonx.ai using your IBM Cloud account.
2. Create a watsonx.ai project. Take note of your project ID in project > Manage > General > Project ID. You'll need this ID for this tutorial.
3. Create a watsonx.ai Runtime service instance (choose the Lite plan, which is a free instance).
4. Generate a watsonx API Key.
5. Associate the watsonx.ai Runtime service to the project you created in watsonx.ai.

Generate and take note of your free Serper API key. Serper is a Google Search API that we will be using in this project.

We need to install the crewAI framework for this tutorial and set our watsonx.ai credentials that we generated in step 2.

To install crewAI, run the following command in your terminal.

In a seperate .env  file at the same directory level as the .env_sample  file, set your credentials as strings like so:

crewAI can be configured to use any open source large language model (LLM). LLMs can be connected through Ollama and several other APIs such as IBM watsonx™ and OpenAI. Users can also leverage pre-built tools available through the crewAI Toolkit as well as LangChain Tools.

Our custom visual search tool is powered by the llama-3-2-90b-vision-instruct model using watsonx.ai. Here is a look at the custom vision tool in the tool directory.

There are many ways you can customize your crew:

• Modify src/my_retail_advisor/config/agents.yaml to define your agents.
• Modify src/my_retail_advisor/config/tasks.yaml  to define your tasks.
• Modify src/my_retail_advisor/crew.py to add your own logic, tools and specific args.
• Modify src/my_retail_advisor/main.py to add custom inputs for your agents and tasks.
• Modify src/my_retail_advisor/tool/custom_tool.py to add custom tools for your agents and tasks.
• Modify src/my_retail_advisor/tool/tool_helper.py to change the custom vision tool based on the multimodal Llama model.
• Replace images/shelf.jpg with an image of your own.

Ensure you are in the proper working directory of this project. You can change directories by running the following command in your terminal.

To kickoff your crew of AI agents and begin task execution, run this command from the root folder of your project. Note, the crew may run for several minutes before returning a final result.

This command initializes themy-retail-advisor Crew, assembling the agents and assigning them tasks as defined in your configuration. This example, unmodified, will use Granite on watsonx.ai to create a report.md file with the output. crewAI can return JSON, Pydantic models and raw strings as output. Here is an example of the output produced by the crew.

Example output:

Action Plan for Rearranging and Improving the Vegetable Shelf

Objective:
To create a visually appealing and organized produce section that showcases the most popular vegetables, increases sales, and enhances customer satisfaction.

Recommendations from the Market Analyst:

1. Create a focal point with a colorful and eye-catching display of the most popular vegetables.
2. Use a variety of colors, textures, and heights to create visual interest.
3. Group similar vegetables together.
4. Consider using baskets, bins, or other containers to add a natural and earthy feel.
5. Use lighting to highlight specific products or promotions.
   ...

Action Plan for Store Manager and Store Buyer:

...

Step 1: Focal Point Display

• Store Manager: Designate a focal point area on the shelf for the most popular vegetables (tomatoes, potatoes, onions, lettuce, carrots, peppers, cucumbers, and celery).
• Store Buyer: Ensure a sufficient stock of these vegetables to maintain a full and appealing display.
• Team: Arrange the vegetables in a visually appealing way, using a variety of colors, textures, and heights.

...

Step 4: Themed Displays and Additional Features

• Store Manager: Plan and implement themed displays (e.g., summer BBQ or holiday-themed) to engage customers and promote related products.
• Store Buyer: Ensure a sufficient stock of themed display materials and related products.
• Team: Use vegetable racks or stands to add extra display space and create a visually appealing display.

As seen in the sample output, the multi-agent system is able to execute the sequential process of processing input, tool calling and formulating findings.

Remember Sarah's fresh produce aisle problem? Here's how the system helped her:

1. Visual analysis: Sarah took a photo of her vegetable aisle with her smartphone. The store manager agent analyzed the image and noticed that the shelves are densely packed and in need of maintenance.
2. Market research: The market analyst agent used the current market trends for vegetable sales and customer preferences to create actionable recommendations for rearranging the vegetables shelf.
3. Action plan: Based on these insights, Sarah received a detailed plan including instructions to:
   • Create a colorful and eye-catching display
   • Use baskets or bins to add a natural feel to the display.
   • Add a themed display, such as a summer BBQ or holiday-themed display, to engage customers in related products.
   • Create space between the different types of vegetables to declutter.
   • Improve the lighting.

In summary, themy-retail-advisor Crew is composed of multiple AI agents, each with unique roles, goals and tools. These agents collaborate on a series of tasks, defined in config/tasks.yaml , leveraging their collective skills to achieve complex objectives. The config/agents.yaml file outlines the capabilities and configurations of each agent in your crew.

With AI-powered tools like this, small and medium-sized retailers can streamline their decision-making process. Just like Sarah, you can transform your store's performance with technology that's accessible, affordable and effective. This architecture also unlocks other AI opportunities across various domains, such as product design and enhanced customer experience. Its flexibility makes it valuable beyond retail, empowering businesses to innovate and excel in industry-specific tasks.