Re-ranking
Overview
Re-ranking is the process of taking an initial set of retrieved documents or search results and reordering them to improve their relevance to a given query. Cross-encoders are models specifically designed for this task, processing the query and each document together to produce highly accurate relevance scores.
Re-ranking with cross-encoders is particularly useful in various information retrieval applications:
- Search result optimization
- Question answering systems
- Content recommendation
- Document retrieval
- Semantic matching
- Information filtering
- Candidate selection in recruiting
- Product search refinement
Example
You can perform re-ranking in IBM watsonx.ai by using cross-encoder models through the API or SDKs. Make sure to use a suitable cross-encoding model.
1. Create an access token
Create an {access_token} by making a request to the IAM Identity Services API with your {api_key}.
— You can access your {api_key} within the Developer Access page.
1curl -X POST \ 2-H 'Content-Type: application/x-www-form-urlencoded' \ 3--data 'grant_type=urn:ibm:params:oauth:grant-type:apikey&apikey={api_key}' \ 4'https://iam.cloud.ibm.com/identity/token'
2. Send request to re-rank endpoint
Make a request to the /ml/v1/text/rerank endpoint using your {access_token} and {project_id}.
— You can access your {project_id} within the Developer Access page.
1curl --request POST \ 2"https://{cluster_url}/ml/v1/text/rerank?version=2023-10-25" \ 3-H 'Authorization: Bearer {access_token}' \ 4-H 'Accept: application/json' \ 5-d '{ 6 "model_id": "cross-encoder/ms-marco-minilm-l-12-v2", 7 "project_id": "{project_id}", 8 "inputs": [ 9 { 10 "text": "In my younger years, I often reveled in the excitement of spontaneous adventures and embraced the thrill of the unknown, whereas in my grownup life, I've come to appreciate the comforting stability of a well-established routine." 11 }, 12 { 13 "text": "As a young man, I frequently sought out exhilarating experiences, craving the adrenaline rush of life's novelties, while as a responsible adult, I've come to understand the profound value of accumulated wisdom and life experience." 14 } 15 ], 16 "query": "As a Youth, I craved excitement while in adulthood I followed Enthusiastic Pursuit.", 17 "parameters": { 18 "return_options": { 19 "top_n": 2 // Number of top results to return 20 } 21 } 22}'
Response
The API returns a JSON response with the following structure:
1{ 2 "model_id": "cross-encoder/ms-marco-minilm-l-12-v2", 3 "results": [ 4 { 5 "index": 1, 6 "score": 0.7461 7 }, 8 { 9 "index": 0, 10 "score": 0.8274 11 } 12 ], 13 "created_at": "2024-02-21T17:32:28Z", 14 "input_token_count": 20 15}