How to deploy a robots.txt in WebSphere

Question & Answer

Question

This technote discusses one way to deploy a robots.txt in WebSphere.

Cause

robots.txt can be used to prevent a search engine web crawler from accessing certain directories or files.

robots.txt are typically deployed in front-end proxies or web servers (like IBM HTTP Server) in production environments.

However, you might occasionally want to deploy a robots.txt directly on WebSphere itself, like in a single server development environment.

Answer

Every "application" in WebSphere needs to be deployed as an EAR (enterprise application). You cannot just put a file in a certain directory (like in Apache) in order to deploy it.

You can create an EAR, which contains a WAR (web application) that is mapped to context root / (that is, context root "forward slash"). The WAR can contain the robots.txt file.

The EAR should be installed in the particular JVM, probably using virtual host default_host. After starting the EAR, you should be able to successfully access http://<hostname>:<port>/robots.txt.

Related Information

Manage Search

[{"Product":{"code":"SSHRKX","label":"WebSphere Portal"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Portal Search Engine","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"6.1","Edition":"","Line of Business":{"code":"LOB31","label":"WCE Watson Marketing and Commerce"}}]

Was this topic helpful?

Document Information

More support for:
WebSphere Portal

Software version:
6.1

Operating system(s):
AIX, HP-UX, Linux, Solaris, Windows

Document number:
406703

Modified date:
03 December 2021

UID

swg21430544

IBM Support

Tips