Support » Knowledge Base » Shop-Script » Useful tips »
To prevent the server from being overloaded by search engine crawlers (e.g., Google, Yahoo!, etc.) indexing your website, it is often advisable to use a robots.txt file containing special directives limiting the range of URLs which crawlers are allowed to index. Besides, this approach allows to disallow indexing of pages containing low-relevant information by search engines.
This method can also be used with WebAsyst Shop-Script. Since the URLs of storefront pages are mostly different depending on whether SEO-friendly URLs are enabled in WebAsyst Installer, below are offered recommendations both for online stores with SEO-friendly URLs enabled and those where this option is turned off.
Note: Use of a robots.txt file is only possible if WebAsyst scripts are installed on your own server or a web hosting server. Users of WebAsyst Shop-Script as a web service are not allowed to upload robots.txt to the server.
SEO-friendly URLs enabled | SEO-friendly URLs disabled | Comments |
| User-agent values | ||
User-agent: * | List of User-agent values of the search engine crawlers, further directives will be applied to. To apply the contents of the file to all crawlers, use the * character. | |
| To lower server load | ||
| Disallow: /pricelist/ | Disallow: /*ukey=pricelist | Price list page |
| Disallow: /search/ | Disallow: /*ukey=search* | Search results page |
| Disallow: /*sort=* | Disallow: /*sort=* | Products sorting results page |
| To prevent irrelevant information from being indexed | ||
| Disallow: /auth/ | Disallow: /*ukey=auth | Customer account login URL |
| Disallow: /cart/ | Disallow: /*ukey=cart | Shopping cart with products |
| Disallow: /checkout/ | Disallow: /*ukey=checkout | Order checkout page |
| Disallow: /feedback/ | Disallow: /*ukey=feedback | Feedback form page |
| Disallow: /installer/ | Disallow: /installer/ | WebAsyst Installer |
| Disallow: /login/ | Disallow: /login/ | WebAsyst user (store administrator)login page |
| Disallow: /logout/ | Disallow: /*ukey=logout | Customer logout link |
| Disallow: /myaccount/ | Disallow: /*ukey=office | Customer account page |
| Disallow: /order_status/ | Disallow: /*ukey=order_status | Order status view link |
| Disallow: /published/SC/html/scripts/ | Disallow: /published/SC/html/scripts/ | Various auxiliary scripts (e.g., RSSfeed) |
| Disallow: /published/wbsadmin/ | Disallow: /published/wbsadmin/ | WebAsyst Installer |
| Disallow: /register/ | Disallow: /*ukey=register | Customer registration page |
| Disallow: /remind_password/ | Disallow: /*ukey=remind_password | Customer password reminder link |
| Disallow: /*view=* | Disallow: /*view=* | Various auxiliary links (e.g., page printed version) |
| Disallow: /*/reviews/ | Disallow: /*ukey=discuss_product | Links to product discussion pages |
Login