Questions tagged [crawler]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
1 answer
24 views

How to set the meta robots NOINDEX tag to header of a specific category page?

I need to insert <meta name="robots" content="NOINDEX,NOFOLLOW"/> in a specific category page of our Magento site. Could you please share an example of how to do it? I don't ...
user avatar
1 vote
0 answers
70 views

Magento 2: How to crawl custom/dynamic URL's?

We need to crawl the custom/dynamic URLs. Is there any way? Suppose we have a URL like https://www.test.com/stores/awt and it's coming from a custom module and /awt is dynamic it changes as per ...
user avatar
2 votes
0 answers
127 views

Change the sitemap.xml size limit in magento 1.x.x

It seems that Magento 1.x.x. generated sitemap.xml is limited to 10MB. Is there any way to change the limit? I do know that I'll need to split the generated sitemap into smaller chunks after that to ...
user avatar
  • 21
0 votes
1 answer
398 views

What is crawler in Magento?

I do not know about crawler in magento.I search but do not know exactly what it do ? there are lots of crawler extension about cache crawler but I want to know about crawler.Anyone can understand ...
user avatar
5 votes
1 answer
3k views

Web Crawler creates a lot of sessions

I recently noted that the table core_session is growing very fast ( +400MB ) I'm quite sure this is something abnormal. I have truncated it but after a couple of days, It is again very big. If I ...
user avatar
  • 6,845