Questions tagged [crawler]

The tag has no usage guidance.

0
votes
0answers
6 views

file get content from iframe [migrated]

I would like to get content of a iframe with id or class of that iframe. $links = "https://www.amazon.co.uk/Book-Secret-Wisdom-Prophetic-Evolution/dp/599054314X/ref=pd_sbs_14_2/257-3608675-7951114?...
2
votes
0answers
95 views

Change the sitemap.xml size limit in magento 1.x.x

It seems that Magento 1.x.x. generated sitemap.xml is limited to 10MB. Is there any way to change the limit? I do know that I'll need to split the generated sitemap into smaller chunks after that to ...
0
votes
1answer
258 views

What is crawler in Magento?

I do not know about crawler in magento.I search but do not know exactly what it do ? there are lots of crawler extension about cache crawler but I want to know about crawler.Anyone can understand ...
5
votes
2answers
2k views

Web Crawler creates a lot of sessions

I recently noted that the table core_session is growing very fast ( +400MB ) I'm quite sure this is something abnormal. I have truncated it but after a couple of days, It is again very big. If I ...