I recently noted that the table core_session is growing very fast ( +400MB ) I'm quite sure this is something abnormal.

  • I have truncated it but after a couple of days, It is again very big.
  • If I switch to file systems backend the problem persists
  • I found out that about every second I get a new session


The session is behaving correctly: I performed some test on our staging server ( protected by .htaccess ):

  • So far everything looks working fine: 1 access produce 1 session.

On live site online customer (Magento backend) shows about 500 customers ( I think each row represent open sessions):

eureka: Bots/Crawlers do not store cookies so everytime the access to the site they produce a new session


  • Is it normal those bots do 100/200 access/session each?
  • How can I limit the related session issue?

1 Answer 1


I'm not sure Bot needs sessions and the answer may be different for every bot for sure they don't store a cookie so it is no possible to re-use the old session matching the session ID stored in the client cookie.

How other E-Commerce fix this
Prestashop folks revolved the problem in this way:

  • Assigning to the bots the old sessions using the IP address to match the old session.

Unfortunately, Magento doesn't store the IP along with the session ID, I guess this is because Magento support more backend for session (files, DB, Redis)

How I fixed it
There is another question actually there is a similar question Magento generating aprox 20 session files per minute
I used the information in the old question to create a solution that integrates the Prestashop approach while it is still compatible with all Magento session backend

All the Magento session magic happens here: app/code/core/Mage/Core/Model/Session/Abstract/Varien.php
This is an abstract that is extended by all the other session models, so you cannot simply rewrite it but only copy it in /local/Mage/Core/Model/Session/Abstract/Varien.php.

So the below code do 3 things:

  1. It adds a more comprehensive list of bot
  2. Check if the client is a bot
  3. in that case, it calculates the session using the IP of the client and forces this id.

In short
Add the following methods to your copy of Mage_Core_Model_Session_Abstract_Varien

public function isBot()
        $isbot = false;
        $bot_regex = '/BotLink|bingbot|AhrefsBot|ahoy|AlkalineBOT|anthill|appie|arale|araneo|AraybOt|ariadne|arks|ATN_Worldwide|Atomz|bbot|Bjaaland|Ukonline|borg\-bot\/0\.9|boxseabot|bspider|calif|christcrawler|CMC\/0\.01|combine|confuzzledbot|CoolBot|cosmos|Internet Cruiser Robot|cusco|cyberspyder|cydralspider|desertrealm, desert realm|digger|DIIbot|grabber|downloadexpress|DragonBot|dwcp|ecollector|ebiness|elfinbot|esculapio|esther|fastcrawler|FDSE|FELIX IDE|ESI|fido|H�m�h�kki|KIT\-Fireball|fouineur|Freecrawl|gammaSpider|gazz|gcreep|golem|googlebot|griffon|Gromit|gulliver|gulper|hambot|havIndex|hotwired|htdig|iajabot|INGRID\/0\.1|Informant|InfoSpiders|inspectorwww|irobot|Iron33|JBot|jcrawler|Teoma|Jeeves|jobo|image\.kapsi\.net|KDD\-Explorer|ko_yappo_robot|label\-grabber|larbin|legs|Linkidator|linkwalker|Lockon|logo_gif_crawler|marvin|mattie|mediafox|MerzScope|NEC\-MeshExplorer|MindCrawler|udmsearch|moget|Motor|msnbot|muncher|muninn|MuscatFerret|MwdSearch|sharp\-info\-agent|WebMechanic|NetScoop|newscan\-online|ObjectsSearch|Occam|Orbsearch\/1\.0|packrat|pageboy|ParaSite|patric|pegasus|perlcrawler|phpdig|piltdownman|Pimptrain|pjspider|PlumtreeWebAccessor|PortalBSpider|psbot|Getterrobo\-Plus|Raven|RHCS|RixBot|roadrunner|Robbie|robi|RoboCrawl|robofox|Scooter|Search\-AU|searchprocess|Senrigan|Shagseeker|sift|SimBot|Site Valet|skymob|SLCrawler\/2\.0|slurp|ESI|snooper|solbot|speedy|spider_monkey|SpiderBot\/1\.0|spiderline|nil|suke|http:\/\/www\.sygol\.com|tach_bw|TechBOT|templeton|titin|topiclink|UdmSearch|urlck|Valkyrie libwww\-perl|verticrawl|Victoria|void\-bot|Voyager|VWbot_K|crawlpaper|wapspider|WebBandit\/1\.0|webcatcher|T\-H\-U\-N\-D\-E\-R\-S\-T\-O\-N\-E|WebMoose|webquest|webreaper|webs|webspider|WebWalker|wget|winona|whowhere|wlm|WOLP|WWWC|none|XGET|Nederland\.zoek|AISearchBot|woriobot|NetSeer|Nutch|YandexBot|YandexMobileBot|SemrushBot|FatBot|MJ12bot|DotBot|AddThis|baiduspider|m2e/i';
        $userAgent = empty($_SERVER['HTTP_USER_AGENT']) ? FALSE : $_SERVER['HTTP_USER_AGENT'];
        $isBot = !$userAgent || preg_match($bot_regex, $userAgent);

        return $isBot;

public function getBotSessionId()
    $s = (int)Mage::app()->getFrontController()->getRequest()->isSecure();
    $botIp = $_SERVER['REMOTE_ADDR'];
    return 'bot' . $s . sha1($botIp);

and change the getSessionId() in this way

public function setSessionId($id = null)
        if (!is_null($id) && preg_match('#^[0-9a-zA-Z,-]+$#', $id)) {
        } elseif ($this->isBot()) {
        return $this;

You may also want to change the behavior in case the bot access via HTTPS inside start() when you see session_regenerate_id(false);

                if (!$this->isBot()) {

The bot session Id will be a little longer, sha1 function:

returned value is a 40-character hexadecimal number

this should not be a problem since PHP allows session to be 128 character long

  • This is an ingenious solution, I did give the other one you linked to a try first but the downside is Google etc get 500 errors when crawling, this solution is the best of both worlds, thanks!
    – Alexander
    Jun 18, 2017 at 23:10
  • See alto this issue if you want to save your SSD magento.stackexchange.com/questions/63295/…
    – Fra
    Jan 19, 2018 at 3:43
  • 1
    I am struggling with the same. I was just wondering where to put the last piece of code if (!$this->isBot()) { session_regenerate_id(false); }
    – Marcel
    Dec 22, 2018 at 8:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.