0

I am attempting to extend the Crawler portion of the Enterprise PageCache module, however, I am getting the following error, I believe it has to do with the line $this->_init('vmr_crawler/crawler'); found in my model, I say this because if I switch out vmr_crawler, with enterprise_pagecache, it works... Here is the error

Fatal error: Call to a member function getUrlsPaths() on a non-object in /jlr/sites/mage/app/code/local/VMR/Crawler/Model/Crawler.php on line 37

Any ideas, snippets, comments are greatly appreciated! Thank you so much, I am posting all of my code below.

My Module Config

app/etc/modules/VMR_Crawler.xml

My Module Config Code

<?xml version="1.0"?>
<config>
<modules>
    <VMR_Crawler>
        <active>true</active>
        <codePool>local</codePool>
        <depends>
            <Mage_Core/>
        </depends>
    </VMR_Crawler>
</modules>

My Config

app/code/local/VMR/Crawler/etc/config.xml

My Config Code

<?xml version="1.0"?>
<config>
<modules>
    <VMR_Crawler>
        <version>0.0.1</version>
    </VMR_Crawler>
</modules>
<global>
    <models>
        <vmr_crawler>
            <class>VMR_Crawler_Model</class>
            <resourceModel>vmr_crawler_resource</resourceModel>
        </vmr_crawler>
        <vmr_crawler_resource>
            <class>VMR_Crawler_Model_Resource</class>
        </vmr_crawler_resource>
        <pagecache>
            <rewrite>
                <crawler>VMR_Crawler_Model_Crawler</crawler>
            </rewrite>
        </pagecache>
        <pagecache_resource_crawler>
            <rewrite>
                <attribute>VMR_Crawler_Model_Resource_Crawler</attribute>
            </rewrite>
        </pagecache_resource_crawler>
    </models>
</global>
</config> 

My Model

app/code/local/VMR/Crawler/Model/Crawler.php

My Model Code

<?php

class VMR_Crawler_Model_Crawler extends Enterprise_PageCache_Model_Crawler
{
protected function _construct()
{
    $this->_init('vmr_crawler/crawler');
}

public function crawl()
{
    if (!Mage::app()->useCache('full_page')) {
        return $this;
    }
    $storesInfo  = $this->getStoresInfo();
    $adapter     = new Varien_Http_Adapter_Curl();

    foreach ($storesInfo as $info) {
        $options = array(CURLOPT_USERAGENT => self::USER_AGENT);
        $storeId = $info['store_id'];
        $this->_visitedUrls = array();

        if (!Mage::app()->getStore($storeId)->getConfig(self::XML_PATH_CRAWLER_ENABLED)) {
            continue;
        }

        $threads = (int)Mage::app()->getStore($storeId)->getConfig(self::XML_PATH_CRAWLER_THREADS);
        if (!$threads) {
            $threads = 1;
        }
        if (!empty($info['cookie'])) {
            $options[CURLOPT_COOKIE] = $info['cookie'];
        }
        $urls       = array();
        $baseUrl    = $info['base_url'];
        $urlsCount  = $totalCount = 0;
        $urlsPaths  = $this->_getResource()->getUrlsPaths($storeId);
        foreach ($urlsPaths as $urlPath) {
            $url = $baseUrl . $urlPath;
            $urlHash = md5($url);
            if (isset($this->_visitedUrls[$urlHash])) {
                continue;
            }
            $urls[] = $url;
            $this->_visitedUrls[$urlHash] = true;
            $urlsCount++;
            $totalCount++;
            if ($urlsCount == $threads) {
                $adapter->multiRequest($urls, $options);
                $urlsCount = 0;
                $urls = array();
            }
        }
        if (!empty($urls)) {
            $adapter->multiRequest($urls, $options);
        }
    }
    return $this;
}
}

My Resource Model

app/code/local/VMR/Crawler/Model/Resource/Crawler.php

My Resource Model Code

<?php

class VMR_Crawler_Model_Resource_Crawler extends Enterprise_PageCache_Model_Resource_Crawler
{
    /**
     * Retrieve URLs paths that must be visited by crawler
     *
     * @param  $storeId
     * @return array
     */
    public function getUrlsPaths($storeId)
    {
        $bWorked = false;
        $collection = array();
        for($i = 0; $i < 3; $i++) 
        {
            echo "hi";
            try {
                $adapter = $this->_getReadAdapter();
                $select = $adapter->select()
                    ->from($this->getTable('core/url_rewrite'), array('request_path'))
                    ->where('store_id=?', $storeId)
                    ->where('is_system=1');
                $collection = $adapter->fetchCol($select);
                $bWorked = true;
            } 
            catch(Exception $ex)
            {

            }

            if($bWorked == true)
            {
                break;
            }
        }

        return $collection;
    }
}

My Resource Model Mysql4 Class

app/code/local/VMR/Crawler/Model/Mysql4/Crawler.php

My Resource Model Mysql4 Class Code

<?php

class VMR_Crawler_Model_Mysql4_Crawler extends Enterprise_PageCache_Model_Resource_Crawler
{

}

My Shell Class For Testing

shell/crawler.php

My Shell Class For Testing Code

<?php
/**
 * Full Page Cache Warming Shell Script
 *
 * @author Jeffrey L. Roberts
 **/

require_once 'abstract.php';

class Mage_Shell_Crawler extends Mage_Shell_Abstract
{
    /**
     * Run script
     *
     */
    public function run()
    {
        // echo "Starting Crawler";

        $crawler = new VMR_Crawler_Model_Crawler();
        $crawler->crawl();

        // echo "Crawler finished";
    }

    /**
     * Retrieve Usage Help Message
     *
     */
    public function usageHelp()
    {
        return <<<USAGE
Usage:  php -f crawler.php 

USAGE;
    }
}

$shell = new Mage_Shell_Crawler();
$shell->run(); 

To execute the shell

cd shell
php -f crawler.php

1 Answer 1

2

Shot in the dark here, as your post is a little dense and I'm not sure I've seen anyone extend both a model resource and override the _construct method in both, but the line 37 that PHP is complaining about looks to be

$urlsPaths  = $this->_getResource()->getUrlsPaths($storeId);

which is backed up by your error message

Fatal error: Call to a member function getUrlsPaths() on a non-object in /jlr/sites/mage/app/code/local/VMR/Crawler/Model/Crawler.php on line 37

So Magento is trying to instantiate your model's resource, but can't. So, if you're looking for a place to start, try tracing the code path of

$o = Mage::getModelResource('vmr_crawler/crawler');

and determine why its returning false.

A few guesses on that. First, you've included a

<deprecatedNode>vmr_crawler_mysql4</deprecatedNode>

but you don't have any vmr_crawler_mysql4 node in your XML file. Why is this here?

Secondly, your resource model for the crawler appears to be initializing with a core/url_rewrite?

protected function _construct()
{
    $this->_init('core/url_rewrite', 'url_rewrite_id');
}

I'm not sure what that's supposed to do, but it looks off (then again, I'm not exactly clear on what you're trying to do here, and I don't have any Enterprise versions available at the moment)

2
  • I am lost with words, I have no idea why it is returning false, I modified my local code to var_dump($o) after loading the resource model, it returns false, and I don't have a clue why, I have read over and double checked every line, I am not sure what else I can do, any ideas on how I can trouble shoot why it is returning false? Commented Mar 19, 2013 at 20:30
  • @jlroberts Trace the system code from getResourceModel. Commented Mar 19, 2013 at 20:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.