3

How to get SOLR working on magento ce 1.7.0.2

Why i need SOLR: I need to use SOLR to reduce my +10seconds loading time on searches. I know there is also sphinx, but i don't like that one.

I have followed instructions from here:

install SOLR: http://inchoo.net/ecommerce/install-apache-solr/

what to do next: magentocommerce.com/boards/viewthread/301508/ (i followed the 2nd post, but i cannot find the conf from step5. i assume i only need to use the schema.xml from the extension and paste it into the conf folder in the SOLR installation)

used this extension: code.google.com/p/magento-community-edition-solr/

Magento version: 1.7.0.2 SOLR version: 4.6.0 Running on VPS (CentOS 6.4) magento-community-edition-solr: 1.1.4

SOLR is running and i can access the Solr Dashboard. I have set the settings in magento system/configuration/Solr Search Engine to my settings like host/path and port.

Problem: I re-index the catalog search indexes and here starting the troubles. I can see activity on my server (mysql. i assume data is re-indexed into the magento database) but when indexes are re-indexed in the sql database i assume it needs to re-index data to/for Solr, right? The "in progress" won't stop and i see nothing been added to Solr collection. When i use the searchbox on my site i get a 400 error "400' Status: Bad Request" I also see some activity in putty when i search on my site, so i think the connection between magento and Solr are OK. If there is no connection i assume Solr cannot do stuff whn i trigger the magento search. No results found because the 400 error so i get the default magento search results.

What i think: I think the problem is in the conf folder (schema.xml). When i use the default schema.xml i get the error "undefined filed store_id". So i need to map the store_id, i know. So i use the schema.xml that i can find in the solr magento extension. When i upload that file i only get more errors and Solr won't work at all.

What i have done: Replaced and edited schema.xml over and over again. Searched google and found people with the same problem but no answers for this problem.

More questions: When i installed Solr i find out that there is a collection1 "core selector". I don't see anything about that in the install instruction of Solr or the Solr extension. Maybe because my Solr version is newer as used in the tutorials? BUt that collection1 i use to store the conf folder with the schema.xml file. In the tutorials people talk to paste conf folder into example folder, but my structure is example/collection1/conf. Am i wrong? How does magento knows what collection it need to use? I think here is something i need to edit (or not) or do i need to add the collection1 to the Solr path i use in the magento extension configuration?

Thanks

edit 1: I have installed (uploaded) solr version 3.4.0 and transfered the schema.xml from the magento extension to the conf folder from solr 3.4.0. I already see 1 problem. The standard schema.xml shows this <schema name="example" version="1.4"> and the schema from magento extension <schema name="example" version="1.5"> i ignored that and headed on. Solr errors and won't start. I removed the new schema and use the standard schema. Error when i start solr "store_id is not a fallid field or something". So i have put the filed code from magento schema.xml into the solr schema.xml. Yes! Search is FAST! But shows no results. hits=0 status=0 I need to tell that the reindex from the catalog search is still running/processing and won't finish and i don't see documents or data added to the solr data/index folder (if that is where documnets are stored).

Another problem is that solr now only runs when i keep the shell connection open. When i close putty, solr will stop. I think this can be solver by using a other command to start solr. Maybe this is a red flag for a other problem, so i mentioned this ;-)

edit 2: I have installed solr for the 3th time, now with this instructions: ctdev.co.za/magento-solr-search-setup-for-community-edition-solr-ce/#comment-86

WHen i reindex the search catalog by shell i get this after a couple minutes:

PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 7 bytes) in /home/admin/domains/test.com/public_html/lib/Zend/Db/Statement/Pdo.php on line 253

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 7 bytes) in /home/admin/domains/test.com/public_html/lib/Zend/Db/Statement/Pdo.php on line 253

Is this error related to the solr index problem? It still won't put data into solr :(

I have enlarged the memory and now i see this when re-indexing:

Jan 26, 2014 5:59:30 PM org.apache.solr.update.DirectUpdateHandler2 deleteAll
INFO: [] REMOVING ALL DOCUMENTS FROM INDEX
Jan 26, 2014 5:59:30 PM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
        commit{dir=/opt/solr/solr/data/index,segFN=segments_3,version=1390755510203,generation=3,filenames=[segments_3]
Jan 26, 2014 5:59:30 PM org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO: newest commit = 1390755510203
Jan 26, 2014 5:59:30 PM org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {deleteByQuery=*:*} 0 6
Jan 26, 2014 5:59:30 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/update params={wt=json} status=0 QTime=6
Jan 26, 2014 5:59:30 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start commit(optimize=false,waitFlush=false,waitSearcher=true,expungeDeletes=false)
Jan 26, 2014 5:59:31 PM org.apache.solr.core.SolrDeletionPolicy onCommit
INFO: SolrDeletionPolicy.onCommit: commits:num=2
        commit{dir=/opt/solr/solr/data/index,segFN=segments_3,version=1390755510203,generation=3,filenames=[segments_3]
        commit{dir=/opt/solr/solr/data/index,segFN=segments_4,version=1390755510204,generation=4,filenames=[segments_4]
Jan 26, 2014 5:59:31 PM org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO: newest commit = 1390755510204
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher <init>
INFO: Opening Searcher@3c76e596 main
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@3c76e596 main from Searcher@39eb305e main
        fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@3c76e596 main
        fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@3c76e596 main from Searcher@39eb305e main
        filterCache{lookups=2,hits=1,hitratio=0.50,inserts=1,evictions=0,size=1,warmupTime=0,cumulative_lookups=2,cumulative_hits=1,cumulative_hitratio=0.50,cumulative_inserts=1,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@3c76e596 main
        filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=2,cumulative_hits=1,cumulative_hitratio=0.50,cumulative_inserts=1,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@3c76e596 main from Searcher@39eb305e main
        queryResultCache{lookups=2,hits=0,hitratio=0.00,inserts=2,evictions=0,size=2,warmupTime=0,cumulative_lookups=2,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=2,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@3c76e596 main
        queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=2,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=2,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@3c76e596 main from Searcher@39eb305e main
        documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@3c76e596 main
        documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: end_commit_flush
Jan 26, 2014 5:59:31 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener sending requests to Searcher@3c76e596 main
Jan 26, 2014 5:59:31 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done.
Jan 26, 2014 5:59:31 PM org.apache.solr.core.SolrCore registerSearcher
INFO: [] Registered new searcher Searcher@3c76e596 main
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher close
INFO: Closing Searcher@39eb305e main
        fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        filterCache{lookups=2,hits=1,hitratio=0.50,inserts=1,evictions=0,size=1,warmupTime=0,cumulative_lookups=2,cumulative_hits=1,cumulative_hitratio=0.50,cumulative_inserts=1,cumulative_evictions=0}
        queryResultCache{lookups=2,hits=0,hitratio=0.00,inserts=2,evictions=0,size=2,warmupTime=0,cumulative_lookups=2,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=2,cumulative_evictions=0}
        documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {commit=} 0 230
Jan 26, 2014 5:59:31 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/update params={wt=json} status=0 QTime=230
Jan 26, 2014 5:59:31 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start commit(optimize=true,waitFlush=false,waitSearcher=true,expungeDeletes=false)
Jan 26, 2014 5:59:31 PM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
        commit{dir=/opt/solr/solr/data/index,segFN=segments_4,version=1390755510204,generation=4,filenames=[segments_4]
Jan 26, 2014 5:59:31 PM org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO: newest commit = 1390755510204
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher <init>
INFO: Opening Searcher@1c6b7657 main
Jan 26, 2014 5:59:31 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: end_commit_flush
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@1c6b7657 main from Searcher@3c76e596 main
        fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@1c6b7657 main
        fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@1c6b7657 main from Searcher@3c76e596 main
        filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=2,cumulative_hits=1,cumulative_hitratio=0.50,cumulative_inserts=1,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@1c6b7657 main
        filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=2,cumulative_hits=1,cumulative_hitratio=0.50,cumulative_inserts=1,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@1c6b7657 main from Searcher@3c76e596 main
        queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=2,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=2,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@1c6b7657 main
        queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=2,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=2,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@1c6b7657 main from Searcher@3c76e596 main
        documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@1c6b7657 main
        documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener sending requests to Searcher@1c6b7657 main
Jan 26, 2014 5:59:31 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done.
Jan 26, 2014 5:59:31 PM org.apache.solr.core.SolrCore registerSearcher
INFO: [] Registered new searcher Searcher@1c6b7657 main
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher close
INFO: Closing Searcher@3c76e596 main
        fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=2,cumulative_hits=1,cumulative_hitratio=0.50,cumulative_inserts=1,cumulative_evictions=0}
        queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=2,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=2,cumulative_evictions=0}
        documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {optimize=} 0 12
Jan 26, 2014 5:59:31 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/update params={wt=json} status=0 QTime=12

And now i get this error (i'm almost there ;-)):

PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 72243474 bytes) in /home/admin/domains/test.com/public_html/lib/Apache/Solr/HttpTransport/FileGetContents.php on line 146

Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 72243474 bytes) in /home/admin/domains/test.com/public_html/lib/Apache/Solr/HttpTransport/FileGetContents.php on line 146

Memory in php.ini is now 2GB. i think that is to much but i get better results. Here they come:

INFO: [] webapp=/solr path=/update params={wt=json} status=0 QTime=13
Jan 26, 2014 6:47:01 PM org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[17637854]} 0 29
Jan 26, 2014 6:47:01 PM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: ERROR: [doc=17637854] unknown field 'product_id'
        at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:340)
        at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:60)
        at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:115)
        at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:157)
        at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:79)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:58)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Jan 26, 2014 6:47:01 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/update params={wt=json} status=400 QTime=29
Catalog Search Index index was rebuilt successfully
[root@server01 shell]# Jan 26, 2014 6:49:51 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select params={fl=product_id,score&start=0&q=jeans&json.nl=map&wt=json&fq=store_id:1&rows=100} hits=0 status=0 QTime=2

I cannot find the data in the solr data index folder. Meybe i need to look somewhere else, but when i search on the shop there are no results :(

2 Answers 2

1

DONE! see last edits from my question. I have added the field product_id to the schema.xml file in the conf folder from solr. The stuff is indexed and i can use the solr search feature (great).

Search pages are now loaded within 4-5seconds before 10-15seconds. Time to optimize the reultpage :-)

Thanks for the support!

0
0

I think the problem is in schema.xml and solr version. Once I tried to use solr 4.x.x for magento enterprise and got many problems because of not compatible schema.xml. So if you read Support Matrix for Solr and Magento EE you will see that magento supports only 1.4.1 to 3.4.0 versions.

My advice, try to install Solr 3.4.0. Installation instructions you can find here.

4
  • i have installed 3.4.0. I now get this when i do a search in the shop Jan 23, 2014 8:46:03 PM org.apache.solr.core.SolrCore execute INFO: [] webapp=/solr path=/select params={fl=product_id,score&start=0&q=huis&json.nl=map&wt=json&fq=store_id:1&rows=100} status=400 QTime=4 reindexing the catalog search in magento still won't finish. It looks to work a bit better, but now i need to make the indexing work.
    – Maarten
    Jan 23, 2014 at 20:50
  • i hvae put back the old schema.xml and added the field store_id (that one shows error when i use standard xml file). Now the search is FAST as - But it shows ZERO products. all searches shows zero results, but they are fast :) Solr shows me hits=0 status=0 QTime=1. Does this mean it cannot find stuff in the index? That is right because there is nothing in the solr indexes. i hope i'm on the right way.
    – Maarten
    Jan 23, 2014 at 21:20
  • Solr is a pain to install, most extensions only work with 3.x, there are various stats but zero indicates it is using the internal search not Solr, which is the fallback for most extensions.
    – user2935
    Jan 24, 2014 at 0:33
  • i have installed solr for the 3th time now with other instructions: see edit2
    – Maarten
    Jan 26, 2014 at 17:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.