URL Rewrite Duplicates and Product URL

Question

My headache begins with a common URL Rewrite issue. I am running Magento 2.3 and rely heavily on an extension called Firebear Import / Export to handle my suppliers CSV feed imports.

Let's say in this example, I currently have no products and then run the import one by one until all 4 of my suppliers products are in the shop (note that a percentage of products will be duplicated as each supplier are in the same trade) - the import runs smoothly and Magento generates the URLs as it should.

Here lies where I am stuck, the URL paths for the products display as such:

https://www.terabytecomponents.co.uk/catalog/product/view/id/6316/s/samsung-200-s22e200b-led-display-54-6-cm-21-5-full-hd-black/category/10/

I need the URLs to be displayed showing the full category path, yes I have changed all the settings in admin such as

Use Categories Path for Product URLs = Yes

This doesn't make a difference even after a reindex and cache clean. Here's the catch, I purchased an extension that regenerates URLs and when I run it (takes about 20 minutes) the category path shows properly in the URL! Fantastic until i run a product import job, it now throws errors like this:

Some URL paths already exist in the url_rewrite table and not related to Product ID: 5013. Please remove them and execute this command again. 

('hp-elitedesk-800-g4-3-2-ghz-8th-gen-intelr-coretm-i7-i7-8700-black-silver-sff-pc-1', 'computing/desktop-computing/hp-elitedesk-800-g4-3-2-ghz-8th-gen-intelr-coretm-i7-i7-8700-black-silver-sff-pc-1');

The import completes but the products that display in these errors will not update, meaning prices and quantity will never change and that's not good.

I am desperate to get the category paths shown but do not use the 3rd party extension as it creates an issue in my product imports.

My questions are:

1. Why isn't Magento allowing the category path to be shown even though I'm telling it to?
2. How can i get it to work properly without using 3rd party extensions?

Answer 1

From much research and a lengthy conversation with a friendly developer at webpanda solutions, i purchased this module https://webpanda-solutions.com/url-rewrites-regenerate-and-customize.html

After installing i still had some problems, but after the settings were customised, I was able to start importing the products using the firebear module, no more errors and the category path is exactly how I wanted it.

This script was also provided to add to the import module:

    $this->collectionFactory - Magento\Catalog\Model\ResourceModel\Product\CollectionFactory
$this->urlPersist - Magento\UrlRewrite\Model\UrlPersistInterface
$this->productUrlRewriteGenerator - Magento\CatalogUrlRewrite\Model\ProductUrlRewriteGenerator

pIds - all product IDs that were processed

$collection = $this->collectionFactory->create();
$collection->addIdFilter($pIds);
$collection->addAttributeToSelect(['name', 'url_path', 'url_key', 'store_id']);

// this should be 1 if you intend to keep redirects to old urls, 0 if you don't want to keep them
$saveRewrites = 1;
foreach($collection->getItems() as $product) {
    $product->setStoreId(0);
    $product->setData('save_rewrites_history', $saveRewrites);

    if (!$saveRewrites) {
        $this->urlPersist->deleteByData([
            \Magento\UrlRewrite\Service\V1\Data\UrlRewrite::ENTITY_ID => $product->getId(),
            \Magento\UrlRewrite\Service\V1\Data\UrlRewrite::ENTITY_TYPE => \Magento\CatalogUrlRewrite\Model\ProductUrlRewriteGenerator::ENTITY_TYPE,
            \Magento\UrlRewrite\Service\V1\Data\UrlRewrite::REDIRECT_TYPE => 0,
            \Magento\UrlRewrite\Service\V1\Data\UrlRewrite::STORE_ID => 0
        ]);
    }
    try {
        $this->urlPersist->replace(
            $this->productUrlRewriteGenerator->generate($product)
        );
    } catch(\Exception $e) {
    }
}

Hope this helps others.

Answer 2

I run into a similar problem with the same error message. I think the problem of the message

Some URL paths already exist in the url_rewrite table and are not related to Product ID:

are duplicate products or identical paths for different products. If you add two products with the same name Magento try to create the same rewrite_urls (the error message occurs for creating the url_rewrite of the second product). I found now a way to fix this problem without an extension.

I also could solve the problem with the commercial extension from https://webpanda-solutions.com/url-rewrites-regenerate-and-customize.html. This extension introduces new a new url-layout that includes the SKU. Now the rewrite_url is different for the same products and a full reorganization at CLI fixes all URLs.