Dieses System wurde erstellt, um ein Tofex Modul zu präsentieren. Eine Anleitung, wie das System verwendet werden kann, ist unter diesem Link zu finden.
Die Dokumentation des Moduls ist unter diesem Link zu finden.

Dies ist eine Demo-Installation. Jegliche Bestellung in diesem Shop wird weder beachtet noch ausgeführt.

Tofex Import

Introduction

This extension provides an advanced import of EAV entities with several layers of caching to avoid unnecessary imports.

Concept

The import was development for two reasons. First to avoid any unnecessary imports and second to avoid a big overhead when importing a small amount of data like a single product. The first reason is imported when you have a huge amount of entities which do not change a lot. This usually happens when an ERP or a PIM exports the whole catalog each day but only a few products have changed data. If you import them all to Magento and then have to run a full index of all data the imports keeps the system busy for hours. Even with the trigger system Magento developed later on, way too much data gets processed unnecessarily.

Caching

To achieve the goals, the import has three cache-phases:

1) Source Cache

The source cache compares the data for each entity you provide to import which may not be in the format of a Magento ready data structure, i.e. the content of a file. In case the same file content gets provided again no further action is required.

2) Transformed Cached

The transformed cache compares the data for each entity after transformation into a Magento ready data structure. This helps in case the formally mentioned file contains information which change the file content but are not relevant to the Magento entity, i.e. current date which is not imported into Magento.

3) Database Comparison

The last step is to compare the data to import with the current data in the database. For this the data of a big chunk of entities is loaded before imported and only the changed data then gets imported. If only a single attribute has changes only this value will be imported. The same loading mechanism is used in the Tofex Product Feed extension to create the data for external use.

These three phases help to allow even imports of 100K products every night in a couple of minutes without any load problems because the system has heavy usage by an import.

Indexing

Magento has introduced their concept of partial indexing in later versions of Magento 1 and continued in Magento 2. It uses database triggers to track changes to entity data. Problem is that these trigger do not have much logic in them because the database does not know what a change in a text value means for the system. So a lot of indexing still occurs which is not really necessary. I.e. if a product has changed value in an attribut which is not relevant to category pages or the search there is no need to run indexing for the product at all. Because the database recognises a change and does not know if this text value is relevant to the search all indexes which could be necessary in case a text value changed will be run.

This extension uses the Tofex Partial Indexing extension to only run the absolute necessary amount of indexing and helps to quickly finish all imports.

Implementation

This is an example of a product import but all EAV entities follow the same pattern. These methods have to be implemented:

  • readSourceData: Method to prepare the source data, i.e. from a file, an external source, etc.
  • validateSourceData: Method to validate the source data and mark single elements as invalid.
  • getSourceElementHashKey: What identifies the source data (e.g. the SKU and the store id) for comparison with later imports.
  • transformData: Method to prepare the data for import in Magento with the Tofex importer.
  • getTransformedElementHashKey: What identifies the transformed data (e.g. the SKU and the store id) for comparison with later imports.
  • displaySourceInvalidElement: How to handle invalid source elements.
  • displayTransformedInvalidElement: How to handle invalid transformed elements.
  • cleanup: In case some action is need after the import.
<?php

class Tofex_ImportProductTest_Model_Import
    extends Tofex_Import_Model_Product
{
    /**
     * @return array
     * @throws Exception
     */
    protected function readSourceData()
    {
        ...
    }

    /**
     * @param array $sourceData
     *
     * @return void
     */
    protected function validateSourceData(array $sourceData)
    {
        ...
    }

    /**
     * @param array $element
     *
     * @return string
     */
    protected function getSourceElementHashKey(array $element)
    {
        ...
    }

    /**
     * @param array $sourceData
     *
     * @return array
     * @throws Mage_Core_Exception
     */
    protected function transformData(array $sourceData)
    {
        ...
    }

    /**
     * @param mixed $element
     *
     * @return string
     */
    protected function getTransformedElementHashKey(array $element)
    {
        ...
    }

    /**
     * @param int    $sourceElementNumber
     * @param mixed  $sourceElement
     * @param string $reason
     * @param string $severity
     *
     * @return void
     */
    protected function displaySourceInvalidElement(
        $sourceElementNumber,
        $sourceElement,
        $reason,
        $severity = 'error')
    {
        ...
    }

    /**
     * @param int    $transformedElementNumber
     * @param mixed  $transformedElement
     * @param string $reason
     * @param string $severity
     *
     * @return void
     */
    protected function displayTransformedInvalidElement(
        $transformedElementNumber,
        $transformedElement,
        $reason,
        $severity = 'error')
    {
        ...
    }

    /**
     * @return void
     * @throws Exception
     */
    protected function cleanup()
    {
        ...
    }
}

Result

This is the console output of an example product import with new products.

2020-05-22T12:35:09+00:00 INFO (6): Tofex_Job:run[91] Running job: import_product_test
2020-05-22T12:35:09+00:00 INFO (6): Tofex_Job_Model_Job:launch[177] Importing file 1/1: /var/www/magento/htdocs/app/code/community/Tofex/ImportProductTest/data/import/product/test/data.xml
2020-05-22T12:35:09+00:00 INFO (6): Tofex_Import_Model_Entity:runImport[114] Read 6 source element(s)
2020-05-22T12:35:09+00:00 INFO (6): Tofex_Import_Model_Import:runImport[153] Found 0 cached source element(s)
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[189] Found 0 invalid source element(s)
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Entity:runImport[114] Transformed 6 element(s)
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Product_Associated_Category:validate[101] Creating category with name: Test Category with path: 1/2/5
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Product_Associated_Category:validate[101] Created category with id: 41
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[203] Found 0 cached transformed element(s)
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[239] Found 0 invalid transformed element(s)
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Found 1 store(s) in elements
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Split data for store with id: 1 in 1 chunk(s)
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Importing chunk: 1/1 containing 6 element(s)
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Creating entity with key: 10000001
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Creating entity with key: 10000002
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Creating entity with key: 10000002-1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Creating entity with key: 10000002-2
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Creating entity with key: 10000003
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Creating entity with key: 10000004
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Entity:importTransformedData[808] Created entity with key: 10000001 has id: 906
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Entity:importTransformedData[808] Created entity with key: 10000002 has id: 907
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Entity:importTransformedData[808] Created entity with key: 10000002-1 has id: 908
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Entity:importTransformedData[808] Created entity with key: 10000002-2 has id: 909
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Entity:importTransformedData[808] Created entity with key: 10000003 has id: 910
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Entity:importTransformedData[808] Created entity with key: 10000004 has id: 911
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Entity:importTransformedData[839] Product with id: 906 has changed values in attributes: type_id, sku, name, description, short_description, price, apparel_type, status, visibility, image, small_image, thumbnail, gallery, attribute_set_id, tax_class_id in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Entity:importTransformedData[839] Product with id: 907 has changed values in attributes: type_id, sku, name, options_container, description, short_description, price, apparel_type, status, visibility, image, small_image, thumbnail, gallery, attribute_set_id, tax_class_id in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Entity:importTransformedData[839] Product with id: 908 has changed values in attributes: type_id, sku, name, description, short_description, price, apparel_type, status, visibility, color, attribute_set_id, tax_class_id in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Entity:importTransformedData[839] Product with id: 909 has changed values in attributes: type_id, sku, name, description, short_description, price, apparel_type, status, visibility, color, attribute_set_id, tax_class_id in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Entity:importTransformedData[839] Product with id: 910 has changed values in attributes: type_id, sku, name, description, short_description, price, apparel_type, status, visibility, image, small_image, thumbnail, gallery, attribute_set_id, tax_class_id in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Entity:importTransformedData[839] Product with id: 911 has changed values in attributes: type_id, sku, name, description, short_description, price, special_price, special_from_date, special_to_date, apparel_type, status, visibility, image, small_image, thumbnail, gallery, attribute_set_id, tax_class_id in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Associated item of product with id: 906 with name: stock has changes in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Associated item of product with id: 906 with name: category has changes in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Associated item of product with id: 906 with name: website has changes in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Associated item of product with id: 907 with name: stock has changes in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Associated item of product with id: 907 with name: category has changes in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Associated item of product with id: 907 with name: super_attribute has changes in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Associated item of product with id: 907 with name: website has changes in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Associated item of product with id: 908 with name: stock has changes in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Associated item of product with id: 908 with name: super_link has changes in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Associated item of product with id: 908 with name: website has changes in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Associated item of product with id: 909 with name: stock has changes in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Associated item of product with id: 909 with name: super_link has changes in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Associated item of product with id: 909 with name: website has changes in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Associated item of product with id: 910 with name: stock has changes in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Associated item of product with id: 910 with name: category has changes in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Associated item of product with id: 910 with name: group_price has changes in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Associated item of product with id: 910 with name: website has changes in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Associated item of product with id: 911 with name: stock has changes in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Associated item of product with id: 911 with name: category has changes in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Associated item of product with id: 911 with name: website has changes in store with id: 1
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Entity:runImport[114] Imported 6 changed products
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Entity:runImport[114] Ignored 0 unchanged products
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[252] Saving 6 source data hash(es)
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Import_Model_Import:runImport[253] Saving 6 transformed data hash(es)
2020-05-22T12:35:11+00:00 INFO (6): Tofex_IndexPartial_Model_Abstract:execute[179] Stock status index was updated for 6 article(s) in website with id: 1
2020-05-22T12:35:11+00:00 INFO (6): Tofex_IndexPartial_Model_Abstract:execute[179] Attribute index was updated for 6 article(s)
2020-05-22T12:35:11+00:00 INFO (6): Tofex_IndexPartial_Model_Abstract:execute[179] Category product index was updated for 6 article(s)
2020-05-22T12:35:12+00:00 INFO (6): Tofex_IndexPartial_Model_Abstract:execute[179] Catalog rules were updated for 6 article(s)
2020-05-22T12:35:12+00:00 INFO (6): Tofex_IndexPartial_Model_Abstract:execute[179] Price index was updated for 6 article(s) in website with id: 1
2020-05-22T12:35:12+00:00 INFO (6): Tofex_IndexPartial_Model_Abstract:execute[179] Category url rewrite index was updated for 1 categories(s) in store with id: 1
2020-05-22T12:35:12+00:00 INFO (6): Tofex_IndexPartial_Model_Abstract:execute[179] Category url rewrite index was updated for 6 article(s) in store with id: 1
2020-05-22T12:35:12+00:00 INFO (6): Tofex_IndexPartial_Model_Abstract:execute[179] Category url rewrite index was updated for 6 article(s) in store with id: 2
2020-05-22T12:35:12+00:00 INFO (6): Tofex_IndexPartial_Model_Abstract:execute[179] Category url rewrite index was updated for 6 article(s) in store with id: 3
2020-05-22T12:35:13+00:00 INFO (6): Tofex_IndexPartial_Model_Abstract:execute[179] Product flat index was updated for 6 article(s) in store with id: 1
2020-05-22T12:35:13+00:00 INFO (6): Tofex_IndexPartial_Model_Abstract:execute[179] Product flat index was updated for 6 article(s) in store with id: 2
2020-05-22T12:35:13+00:00 INFO (6): Tofex_IndexPartial_Model_Abstract:execute[179] Product flat index was updated for 6 article(s) in store with id: 3
2020-05-22T12:35:13+00:00 INFO (6): Tofex_IndexPartial_Model_Abstract:execute[179] Category flat index was updated for 1 category(s)
2020-05-22T12:35:13+00:00 INFO (6): Tofex_IndexPartial_Model_Abstract:execute[179] Catalog search index was updated for 6 article(s) in store with id: 1
2020-05-22T12:35:13+00:00 INFO (6): Tofex_IndexPartial_Model_Abstract:execute[179] Catalog search index was updated for 6 article(s) in store with id: 2
2020-05-22T12:35:13+00:00 INFO (6): Tofex_IndexPartial_Model_Abstract:execute[179] Catalog search index was updated for 6 article(s) in store with id: 3
2020-05-22T12:35:10+00:00 INFO (6): Tofex_Job:run[91] Finished job: import_product_test
2020-05-22T12:35:13+00:00 INFO (6): Tofex_Job:run[91] Duration: 0 minute(s), 4 second(s)
2020-05-22T12:35:13+00:00 INFO (6): Tofex_Job:run[91] Max memory usage: 22 MB

Source Cache

In case the source cache matches the data to import, no import is performed.

2020-05-22T12:36:27+00:00 INFO (6): Tofex_Job:run[91] Running job: import_product_test
2020-05-22T12:36:27+00:00 INFO (6): Tofex_Job_Model_Job:launch[177] Importing file 1/1: /var/www/magento/htdocs/app/code/community/Tofex/ImportProductTest/data/import/product/test/data.xml
2020-05-22T12:36:28+00:00 INFO (6): Tofex_Import_Model_Entity:runImport[114] Read 6 source element(s)
2020-05-22T12:36:28+00:00 INFO (6): Tofex_Import_Model_Import:runImport[153] Found 6 cached source element(s)
2020-05-22T12:36:28+00:00 INFO (6): Tofex_Import_Model_Entity:runImport[114] No products to transform
2020-05-22T12:36:28+00:00 INFO (6): Tofex_Job:run[91] Finished job: import_product_test
2020-05-22T12:36:28+00:00 INFO (6): Tofex_Job:run[91] Duration: 0 minute(s), 1 second(s)
2020-05-22T12:36:28+00:00 INFO (6): Tofex_Job:run[91] Max memory usage: 10 MB

Transformed Cache

In case the source cache does not match, but the transformed cache matches the data to import, no import is performed.

2020-05-22T12:37:50+00:00 INFO (6): Tofex_Job:run[91] Running job: import_product_test
2020-05-22T12:37:50+00:00 INFO (6): Tofex_Job_Model_Job:launch[177] Importing file 1/1: /var/www/magento/htdocs/app/code/community/Tofex/ImportProductTest/data/import/product/test/data.xml
2020-05-22T12:37:51+00:00 INFO (6): Tofex_Import_Model_Entity:runImport[114] Read 6 source element(s)
2020-05-22T12:37:51+00:00 INFO (6): Tofex_Import_Model_Import:runImport[153] Found 0 cached source element(s)
2020-05-22T12:37:51+00:00 INFO (6): Tofex_Import_Model_Import:runImport[189] Found 0 invalid source element(s)
2020-05-22T12:37:51+00:00 INFO (6): Tofex_Import_Model_Entity:runImport[114] Transformed 6 element(s)
2020-05-22T12:37:51+00:00 INFO (6): Tofex_Import_Model_Import:runImport[203] Found 6 cached transformed element(s)
2020-05-22T12:37:51+00:00 INFO (6): Tofex_Import_Model_Entity:runImport[114] No products to validate
2020-05-22T12:37:51+00:00 INFO (6): Tofex_Import_Model_Import:runImport[252] Saving 6 source data hash(es)
2020-05-22T12:37:51+00:00 INFO (6): Tofex_Job:run[91] Finished job: import_product_test
2020-05-22T12:37:51+00:00 INFO (6): Tofex_Job:run[91] Duration: 0 minute(s), 1 second(s)
2020-05-22T12:37:51+00:00 INFO (6): Tofex_Job:run[91] Max memory usage: 13 MB

Database Comparison

In case the source and the transformed cache does not match, the import still detects no changes by comparing the import values with the current database values. Even though no import was done, the process still saves the source and transformed cache to its cache to avoid further comparison of database values when the same data gets imported again.

2020-05-22T12:40:08+00:00 INFO (6): Tofex_Job:run[91] Running job: import_product_test
2020-05-22T12:40:08+00:00 INFO (6): Tofex_Job_Model_Job:launch[177] Importing file 1/1: /var/www/magento/htdocs/app/code/community/Tofex/ImportProductTest/data/import/product/test/data.xml
2020-05-22T12:40:08+00:00 INFO (6): Tofex_Import_Model_Entity:runImport[114] Read 6 source element(s)
2020-05-22T12:40:08+00:00 INFO (6): Tofex_Import_Model_Import:runImport[153] Found 0 cached source element(s)
2020-05-22T12:40:08+00:00 INFO (6): Tofex_Import_Model_Import:runImport[189] Found 0 invalid source element(s)
2020-05-22T12:40:08+00:00 INFO (6): Tofex_Import_Model_Entity:runImport[114] Transformed 6 element(s)
2020-05-22T12:40:08+00:00 INFO (6): Tofex_Import_Model_Import:runImport[203] Found 0 cached transformed element(s)
2020-05-22T12:40:08+00:00 INFO (6): Tofex_Import_Model_Import:runImport[239] Found 0 invalid transformed element(s)
2020-05-22T12:40:08+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Found 1 store(s) in elements
2020-05-22T12:40:08+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Split data for store with id: 1 in 1 chunk(s)
2020-05-22T12:40:08+00:00 INFO (6): Tofex_Import_Model_Import:runImport[245] Importing chunk: 1/1 containing 6 element(s)
2020-05-22T12:40:09+00:00 INFO (6): Tofex_Import_Model_Entity:runImport[114] Imported 0 changed products
2020-05-22T12:40:09+00:00 INFO (6): Tofex_Import_Model_Entity:runImport[114] Ignored 6 unchanged products
2020-05-22T12:40:09+00:00 INFO (6): Tofex_Import_Model_Import:runImport[252] Saving 6 source data hash(es)
2020-05-22T12:40:09+00:00 INFO (6): Tofex_Import_Model_Import:runImport[253] Saving 6 transformed data hash(es)
2020-05-22T12:40:09+00:00 INFO (6): Tofex_Job:run[91] Finished job: import_product_test
2020-05-22T12:40:09+00:00 INFO (6): Tofex_Job:run[91] Duration: 0 minute(s), 1 second(s)
2020-05-22T12:40:09+00:00 INFO (6): Tofex_Job:run[91] Max memory usage: 15 MB

License

Tofex Import is licensed under the MIT License - see the LICENSE file for details.