Hallhuber, like many other companies, was faced with the great challenge of taking an eCommerce-System that had grown over the years and was already functioning very well to a new level and thus laying the foundation for further growth and corresponding future security. The switch from Magento 1 to Magento 2, which is associated with a multitude of innovations and technical innovations, is intended to offer Hallhuber customers a new shopping experience in the future.

Challenge

By far the greatest challenge has been the data import/export from and back to TB.One as well as the data migration from Magento 1. Whereas TB.One provides separate XML files, with two different XML structures, option and category IDs, for each of the 6 websites. These files has to be merged and extracted into separate CSV files in the default M2IF CSV formats for Attributes + Attribute Options as well as products and product prices. Beside the frequent import process that will run once a day, it should also be possible to manually trigger the import for exactly one website. Additionally product prices and stock information from a separate TB.One feeds, also providing XML files, has to be processed in intervals of 60 and 15 minutes.

Solution

To improve performance and optimizer indexing processes, product, stock + price imports supports full as well as delta import modes, whereas the delta mode is used to update stock and price information during the day. In all cases the XML contains a timestamp to re-capture the delta imports up from the last timestamp. The implemented TB.One connector has been optimized for high performance, memory efficient XML processing which merges and enriches the XML files with mappings for the categories, which are maintained on store view level, and extracts the data into the default CSV target files needed for the attribute/attribute option and product import. The process pipelines streamlines the import processes and takes care that the data consistency between the import steps will be ensured. Beside the frequent import that happens nightly it is possible to trigger the import for a dedicated website by simply selecting the apropriate pipeline in the admin backend. The indexing process for the high frequent price and stock import, also streamlined by the process pipelines, has been optimized to avoid complete cache invalidation after each import.

Result

The data crunching for the 6 XML files with several hundreds of thousands lines in each file will take around 26 minutes and needs less then 1 GB of memory. The import process, that follows the conversion step, finally needs round about 10 minutes to import the attributes as well as the products into the eCommerce-System. Finally another step updates the product status which again runs nearly 10 minutes. The main import will be processed in general once per night and does NOT affect the daily business as the shop will be fully functional meanwhile. The single transaction mode ensures that product, price and stock updates, indepenedent in full or delta mode, will always be consistent and will either be complete in its entirety or have no effect whatsoever. Additionally the single transaction mode can massively speed up the import process, because all DML statements will be processed at once to persist the data to the database, when the transaction will be committed. More details can be found in the customer success story on the solution partners website.

IMAGE

Components

EIP 1.0 Product Batch
Performance Booster (necessary for Single Transaction Mode when using EE)
M2IF 1.0 Single Transaction Mode
Attribute
Product

Entities / Frequency

Imported entities Attributes
Attribute Options
Products
Prices
Import frequency Stock [Frequency: every 30 minutes, Runtime: ~3 seconds]
Attribute [Frequency: once per day, Runtime: ~6 seconds]
Product + Prices [Frequency: once per day, Runtime: ~10 minutes]

System

Magento Version Magento EE 2.1
ERP TB.One

Amount of data

Websites 8
Stores 8
Store Views 10
Attributes ~213
Attribute options ~680
SKUs ~30.000
~25.000 Simples
~5.000 Configurables
Categories ~150
Load Comments

Please note that by clicking "Load Comments", your browser will establish a connection with servers from Disqus.
For more information, please visit our privacy statement.