As M2IF uses the goodby/csv streaming library to read the content of a CSV file row by row, it has already been optimized for a very low memory consuption. But up with version 3.5.0 we've reduced the memory footprint again signifacantly.

This has been possible by removing debug information, that usually will not be necessary in production. The result is a round about 70 % lower memory consumption. To give you a better impression what we're a talking about we'll use the sample data with round about 1.891 simple, 147 configurable and 1 bundle products. Instead of 80 MB, now the peak memory usage for the product import is at 22 MB, whereas the performance improvement (on my local development machine) of 1 second, from 24 to 23 seconds, is not really noticable.

The debug information is useful, if data from the CSV file that has not been processed during the first step (importing the simple products) but in one of the further steps, e. g. when the variants will be created. In that case, after the simple products has been imported, a new artefact with the information which is necessary to create the variants will be created. Before version 3.5.0, each artefact will contain the data on row/column level where it has been located in the original file. This data blows up memory but will usually not necessary in production mode. If an error occurs, this information can easily be reproduced, by enable the debug mode.

To make memory consumption visible, simply extend your logger configuration by adding the MemoryPeakUsageProcessor to your configuration (up from 3.5.0 this is the default configuration) like

{
  ...,
  "loggers": [
    {
      "name": "system",
      "channel-name": "logger/system",
      "type": "Monolog\\Logger",
      "handlers": [
        {
          "type": "Monolog\\Handler\\ErrorLogHandler",
          "formatter": {
            "type": "Monolog\\Formatter\\LineFormatter",
            "params" : [
              {
                "format": "[%datetime%] %channel%.%level_name%: %message% %context% %extra%",
                "date-format": "Y-m-d H:i:s",
                "allow-inline-line-breaks": true,
                "ignore-empty-context-and-extra": true
              }
            ]
          }
        }
      ],
      "processors": [
        {
          "type": "Monolog\\Processor\\MemoryPeakUsageProcessor"
        }
      ]
    }
  ],
  ...,
}

which should finally end-up in a logger output (in your PHP error log) like

logger/system.INFO: Successfully finished import with serial 3f8d0cd1ff43 in \
  23,550142 s  {"memory_peak_usage":"22 MB"}

Next Post Previous Post

Load Comments

Please note that by clicking "Load Comments", your browser will establish a connection with servers from Disqus.
For more information, please visit our privacy statement.