Have you ever wondered why Magento 2 import and export processes are so slow? If you move data to and from your store on a regular basis, low import/export speed may become really annoying. But what is the reason for this problem and how to solve it? You will find the answer below.
Also, the article provides an alternative solution for the Magento 2 import and export performance issue. The default data transfer solution is associated not only with relatively low speed stats but also with a plethora of other hurdles that don’t let your admins manage data most efficiently. However, there is a universal solution that addresses all the issues associated with Magento’s built-in tools – the Improved Import & Export extension. You will also find its core features described below.
Download / Buy Firebear Improved Import Export Magento 2 Extension
Table of contents
Magento 2 Import Performance Issues
The first reason is related to Object-Relational Mapping. It is a programming technique designed to converting data between incompatible type systems. In case of Magento 2, ORM must be flexible and as a result complex – this makes access to database much slower.
The second reason is the database itself. Due to the nature of the ecommerce platform, the database has a complex structure. Thus, the Entity Attribute Value model on the database causes the situation when product and other entities data is distributed by dozens of tables, so lots of information need to be collected and bundled on each request. So, how to enhance the default Magento 2 platform and increase import and export speed?
Magento 2 Import Speed Improvement
The answer is very simple: you need a reliable third-party extension that improves Magento 2 import and export performance. And there is the appropriate module in our portfolio. With the Firebear Improve Import & Export for Magento 2, you will dramatically speed up both processes. For instance, an average import speed of Magento 2.1 on 4 core CPU and 8 RAM per 1k simple products takes approximately 3 minutes. In case of our Magento 2 import extension, it is decreased to ~20 seconds. And this is not a final result. We constantly work on improving our import/export solution, so in the nearest future it will provide even better performance.
High productivity results get the highest priority when we work on our products, and Improved Import & Export for Magento 2 is not an exception. The extension leverages the sami principle that was used in Magmi for Magento 1: it executes more direct SQL without ORM. So if you are thinking about Magmi on Magento 2, freely choose our Magento 2 import/export extension, since it is the closest possible Magmi replacement (and analog) for Magento 2.
The recent speed tests conducted on March 2019 displayed the following results for different entities.
| Behavior | Entity | Entity count | File size | Minutes | Memory usage | Entities/minute |
| Import | Products | 500k | 645 Mb | 200 | 3.5 Gb | 2500 |
| Import | Customers + one address | 1m | 164 Mb | 45 | 2.5 Gb | 11111 |
| Import | Customers + one address | 1m | 330 Mb | 117 | 6.18 Gb | 8574 |
| Export | Products | 500k | 710 Mb | 159 | 500 Mb | 3145 |
| Export | Customers | 1m | 240 Mb | 6 | 3.72 Gb | 166667 |
| Export | Addresses | 1.4m | 244 Mb | 13 | 5 Gb | 107692 |
NOTE: native Magento 2 import/export tool couldn’t manage such data volumes. The import process either halts and imports only a part of the data with some of the entities missing, or has troubles validating the imported files.
Import and export performance tests setup
Import and export performance tests have been conducted on the following setup:
- PHP 7.2.6
- 8 core CPU
- 64 Gb RAM
- MariaDB 10.1.26 (MySQL 5.7)
- Intel(R) Xeon(R) CPU E3-1275 v5 @ 3.60GHz
- DDR3 ECC 16Gb
- 2 x Crucial MX550 SSD
Final Words
We are working with Magento 2 merchants from all over the world and there are both small merchants and huge ecommerce enterprises among the users of Improve Import and Export for Magento 2. Thus, millions of products are imported and updated daily with the help of our module. And due to the collected feedback, we always make import/export processes faster as well as fix core bugs and bottlenecks of the default Magento 2 procedures related to the functionality of our module.
For further information about the Magento 2 Improved Import Export extension, check its manual: Firebear Improved Import Export Extension Manual. You can purchase the module here:
Download / Buy Firebear Improved Import Export Magento 2 Extension
Improved Import & Export FAQ
Let’s take a look at other nuances of transferring data to Magento 2 with the help of the Improved Import & Export Magento 2 extension.
Which data sources does Improved Import & Export support?
While the default import solution works with your local server and file uploads, the module lets you transfer data between your e-commerce store and external systems via direct file uploads, FTP/SFTP servers, Google Drive, OneDrive, DropBox, direct URL uploads, Google Sheets, or via REST or SOAP APIs.
Which file types does Improved Import & Export for Magento 2 support?
By default, Magento 2 works only with CSV files. As for the Improved Import & Export extension, it supports CSV, XML, XLSX, ODS, JSON, ZIP, and TAR.
How to import and export files in Magento 2 automatically?
If you need to transfer any files between Magento 2 and third parties on a regular basis automatically, the Improved Import & Export extension lets you create a schedule of updates. The plugin uses cron to transfer all the updates for you.
How to import data with third-party attributes to Magento 2?
The performance of your admins is dramatically slowed down due to third-party attributes. In the case of the default import tools, they have to edit each data file manually to make it suitable for the requirements of Magento 2. With the Improved Import & Export extension, it is possible to create a mapping scheme where you match third-party attributes and values to the corresponding ones used within Magento 2. Consequently, any third-party data is moved to your database without editing. It is also possible to apply mapping to export processes to create any data output your partners require.
How to import and export data via ODS files to Magento 2?
Unsupported data standards also slow down the performance of your administrators since they have to convert ODS files to CSV. However, Improved Import & Export lets you import and export them without conversion.
How to import and export data via XLSX files to Magento 2?
Unsupported data standards also slow down the performance of your administrators since they have to convert XLSX files to CSV. However, Improved Import & Export lets you import and export them without conversion.
How to import and export data via XML files to Magento 2?
Unsupported data standards also slow down the performance of your administrators since they have to convert XML files to CSV. However, Improved Import & Export lets you import and export them without conversion.
How to import and export data via JSON files to Magento 2?
Unsupported data standards also slow down the performance of your administrators since they have to convert JSON files to CSV. However, Improved Import & Export lets you import and export them without conversion.
How to import data to Magento 2 from Google Sheets?
Storing data in Google Sheets may consume tons of time if you try to import it to Magento 2 unless the Improved Import & Export module is utilized. In the case of the default tools, you have to export an update from Google Sheets, edit it according to the requirements of Magento 2, and run the import process. Our module lets you establish a direct connection between Magento 2 and Google Sheets. No files are necessary. No conversion takes place.
How to import and export files between Magento 2 and Dropbox?
The Improved Import & Export module lets you transfer entities to Magento 2 from Dropbox and back. You can use CSV, JSON, ODS, XML, or XLSX files. If they are compressed, ZIP and TAR archives are supported.
How to import and export data files to Magento 2 from Google Drive?
You can use our extension to leverage Google Drive as a data file exchange point between Magento 2 and external systems. Use CSV, JSON, ODS, XML, or XLSX files and ZIP or TAR archives.
How to import and export data between Magento 2 and external systems via API?
The best way to improve the performance of your admins when it comes to import and export processes is to create API connections between your e-commerce website and external systems. The Improved Import & Export extension lets you do that after a few simple manipulations. Each import and export job provides a place where you can apply the parameters necessary to create an API connection for moving data.
