I'll write in detail about the whole import process in another post. Since the uploaded CSV is expected to have tens or even hundreds of thousands of rows, all of the operations need to be done in a memory-efficient way, otherwise, the app would break from running out of memory. If everything was fine, the mapped data from the first JSON file is converted into database records, which in this case span several connected tables.There can be A LOT of validation errors for large CSV files. Validation errors are saved to different JSON file so they can be fetched later from the frontend without additional processing by the application. If there are any validation errors, we don't want to save anything to the database, we want to present all of the errors for each row. Finally, if there are no validation errors the data is read from the JSON file again and saved to the database. In the second step the JSON file is read and each item from the collection is validated against the defined rules.This allows us to not worry about parsing the data again in the following steps. First, the CSV file is read, columns are mapped, and saved to a JSON file.The import then goes through several stages:
The user selects a CSV file, maps columns to supported fields so the app can tell what is what, and submits it. Using PHP streams to encode and decode large JSON collectionsĪ while ago, I was working on a way to import large datasets in a hobby project of mine Biologer.
0 kommentar(er)
