iii Means to Read Multiple CSV Files: For-Loop, Map, List Comprehension

Written by Matt Dancho on September 21, 2021

Reading many CSV files is a common chore for a information scientist. In this free tutorial, we show you lot three ways to streamline reading CSV files in Python. You'll read and combine 15 CSV Files using the top 3 methods for iteration.

Python Tips Weekly

This article is part of Python-Tips Weekly, a bi-weekly video tutorial that shows y'all step-past-step how to exercise mutual Python coding tasks.

Hither are the links to get set. 👇

  • Get the Lawmaking
  • YouTube Tutorial

Video Tutorial
Follow along with our Full YouTube Video Tutorial.

This 5-infinitesimal video covers reading multiple CSV in python.

(Click paradigm to play tutorial)

Read xv CSV Files [Tutorial]

This FREE tutorial showcases the awesome power of python for reading CSV files. We'll read 15 CSV files in this tutorial.

Before we get started, get the Python Crook Sheet

The Python Ecosystem is Big. To help, I've curated many of the eighty/twenty Python Packages, those I use most ofttimes to get results. Merely Download the Ultimate Python Crook Sail to admission the entire Python Ecosystem at your fingertips via hyperlinked documentation and crook sheets.

(Click prototype to download)

Onto the tutorial.

Project Setup

Showtime, load the libraries. We'll import pandas and glob.

  • Pandas: The chief information wrangling library in Python

  • glob: A library for locating file paths using text searching (regular expressions)

2nd, use glob to extract a listing of the file paths for each of the fifteen CSV files we need to read in.

Libraries

Get the code.

Method 1: For-Loop

The about common fashion to repetitively read files is with a for-loop. It's a cracking way for beginners but it'south non the most curtailed. We'll bear witness this fashion commencement.

Python For-Loop for Reading CSV Files

Get the lawmaking.

We can see that this involves 3-steps:

  1. Instantiating an Empty List: We do this to shop our results every bit nosotros make them in the for-loop.

  2. For-Each filename, read and append: Nosotros read using pd.read_csv(), which returns a data frame for each path. Then we append each information frame to our listing.

  3. Combine each Data Frame: We use pd.concat() to combine the list of data frames into one big information frame.

PRO-TIP: Combining data frames in lists is a common strategy. Don't forget to utilize axis=0 to specify row-wise combining.

Method 2: Using Map

The map() function is a more concise way to iterate. The advantage is that we don't accept to instantiate a list. Still, information technology can be more than disruptive to beginners.

How it works:

Map takes in two general arguments:

  1. func: A office to iteratively apply

  2. *iterables: One or more iterables that are supplied to the function in lodge of the functions arguments.

How Map Works (Python)

Become the lawmaking.

Let's use information technology.

Ok, then let's endeavour map().

Python Map for Reading CSV Files

Get the lawmaking.

We use three-steps:

  1. Brand a Lambda Role: This is an anonymous function that nosotros create on the fly with the first argument that will accept our iterable (each filename in our list of csv file paths).

  2. Supply the iterable: In this case, we provide our listing of csv files. The map function will and then iteratively supply each element to the part in succession.

  3. Catechumen to Listing: The map() function returns a map object. We can so convert this to a list using the listing() office.

PRO-TIP: Beginners can exist confused past the "map object" that is returned. But simply apply the list() part to extract the results of map() in a list structure.

Method iii: List Comprehension

Because we are returning a listing, even easier than map(), nosotros tin utilize a List Comprehension. A list comprehension is a streamlined way of making a for-loop that returns a list. Here'south how it works.

Python List Comprehension for Reading CSV Files

Get the lawmaking.

  1. Exercise this: Add together the part that you desire to iterate. The parameter must match your looping variable proper noun (side by side).

  2. For each of these: This is your looping variable name that you lot create inside of the list comprehension. Each of these are elements that will go passed to your office.

  3. In this: This is your iterable. The list containing each of our file paths.

Summary

There you have it. You at present know how to read CSV files using 3 methods:

  1. For-Loops
  2. Map
  3. List Comprehension

Only there'south a lot more to learning data science. And if you're similar me, you're interested in a fast runway system that will advance you without wasting time on information yous don't demand.

The solution is my form, Information Scientific discipline Automation with Python

Data Scientific discipline Automation with Python Class

Tired of struggling to learn data science? Getting stuck in a bounding main of neverending resources? Eliminate the confusion and speed up your learning in the process.

Businesses are transitioning manual processes to Python for automation. We teach yous skills that organizations demand correct now.

Larn how in our new course, Python for Data Science Automation. Perform an finish-to-end concern forecast automation using pandas, sktime, and papermill, and learn Python in the procedure.