One CSV read cost me half a second
On September 23 at 10:00 am Eastern US time, I'm hosting a free webinar. In it, you'll found out the fastest route to becoming a Python quant:
- Master the 20% of Python that drives 80% of quant results.
- Build professional-grade backtests in under 30 minutes.
- Automate execution by connecting Python directly to broker APIs.
- Clean messy market data using the exact workflows used at J.P. Morgan.
- Get the step-by-step roadmap to transition from “tutorial hell” to deploying real trading strategies.
It's based on my flagship course, Getting Started With Python for Quant Finance.
Register for free here 👉
Only the first 1,000 registrants will get a spot.
"
A top-tier introduction to Python and quant finance.
"
– Zac T
|
|
You get maybe 40 minutes at the laptop after work.
Mine went like this. Download 30 years of daily bars, restart the kernel (the Python session running behind your notebook) because something was stuck in memory, re-read the CSV, then find a column parsed as text and restart again. Half a second per read isn't much on its own.
But I ran that loop most of one evening in October and never typed the moving-average crossover test I sat down to write.
Here are the numbers from a test I ran on my own laptop.
The file holds 3.8 million daily bars for the 500 companies currently in the S&P 500, 1994 through 2024, saved uncompressed with a date column, a ticker column, and five price and volume columns.
Current members only, so every company delisted since 1994 is missing from it. On an SSD with default settings and no dtype hints, pandas.read_csv averaged 458 milliseconds over three runs and polars.read_csv averaged 3.57 milliseconds.
I was paying that 458 milliseconds a dozen times an evening without once measuring it.
Time each step of your loader separately, the network download and the parse and the cleanup, then fix whichever one dominates.
Write the cleaned table to disk in a format that needs no parsing, and a restart costs you one read instead of a re-download.
That's the first piece of a system that pulls market data and keeps a clean copy on your machine.
Here's a Python example you can use now.
Assumes pd (pandas), pl (polars), and perf_counter from Python's time module are already imported, and that prices.csv sits in your working directory.
This prints two numbers in milliseconds, one for each library, on your file and your machine.
Wrap any other slow line in time_call the same way, a date parse or a group-by that averages rows bucket by bucket, and you'll see which one actually costs you the wait.
The last line writes the table to Parquet, a binary file format that stores each column separately, so the next restart reads binary instead of parsing 3.8 million lines of text.
Polars ran slower than pandas when I forward-filled missing closes grouped by ticker and sorted by date, both in eager mode, and I never worked out why, so time your own operations before you switch libraries for them.
[Content]
Your next steps
Time each step in your current loader separately, fix the slowest one, then write the result to Parquet and time that read too, because kernel startup is a separate cost you won't get back.
From there, write a script that pulls yesterday's bars, checks them against the last date already on disk, and rewrites the file (Parquet won't append in place, so you rewrite it or keep one file per year).
At 40 minutes a session, a month of evenings gets you a local price history you can query without waiting on a download.