34 Instruments
23 Forex pairs, two commodities (WTI and Brent) and nine indices, including the S&P 500, Nikkei and DAX.
Open Source
histdata: Historical Data Downloader
Download historical market data from histdata.com and convert M1 bars to compressed Apache Parquet. Choose from 34 instruments for local research.
Python 3.10+Apache ParquetzstdMIT License34 Instruments
$ pip install git+https://github.com/StratCraftsAI/histdata.git
$ histdata --pairs EURUSD GBPUSD --year-start 2020 --year-end 2025
Illustrative output: one Parquet file per instrument
output/EURUSD_M1.parquet
output/GBPUSD_M1.parquet
Features
Build a local market dataset and review its coverage before quantitative analysis.
Apache Parquet Output
Parquet with zstd compression, readable with pandas, Polars, DuckDB and other Arrow-compatible tools.
CLI Tool
Install with pip and select instruments and years with the histdata command.
Archive Selection
M1 downloads select available annual or monthly archives for the requested date range and reuse validated local archives.
Typed Schema
OHLCV columns use timestamp[ms] and float64. Review the source timezone and the selected CLI workflow before analysis.
Python Dependencies
Uses requests, beautifulsoup4, pyarrow and filelock. Python 3.10 or later is required.
Pipeline
Download archives, parse M1 data and write typed Parquet files.
1
Extract Token
→
2
Download ZIP
→
3
Parse M1 CSV
→
4
Write Parquet
histdata.com → ZIP (CSV) → list[OHLCVRow] → Parquet (zstd)Supported Instruments
34 instruments across Forex, commodities and global indices. Available history varies by instrument.
Forex 23
Major Pairs
EURUSDGBPUSDUSDJPYUSDCHFUSDCADAUDUSDNZDUSDCross Pairs
EURAUDEURCADEURCHFEURGBPEURNZDGBPAUDGBPCADGBPCHFGBPNZDAUDNZDJPY Crosses
AUDJPYCADJPYCHFJPYEURJPYGBPJPYNZDJPYCommodities 2
BCOUSDBrent Crude Oil
WTIUSDWTI Crude Oil
Indices 9
AUXAUDASX 200
FRXEURCAC 40
GRXEURDAX 30
HKXHKDHang Seng
JPXJPYNikkei 225
NSXUSDNASDAQ 100
SPXUSDS&P 500
UDXUSDUS Dollar Index
UKXGBPFTSE 100
Output Schema
Typed Parquet columns. Validate timestamps and data quality before combining datasets.
| Column | Type | Description |
|---|---|---|
| timestamp | timestamp[ms] | Bar time; normal M1 downloads use UTC, --skip-download retains source EST |
| open | float64 | Open price |
| high | float64 | High price |
| low | float64 | Low price |
| close | float64 | Close price |
| volume | float64 | Source volume field |
Sourcehistdata.com — Generic ASCII CSV, semicolon-delimited; no API key is required
Source TimezoneEST (UTC-5, no daylight saving adjustment)
FormatApache Parquet + zstd
Quick Start
Install from GitHub and select the instruments and date range you need.
Installation
pip install git+https://github.com/StratCraftsAI/histdata.gitCLI Usage
# Download EURUSD 2020-2025 and convert to Parquet
histdata --pairs EURUSD --year-start 2020 --year-end 2025
# Multiple pairs
histdata --pairs EURUSD GBPUSD USDJPY --year-start 2015 --year-end 2025
# All 34 instruments
histdata --pairs all --year-start 2020 --year-end 2025Read with pandas
import pandas as pd
df = pd.read_parquet("output/EURUSD_M1.parquet")
print(df.head())
# timestamp open high low close volume
# 0 2023-01-02 17:00:00 1.07045 1.07048 1.07045 1.07048 0.0FAQ
Do I need an API key?
No API key is required. The downloader retrieves an archive token from histdata.com before downloading.
How long does a download take?
Duration depends on the number and size of available archives, the network and local processing. The downloader throttles requests; no completion time is guaranteed.
Can I use the output for quantitative research?
The M1 Parquet files can be read with pandas or other Parquet-compatible tools. Check data coverage, quality, column mapping and timezone requirements before importing into a research platform.
What timezone are the timestamps in?
Source CSV uses fixed EST (UTC-5) without daylight saving adjustment. Normal M1 downloads convert to UTC. The --skip-download path retains parsed source EST values. Parquet timestamp[ms] has no timezone metadata.
Does it support tick data?
The --timeframe tick option can download tick archives, but tick parsing and Parquet conversion are not implemented. Use M1 for the documented OHLCV output.
Get Started
Open-source software under the MIT license. Review the source and validate a small dataset before wider use.