Data Collection in Data Science
Press Next → or use ← → arrow keys
Why It's The Most Critical Step
It's the very first stage of the pipeline, and everything downstream inherits its quality. Perfect cleaning, brilliant features, a state-of-the-art model — none of it can undo data that was biased or badly gathered at the source.
A model trained on poorly collected or biased data produces poor, biased predictions — even if every later step is flawless. Get collection right and the rest of the pipeline has a fighting chance; get it wrong and nothing else can save you.
The Data Determines The Destiny
The identical model gives trustworthy answers on clean, representative data and biased garbage on dirty data. The lever that decides which you get isn't the algorithm — it's the care you put into collection. That's why this is where a project lives or dies.
Five Places Data Comes From
Internal databases for company records, APIs for live third-party feeds, flat files for exports and hand-offs, web scraping for data that only exists on pages, and open datasets for benchmarks and public information. Most real projects blend several.
Databases — SQL & NoSQL
import pandas as pd, sqlite3 conn = sqlite3.connect("sales.db") query = "SELECT customer_id, revenue FROM sales WHERE year = 2024" df = pd.read_sql(query, conn) # query result → DataFrame in one step
APIs — Data On Request
import requests, pandas as pd resp = requests.get(url, params={'apikey': KEY}) if resp.status_code == 200: # 200 OK · 401 auth failed · 429 rate limited data = resp.json() # APIs usually return JSON df = pd.json_normalize(data['results']) # flatten nested JSON → table
APIs (OpenWeatherMap, Alpha Vantage, NASA, REST Countries, and countless others) serve fresh data on
demand, usually as JSON. Always check the status code — a 429 means you've
hit the rate limit. Respect those limits and add exponential backoff so a burst of
requests retries politely instead of getting you blocked.
Most useful APIs need an API key. Store it in an environment variable or secrets manager — never hard-code it into a notebook you might share or commit to Git.
Flat Files — CSV To Parquet
| Format | Strengths | Watch Out |
|---|---|---|
.csv | Universal, human-readable | Slow & bulky on large files; no types |
.xlsx | Familiar, multi-sheet | Row limits; not for big data |
.json | Nested / semi-structured | Needs flattening for tables |
.parquet | Columnar, typed, compressed | Not human-readable |
Parquet stores data by column, keeps types, and compresses hard — up to ~80% smaller and 10–100× faster to read than CSV on large datasets. CSV is fine for small, shareable files; past ~100 MB, Parquet saves real time and disk.
Web Scraping — When Data Lives On A Page
Tools like BeautifulSoup (static pages) and Playwright / Selenium
(JavaScript-rendered pages) turn HTML into data. But always check robots.txt and
the site's Terms of Service first, throttle your requests, and never scrape personal data you
have no right to. "Technically possible" is not the same as "allowed."
Open & Public Datasets
# Kaggle CLI pip install kaggle kaggle datasets download -d username/dataset-name # then, in Python: df = pd.read_csv("data.csv")
"Open" doesn't mean "anything goes" — confirm the licence permits your use (commercial vs research), and check how recent the data is. A benchmark from 2010 may no longer represent today's world.
Structured, Semi-Structured & Unstructured
Only about 20% of enterprise data is neat, tabular structured data; roughly 10% is semi-structured (JSON, XML, logs — flexible but parseable); and a whopping 70–80% is unstructured — text, images, audio, video — usually parked in data lakes like S3 or HDFS. The richest data is also the messiest to collect.
Different Data, Different Tools
| Data | Typical Tools | What It Takes |
|---|---|---|
| 📝 Text | spaCy, NLTK, Transformers | NLP / tokenization |
| 🖼️ Images | OpenCV, torchvision, PIL | Computer vision |
| 🔊 Audio | librosa, SpeechRecognition | Signal processing |
| 🎬 Video | OpenCV, ffmpeg | Frame extraction + CV |
| 📄 Documents | pdfplumber, PyMuPDF | Text/table extraction |
For unstructured data, "collecting" it is only half the job — turning a folder of images or a pile of PDFs into model-ready features needs the right toolkit (CV, NLP, or signal processing). Plan for that effort when you choose an unstructured source.
Quality, Bias & Ethics
Cleaning and preprocessing typically discard 30–50% of raw data — bad rows, duplicates, missing values. Gathering a healthy surplus up front means you still have enough left to model with once the mess is removed. Collection is an ongoing, deliberate process, not a one-off grab.
Six Rules For Data Collection
You Can Now Gather Data Well
Data collection is the first and most decisive step: gather from the right sources (databases, APIs, files, the web, open data), know whether it's structured or not, document it, and guard against bias and privacy risk. Because everything you build later inherits the quality of what you collect here.
With raw data in hand, the pipeline moves on to data cleaning (missing values, duplicates, outliers), then transformation and feature engineering — turning the raw material you just collected into something a model can learn from.
🗄️ End of tutorial · Press ← to review, or click Restart