Data Science Workflow: From Problem to Solution – A Complete Step-by-Step Journey for Beginners

📗 Chapter 2: Data Collection and Acquisition

Gathering the Right Data to Power Your Data Science Project

🧠 Introduction

Once you’ve defined your problem clearly (as discussed in Chapter 1), the next step is to gather the right data — because a great model built on irrelevant or low-quality data is still a bad solution.

Data collection is where the practical meets the strategic. You need to know where to find data, how to access it, and what format it’s in — while also understanding the ethics, legality, and scalability of your sources.

This chapter will walk you through:

• Types of data and their sources
• Methods of acquiring data (APIs, web scraping, databases, etc.)
• Loading data in Python using pandas
• Common challenges in data acquisition
• Real-world examples and best practices

🔍 1. What is Data Collection?

Data collection is the process of sourcing raw information that will be used to solve your problem. It could be structured (CSV files, databases) or unstructured (text, images, videos).

🧩 2. Types of Data Sources

📦 3. Common Data Formats

🧰 4. Reading Local Files in Python

▶ CSV Files:

python

import pandas as pd

df = pd.read_csv('data/customers.csv')

df.head()

▶ Excel Files:

python

df = pd.read_excel('data/sales.xlsx', sheet_name='2023')

▶ JSON Files:

python

df = pd.read_json('data/config.json')

🖧 5. Collecting Data via APIs

APIs allow you to pull real-time or fresh data from external providers.

▶ Example: OpenWeatherMap API

python

import requests

url = "http://api.openweathermap.org/data/2.5/weather?q=London&appid=YOUR_API_KEY"

response = requests.get(url)

data = response.json()

print(data['weather'][0]['description'])

▶ Tools:

• requests, http.client
• json for parsing
• API keys and authentication

🌐 6. Web Scraping for Custom Data

Web scraping extracts data directly from websites. Always check site terms of use before scraping.

▶ Example using BeautifulSoup:

python

import requests

from bs4 import BeautifulSoup

url = "https://example.com/products"

html = requests.get(url).text

soup = BeautifulSoup(html, 'html.parser')

titles = [tag.text for tag in soup.find_all('h2', class_='product-title')]

print(titles)

▶ Popular Libraries:

• BeautifulSoup
• Scrapy
• Selenium (for dynamic pages)

🗃 7. Extracting from Databases (SQL)

For internal enterprise data, you’ll often query a database.

▶ Example using SQLite:

python

import sqlite3

conn = sqlite3.connect('sales.db')

df = pd.read_sql_query("SELECT * FROM orders", conn)

df.head()

▶ Other tools:

• PostgreSQL: psycopg2
• MySQL: mysql.connector
• SQLAlchemy: works with most relational databases

☁️ 8. Cloud-Based Data Sources

Data can also live in the cloud:

• Google BigQuery
• Amazon S3
• Firebase
• Snowflake

▶ Example: Reading from S3 using boto3

python

import boto3

import pandas as pd

s3 = boto3.client('s3')

s3.download_file('bucket-name', 'folder/file.csv', 'local_file.csv')

df = pd.read_csv('local_file.csv')

📋 9. Considerations When Acquiring Data

🧠 10. Best Practices for Data Acquisition

✅ Case Study Example: Acquiring Customer Data

Scenario:

A retail business wants to predict customer churn. You’ve been given access to internal CRM data and user interaction logs. You also need supplementary demographic data from an open API.

Breakdown:

🔄 Automation for Regular Collection

Set up scheduled scripts to fetch and clean data periodically.

bash

# Use cron (Linux) or Task Scheduler (Windows)

0 * * * * /usr/bin/python3 /home/project/scripts/fetch_data.py

You can also integrate tools like:

• Apache Airflow
• Prefect
• Luigi

📊 Summary Table: Data Acquisition Methods