GETTING THE DATA INTO THE WAREHOUSE: EXTRACT, TRANSFORM, LOAD MIS2502

GETTING THE DATA INTO THE
WAREHOUSE: EXTRACT,
TRANSFORM, LOAD
MIS2502
Data Analytics
Getting the information into the data mart
The data in
the
operational
database…
…is put into
a data
warehouse…
…which
feeds the
data mart…
Now let’s address this part…
…and is
analyzed as
a cube.
Extract, Transform, Load (ETL)
• The process of copying data from the
transactional database to the analytical database
• Going from relational to dimensional
• Basically, it’s a matter of identifying where the
data should come from to fill the data mart
ETL Defined
Extract
Transform
Load
from the
operational
data store
(the
relational
database)
it into an
analysisready format
it into the
analytical
database
(the
dimensional
database)
The Actual Process
Extract
Transform
Load
Query
Transactional
Database 1
Query
Data conversion
Data Mart
Transactional
Database 2
Query
Data conversion
Query
Main ETL Issues: Conversion Stage
Data
• What if the data is in different
formats?
Consistency
Data Quality
• How do we know it’s correct?
• What if there is missing data?
• What if the data we need isn’t
there?
Data Consistency:
The Problem with Legacy Systems
• An IT infrastructure evolves over time
• Systems are created and acquired by different
people using different specifications
This can happen through:
•
•
•
•
Changes in management
Mergers & Acquisitions
Externally mandated standards
General poor planning
This leads to many issues
• Redundant data across the organization
• Customer record maintained by accounts
receivable and marketing
• The same data element stored in different
formats
• Social Security number (123-45-6789 versus
123456789)
• Different naming conventions
• “Doritos” versus “Frito-Lay’s Doritos” verus
“Regular Doritos”
• Different unique identifiers used
• Account_Number versus Customer_ID
What are the
problems
with each of
these
?
What’s the big deal?
• This is a fundamental problem for creating data
cubes
• We often need to combine information from
several transactional databases
• How do we know if we’re talking about the same
customer or product?
Now think about this scenario
Hotel Reservation Database
Countries
Country_code
Country_currency
Country_name
Hotel rooms
Room_number
Hotel_id
Room_type
Room_floor
Room types
Room_type_code
Room_standard_rate
Room_description
Smoking_YN
Hotels
Hotel_id
Country_code
Hotel_name
Hotel_address
Hotel_city
Hotel_zipcode
Room Bookings
Booking_id
Room_type_code
Hotel_id
Checkin_date
Number_of_days
Room_count
Café Database
Hotel Amenities Lookup
Characteristic_id
Characteristic_description
Hotel Amenities
Characteristic_id
Hotel_id
Guest Bookings
Booking_id
Guest_number
Guests
Guest_number
Guest_firstname
Guest_lastname
Guest_address
Guest_city
Guest_zipcode
Guest_email
Customer
Customer_number
Customer_name
Customer_address
Customer_city
Customer_zipcode
Food item
Order number
Food_item_id
Order_date
Order_time
Order
Order_number
Customer_number
Hotel_id
Food_item_id
Order_date
Order_time
Table_number
Hotels
Hotel_id
Country_code
Hotel_name
Hotel_address
Hotel_city
Hotel_zipcode
What are the differences between a “guest” and a “customer”?
Is there any way to know if a customer of the café is staying at the hotel?
Solution: “Single view” of data
• The entire organization understands a unit of
data in the same way
• It’s both a business goal and a technology goal
and really
more this…
..than this
Closer look at the Guest/Customer
Guests
Guest_number
Guest_firstname
Guest_lastname
Guest_address
Guest_city
Guest_zipcode
Guest_email
vs.
Customer
Customer_number
Customer_name
Customer_address
Customer_city
Customer_zipcode
Getting to a “single view” of data:
How would you
represent
“name?”
What would you
use to uniquely
identify a
guest/customer?
Would you
include email
address?
How do you
figure out if
you’re talking
about the same
person?
Organizational issues
• Why might there be resistance to data
standardization?
• Is it an option to just “fix” the transactional
databases?
• If two data elements conflict, who’s standard
“wins?”
Data Quality
• The degree to which the data reflects the actual
environment
Do we have
the right
data?
Is the
data
accurate?
Is the
collection
process
reliable?
Finding the right data
Choose data consistent with the goals of
analysis
Verify that the data really measures what
it claims to measure
Include the analysts in the design process
Adapted from http://www2.ed.gov/about/offices/list/os/technology/plan/2004/
site/docs_and_pdf/Data_Quality_Audits_from_ESP_Solutions_Group.pdf
Ensuring accuracy
Know where the data comes from
Manual verification through sampling
Use of knowledge experts
Verify calculations for derived measures
Adapted from http://www2.ed.gov/about/offices/list/os/technology/plan/2004/
site/docs_and_pdf/Data_Quality_Audits_from_ESP_Solutions_Group.pdf
Reliability of the collection process
Build fault tolerance into the process
• Check logs (if you can)
Periodically run reports and verify results
Keep up with (and communicate)
changes
Adapted from http://www2.ed.gov/about/offices/list/os/technology/plan/2004/
site/docs_and_pdf/Data_Quality_Audits_from_ESP_Solutions_Group.pdf