ETL Using Core Python

What will you learn?

Extraction

•Create connections with databases (Oracle, SqlServer, PostgreSql)

•Extract data from the database

•Extract data from AWS S3 buckets

•Extract the data from files

Transformation

•Data Type conversion

•Data Manipulations using Expressions

•Joins

•Union

•Filter

•Sorter

•Transform CSV to Json

•Transform Json to CSV

Load

•File to Database

•Database to File

•Oracle to PostgreSql

•PostgreSql to SqlServer

•Load the files to S3 buckets

In computing, extract, transform, load (ETL) is a three-phase process where data is extracted, transformed (cleaned, sanitized, scrubbed) and loaded into an output data container. The data can be collated from one or more sources and it can also be outputted to one or more destinations. ETL processing is typically executed using software applications but it can also be done manually by system operators. ETL software typically automates the entire process and can be run manually or on reoccurring schedules either as single jobs or aggregated into a batch of jobs.

Conventional ETL diagram[1]

A properly designed ETL system extracts data from source systems and enforces data type and data validity standards and ensures it conforms structurally to the requirements of the output. Some ETL systems can also deliver data in a presentation-ready format so that application developers can build applications and end users can make decisions.[1]

The ETL process became a popular concept in the 1970s and is often used in data warehousing.[2] ETL systems commonly integrate data from multiple applications (systems), typically developed and supported by different vendors or hosted on separate computer hardware. The separate systems containing the original data are frequently managed and operated by different stakeholders. For example, a cost accounting system may combine data from payroll, sales, and purchasing.