Preparation and persistence
Type conversion, Parquet file creation and loading of clean tables into SQLite.
Applied Python · Data engineering
E-commerce analytics pipeline in Python
Turn raw files into controlled data, calculate reliable indicators and publish a lightweight output to bring technical preparation closer to business use.
01
Raw sources
Customer, order, product and payment files feed the process.
02
Preparation
Pandas standardises names, types, dates and amounts.
03
Checks
Duplicates, missing values and inconsistent references are measured.
04
Metrics
Revenue, orders, active customers and average order values are aggregated.
05
Output
Results are exported and made available in a lightweight application.
Business context
An e-commerce team receives several operational files whose formats and quality can vary. Before any reporting, the data must be harmonised, linked and checked in a reproducible way.
Challenge
How can heterogeneous files be turned into usable metrics without multiplying manual operations, while retaining a record of anomalies and each execution?
Approach
The prototype clearly separates preparation, control, calculation and delivery. Pandas processes the data, Parquet provides a compact analytical format, SQLite supports simple queries and Streamlit exposes the results in a lightweight interface.
Reproducible pipeline
The same rules are replayed on every run instead of being applied manually.
Measurable quality
Each check returns an affected volume and an explicit severity level.
Versatile outputs
Parquet for analysis, SQLite for SQL and CSV for exchange with other tools.
Traceable execution
A log retains the status, detected anomalies and message for each run.
What the demo shows
Explore three possible pipeline behaviours
The simulation illustrates a normal run, a batch containing several anomalies and a controlled stop caused by an unavailable source. It highlights the metrics, checks and logging expected around a Python pipeline.
Local front-end simulation: it reproduces the prototype states and checks without running Python on this server or sending data to an external service.
Choose a scenario and run the simulation to display the stages, checks and log.
Processing result
Completed
Rows received
0
Rows retained
0
Orders
0
Average order value
0 €
Pipeline progress
Quality checks
Prototype deliverables
The screenshots below come from the original Python prototype. They document the code, analytical outputs, checks and Streamlit application. All data is fictitious.
The script enforces the expected types, produces clean tables and persists them in Parquet and SQLite.
Parquet outputs
Five files separate customers, orders, order lines, products and payments for downstream analysis.
Monthly aggregations feed exports that can be reused in a report, presentation or BI tool.
Revenue trend
A monthly series helps identify variations and prepare a trend analysis.
Order volume
Tracking volume in parallel helps distinguish a frequency effect from an average order value effect.
KPI export
The monthly file retains revenue, orders, active customers and average order value.
Execution trace
Output paths and a preview of the results make the run easier to review.
Quality rules make anomalies visible before they affect metrics or outputs.
Rules and severities
Duplicates, missing values, negative amounts and orphan references are counted.
Processing log
Status, anomaly count and message are retained for audit and diagnosis.
The lightweight application brings together the KPIs and log. It also shows failures in external calls so that monitoring does not hide incomplete processing.
Streamlit application
Monthly table, charts and log brought together in a lightweight interface.
External integration
An API call enriches the flow and returns a clear message in the interface.
Writing the result
The log retains the integration status to make errors visible and diagnosable.
Python
Pandas
Parquet
SQLite
Matplotlib
Streamlit
Business value
Consistent preparation
Typing and cleaning rules are documented and then applied consistently.
More reliable reporting
Anomalies are detected before metrics are calculated and distributed.
Reusable analysis
Standardised outputs can feed Python, SQL, Excel or a BI solution.
Easier diagnosis
The log helps explain a failure and identify the affected stage.
From demo to production
For real-world use, the architecture mainly depends on volume, frequency and the required level of availability. Moderate processing can be containerised and orchestrated in Azure. Larger volumes may justify a distributed engine such as Spark.
Orchestrated ingestion
Azure Data Factory or Fabric Data Factory collects files and schedules dependencies.
Industrialised processing
Python code is tested, versioned and executed in a container or a service suited to the volume.
Quality and monitoring
Pandera or Great Expectations formalises the rules, while Azure Monitor centralises alerts.
Controlled consumption
A lakehouse, SQL database, Power BI or business application exposes only validated data.
Pandas and SQLite suit this local prototype. They are not a universal recommendation: data volume, concurrency, security, disaster recovery and cost must guide the choice of production architecture.
Going further
This type of pipeline can support operational reporting, billing control, BI data preparation or the automation of business monitoring, with quality rules suited to downstream decisions.