Solving a real problem with pipelines and semaphores

In this repository I have recreated/simulated a problem I faced at work some time ago.

Here I show three different ways to solve the problem:

1. Linear approach (Less optimal solution)
2. Using pipelines
3. Using pipelines and semaphores (Most optimal solution)

⚠️ Remember: the patterns are a conceptual idea more than a strict way to do something.

Problem

Calculate the net profit for each closing in a company's stores.

Context

A company operates stores

Each store has multiple closings (accounting periods, such as monthly or quarterly)

For every closing, the company tracks:

• sales Revenue generated during the period.
• costs Expenses incurred to run the store during the period.

Objective

Develop a process to calculate and store the net profit of each close, where net profit = sales - costs

⚠️ Some details have been changed to avoid revealing confidential information but the problem remains the same.

Limitations

The process runs every 6 hours on weekdays, so it runs on AWS Lambda (EventBridge) to save costs, and as is well known, AWS Lambda has a maximum timeout of 15 minutes. So, the idea is not to exceed the timeout. See how to determine the appropriate timeout value for a Lambda function

Solutions

Linear (See the code)

How to run

make lineal
time ./build/lineal

Benchmarking

make benchmark_lineal

Diagram

Pipelines (See the code)

How to run

make pipeline
time ./build/pipeline

Benchmarking

make benchmark_pipeline

Pipelines + Semaphores (See the code)

How to run

make semaphore
time ./build/semaphore

Benchmarking

make benchmark_semaphore

Resumen

Linear solution vs. Pipelines + Semaphores

goos: darwin
goarch: arm64
pkg: github.com/yael-castro/pipelines/internal/logic
cpu: Apple M4
                      │   lineal.out   │            semaphore.out            │
                      │     sec/op     │   sec/op     vs base                │
Logic_CalculateProfit   328497.6m ± 3%   272.8m ± 4%  -99.92% (p=0.000 n=10)

                      │  lineal.out  │            semaphore.out             │
                      │     B/op     │     B/op      vs base                │
Logic_CalculateProfit   41.31Mi ± 0%   47.29Mi ± 0%  +14.48% (p=0.000 n=10)

                      │ lineal.out  │            semaphore.out            │
                      │  allocs/op  │  allocs/op   vs base                │
Logic_CalculateProfit   129.9k ± 0%   188.0k ± 1%  +44.70% (p=0.000 n=10)

⚠️ The benchmarks were performed using only one logic CPU at a time.

_ = runtime.GOMAXPROCS(1)