Goals

The MAKARA project (Multi-Architecture Knowledge Analysis and Report Assessment) tests how deeply Large Language Models (LLMs) understand the foundational principles of computer architecture.

The core goal is to determine if an LLM can accurately predict performance behavior across different hardware. We challenge an LLM with a program's source code and its performance counter metrics from an initial architecture (Architecture A). The central objective is to query the LLM to predict how those same performance counters would appear if the identical program were executed on a distinct target architecture (Architecture B).

This cross-domain performance projection serves as a crucial metric for assessing an LLM's true architectural reasoning, moving beyond pattern recognition to validate its grasp of hardware-software interaction.

---

Help Us Build a Global Performance Dataset!

We are building a comprehensive dataset of performance counters, and we need your help! This dataset will map programs to their performance metrics on specific hardware.

By contributing, you'll help create a robust, open resource for researchers worldwide. As a collaborator, you will receive full access to the entire dataset!

Contributing is Easy: We've created an automated script that handles everything:

• It downloads and compiles our benchmark collection (Makara/Jotai).
• It runs the benchmarks and efficiently collects performance counter values.
• It captures system architecture details (CPU, caches, etc.).
• It automatically packages all the results for easy submission.

Ready to contribute? Just run the script (see instructions below) and send your data via this Google Form. Running the script takes about five to six minutes on a standard machine.

---

Overview

The Makara Data Pipeline automates the process of compiling, executing, and profiling programs using perf, aggregating the results into a structured dataset. It supports distributed execution, making it suitable for large-scale performance analysis or research in compiler optimization and program behavior characterization.

---

Motivation

We can use the data in the Makara project in several ways. For instance, this data lets us build cost models that guide compiler optimization heuristics. They also lets us predict the performance of programs on some architecture, given that we know this performance on another. The figure below illustrates this trend, showing that the number of instructions executed on an Intel Core i7-1355U has a strong linear correlation with the number of instructions that runs on an AMD Ryzen 5 3500U:

---

Results and Dataset

You can browse and download the available datasets of different archies here:

• 13th_Gen_Intel(R)_Core(TM)_i7-1355U.csv
• AMD_A12-9720P_RADEON_R7,_12_COMPUTE_CORES_4C+8G.csv
• AMD_Ryzen_7_9800X3D_8-Core_Processor.csv
• AMD_Ryzen_3_3200G_with_Radeon_Vega_Graphics.csv
• AMD_Ryzen_5_3500U_with_Radeon_Vega_Mobile_Gfx.csv
• AMD_Ryzen_5_5600G_with_Radeon_Graphics.csv
• AMD_Ryzen_Threadripper_7970X_32-Cores.csv
• Intel(R)_Core(TM)_i5-9300H_CPU__2.40GHz.csv
• Intel(R)_Core(TM)_i5-9400F_CPU__2.90GHz.csv
• Intel(R)_Core(TM)_i7-6700T_CPU__2.80GHz.csv
• Intel(R)_Xeon(R)_CPU_E5-2620_0__2.00GHz.csv

---

Dependencies

Ensure the following dependencies are installed on all machines participating in data collection:

• GCC – for compiling C/C++ programs
• Python 3.x – for orchestration and automation
• Linux perf – for collecting performance metrics

---

Installation

You can install the dependencies using your package manager. For Ubuntu/Debian-based systems:

sudo apt update
sudo apt install build-essential python3 linux-tools-common linux-tools-$(uname -r)

(If perf is not found after installation, you may need to create a symbolic link to /usr/bin/perf.)

You can test if perf is installed with

perf --version

Configuration

To allow perf to collect data without restrictions, please run the following one-time command:

sudo sysctl -w kernel.perf_event_paranoid=1

---

▶️ How to Run

To start the data collection process, simply execute:

python3 collect_data.py

Note: On a typical machine, such as an Intel i7 at 2.8 GHz, this script takes 5 to 6 minutes to run.

The Makara Data Pipeline will:

1. Compile all source programs;
2. Execute each one while recording performance statistics with perf;
3. Store the results in a structured results/ directory;
4. Automatically compress the collected data into a .zip archive for easy sharing.

Once the .zip file is created, please submit your results via the Makara - Results Submission form.

---