Performance studies #142

plexoos · 2025-08-04T17:04:05Z

No description provided.

outdated suggestion

github-actions

Cpp-linter Review

Used clang-format v18.1.3

Click here for the full clang-format patch

diff --git a/src/g4appmt.h b/src/g4appmt.h
index f0f0020..88258fc 100644
--- a/src/g4appmt.h
+++ b/src/g4appmt.h
@@ -125,2 +125 @@ struct PhotonHit : public G4VHit
-        G4cout << "Detector id: " << fid << " energy: " << fenergy << " nm"
-               << " time: " << ftime << " ns"
+        G4cout << "Detector id: " << fid << " energy: " << fenergy << " nm" << " time: " << ftime << " ns"

Have any feedback or feature suggestions? Share it here.

src/g4appmt.h

outdated suggestion

plexoos · 2025-12-15T18:45:36Z

optiphy/tools/run_performance.py

 /process/optical/cerenkov/setStackPhotons {flag}
 /run/initialize
-/run/beamOn 50000
+/run/beamOn 500


Why was this number reduced? What small GPU RAM are you referring to? For consistency, we should be able to reproduce the published results using the same number of events.

In this repository, the event count is set to a conservative default because the goal of this example is to help users get a first successful run with EIC-Opticks and understand the workflow, not to serve as a paper quality reproduction test. Since users run on a wide range of hardware, a high default can fail on GPUs with limited VRAM and lead to crashes/segfaults that are frustrating to debug as a first step.

Also, because we swapped the geometry, the performance and scaling will not match the paper one-to-one anyway, the achievable speedup depends on geometry and workload characteristics.

To keep things consistent and reproducible, the README explicitly notes that users should set the event count themselves.

So, what kind of speedup do you see with this geometry? Is it far from the public numbers?

If I use pfrich_min_FINAL.gdml and the patch for open cones, should I expect to reproduce the numbers in the paper?

I did not check. Can you check how much speedup you get?

Yes. I tested it and sent you the example result Dec 8. 3:58 PM in mattermost chat.

This reverts commit f3d977e.

github-actions

Cpp-linter Review

Used clang-format v18.1.3

Click here for the full clang-format patch

diff --git a/CSG/csg_intersect_leaf_newcone.h b/CSG/csg_intersect_leaf_newcone.h
index 9801725..3bc042d 100644
--- a/CSG/csg_intersect_leaf_newcone.h
+++ b/CSG/csg_intersect_leaf_newcone.h
@@ -101 +101 @@ void intersect_leaf_newcone( bool& valid_isect, float4& isect, const quad& q0, c
-    const float t_cand = fminf(roots) ;
+    const float t_cand = fminf(roots);

Have any feedback or feature suggestions? Share it here.

CSG/csg_intersect_leaf_newcone.h

plexoos · 2025-12-15T23:00:15Z

Here’s what I’m getting when I run this branch as it is right now:

c6b7d2a3be6c:/src/eic-opticks# run-performance -g ./tests/geom/pfrich_min_FINAL.gdml -o ./out_pfrich_min_FINAL/rel/

ggalgoczi · 2025-12-16T01:36:44Z

You need to modify the eic-opticks code by running the cone modification script , building eic-opticks and the executable again if you want to run the performance script on pfrich. Did you do all? If yes I will check this out.

plexoos · 2025-12-16T14:37:39Z

You need to modify the eic-opticks code by running the cone modification script , building eic-opticks and the executable again if you want to run the performance script on pfrich. Did you do all? If yes I will check this out.

A cone modification script? Do you mean there’s something in addition to applying this patch 04f4d4a that I should run?

plexoos · 2025-12-16T15:45:37Z

Okay, I rebuilt everything from scratch to make sure there were no leftovers from previous tests, and I’m still seeing a speed-up factor of about ~30 between single-threaded G4 and OptiX, as shown in the plot above. Please check.

ggalgoczi · 2025-12-16T16:42:58Z

Why did you do those modifications in 04f4d4a ? It is different from I did when I got the code working. See my modification script:
https://github.com/BNLNPPS/esi-g4ox/blob/PerformanceAnalysis/scripts/cone_opticks_intersection_modify.py

Now I am seeing the following speedup: 170x speedup wrt to single thread G4 and 10x wrt to 20 MT G4. How did I get these number:

I fixed the cone to what I mentioned before.
I modified the momentum of electrons to 5 MeV that was used in the paper.
I ran optiphy/tools/run_performance.py and I checked files in timing_geant.txt and timing_optix.txt.
Make sure to simulate 50k electrons.
You can check if you set things up correctly with running the G4 code and seeing similar numbers when running:

time simg4oxmt -g /src/eic-opticks/tests/geom/pfrich_min_FINAL.gdml -m /src/eic-opticks/run.mac

Simulation time: 0.519579 seconds
Opticks: NumCollected:  540880
Opticks: NumCollected:  46927986
Opticks: NumHits:  8865899

Also if you modify eic-opticks or the example source code do not forget to build, eg. in my case I used:

cmake -S $OPTICKS_HOME -B $OPTICKS_BUILD -DCMAKE_INSTALL_PREFIX=$OPTICKS_PREFIX -DCMAKE_BUILD_TYPE=Release && cmake --build $OPTICKS_BUILD --parallel --target install

This reverts commit 04f4d4a.

outdated suggestion

github-actions

Cpp-linter Review

Used clang-format v18.1.3

Click here for the full clang-format patch

diff --git a/CSG/csg_intersect_leaf_newcone.h b/CSG/csg_intersect_leaf_newcone.h
index 01a9845..fe8068e 100644
--- a/CSG/csg_intersect_leaf_newcone.h
+++ b/CSG/csg_intersect_leaf_newcone.h
@@ -101,2 +101,2 @@ void intersect_leaf_newcone( bool& valid_isect, float4& isect, const quad& q0, c
-    const float t_cand = fminf(roots) ; 
-    
+    const float t_cand = fminf(roots) ;
+
@@ -106,11 +106,3 @@ void intersect_leaf_newcone( bool& valid_isect, float4& isect, const quad& q0, c
-        float3 intersection_point = make_float3(
-            o.x + t_cand * d.x,
-            o.y + t_cand * d.y,
-            o.z + t_cand * d.z
-        );
-        float3 n = normalize(make_float3(
-            intersection_point.x,
-            intersection_point.y,
-            (z0 - intersection_point.z)*tth2
-        ));
-
+        float3 intersection_point = make_float3(o.x + t_cand * d.x, o.y + t_cand * d.y, o.z + t_cand * d.z);
+        float3 n =
+            normalize(make_float3(intersection_point.x, intersection_point.y, (z0 - intersection_point.z) * tth2));

Have any feedback or feature suggestions? Share it here.

CSG/csg_intersect_leaf_newcone.h

This reverts commit 8936907.

outdated suggestion

Move gun so the opticks_raindrop.gdml has photons created

ggalgoczi · 2025-12-23T21:57:45Z

I confirm that when moving the particlegun to 0,0,0 mm (I just pushed this change into the branch) and running
optiphy/tools/run_performance.py

I see performance data created in timing_optix.txt and timing_geant.txt files. Namely the simulation times are respecitvely 0.8 s and 11.7 s. Let me know @plexoos if we can close this PR.

ggalgoczi · 2026-01-02T17:22:17Z

@plexoos could you take a look?

plexoos · 2026-01-06T22:00:55Z

Here is a basic test I use in CI to compare the G4 and OptiX outputs:
https://github.com/BNLNPPS/eic-opticks/blob/main/tests/compare_ab.py

Can we run it on the output from these performance jobs?

Fix ordering so EIC-Opticks hits match up

ggalgoczi · 2026-01-06T22:25:53Z

Since we use the "Minimal" setting there are no such record files in my understanding, are there?

I found a bug in the geometry. The ordering of the surfaces was incorrect, fixed it in 4101e08

Also I added printing out number of Geant4 hits in the executable in 7fd2968

Now when I run 2 events I get matching number of hits from EIC-Opticks and Geant4:

G4WT0 > PhotonSD::EndOfEvent Number of PhotonHits: 14198
G4WT0 > PhotonSD::EndOfEvent Number of PhotonHits: 11896

Opticks: NumHits: 26076

We agreed that this the requirement to merge this PR. If you need any more details please open an other PR.

plexoos · 2026-01-06T22:45:32Z

Have you verified that the hits have the same or similar coordinates?

ggalgoczi · 2026-01-06T22:47:33Z

No, this PR should be for performance study, not a full validation.

ggalgoczi · 2026-01-06T22:49:27Z

If you want feel free to open a "raindrop geometry validation" PR.

plexoos · 2026-01-06T23:10:53Z

How do we know that these jobs produce reasonable hits?

ggalgoczi · 2026-01-06T23:13:57Z

Define reasonable

plexoos · 2026-01-07T00:18:05Z

Sure — here are a couple of metrics we could use, for example: The hit coordinate distributions statistically agree and are constrained by the sensitive detector

This comment was marked as resolved.

Sign in to view

plexoos force-pushed the performance-studies branch from 5c85f89 to 7b8a75d Compare August 18, 2025 23:55

plexoos force-pushed the performance-studies branch from 5142852 to b60690d Compare September 3, 2025 14:48

github-actions bot previously requested changes Sep 3, 2025

View reviewed changes

src/g4appmt.h Outdated Show resolved Hide resolved

plexoos force-pushed the performance-studies branch 2 times, most recently from c50baae to c9db7a7 Compare October 13, 2025 17:12

plexoos force-pushed the performance-studies branch from c9db7a7 to fb9a812 Compare November 7, 2025 17:54

plexoos added 5 commits December 5, 2025 15:28

docs: update README

d43b18d

move code from esi-g4ox and rename files

c22acd1

build: add new executable simg4oxmt

752f10a

add run_performance.py script

7940314

add geom file pfrich_min_FINAL.gdml from esi-g4ox

22ee1ae

plexoos commented Dec 15, 2025

View reviewed changes

plexoos added 4 commits December 15, 2025 14:21

docs(README): fix formatting/line-wrapping

96e1e8d

style(opticks_raindrop.gdml): whitespace and indentation cleanup only

ff34c93

workaround for incorrect ray intersections with open cones

04f4d4a

Revert "Move particle gun to create photons in raindrop geom"

8be5074

This reverts commit f3d977e.

github-actions bot previously requested changes Dec 15, 2025

View reviewed changes

CSG/csg_intersect_leaf_newcone.h Outdated Show resolved Hide resolved

plexoos added 2 commits December 16, 2025 11:52

Revert "workaround for incorrect ray intersections with open cones"

6e16acb

This reverts commit 04f4d4a.

workaround for incorrect ray intersections with open cones - 2

8936907

github-actions bot previously requested changes Dec 16, 2025

View reviewed changes

CSG/csg_intersect_leaf_newcone.h Show resolved Hide resolved

CSG/csg_intersect_leaf_newcone.h Outdated Show resolved Hide resolved

Revert "workaround for incorrect ray intersections with open cones - 2"

a4bb33f

This reverts commit 8936907.

Update position in GeneratePrimaries method

683cf3a

Move gun so the opticks_raindrop.gdml has photons created

ggalgoczi added 2 commits January 6, 2026 17:19

Reorder physvolref elements in opticks_raindrop.gdml

4101e08

Fix ordering so EIC-Opticks hits match up

Log number of PhotonHits at end of event

7fd2968

Performance studies #142

Are you sure you want to change the base?

Performance studies #142

Uh oh!

Conversation

plexoos commented Aug 4, 2025

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Cpp-linter Review

Used clang-format v18.1.3

Uh oh!

Uh oh!

plexoos Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

ggalgoczi Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

plexoos Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

ggalgoczi Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Cpp-linter Review

Used clang-format v18.1.3

Uh oh!

Uh oh!

plexoos commented Dec 15, 2025

Uh oh!

ggalgoczi commented Dec 16, 2025

Uh oh!

plexoos commented Dec 16, 2025

Uh oh!

plexoos commented Dec 16, 2025

Uh oh!

ggalgoczi commented Dec 16, 2025

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Cpp-linter Review

Used clang-format v18.1.3

Uh oh!

Uh oh!

Uh oh!

ggalgoczi commented Dec 23, 2025

Uh oh!

ggalgoczi commented Jan 2, 2026

Uh oh!

plexoos commented Jan 6, 2026

Uh oh!

ggalgoczi commented Jan 6, 2026

Uh oh!

plexoos commented Jan 6, 2026

Uh oh!

ggalgoczi commented Jan 6, 2026

Uh oh!

ggalgoczi commented Jan 6, 2026

Uh oh!