PHENIX GEANT3 Simulations (PISA)
at the UNM HPC
The New Mexico Center for Particle Physics
thomas@phys.unm.edu
I propose to utilize the Los Lobos system for three days (72 hours; all 512 CPUs) in order to perform a monte carlo simulation of the PHENIX Detector System using the PISA simulation code. This time period includes an approximately 30% contingency factor, which could also be fully utilized to produce even more events than my baseline target of 65,000. The simulated data generated by these runs will be transferred to Brookhaven National Laboratory and shared among the over 350 PHENIX collaborators, who will use it to analyze and interpret in various ways the real data sample being collected by the experiment this year. (For more details, see sections below.) Several test runs have already been successfully completed on Los Lobos, including a 100-event (1.6 day) "unit" run that could be replicated 2 x 512 times to produce a data set containing almost 100,000 events. This would be the shortest time to produce a simulated data set of that size in the PHENIX collaboration's history.
Preliminary calculations (see appendices) to demonstrate feasibility of this challenge calculation were performed using the PISA code on the National Computational Science Alliance/University of New Mexico RoadRunner and LosLobos superclusters. Computational resources were provided by a starter allocation awarded by UNM.
The PHENIX experiment is the most complex of the four experiments located at the Relativistic Heavy Ion Collider (RHIC) accelerator complex at the Brookhaven National Laboratory (BNL) on eastern Long Island. Our group, within the University of New Mexico's Center for Particle Physics, is collaborating on the PHENIX experiment and is responsible for a significant component of the Muon Tracking System hardware, as well as for related analysis software. The RHIC/PHENIX research program explores the detailed structure of the strong nuclear interaction. The primary focus is on discovery and characterization of a hadron<-->QGP (Quark-Gluon Plasma) phase transition, through high-energy heavy-ion collisions, and on measurements of the spin structure of the proton wave function using high-energy spin-polarized proton collisions. There is a document summarizing recent results from PHENIX. Other recent references can be found at the Los Alamos preprint archive.
The "PHENIX Integrated Simulation Application" (PISA) is based on the GEANT3 code, a detector description and simulation tool that has been under continuous development for more than 25 years at CERN, the European Organization for Nuclear Research in Geneva, Switzerland. GEANT3 contains an enormous amount of information about the physics of particle propagation and interaction in materials. There are documents giving an introduction to the concept of GEANT3 (see first link) and a brief review of its history. (There is a modern re-engineering of this code, GEANT4, still under development at CERN, which PHENIX plans to utilize in the future, after it has stabilized.)
There are many analyses that can be performed on a data set collected
with a large multi-purpose particle detector like PHENIX. Each analysis
typically makes use of at least one and often several types of monte
carlo-simulated data sets.
The PISA-simulated data set I propose to generate will be useful for many
different analyses of the real data, since it will be a relatively generic
one: the only "bias" is that I will simulate only head-on ("central")
collisions. These typically constitute only a few percent of the events
in a real data set, so this simulated data will have a statistical power
equivalent to a real data set several hundred times its size. Despite
this bias, it will reflect those features of the PHENIX detector that
any real data set exhibits, no matter what extra signals (such as a QGP
signature) might also be present.
By mixing it with other, more specialized simulated-signal data sets
that already exist, it will be used to understand features of signals
and backgrounds, to estimate signal-to-noise levels, and to study the
"acceptance" and efficiency for various signals. These analyses will
be performed on computers located at BNL, where the real data are
now being collected.
Heavy ion collision events, especially those that involve a "central"
(head-on) collision, are very CPU-intensive to simulate...
From a Los Lobos test of 100 "central" Au-Au events:
========================================================
Event characteristics: 22.79 CPU min/evt (Los Lobos)
3.04 MB/evt of output
ASSUMING 512 CPUs:
EVENTS TIME SPACE
-------- -------- -------
50K 1.6 days 150 GB
65K 2.0 days 200 GB
100K 3.1 days 300 GB
========================================================
Below are some pictures illustrating operation of the PISA code. First, about the geometry...
The pictures below are from an interactive run, done at the AHPCC,
simulating a single head-on ("central") gold-on-gold ion collision event
at a beam energy of 100 Giga electron Volts per nucleon ("GeV/A"), which
corresponds to a center-of-mass energy of 200 GeV per nucleon-nucleon
collision.
The event actually contains more than 9000 charged and neutral "primary"
particles - the original particles produced by the collision or by the
decays of very short-lived original particles. EACH of these primaries may
produce dozens or even hundreds of "secondaries" via interaction and/or decay
during propagation through the detector materials. These physical effects -
propagation, interaction, and decay - are what PISA is simulating.
For memory management purposes, the program breaks the complete event up into
"sub-events", chunks comprising 125 of the primary tracks, along with all the
resulting secondaries. The pictures below show only the results of the last
such sub-event, which is why the tracks only appear near the beam axis: these
happen to be the last tracks processed. Keep this in mind as you try to
envision the entire event, WITH OVER 9000 PRIMARY TRACKS AND THE MULTITUDE
OF RESULTING SECONDARIES, spread over all theta and phi angles! (I am still
working on making a digestible picture of an entire event.)
This single event took about 25 minutes to compute on a desktop 650 MHz
Pentium III system, part of the Truchas Cluster at the AHPCC...
