key: cord-0550749-dx3amsd7 authors: Kafkes, Diana; John, Jason St. title: BOOSTR: A Dataset for Accelerator Control Systems date: 2021-01-20 journal: nan DOI: nan sha: 252a2c7007e271d238f065b68b09eae75621fc0c doc_id: 550749 cord_uid: dx3amsd7 BOOSTR (Booster Operation Optimization Sequential Time-Series for Reinforcement) was created to provide cycle-by-cycle time series of readings and settings from instruments and controllable devices of the Booster, Fermilab's Rapid-Cycling Synchrotron (RCS) operating at 15 Hz. BOOSTR provides a time series from 55 device readings and settings which pertain most directly to the high-precision regulation of the Booster's Gradient Magnet Power Supply (GMPS). To our knowledge, this is the first known dataset of accelerator device parameters made publicly available. We are releasing it in the hopes that it can be used to demonstrate aspects of artificial intelligence for advanced control systems, such as reinforcement learning and autonomous anomaly detection. Tuning and controlling particle accelerators is challenging and time consuming. Even marginal improvements to accelerator operation can translate very efficiently into improved scientific yield for an experimental particle physics program. The data released here was collected in the hopes of achieving improvement in precision for the Fermilab Booster Gradient Magnet Power Supply (GMPS) regulatory system, which is detailed below. The Fermilab Booster receives a 400 MeV proton beam from the Linear Accelerator and accelerates it to 8 GeV through synchronously raising accelerator cavity radiofrequency and instigating a controlled magnetic field to steer the beam with combined-function bending and focusing electromagnets, known as gradient magnets. These magnets are powered by the GMPS, which operates on a 15 Hz cycle between a minimum current (at injection) and a maximum current (at beam extraction). The GMPS is realized as four power supplies, evenly distributed around the Booster, and each powers one fourth of the gradient magnets. The role of the GMPS regulator is to calculate and apply small compensating offsets in the GMPS driving signal in order to improve the agreement of the resulting observed minimum and maximum currents with their set points. Without regulation, the fitted minimum of the magnetic field may vary from the set point by as much as a few percent. At beam injection, a perturbation of only a percent is enough to significantly decrease beam transfer efficiency and thereby reduce the beam flux ultimately available to the high-energy particle physics experiments run at the lab. Disturbances to the magnet current can occur due to ambient temperature changes, other nearby high-power pulsed radio-frequency systems, and electrical ground movement induced by power supplies of other particle accelerators at the complex. The current GMPS regulation involves a PID (Proportional-Integral-Derivative) control scheme (see Figure 1 for schematic). The regulator calculates estimates for the minimum and maximum currents of the offset-sinusoidal magnetic field from the previous 15 Hz cycle. These values are then used to adjust the power supply program and decrease systemic error in the next cycle's current, such that it more closely matches the set point. Presently, the PID system achieves regulation errors corresponding to roughly 0.1% of the set value. Although some 200,000 entries populate the device database of Fermilab's accelerator control system 1 , the 55 device value time series presented here in BOOSTR 2 were collected in accordance with suggestion by Fermilab accelerator subject matter experts (SMEs). These values exhibit correlations with GMPS power supply perturbations. The full data were collected during two separate periods: from June 3, 2019 to July 11, 2019 -when the accelerator was shut down for regular maintenanceand from December 3, 2019 to April 13, 2020 -when the accelerator was shut down in response to the Covid-19 Pandemic. Data from a single day of BOOSTR was previously described in a Datasheet 3 . A proof-of-concept paper 4 (submitted to Physical Review Accelerators and Beams) used this subset of BOOSTR and demonstrated the viability of training a reinforcement learning agent to control GMPS regulation better than the existing PID system. Relative to the original Datasheet 3 , this manuscript is expanded with more SME input, describes more than 100 times more data, and includes documentation of validation not presented in the original Datasheet. A data collection node was created and set to request data at 15 Hz from the Data Pool Manager of the Accelerator Control Network (ACNET) 1 . The created scheme involved front-end nodes, each managing their respective devices, replying with timestamped values at the stated rate barring differences of clock speed, input-output (I/O) lag time variations due to network traffic fluctuations, and higher-priority interruptions from competing processes on the front-end node. These inconsistencies were later addressed through a time-alignment process discussed in the Data Processing Section. The collection node stored the data in a circular buffer approximately 10 days deep. A Python script managed by a nightly cron job polled the data collection node for the most recent midnight-to-midnight 24 hours of timestamped data for each of the 55 time series identified by SMEs. A second cron-managed script did the same for relevant accelerator control events issued in the same period. These event data correspond to important cycles achieved through controlling the devices at the accelerator. Event data were requested by a separate data collection node. Each day's data harvest was originally stored in HDF5 (Hierarchical Data Format Version 5) files. Any data instances missing from the parquet files released here were not included in the original data buffers from which this dataset was drawn. Each instance was created through a concatenation of each device's timestamp data table within every HDF5 file and then saved in parquet format. A similar procedure was undertaken for one of the accelerator control event signals polled, Event0C, as its broadcast is synchronized with the GMPS magnetic field minimum. Event0C was collected to correct a problem in the observed sampling frequency: there was an issue of the sampling of each device being nominally at 15 Hz, but in reality synchrony was demonstrably imperfect, and the time intervals between successive timestamps display varying lags. Since Event0c serves as the baseline or heartbeat of the Booster at approximately 15 Hz and is synchronized with the smoothly varying electrical current GMPS regulates, we used Event0c to time-align our data. The alignment approximates the data available to the GMPS regulator operating in real time. We used the GMPS-synchronized Event0C's timestamp as the moment to begin forward inference, taking the value for each device time series which had the most recent corresponding timestamp. In practice, this required timestamp-sorted series for each device and finding the most recent device value, relative to Event0c timestamp, in a lookback window equal to the maximum interval between device timestamps (necessarily excluding the five month gap between our two data collection periods). This time-alignment step was run over the whole dataset in multiple parallel processes using Apache Spark. Notably, the data recorded for Event0c was missing the period from July 1 to 11, 2019. Therefore aligning on this variable discarded some of the data collected during our first period of collection. The data release is stored on Zenodo 2 . Each instance is a zip compressed parquet of one of the 55 aligned time series with columns corresponding to the aligned time stamp, original time stamp, difference between time stamps, and the reading/setting value 2 . The original timestamp and time difference is included to demonstrate the mechanics of our alignment process and enable a check for reproducibility. All timestamps are in Greenwich Mean Time (UTC). Our data release contains device data from each of the four gradient magnet power supplies, the GMPS PID regulator, and the Main Injector, where the beam is directed after acceleration via the Booster. Minimum and maximum current information readings and settings, the feedback and transductor gain settings, and the feed-forward start trigger are collected as part of the current PID regulation scheme. The "inhibit" value controls whether the GMPS regulator accepts settings changes for parameters other than the minimum and maximum current, such as the gain settings (any positive value will prevent changes). Additionally,˙ B, the rate of change of the magnetic field, is recorded as a proxy for the magnetic field we are interested in regulating. Timing information derived from˙ B = 0 synchronizes the current PID regulator system. We acknowledge that the ACNET parameter names are by no means standardized across different particle accelerators and that they will appear especially abstruse for those well-versed in control systems who are new to working with accelerators. In Table 1 , we detail explanations of each of the parameters read from devices (devices whose first letter is followed by :) and indicate whether the device setting was included in the dataset (devices whose first letter is followed by _ and appear in Setting column), since describing these corresponding pairs would be redundant. In Figures 2 and 3 , we visualize metadata trends for each "nonconstant" parameter in each data collection period (see Table 2 for a list of values we considered to be virtually unchanging within the two periods) and also provide the mean and standard deviation of device readings across the two collection periods in Table 1 . Furthermore, Table 1 includes dates missing in the data recorded for each reading. As a reminder the data were collected during two separate periods: from June 3, 2019 to June 30, 2019 (July is missing due to time-alignment 2/11 with Event0C) and from December 3, 2019 to April 13, 2020. Finally, in Figure 4 we demonstrate the centrality of each recorded parameter with a heatmap of histogram values. Additionally, we provide the PID regulator status values B|GMPSSC (ACNET status parameters include |), whose 16 bits contain various motherboard states. Here we are concerned with bit 3, which indicates whether or not the GMPS regulator was on (1), and bit 7, which indicates whether the Booster is in its normal alternating current (AC) mode (1) or "coasting beam" direct current (DC) mode at constant beam energy (0). Unlike the rest of the devices, this status value is presently recorded at only 1 Hz because it was not included in our initial data node request and was relegated to an archived data node at lesser frequency. While the same time-aligning described above was applied to align B|GMPSSC, due to the slow sampling rate, we caution the user to refer closely to the original timestamp such that they might make decisions about whether to use data when GMPS was off and to inform them of potential problems when interpolating in a region immediately before or after a status change. See Figure 5 for more details on B|GMPSSC values. In order to verify the quality of this dataset, we pored over the electronic logbook (Elog) 5 that Fermilab Booster technicians and operators use to record changes to device settings as well as observations while in operation. We used these Elog entries to authenticate our data's viability across timescales. First, we used the Elog to corroborate expert acknowledgement of the major spikes observed in Figures 2 and 3 . These outlier changes, typically seen in the value's mean and standard deviation, represented major changes made on that specific day, including when the Booster was switched from alternating current (AC) to direct current (DC) mode (see Figure 5 ) as well as when the GMPS regulator was turned off altogether. These reconciliations are presented in Table 3 . Furthermore, we pinpoint changes in the AC vs. DC settings according to the Elog 5 for June 24, 2019 and March 11, 2020 in Figure 6 . Here applying a bitmask reveals that a B|GMPSSC value of 159 indicates AC mode/GMPS on, while 31 indicates DC mode/GMPS on, and 407 indicates AC mode/GMPS off. In this figure, the plotted timestamps were offset to Central Time in order to align with times given in the Elog, which were not recorded in UTC. On June 24, the trace of B|GMPSSC clearly shows GMPS regulation briefly switching off before commencing DC studies from 8:00 AM -6:00 PM with a value of 31, then being turned back to 159. On March 11, B|GMPSSC is at 159 before 6:00 AM, off at 407 from 6:00 -9:50 AM, and then is set to AC mode from 9:50 AM -12:45 PM, to DC mode from 12:45 -3:49 PM, and back to AC mode for the rest of the day, as per the Elog 5 . The close correspondence of these changes in our data to the recorded actions and observations of Booster personnel boost our confidence in the quality and relevance of the collected dataset. Additionally, we plot settings changes on March 10 and 11 documented in the Elog 5 in Figure 7 . The blip in B:ACMNPG from 6.5 to 13.5 is visible as is the slight decrease in B_VIMIN around 4:00 PM CST, which were mentioned in Table 3 . BOOSTR could be used to train various control networks for accelerator regulation, to construct "digital twins" of the Fermilab Booster regulator's control environment, or to develop anomaly detection/categorization capabilities. Please note: there are no legal or ethical ramifications of using this data as it was collected from a machine, and not collected from or representative of people. In the future, the dataset could feasibly be expanded to include more of the 200,000 available ACNET system parameters and therefore be used to control, mimic, or monitor further aspects of the particle accelerator complex. One could argue that this initial dataset might become the foundation upon which substantial fine-tuning of particle accelerators could depend. The preprocessing code is available here. When using BOOSTR data, the authors recommend ordering by time immediately, as the parquet files do not store the data entries sequentially 2 . Table 3 entries for June 24, 2019 and March 11, 2020 (timestamps were put in Central Time to align with Elog). Recall: a value of 159 indicates AC study/GMPS on, 31 indicates DC study/GMPS on, and 407 indicates AC study/GMPS off. These traces display this value at a much greater granularity than Figure 5 . Table 3 entries for March 10, 2020 and March 11, 2020: B:ACMNPG changed from 6.5 to 13.5 and B_VIMIN decreased from 103.418 to 103.386 (timestamps were put in Central Time to align with Elog). The sudden large increase in B_VIMIN from 12:45 -3:49 PM CST to a value off the plotted region corresponds to the DC mode observed in Figure 6 . The fermilab accelerator control system BOOSTR: A Dataset for Accelerator Control Systems BOOSTR: A Dataset for Accelerator Control Systems Real-time artificial intelligence for accelerator control: A study at the fermilab booster Fermilab electronic logbook Dates Collected Variable Metadata B:ACMNIG B:ACMXIG B:VINHBT B:VIPHAS Dates Collected Variable Metadata B:ACMNPG B:ACMXPG B:GMPSFF B:IMAXXO B:IMINER B:NGMPS Dates Collected Variable Metadata B:DCPG B:IPHSTC B:LINFRQ B:PS2VGM B:PS3VGM B:PS4VGM Dates Collected Variable Metadata B:PS1VGM B:PS1VGP B:PS2VGP B:PS3VGP B:PS4VGP B:VIMIN B_VIMIN Dates Collected Variable Metadata B:GMPS1V B:GMPS2V B:GMPS3V B:GMPS4V Dates Collected Variable Metadata I:IB I:MDAT40 I:MXIB The graphs show the mean for each variable Dates Collected Variable Metadata B:ACMNIG B:ACMXIG B:VINHBT B:VIPHAS Dates Collected Variable Metadata B:ACMNPG B:ACMXPG B:GMPSFF B:IMAXXO B:IMINER B:NGMPS Dates Collected Variable Metadata B:DCPG B:IPHSTC B:LINFRQ B:PS2VGM B:PS3VGM B:PS4VGM Dates Collected Variable Metadata B:PS1VGM B:PS1VGP B:PS2VGP B:PS3VGP B:PS4VGP B:VIMIN B_VIMIN Dates Collected Variable Metadata B:GMPS1V B:GMPS2V B:GMPS3V B:GMPS4V Dates Collected Variable Metadata I:IB I:MDAT40 I:MXIB We are extremely grateful for Brian Schupbach and Kent Triplett for lending their Booster technical expertise, without which we could not have validated our dataset. Additionally, we would like to acknowledge Burt Holzman for guidance on The authors declare no competing interests.