key: cord-0255404-lnpdsrtk
authors: Anugu, Narsireddy; Brummelaar, Theo ten; Turner, Nils H.; Anderson, Matthew D.; Bouquin, Jean-Baptiste Le; Sturmann, Judit; Sturmann, Laszlo; Farrington, Chris; Vargas, Norm; Majoinen, Olli; Ireland, Michael J.; Monnier, John D.; Mourard, Denis; Schaefer, Gail; Gies, Douglas R.; Ridgway, Stephen T.; Kraus, Stefan; Petit, Cyril; Tallon, Michel; Lim, Caroline B.; Berio, Philippe
title: CHARA Array adaptive optics: complex operational software and performance
date: 2020-12-21
journal: nan
DOI: 10.1117/12.2561560
sha: 5bfdcd4c86a3a7fa366641b296a426681ad2943a
doc_id: 255404
cord_uid: lnpdsrtk

The CHARA Array is the longest baseline optical interferometer in the world. Operated with natural seeing, it has delivered landmark sub-milliarcsecond results in the areas of stellar imaging, binaries, and stellar diameters. However, to achieve ambitious observations of faint targets such as young stellar objects and active galactic nuclei, higher sensitivity is required. For that purpose, adaptive optics are developed to correct atmospheric turbulence and non-common path aberrations between each telescope and the beam combiner lab. This paper describes the AO software and its integration into the CHARA system. We also report initial on-sky tests that demonstrate an increase of scientific throughput by sensitivity gain and by extending useful observing time in worse seeing conditions. Our 6 telescopes and 12 AO systems with tens of critical alignments and control loops pose challenges in operation. We describe our methods enabling a single scientist to operate the entire system.

The CHARA Array 1-3 is the world longest baseline optical or near-infrared interferometer with six 1-meter diameter telescopes and baseline up to B = 331 m delivering sub-millisecond angular resolution (e.g., λ/2B ∼ 0.6mas at H-band). The six telescopes of the CHARA Array are arranged in a Y-shaped configuration, and they provide 15 interferometric baselines and 20 closure phases. The CHARA Array has been a very successful observatory so far by delivering landmark results in the areas of stellar imaging, 4, 5 binaries, 6, 7 expansion phase of a nova explosion, 8 and stellar radii measurements 9 with a sub-millisecond angular resolution. However, to achieve the challenging goals of imaging faint targets such as young stellar objects and active galactic nuclei, and measuring radii of TESS exoplanet host targets require higher instrument sensitivity.

Since obtaining the first-fringes, the beam combiner laboratory of the CHARA Array is equipped with fast tip-tilt systems, 10 although the original goal was to deploy adaptive optics systems (AO) for the telescopes and in the laboratory. Since 2012, the CHARA Array adaptive optics programs are under development in the beam combiner laboratory and at the telescopes. They are developed in two phases, accommodating funding realities (PI: Theo ten Brummelaar). These adaptive systems enable higher throughput by increasing the Strehl ratio of each beam. The light collected by the telescopes is injected into single-mode fibers, is a standard practice currently in the field, for spatial filtering 11 before the beam combination inside a beam combiner (e.g., MIRC-X, [12] [13] [14] and SPICA 16, 17 ). The sensitivity of optical interferometers is an order of a magnitude lower than those in most classic telescopes is due to:

• Too many mirrors in the light path from the sky to a beam combiner, e.g., the CHARA Array contains 21 mirrors. Furthermore, telescope and instrument vibrations play a huge role.

• Non-common-path errors in an interferometer are orders of magnitude greater than those in the most telescope AO systems.

• Interferometers combine coherent light to make fringes, but because of atmospheric turbulence, only short coherent integrations are allowed -smaller than the atmospheric coherence time, typically 10-25ms.

• Atmospheric turbulence severely limits the light coupling into the single-mode fibers with a distorted point spread function. 11 For a pupil with a central obstruction of 25% of the pupil diameter, the expected coupling efficiency into a single-mode fiber is 0.69 times the Strehl ratio. 18 To improve the instrument throughput, we need high Strehl ratio beams at the tip of single-mode fibers.

Without adaptive optics, the sensitivity of observing the faintest star with any beam combiner at the CHARA Array is lower than eight magnitudes in H or K-bands.

The CHARA Array has twelve adaptive optics systems in total :

• Tel-AO (in Phase II) [19] [20] [21] [22] Six at the telescope sites to correct the tip-tilt and high-order aberrations induced by the atmospheric turbulence. These AO systems provide high Strehl ratio beams to the vacuum tubes, which transport the light to the beam combiner lab.

• Lab-AO (in Phase I) 23, 24 Six at the beam combiner lab to correct significant non-common path aberrations that arose between the telescope and the beam combiner instruments. The non-common path aberrations errors are due to several hundred meters of the light path in between the telescope and the beam combiner, including several mirrors and delay lines.

The Lab-AO systems were commissioned already in Phase I 23 and are in operation since 2014 for science observations as reported in the previous SPIE proceedings. [21] [22] [23] [24] The Tel-AO systems have been commissioned since 2018, in incremental steps, one telescope after another. The preliminary optical design of the Tel-AO systems is presented in 2018 SPIE proceedings. 19 In this presentation, we report the software architecture, operation, and results.

A few challenges we face in the operation of the twelve adaptive optics systems are:

• Obtain high-Strehl ratios by making a good AO correction For this, the software development requires acquiring low-latency frames from the wavefront sensor camera, computation of slopes, computation of control matrix, correction of high-order aberrations with the ALPAO deformable mirrors (DM), and tip-tilts with the telescope M2.

• Alignment between Tel-AO and Lab-AO The Lab-AO is connected to the Tel-AO using a metrology system. The Lab-AO wavefront sensor (WFS) records beacon light sent from the telescope that uses a wavelength outside that of the science band of interest to measure the non-common path aberrations. 19 A dichroic at the telescope reflects some visible wavelengths of starlight into the Tel-AO WFS, and it also sends the beacon light into the Lab-AO WFS. The beacon light must follow the same light path as the star. We require an automatic alignment in pupil and focus.

• Complexity in operation Control of six telescopes and twelve adaptive optics systems by an operator is a challenge. Several Graphical User Interface (GUI)s execute the control of these systems. Each system has its own GUI, for instance, GUI for the telescope, acquisition camera, telescope tip-tilt, Tel-AO, and Lab-AO. Many GUIs invite inefficiency and error. Our challenge is to present a manageable interface for the operator. For example, the Tel-AO and Lab-AO systems alone require six monitors to fit their GUIs. For each slew to a new star target, the sequence includes the telescope pointing and tracking, star acquisition with the acquisition camera, locking the closed-loops of the tip-tilt, Tel-AO, and Lab-AO. We required an efficient software system and an integrated, stable, and easy operable GUI for each telescope to do all these steps in a small amount of time to maximize the time for the fringe finding and recording. Figure 1 shows the Tel-AO and Lab-AO systems as built. A brief description of the optical design of each system is followed below. 

The Tel-AO is designed to correct the atmospheric turbulence above the telescope and improve the Strehl ratio of the beams in H or higher wavelengths. Each Tel-AO system consists of (a) a 7 × 7 lenslet based Shack-Hartmann wavefront sensor and a 500 Hz fast frame rate and sub-electron readout noise, Andor 897 EMCCD camera 21 and (b) an ALPAO voice-coil based deformable mirror with 61 actuators. 19 The Tel-AO DMs are placed in the position of the M4 mirror to limit new reflections to the beam train. This mirror position is at 45 degrees, and this orientation results in an ellipse pattern pupil. The DM and the Shack-Hartman match this ellipse shape. The Tel-AO has three dichroic glasses: BARE, VIS, and YSO. Based on a science requirement, an appropriate dichroic filter is chosen to maximize flux into the Tel-AO WFS. Table 1 presents the dichroic filter and their percentage of share into the Tel-AO WFS.

The expected performance of Tel-AO is (i) more than one magnitude sensitivity improvement in good seeing conditions (Fried parameter r 0 > 15cm at λ = 0.5µm), and (ii) 1-3 magnitude sensitivity improvement for 5 < r 0 < 15cm and that means three times of more high quality observing time. 

The Lab-AO systems are placed after the delay lines in the beam combiner laboratory and just before the beam combiner instruments. 23 The Lab-AO is intended to correct for non-common path errors in the system, which are significant as there are several hundred meters of the path in each beam, each including several mirrors and delay line carts.

To reduce the cost of the Lab-AO systems, use off-the-shelf DM (OKO MMDM with 37 actuators) and 6 × 6 lenslet imaged with a USB CCD camera. Since the Lab-AO uses the beacon from the telescope, there is enough light, and a high-readout noise camera is not a problem. The performance of the Lab-AO systems are not at the level of a standard adaptive optics system (see Table 2 and 3), and as it is used only for quasi-static aberrations: (i) it has a lower number of actuators, and its WFS operates at a lower speed 100Hz, and (ii) The measured wavefront errors are in a low order compared to the Tel-AO WFS estimated wavefront errors.

The computer server plays a critical role in the adaptive system's performance. It needs to read wavefront sensor images with a low-latency and process them for wavefront slopes and compute the command matrix without much additional delay. The Tel-AO computer hardware is entirely commercial off-the-shelf (COTS), utilizing standard components. There are six computers, one for each telescope, and the fundamentals of these computers and data acquisition frame grabbers are identical, meaning they use the same motherboard, processor, and RAM. The computers' selection is driven by low-power consumption as the WFS camera is placed inside the telescope The camera link server reads the frames from the WFS Andor EMCCD camera through BitFlow Neon frame grabber and writes them to an instrument shared memory in a circular buffer so that the main WFS main server uses them. The WFS main server reads the frames from the instrument shared memory and calculates the wavefront slopes from each new science frame. Next, the DM commands are computed by multiplying the slopes matrix with the slopes-to-actuator reconstructor matrix. The tip/tilt is computed by taking the average of slopes and then sent to telescope M2 for correction with appropriate gain. The rest higher-order corrections are sent to the ALPAO deformable mirror. The wavefront sensor images and deformable mirror shapes are sent to the client GUI for monitoring and operation.

The software development of the Tel-AO systems (see Figure 2 ) involves acquiring low-latency wavefront sensor images, computation of slopes, 25 computation of control matrix, correction with the ALPAO deformable mirrors, and telemetry data recording. Furthermore, the software handles the automated stabilization of pupil and beacon drifts. We follow a modular design, whereby multiple processes/threads interact through shared memory to process the camera data at a full-frame rate with minimum latency.

We realized the default USB-based data acquisition of the Andor camera is missing frames (see Figure 3 ), and as a solution, we implemented a low latency frame grabbing solution with a BitFlow Neon CLB frame grabber. Before trying BitFlow frame grabber, we tried Matrox Radient eV-CL frame grabber, and it did not work as the Andor camera link out is non-standard in the market: base configuration, 3-tap interface, and 16-bit greyscale. We were able to find another working frame grabber, Epix, but the latency jitter was high compared to the BitFlow Neon CLB frame grabber. The frames produced by the camera are written to an instrument shared memory, and those are accessed directly by the main wavefront sensor server. The main server computes slopes from raw images, commands from slopes matrix, and controls the deformable mirror for the correction of high-order aberrations and the telescope M2 mirror for the correction tip-tilts (see Figure 2 ).

In Figure 3 , we compare the USB based data acquisition and camera-link based data acquisition. The time difference between two consecutive frame grabs is recorded 1000 times. In a low latency situation, one expects to get one frame for each frame grab request, and the time difference between two consecutive grabs is more or less equal. In a jitter frame grabbing situation, we get more than one frame in a grab, and the time difference between two consecutive grabs is not the same (sometimes considerable big time and some less time). The USB data acquisition has many jitters in frame arrival, and the average consecutive frame arrival is for every 10ms for an integration time of 3ms. In the worst case, a USB frame grab gets eight frames in a chunk (i.e., eight frames of delay, which means a delay between consecutive frames is more significant than 24ms). In the camera-link case, there is jitter in the frame arrival but very small -average consecutive frames arrival for every 3.1ms for the integration of 3ms. Here we do not report the pure delay.

We list below the Andor functions used in our software. These were critical in getting a low-latency arrival of frames from the Andor camera; otherwise, we noticed an additional pure delay of two frames, at 2ms integration. The measure of pure delay of the WFS Andor camera is attempted by sending a series of impulse pokes on the ALPAO deformable mirror and noticing the resulting shapes on the wavefront sensor, and computing the time delay. The known deformable mirror delay is subtracted from this measurement. For a 2ms integration time, we measure 1.8ms pure delay, which is the sum of the camera frame transfer time and readout time. For comparison, this is almost similar to NAOMI, 26 and their reported value is 1.54ms. They use the same Andor 897 camera, but the difference of lower 0.26ms maybe due to the usage of their hard real-time operating machine for frame grabbing than our COTS machine.

-> 0.5 microseconds hspeed(2) -> 10MHz SetFrameTransferMode (1) -> readout in Frame Transfer Mode SetReadMode(4) -> readout mode image SetIsolatedCropMode(1, 90, 90, 1, 1) -> read cropped 90x90 pixels with starting vertical and horizontal bins SetIsolatedCropModeType(1) -> low latency mode saves 2 frames of delay SetAcquisitionMode (5) -> acquisition run till abort StartAcquisition() -> starts the acquisition

The influence functions of all DM actuators were measured in a standard method using the wavefront sensor. Each actuator is "poked" and the resulting influence on the wavefront is measured. The actuator poking was done in positive and negative directions to minimize the measurement's noise by taking the average of both amplitudes. Furthermore, 100 wavefront measurements were taken for each poke to average out and reduce the noise of the WFS detector.

A challenge for telemetry data recording is management since we have twelve systems. We only save telemetry data when requested by the user as all the AO systems generate more than 1TB of data per night. An important feature of our telemetry data recording and saving is that the data is recorded into memory and saved to disk without interrupting priority tasks. The telemetry data includes time-stamped wavefront sensor images, measured slopes, aberrations, and actuator-to-slopes reconstructor matrix. This telemetry data used to estimate the performance of the AO system, and as well as it is used to correlate with science fringe data during postprocessing. Figure 3 . USB data data acquisition of frames (black color) in a comparison to the camera-link data acquisition (blue color). The integration is 3ms but the data points are the time difference between two consecutive frame grabs. The USB acquisition has a lot of jitters, in a worst-case, a grab gets 8 frames in a chunk (i.e., 8 frames of delay). In the case of camera-link, there is also jitter but very small. Figure 4 . A photograph of the main observing GUI at the CHARA Array, obsgtk. Each obsgtk controls one telescope and has all the controls for the telescope and its acquisition camera, tip-tilt, Tel-AO, and Lab-AO, including the real-time displays of the acquisition camera and wavefront sensors of Tel-AO and Lab-AO. This GUI is optimized for screen space -all six GUIs fit in a single screen monitor, time-efficient usage, and stability by keeping only the required observing controls together. This photograph is taken with telescope beacon was ON.

The adaptive optics systems software is built leveraging on the CHARA Array client/server architecture. 1 Servers publish the information, and appropriate clients subscribe to the servers to get the information. We do not have a central database system. The hardware (cameras, DMs, and other actuators) is controlled by the servers installed at the site of AO systems. We use GNU Tool Kit (GTK) Graphical User Interface (GUI) clients installed on the computers at the control room to send user configurations to the servers and monitor the real-time display of acquisition cameras and wavefront sensor cameras.

The control of six telescopes, six acquisition cameras, twelve adaptive optics systems, and six tip-tilt systems is challenging for an operator considering several GUIs. The GUIs of twelve adaptive optics systems take a lot of computer screen space and would require around six monitors. Requiring hands-on interaction with a large number of subsystem GUI's is not a practical solution for a telescope facility that must work with a single operator. A software solution is required that simplifies the control problem. This problem is approached with an integrated GUI for each telescope for saving screen space, operational efficiency, time-saving, and easy usability. We took critical real-time displays, buttons, and features from the engineering GUIs of different systems and put them into the integrated GUI. In the end, the GUIs of all the telescopes, acquisition cameras, adaptive optics systems, and tip-tilts systems are fit into a single computer display monitor, as shown in Figure 4 .

The integrated GUI, obsgtk, connects and communicates with several servers including telescope (e.g., S1), acquisition camera (e.g., S1 ACQ), dome (e.g., dome S1), motors server (e.g., S1 HUT), Tel-AO (e.g., wfs S1) and Lab-AO (e.g., LABAO S1). Where S1 being the name of a telescope and names of other telescopes are S2, W1, W2, E1 and E2. The display of real-time images from the cameras are built with GTK + XWindow system and they are fast and use very less CPU and memory usage.

The Lab-AO systems are connected to the Tel-AO systems using a metrology system. A dichroic at the telescope reflects some visible wavelengths of starlight into the Tel-AO WFS and also sends the beacon light into the Lab-AO WFS. It is very critical for the beacon light to follow the same path as the starlight.

An automatic alignment software tracks telescope pupil and focus drifts in both the Tel-AO and Lab-AO WFS that can be caused by beacon position drifts and by the temperature changes during the course of observing. These pupil and focus drifts are corrected with several actuators, including M1, M2, M7, and M10 mirrors of the telescope as the Tel-AO's alignment is strongly coupled to the alignment of the telescope itself. The beacon alignment is strongly coupled to the quality of the Coudé path alignment. The AO system has several automatic alignment controls, and a few are listed below:

• Align the focus of the WFS feed parabola.

• Align the beacon to the WFS.

• Focusing the telescope sending starlight to the WFS.

• Align beacon focus.

• Align of M7 of the telescope.

• Align of M10 of the telescope. Figure 5 shows the performance of the Tel-AO for an integration time of 2ms. For this integration, the camera operates at a 440Hz measured frame rate and has a -3dB closed-loop bandwidth of 19Hz. For comparison, this is almost similar performance to the adaptive optics system of the Auxiliary Telescopes of the VLTI (NAOMI) 26 that runs at 500Hz and has -3dB closed-loop bandwidth of 18Hz. To verify the system performance, we have modeled the transfer function by putting the total delays of the control system, T D (see Figure 5 ). The total cycle time is the sum of the integration (2ms) and frame-transfer and readout time (1.8ms) that is T C = 3.8ms for 2ms integration. The computational time for reading raw images from the instrument shared memory, computation of slopes, and computation of DM commands from slopes is T RTC = 0.7ms. The total delay is T D = T C + T RTC .

The Lab-AO systems have been used since 2014 to measure a good "default flat" of the sky close to the target star, and this has proved to be very helpful in coupling the light into the single-mode fibers used in many beam combiners. 27 We yet to measure the Tel-AO and Lab-AO's final performance in terms of sensitivity due to the COVID-19 and 2020 Bobcat Fire, but these systems already show increased throughput more than a magnitude.

We here report the CHARA Array adaptive optics systems from the software perspective -highlighting operational complexity, solutions, and performance. An adaptive optics system's success depends on a lot of "end-to-end" closed-loop testing with its calibration source and the on-sky. We have tested all the subsystems, including simultaneous operation of Tel-AO and Lab-AO since 2018 in various observing conditions. The Tel-AO's -3dB closed-loop transfer function bandwidth is measured at 19Hz, and this performance is almost similar to NAOMI instrument, 26 which also uses the same model Andor 897 camera. The on-sky science observations with AOs are routine since the beginning of 2020. The on-sky tests are very promising, with more than a magnitude sensitivity improvement. We calculate more than 50% Strehl ratio achieved in the H and K-bands. These results push observations of faint young stellar object disks and active galactic nuclei, and other interesting systems. The AO also push the observations in worse seeing conditions, which were not possible without the adaptive optics system before. The sensitivity improvements will be enjoyed by beam combines operating in the H and K-bands, for instance, MIRC-X, 13 CLASSIC/CLIMB 28 and upcoming MYSTIC. 15 The adaptive optics systems are used not only for improving the throughput but also for other science cases:

• The Tel-AO and Lab-AO WFS pupil images can also be used to track the pupil between the telescope and the laboratory separated by many hundreds of meters of the optical path using starlight. This information is important for the estimation of accurate astrometry (e.g., MIRC-X astrometric mode 29 ) by computing pupil shifts projected on the telescope space 30 during the course of observing. The Tel-AO and Lab-AO images from the telemetry will be used to measure pupil shifts in the post-processing.

• The Telescope DMs will be used to take sky thermal backgrounds for the upcoming MYSTIC 15 instrument by exploiting the large stroke of the ALPAO deformable mirrors.

We had to return two deformable mirrors to the ALPAO company for repairs. One for an upgrade and another to replace a bad actuator. They are expected to arrive in 2021 and will be installed once the COVID-19 situation is under control. During the first days of commissioning, the time taken to slew to a new target, acquisition, and locking all AO systems was almost double the slew time without the adaptive optics systems. However, since then, we have gained experience in our operations and reduced the total acquisition times. We are still learning and improving the better strategies for the interplay between the Tel-AO and Lab-AO systems and simplification of the operational aspects.

Our software is on the CHARA GitLab but private repository (https://gitlab.chara.gsu.edu/chara). If the reader is interested in accessing the software or discussing it, please feel free to contact the authors.

First Results from the CHARA Array. II. A Description of the Instrument

An update on the CHARA array

Recent highlights from the CHARA Array

Imaging the Surface of Altair

No Sun-like dynamo on the active star ζ Andromedae from starspot asymmetry

Infrared images of the transiting disk in the ∈ Aurigae system

A triple-star system with a misaligned and warped circumstellar disk shaped by disk tearing

The expanding fireball of Nova Delphini

Stellar Diameters and Temperatures. II. Main-sequence K-and M-stars

Nine-channel tip/tilt detector at the CHARA Array

Single-mode fiber optics in a long-baseline interferometer

MIRC-X/CHARA: sensitivity improvements with an ultra-low noise SAPHIRA detector

MIRC-X: A Highly Sensitive Six-telescope Interferometric Imager at the CHARA Array

CHARA/MIRC-X -a high-sensitive six telescope interferometric imager concept, commissioning, and early science

MYSTIC: Michigan Young STar Imager at CHARA," in [Optical and Infrared Interferometry and Imaging VI

SPICA, Stellar Parameters and Images with a Cophased Array: a 6T visible combiner for the CHARA array

Progress of the CHARA/SPICA developments: Compensation of path length fluctuations and chromatism in the visible

A study of degraded light coupling into single-mode fibers

The CHARA array adaptive optics program

The CHARA Array Adaptive Optics Program," in [American Astronomical Society Meeting Abstracts #227

Optical and Mechanical Design of the CHARA Array Adaptive Optics

Adaptive optics for the CHARA array

CHARA array adaptive optics II: non-common-path correction and downstream optics

The CHARA array adaptive optics I: common-path optical and mechanical design, and preliminary on-sky results

Peak-locking centroid bias in shack-hartmann wavefront sensing

Fibered visible interferometry and adaptive optics: FRIEND at CHARA

The Classic/climb Beam Combiner at the CHARA Array

Armada i: Triple companions detected in b-type binaries alpha del and nu gem

Methods for multiple-telescope beam imaging and guiding in the near-infrared