At 1 a.m. on Monday, 17 July, the LHC beams were dumped due to an electrical perturbation. Approximately 300 milliseconds later, several magnets lost their superconducting state (“quenched”). During a quench, the magnet warms up, which in turn warms and pressurises the liquid helium that surrounds it.
While not common, this sequence of events is a normal occurrence that protects the superconducting cable of the magnet when an electrical glitch occurs; the mechanical stress exerted on different parts of the magnet can be quite strong.
Among the magnets that quenched on 17 July were the inner triplet magnets located to the left of Point 8 of the LHC, which play a crucial role in focusing the beams for the LHCb experiment. Unfortunately, on this occasion, the quenches led to a helium leak in these magnets and put a halt to regular LHC operations.
Scroll through the photo diary below to relive the ten-day race against the clock to repair the leak.
The reason for the electrical glitch that caused the safety systems in the LHC to dump the beam and several magnets to quench was found: a tree on the Swiss side (about 55 km from CERN in the Canton of Vaud) fell on the power lines and disrupted the power system.
Ten hours later, on entering the tunnel, the investigating team found that the cryostats* of the triplet magnets near Point 8 were partly covered in ice. Tests quickly confirmed that a small amount of helium had escaped through a leak and filled the insulation vacuum.
Action was taken immediately: the adjacent magnets were electrically isolated, circuits were locked off and grounded, and the quench heaters for this sector were switched off. Additionally, to allow work to be done on the triplet, the 3 km of superconducting magnets in the affected sector were stabilised at a temperature of 20 K, instead of their usual 2 K (-271°C).
*All LHC superconducting magnets are housed in cryostats. During normal operation the external wall of the cryostat is at room temperature, while the magnet operates at 2 K. The cryostat is designed to maintain the magnet at such a low temperature by minimising the in-flow of heat – and insulation vacuum is essential to achieve that.
The exact position of the helium leak in the 50-m-long cryostat was still unknown. By Tuesday, 18 July, vibration and acoustic tests had been performed. Attaching accelerometers and microphones, the intervening team detected a clear signal in the interconnection zone between the first quadrupole magnet (Q1) and the second (Q2). Additional X-ray scans showed that the spacing of the bellows ridges on one of the pipes in the superconducting magnets appeared to be stretched. Bellows are employed in the physical connections between two magnets, giving flexibility. In this case the stretched bellows were on the M2 pipe, which contains the instrumentation connections.
The intervening teams agreed that the Q1-Q2 interconnection between the two quadrupole magnets would have to be opened for further investigation and repairs. To make a safe intervention possible, the sector around the magnets was emptied of liquid helium. In parallel, an electrical quality assessment showed that the electrical circuits of the triplet were fine – the problem was thus elsewhere.
Teams of experts from different CERN groups (safety, vacuum, cryogenics, magnets, engineering, powering, magnet protection, survey, beam instrumentation, operations) discussed how to tackle a problem that had never been encountered before on a 15-year-old string of magnets and established a procedure on the spot.
The whole triplet cryostat reached room temperature. The external bellows and inner thermal shields at the Q1-Q2 interconnection were removed so that the inner helium lines could be inspected.
The leaking bellows in the M2 pipe.
The teams located the M2 bellows that were suspected to be at the root of the problem and, indeed, found that there was a 1.6-mm-long crack on it, which was the source of the helium leak. An action plan was put in place: remove the broken bellows, replace them, do all the necessary tests, close up again and start the cooldown…
…all in under 10 days. Otherwise, a complete warm-up of the affected LHC sector could not be avoided, and this would put an end to the whole LHC physics programme for 2023.
While the broken bellows were cut out, the vacuum team conducted pressure and leak tests on spare bellows to test their resilience and provide a replacement unit for the tunnel repair.
Experts led by Sandrine Le Naour and Said Atieh discussed the possible repair solutions on site.
The new bellows were installed. On the far left: threading the instrumentation through the new bellows. In the middle: many hands make light work! On the right: skilled welders do their magic.
Graeme Barlow looking at the open interconnection, with the various pipes inside visible. The M lines allow the helium to be transported between magnets (M1 contains the busbar for the electrical connection, M2 contains the instrumentation connections, and M4 has a cryogenic function). In the middle sits the beampipe where the particles circulate. The M2 bellows are just visible between the M1 and the beampipe.
The vacuum and mechanical teams discussed the action plan while the repairs were in progress.
Two teams were often at work at the same time: on the left, reinstalling beam position monitor (BPM) cables, on the right, starting the leak test on the new bellows.
The vacuum team installing the leak-test tooling.
During the opening of the Q1-Q2 interconnection, the beam position monitoring (BPM) cables had to be removed. Here, the cable reinstallation is under way.
Sandrine Le Naour (far right) assessing the progress. She coordinated the mechanical interventions to open the magnet interconnections and then had to prepare the careful reclosure.
Wim Maan and Marcel Knoch checking the tightness of the final weld.
The M2 bellows fully repaired. The bellows are surrounded by external shells to support and guide them when the helium is pressurised during different operational phases of the LHC.
The bellows were repaired and the leak test was successfully completedwithin the ten day deadline. Although there’s still plenty to do to reclose the interconnection, the light at the end of the tunnel is in sight! After the teams have repumped the vacuum and cooled down the magnets, the LHC can restart.
The LHC operations team is confident of seeing the first beam back in early September.
For more information on this story, watch a video interview with Paul Cruikshank, one of the coordinators of the repair operation: