Informed Sources
A fall in frequency in the traction power supplied stopped Thameslink trains; TfL also affected
After a power cut on 9 August affecting a wide area of southern England, power was reconnected within 45 minutes.
Despite the overhead line supply remaining live, on Thameslink services north of Farringdon, King’s Cross and St Pancras passengers were trapped for up to eight hours, because around 60 Class 700/717 electric multiple-units had shut down and 31 could not be booted-up without the assistance of a technician.
Until these trains could be restarted, they blocked lines out of Farringdon and King’s Cross, trapping other services. Blue-light escorts were provided where possible to get engineers to the stranded GTR trains to reset the software. There were 17 technicians available at the time of the event and a further 24 technicians were mobilised within the next hour.
Thousands of passengers were delayed. In addition to 23 train evacuations, 371 trains were cancelled, 220 part-cancelled, and 873 trains delayed. London St Pancras and King’s Cross stations had to close for several hours due to overcrowding and London Euston went exit-only for a period of time.
Lines were reopened between St Pancras and Bedford by 20.54. Between King’s Cross and Peterborough services did not restart until 23.05, including running around still-stranded trains.
WHAT CAUSED THE POWER CUT?
At 16.51 on 9 August, lightning strikes on the electricity transmission system north of London caused a short-lived disconnection. This was followed by a loss of power from an off-shore wind farm and a power station at Little Barford.
Together, the two generators coming off-line represented a greater power loss than the nominal reserve of 1,000MW maintained by the Electricity Supply Operator, the National Grid. The loss of generated power was so large that within a minute it dragged down the standard 50Hz frequency of the grid to 48.8Hz.
At this point, demand is disconnected to protect the integrity of the supply. This is known as Low Frequency Demand Disconnection (LFDD). Relays across the Distribution Network Operators (DNO), which are responsible for regional power supplies, opened to disconnect approximately 1MW of demand from the electricity network. This represented about 5% of the total demand.
Network Rail had not been informed its assets were involved in the LFDD scheme, either by the DNOs or National Grid. Reportedly, other users were also unaware their loads were potentially subject to LFDD.
Cutting demand stopped the fall in frequency. With the available reserve, plus additional generation capacity, being brought on line, system security was restored within five minutes.
Network Rail was not contacted by the DNOs during the outage and at the time of writing had yet to receive any information from them about the event.
Network Rail asset specialists in devolved operational businesses are contacting DNOs to establish senior points of contact to ensure a more collaborative approach during any future incidents.
WHAT HAPPENED ON THE RAILWAY?
According to the investigation by National Grid, ‘the built-in resilience of Network Rail’s electrical power infrastructure meant traction power was maintained to the vast majority of the railway throughout the incident’. The drop in frequency caused protective circuits to trip at two DC traction locations. Eight signal power supplies ‘in principally rural locations’ suffered minor outages with minimal passenger impact.
However, the Class 700 and 717 trains shut down when their low frequency protection systems were triggered. The report emphasises that ‘the Network Rail overhead line power supply operated continually’.
An eye witness in a Class 700 stopped in the tunnel between St Pancras and the Midland main line could hear the driver talking to the signaller. This source reports that ‘a bit after 18.30’ the driver reported that the power was restored but the train could not draw power.
He was advised to switch everything off, wait two minutes and switch everything back on again. Passengers were warned that this would involve turning off the interior lighting. After three unsuccessful attempts and a further wait, evacuation began around 19.30.
WHY DID LOW FREQUENCY CAUSE THE TRAINS TO SHUT DOWN?
Three-phase alternating current traction drives depend on high frequency switching of direct current to create a variable voltage/ variable frequency alternating current. When AC traction was being introduced this was recognised as a threat to track circuits.
Harmonics of the switching frequency would appear in the traction return currents in the running rails and, potentially, this could cause wrong-side track circuit failures. Switching algorithms were developed which ensured that in normal operation the drives avoided the critical frequencies.
However, there remained the remote possibility that a fault in the software or hardware could generate harmonics at frequencies used by signalling. To protect against such malfunctions, Interference Current Monitoring Units (ICMU) were developed. An ICMU monitors the traction current and, if it identifies a critical frequency, shuts down the offending traction package.
With over two decades of service experience, ICMUs are no longer ‘bolted on’ and traction packages are self-monitoring. They will shut down if potentially dangerous frequencies are identified.
At the front-end of the traction package is the 4 Quadrant Converter (4QC) which provides the DC supply to the traction inverters feeding the AC motors. The 4QC chops at a fixed frequency, say 39 cycles/second (39Hz) for illustrative purposes. When combined with the 50Hz of the AC supply this gives a combined frequency of 1,950Hz with known harmonic frequencies.
However, if the supply frequency drops while the chopping frequency of the 4QC remains constant, the combined frequency changes.
For example, a fall to 48Hz gives a frequency of 1,872Hz. Typically traction packages monitor the supply frequency and if it strays outside pre-set limits open the circuit breaker between the pantograph and the transformer to shut off the power.
However, while this protection was activated on the Class 700 and 717 stock, plus certain Bombardier traction packages on Transport for London trains, Siemens has discounted EMC protection issues as the cause of drivers being unable to restart their trains.
With the Siemens trains, tripping involved permanent lock-out of the 4QC that occurred when the frequency had fallen to around 49Hz. However, the system should not have tripped until the frequency was down to 48.5Hz.
Nor, according to Siemens, should there have been a permanent lockout following a protective shutdown caused by a supply voltage frequency drop. All trains should have been recoverable by a battery reset.
ARE THERE SPECIFICATIONS FOR FREQUENCIES WHICH ROLLING STOCK SHOULD TOLERATE ON THE MAIN LINE NETWORK?
Yes there are, in particular Railway Group Standard GM/RT2111 ‘Rolling stock subsystem and interfaces to AC energy subsystem’. This was first issued in December 2014, so could post-date the Class 700 fleet for which the Train Infrastructure Interface Specification (TIIS) was signed off in November 2008.
According to the TIIS ‘The Unit shall be capable of operating on traction power supplies within the range of voltages and frequencies defined in Network Rail Company document NR/GN/ELP/27010 “Guidance for compatibility between electric trains and electrification systems”‘. This noted that the supply frequency can fall to 48.5Hz in extreme conditions. It was withdrawn and replaced by GM/RT2111.
In the case of 25kV electrification Section 3.1 of GM/RT2111 requires: ‘25kV electric rail vehicle(s) shall be able to operate continuously over the frequency range 47Hz to 52Hz, as set out in EN 50163:2004+A1:2007 clause 4.2’.
Which brings us to EN (Euro Norm) 50163:2004+A1:2007 which pre-dates the Class 700 tendering. It refers to 49.5Hz to 50.5Hz for 99.5% of a year, but with the possibility of 47Hz to 52Hz.
However, Clause 4.2 Note 2 of this EN states that frequency variation is much more closely controlled in Europe, where vehicles will operate only within the frequency tolerances between 49Hz and 51Hz. ‘If the frequency is out of this range, the vehicles’ performance may be reduced or the vehicles’ drives shall be disconnected’.
A review of Siemens’ submission for Class 700 to the Notified Body, which validates a new train design, has shown that compliance with both NR/GN/ELP 27010 and EN 50163 would be demonstrated. However, according to Siemens, the supply frequency response of the Class 700 traction was designed to comply with the EN 50163 Clause 4.2 Note 2 that allows traction packages to be disconnected at 49Hz.
Clearly, the protective system, while meeting the EN, did not comply with UK requirements.
There is a close parallel with the initial incompatibility of the Hitachi Class 800 traction packages with Solid State Interlocking lineside equipment and other lineside communications.
WHY DID SOME CLASS 700/717 UNITS FAIL TO RESET AFTER THE SHUT-DOWN?
When the frequency dropped to 49Hz, the 4QC on the Class 700/717 units was shut down and locked out.
Following failed attempts to repower the trains, the established first response is for the driver to reboot the train using the battery reset procedure. This takes 10 minutes.
GTR Fleet Control diagnosed the problem rapidly and a global GSM-R broadcast instructing drivers to perform a battery reset saw traction power restored to 27 of the affected units. This left 31 units that could not be reset by the driver and had to wait to be rebooted by a technician using a laptop.
A feature of the latest software enabled trains is a continuous process of software updates, or ‘drops’ to the Train Control Management System.
These are intended to resolve issues or enhance performance.
On 9 August Siemens was partway through implementing software version 3.27. Critically, this removed the ability of the driver to recover from a permanent lockout by using the battery reset process.
All the stalled trains had version 3.27 or above, while those reset by the driver were at version 3.25 or below. I understand that removal of the battery reset facility was intended to protect some electronics components in the traction package.
Immediately following the incident Siemens commissioned a software patch that will allow units which protectively shutdown below 49Hz to recover themselves without the need of a reboot or laptop when the frequency rises to 49.5Hz. At the beginning of September, this patch was being verified by Siemens software engineers at Erlangen in Germany. It was then expected to be trialled on a Class 717 before being downloaded to those trains with the later Mod Standard software.
Siemens can’t revert to version 3.25, because version 3.27 includes a number of ‘fixes’ which, if reversed, could affect availability.
One example is the CCTV system.
WERE OTHER TRAINS AFFECTED?
Not on Thameslink, where the Class 387, Class 365 and, not unsurprisingly, the software-free Class 313 were unaffected.
On Transport for London, the Victoria Line’s 2009 Tube Stock (09TS) and the Sub-Surface Lines S Stock, both with Bombardier traction equipment, were affected.
These units are fitted with Correct Operation Protection (COP) relays that monitor traction voltage and current for critical frequencies.
Successive multiple operation trips these relays, locking out the traction converters.
These relays are set around the normal AC harmonic ripple signature in the direct current traction power supply. The setting at each frequency assumes that the frequency will stay within approximately 1%. For example, that the 1,200Hz harmonic generated by the traction transformer-rectifiers in substations will stay within the range 1,188–1,212Hz.
On 19 August COP relays on 09TS and S Stock trains recorded successive multiple trips in the range 1,171–1,175Hz. Similar lower frequency trips were recorded for other characteristic harmonics.
This disturbance resulted in Victoria Line train services being suspended. Service was restored at 17.35 after train operators carried out ‘traction fault reset’ on each affected train. Running began in ‘restricted manual’ mode before automatic train operation (ATO) was resumed.
Transport for London notes that while the S Stock trains were reset and the service continued without reported delay, it took about 30 minutes to reset the traction converter systems on the 09TS trains and restart the service.
HOW DO OTHER ELECTRIC TRAINS HANDLE FREQUENCY DROPS IN THE 25KV SUPPLY?
On the Alstom Class 334 the VCB will open if the frequency is outside the range 48Hz-51Hz for more than one second. The VCB will close automatically when the frequency returns to within the range 48Hz-51Hz for more than one second.
With the Class 390 Pendolino, the operating frequency range is 46.5Hz to 52.5Hz. A fault is logged if frequency is out of range for 500ms. The fault resets when frequency is restored to the range 48.5Hz-50.5Hz.
