PremDay #2 - Prem'Fail - Quick definition of (bad) luck

As part of the Prem'Fail track, Hugo Collignon from Scaleway presents a set of bad luck stories happening to him and especially when an ATS (Automatic Transfer Switch) tried to kill him.

Transcript

1. Quick definition of (bad) luck Hugo Collignon 02/04/2025
2. 2 I am probably not lucky
3. 3 Fiber cabling Around 10 optical fibers in hand (coming from others rack via patch panels) … obviously no label Letʼs try one by one and monitor the opposite interface Physics and math can explain this? Well, letʼs admit The good one was the latest BUT 9 remaining… was the 9th then the 8th … Almost the last one each time
4. 4 Birds How many times? 0? 1? 2?? Me, 4 Including one in a date (true story)
5. 5 Back to IT infrastructure Firmware issues? Missing an API? Bug found somewhere? Random reboot? Underlay infrastructure Under anything related to software Even under what we call hardware Letʼs go to electrical topic (and issues)
6. 6 When an ATS tried to kill me ATS for Automatic Transfer Switch Useful with monofeeded devices Switch between Feed A and Feed B
7. 7 When an ATS tried to kill me Rack ATS Feed A Feed B PDU X X Composite housing No continuity
8. 8 When an ATS tried to kill me
9. 9 When an ATS tried to kill me Rack ATS Feed A Feed B PDU X X Screws ~ 150 VAC
10. 10 Remediation Breath a little Check yourself (and the following hours) Ask someone to check you Then investigate
11. 11 Remediation Asked the manufacturer For them (electrical devices manufacturer point of view): Human injury / human risk / P1 / Critical
12. 12 Root cause Screws were not screwed on terminal ATS side) Earth wire was out Thatʼs it Thatʼs all This line
13. 13 Any others? We have tens / hundreds of those devices They made the SN list of possible involved parts (total: 96 Inventory everywhere Ship everything (possible) to them (~ 30 → ~ 30 % Include SN1 & SN1 from Patient zero Analysis
14. 14 Never happen again Perfect professionalism from manufacturer Connected screwdrivers with torque measurement When doing an ATS assembly, unable to go to the next step without having the good torque
15. 15 Any others? Nope. Only one. The one we had installed in the lab.
16. 16 Still alive Life is just trolling me.
17. Thank you!