One of the telecoms engineers in my workplace spoke to a source inside BT and found out the true story behind the two outages. Last Friday was not a power cut. A card in a router locked up, and rather than replacing it BT took the decision to simply reboot it and hope it doesn’t happen again.

This goes against general practices in any large datacentre where you have a multitude of people relying on a single device to be working properly 100% of the time – when something that critical fails, you replace it immediately. It also means they really don’t have any redundancy as it appears there was no second card or router to take over when the first failed, and that means a 2-6 hour outage while engineers are gotten out of their beds, travel to site, diagnose and fix.

So it seems I was right, given that this morning once again the entire of Northern Ireland, Scotland and parts of England all lost their broadband connections again. Except this time BT is blaming an unnamed hardware vendor for the issue, again at the Edinburgh datacentre.

Imagine the broadband access for the entire of California, of even just a large city like New York going out for hours on end. There would be a senate hearing on the cause, loss of manhours for businesses and public outrage. In the UK… our government just gives the incumbent telco some more money and a tells them to try and not do it again.

There was an overnight outage where BT’s Edinburgh datacentre lost power. Now bearing in mind that the BRAS’s in this datacentre service pretty much the entire of Northern Ireland and Scotland, and parts of the North of England, why would they claim that only 20,000 homes were affected?

My workplace specializes in providing DSL to businesses in Northern Ireland, and I know for a fact that every single one of them was down aside from a small of handful of lines that we have in England. These are lines spread across the entire province, from Belfast to Derry. None of my neighbors had working broadband either, except for the two that were using Sky LLU. My Dad didn’t and he’s on Plusnet, many miles from where I live.

20,000 homes is about an 8th of the numbers of homes in Belfast alone, never mind across Northern Ireland, Scotland, or the North of England. Why won’t BT admit to the real number of homes affected? Because then people might realize how utterly incompetent they are to design such a widely used and relied upon system to be able to be take down nearly half the country because of a supposed power outage in a single location (which in a properly built datacentre, should be nearly impossible).

BT – Liars and idiots. Can someone remind me again why we gave these people the telecoms monopoly?

They tried to requote us for their 21CN broadband platform, assuming we take it in London as they wanted to charge us 50p per meter all the way from Manchester to Belfast (totaling some £250k). After carefully examining our current installation they decided that we should replace our pair of 34Mb pipes with a single pipe containing:

• 20 users on 24Mb ADSL sharing just 1Mb of bandwidth.
• All other users sharing 30Mb of bandwidth on old 20CN 8Mb ADSL.
• Added in enhanced care for all users at £8 a go.
• Forgot monthly broadband line rental charges at £7.90 a go.
• Will charge us for bandwidth across the 21CN network, plus charges for 3km of fibre across the London Docklands and we have to provide the BRAS – but yet they still have the balls to charge us £24k a year just for the privilege of doing business with the almighty BT Wholesale. Seriously, noone can explain what this charge is for given that they have separate charges for both bandwidth and fibre.

When I entered the correct figures into their shitty little price sheet, added in all the things they forgot, it came to a whopping £26 per user before any profit margin is added.

By comparison, Be/Fluidata is charging a non-recurring £3k to setup a simple crossconnect in any London Telehouse, and then all we pay are simple line charges depending on the product used, the average one of which is £16 per month.

It’s quite clear that BT Wholesale is not interested in providing any sort of service to other service providers. The ridiculous ordering/faults system, the outright denial of clear area-wide faults and now these ridiculous and quite arbitrary charges for access to their so-called 21st Century Network that still doesn’t properly support IPv6 are all very telling.

A new level of fail from our friends at BT Wholesale. They have actually willingly provided proof that they do not read fault reports the first time around:

Yes.. that is a grand total of 43 seconds from reporting the fault to BT Wholesale rejecting it. This was even reported via KBD, which lets you confirm that the user has already attempted to replace his router, cables, filters and even tried from the test socket. 43 seconds is not enough time for most people to type that long-winded reply about SFI appointments, let alone for BT to run the necessary diagnostics to determine if there is a fault or not.

At my place of work we have suspected that BT was doing this for a long time as all too often, and 9 times out of 10 blatently obvious faults are rejected with the message “not due to a network fault”. Now I have a handful of faults, some where it was customer some, but some where there was genuine faults such as the DSLAM being faulty where BT has denied anything being wrong and cleared the fault in less than a minute.

A few further thoughts on things that people forget to take into account when attempting to measure bandwidth:

• When measuring bandwidth, attempt to use a site or tool that is close to your ISP. If you’re in the UK and you try to test your connection using a site hosted in the US it’s never going to give you a decent idea of your speed. I recommend Speedtest.net, as it’s a single tool that can test to a multitude of different locations and will give you a much better idea of exactly how your line is performing.
• Remember to allow around 10% for overheads. An 8Mb ADSL line will top out at 7.2Mbps. This is due to overheads for the ADSL line itself: a certain amount of bandwidth is required to manage your packets that will not be visible on any web-based bandwidth test.
• Any download requires a certain amount of packets to be sent in the opposite direction. Usually these are acknowledgement packets to assure the server you are downloading from that everything is being received okay (or not, as the case may be). Again, that magic 10% figure is the one to watch out for. A 1Mbps download will roughly need a 100Kbps upload. If you are using up all your upload bandwidth, your download bandwidth will be poor.

An issue that comes up for me at work time and time again is customers misunderstanding how bandwidth is measured.

Data is traditionally measured in Bytes. A CD contains 650MBytes of data. Bandwidth is measured in bits however, and this is what most customers misunderstand. A CD measured in terms of bandwidth, is 5,200Mbits (there are 8 bits per byte). Note that in writing, you use a capital ‘B’ to denote Bytes, and a lower-case ‘b’ to denote bits.

The issue is that bandwidth is traditionally measured in bits, not bytes. A 1Mbit circuit lets you download at 100KBytes/second. A lot of people make the mistake of thinking that a 1Mbit circuit is the same as downloading at 1MBytes/second.

This becomes a problem when a customer – as has happened today – complains of a slow speed problem. The systems (which I built and maintain) show this customer as downloading up to 32Gbits per day. They dispute this via the phone, proclaiming that they only downloaded “4 gig” (in a 5 hour window, I’ll add). If you do the math: 4*8 = 32. 32Gbits. On a standard ADSL line, that’s a crazy amount of usage – averaging around 1.8Mbit/s for that 5 hour window. During peak hours, an ADSL Max line (due to contention) may only be able to achieve 2Mb/s. It’s a classic case of someone mistaking Bytes for bits… of course, explaining that to them is another matter.

A common problem with ADSL in the UK is that most connections are still using PPPoA. This means that if you want a computer to have a public IP address on one of these connections, you need to either have a block of IP addresses routed by your ISP to your router (at extra) cost, or you use a USB modem. There’s no real option for those folks that want to connect something like a SonicWall or any other firewall device directly to the line.

However, there is a poorly documented hidden “half-bridge” mode in the ZyXEL 660R routers. These cheap little single-port routers have the ability to push the public IP address and all it’s traffic onto a single device connected to the LAN port.

To do this, set the router up as normal with the username and password for your connection, then logout from the web interface. You’ll now need to telnet to the device, and enter the following:

poe bridge switch on
ip dhcp enif0 server lease 120
sys save

After this, reboot the router. Once it boots up and logs into your ISP, you should find that it gives you a single IP address on DHCP and that address will be an external fully public IP address.

Update: IntoTheUnknown uses this to build a SIP VoIP system, which of course can have problems when passing through any firewall or NAT conversion.