nanog mailing list archives

Re: East Coast Earthquake 8-23-2011 - comment and a bit of a Christchurch Telco report :)


From: Don Gould <don () bowenvale co nz>
Date: Thu, 25 Aug 2011 10:53:08 +1200



On 25/08/2011 12:27 a.m., Scott Morris wrote:
Also, the quake on the east coast was much closer to the surface than
most west coast quakes, which could account for the feeling.

Scott (not a geologist)

/me is also not a geologist, but does live in the east part of Christchurch, New Zealand. Our CBD has been closed for 6 months now as a result of a 6.3 on February 22, 2011.

A number of factors effect impact on buildings from our experience. Depth is a big issue. A quake 80km deep at 5.8 will have far less impact than one 1km deep. When I read yours was 1km deep, I started looking for the impact stories.

Direction of the movement is also a big issue. In our case the ground was going up and down under our tall(ish) buildings faster than the buildings. The ground would start coming up again before the building had finished coming down again from the last movement cycle. This just smashes buildings to bits.

Our tallest building is 26 stories, will take a year to bring down and is on a lean. Interesting most of the damage was caused by the building next to it being hammered into the side. We have many ~10 story buildings to come down.

Anyway enough about buildings... here's some comment on networks that I thought some might find interesting...


Networks - Mobile

Mobile traffic does go nuts. We have an average (iirc) of less than 30 minutes voice traffic per month per user. So out networks simply are not designed with a large load in mind.

We also don't have the 'confirmation' aspect of the sms (texting) protocol turned on. This means that the senders mobile phone doesn't know if the message has been delivered to the receiver.

In our case, we have 3 mobile networks in Christchurch. We discovered that we simply didn't have enough inter carrier capacity provisioned for sms traffic. While users could send messages 'same network to same network', they couldn't send them 'off network'.

We also have full number portability. This means that senders don't know which network they're even sending messages to any more.

+64 021 - Vodafone
+64 022 - 2Degrees
+64 027 - Telecom

Fail - not any more.

The up shot is that when disaster struck us, text/sms messages were not getting though and now one knew what the problem was.

So people started to attempt phone calls (on a network provisioned for an average load of ~30 minutes a month - you can see how this is heading down hill fast!).

The really ironic part of the lack of inter-carrier capacity for sms was that the 3G data capacity never failed for me. So while I couldn't send a 200byte message, I could send a 300kbyte email with photos.

!!!POWER!!!

Battery life also quickly becomes a problem for both network operator and user. As towers start shutting down, phones move to more distant towers, which mean everything uses more power.

Telco's are then presented with the problem of getting generators to towers. "Remember your chains and padlocks!" - when disaster strikes, idiots think stealing telco generators is cool. You don't want to have to revisit a tower just to replace the generator.

Home users quickly realise they have no way to recharge their mobiles. New, cheap, smart phones only last a day and even less when they're trying to talk to distant towers. (You should see the pile of hand held 'crank' torch/radio/mobile phone chargers that our local hardware store now has!)

People are also asking about inverters on local message boards.

Population Movement...

Another problem we've found is that population movement causes load issues. Thousands of people left the city area to towns up to 3 hours away to escape the aftershocks. They take their phones and mobile broadband and then spent lots of time calling back into the city to local friends and family. Suddenly everyone's doing much more calling from small towns back into the city over the mobile network than normal.


Networks - Fixed

Our fixed lined networks did stay up reasonably well. (We have two separate copper networks in 50% of the city - Telecom & TelstraClear, and one incumbent who covers the whole city - Telecom.)

However, the power went out (in many cases simply because the street side transformers have sensors in them to detect if the oil reservoir has moved. The earth quakes trip the sensors and they either go out for a while or you have to wait until someone comes and resets them I think - either way, it resulted in 9 hours with no power for me on the 22nd).

Many many people use cordless phones and don't have a non-powered/corded phone any more.

After the 22nd we had a national campaign to get old corded phones to Christchurch. 5,000 were collected. (Now when you consider the country has a population of 4m and Christchurch has ~360k, 5,000 phones is quite a few).

We are moving forward into an FTTN/H world. This means that homes need power to keep voip units up. Simple rule - when disaster strikes, mobile is going to be 'it', so make sure you've got enough capacity provisioned!

Our road side FTTN cabinets only have battery life for between 4 to 8 hours. Each cabinet services ~200 customers. So you can see a new problem unfolding... how do you get generators to those? The FTTN nodes currently do midpoint injection, so the PSTN stayed going but DSL failed after time, expect where the 'E' side cables where damaged.

The newer TelstraClear fixed network is a combination HFC/UTP POTS network delivered over head or in street ducting. Interestingly the HFC didn't miss a beat. The POTS did eventually drop out, and when it came back on line, the system only seemed to have a basic configuration and the lines wouldn't support appliances like dialup EFTPOS machines any more. (This was fixed a few days later.) - oh how I long for a total IP world! :)

EFTPOS is very important in local socio-economic high crime areas. Small local store owners start to stress when they start to collect to much cash in the till as it makes them a target. EFTPOS means they can 'give out cash' at the same rate they collect it. Many people on public benefits, such as the old age pension, also don't have cash and use a plastic card to buy something as simple as milk.

While people had 'money in the bank', they couldn't get to it to spend it end they needed to. IP EFTPOS means you can transfer the terminal from a fixed service to another network or mobile solution quickly.

My local store owner only had fixed lines from the one provider whos service failed, so I had to string cables down 4 buildings to get a working phone service from 'the other network'. (We were lucky that one building in the block did still have a working service from 'the other provider'. $5 difference a month means that 70% of customers in the area are using the cheaper provider).

Ground Cables ---

The older Telecom network is 'direct bury' and the cables in my area are ~60 years old.

This causes a number of problems. With 7,000 after shocks since our 7.1 quake on Sept 4, 2010, the cables are getting pulled apart and pulled away from street side termination. We now have holes on our road because of earthquake damage and holes in our foot paths from telco contractors digging up bits of cable all over the suburb to trace faults.

While the old cables are armoured, you only have to touch them with a digger and you get 4 nice cable techs parked out side your house for 2 days drinking your coffee while they dig up and fix 85 pairs. (Which I might note only supplies service to 1 customer now)

City wide we've had many buildings pulled down. This also causes problems where cables run across buildings. Local techs told me that they've had many call outs where the cable to an adjacent building has simply been removed when a building is demolished in a hurry (because it's likely to fall on someone if there's another aftershock.)

We also had inter-exchange load issues. People could call 'out of town' but not exchange to exchange - oh how I long for a total IP world! :)





--
Don Gould
Christchurch, New Zealand



Current thread: