IP: REPORT OF BELL ATLANTIC ON INTERNET TRAFFIC -- LONG but an

[Dave Farber <farber () central cis upenn edu>]

From: Lyle Evans <<evansl () vt edu






<underline><color><param> 0 0, 0


0,ffff</param>http://www.ba.com/ea/fcc/report.htm


</color></underline>


REPORT OF BELL ATLANTIC ON INTERNET TRAFFIC






Bell Atlantic submits data today to assist the Commission staff in the 


evaluation of  current arrangements which are available to Internet 


Service Providers (ISPs) to obtain access to their customers.  Through 


the Enhanced Service Provider (ESP) exemption, ISPs are permitted to 


obtain their access services from local service tariffs.  Local services






are provided via network technology which was designed in anticipation of






far different traffic characteristics than are currently being 


experienced.  Bell Atlantic, through submission of these data,  


encourages the Commission to consider elimination of or modifications to






the ESP exemption.




This submission is organized into five sections.  The first section 


describes how the ISPs purchase services from Bell Atlantic today, 


including general information on prices for these services.  The second 






section describes a traffic study which Bell Atlantic conducted on seven






ISPs in the Virginia, Maryland and Washington, D.C area.  The third 


section describes how ISP traffic loads have affected the network.  The 






fourth section describes cost impacts.  The final section describes 


suggestions as to how the FCC may address the issues raised by the ESP 


exemption, including a discussion on how correct economic pricing may 


provide better incentives for the ISPs to subscribe to services based 


technologies which can more efficiently handle ISP traffic patterns.




At the outset, Bell Atlantic views the recent development of the 


Internet, including use of the World Wide Web, as very positive for 


society and the telecommunications industry. While changes are needed to






address problems caused by current access arrangements, we are committed






to finding solutions which will sustain continued growth and interest in






use of the Internet by the public at large. In fact, the changes Bell 


Atlantic is seeking to define, should help sustain growth by providing 


incentives for ISPs to embrace more efficient emerging technologies by 


eliminating artificial pricing arrangements that discourage use of 


improved technology. At the same time, Internet growth should not occur 






in ways that require cross-subsidization by non users of the Internet, or






worse, lead to potential disruption of vital public safety services such






as 911 emergency call service.




SECTION ONE - Current ISP Access Arrangements




ISPs currently use several interconnection arrangements purchased from 


local service tariffs to transport calls from their subscribers to their






centrally located network aggregation centers.  These network centers 


typically house modems, routers, WWW servers, authentication servers, 


mail servers, etc.  Traffic collected at these network centers is routed






to the Internet backbone over dedicated facilities, or to other on-line 






services.  Placement of these centers often is a function of minimizing 






local access costs, i.e. maximizing the number of subscribers that can be






reached on a flat rate, local call, untimed basis.




The most common interconnection arrangement is to use the existing DDD 


network to provide dial-in access to an analog "modem pool" for 


those customers who can reach the hub central office on a local call 


basis.  The ISP's subscribers dial in to the lead number of the multiline






hunt group serving the ISP, and the DDD network makes the connection.  


Depending upon the size of the multiline hunt group, and the features and






functionality desired, many ISPs decide to purchase either Business Dial






Tone Line, Engineered CENTREX or CustoFLEX 2100 from the hub central


office.




The price for Business Dial Tone Line service in Virginia, for example 


(including Subscriber Line Charge), varies from a low of $16.93 to a high






of $18.93 per month, per line.  There are no usage charges to the ISP 


since all traffic is incoming from the ISP's subscribers, and local 


tariffs contain no charges for terminating usage.




In order to obtain additional functionality and higher transport speeds,






and to reduce their costs of operation, many ISPs are beginning to 


purchase ISDN PRI (Primary Rate Interface) service.  Each PRI facility is






equipped to handle 24  circuits (in most cases one circuit is required 


for signaling).  The price of an ISDN PRI facility in Virginia


(including


Subscriber Line Charge) is 


$455.93 per month. Again, there are no usage charges billed to the ISP 


since all traffic is incoming from the ISPs subscribers.




SECTION TWO - Traffic Study




While field operations personnel report on a firsthand basis the effects






of  heavy traffic patterns associated with access into ISP facilities


(<<a 


see SECTION THREE below), we concluded that a more 


systematic approach to measuring traffic loads was needed to substantiate






impact on our network.  ISPs submit orders for local services (


see SECTION ONE ) through our business offices in 


similar fashion to any other end user business customer, therefore their






facilities cannot be separately identified for collection of usage data. 






Orders flow through our provisioning systems on a mechanized basis, and 






are not screened to enable identification of ISP facilities as such.


<<P


Thus the first step for the study was to ask the sales account teams in 






our Large Business Sales organization (the sales unit that has 


responsibility for the larger ISPs) to randomly select a number of ISPs 






who were known to have ordered service in the Virginia, Maryland, 


Washington, D.C. area.  We selected these three jurisdictions due to the






ready availability of trained personnel who had the knowledge of how to 






perform a usage study of the type required.




Even though calls from end users (for example, people dialing up over 


modems from their homes) increase traffic loads on the central offices 


that serve the end user, the heaviest concentration of traffic loads are






occurring in the central offices that serve the ISPs.  These central 


offices are where the earliest impacts on network resources are being 


observed.  Therefore we selected lines from the ISPs that terminate into






their serving central offices as the point at which to perform our 


traffic study.  




We determined that by measuring traffic continuously over a 24 hour 


period for 7 days a week, for a period of four weeks we could obtain a 


meaningful sample of the overall impact within the region.  Given that 


our computer systems were collecting information for each call made 


during this sample period, the quantity of data to be collected was quite






large.  Thus we determined that we did not want to exceed ten central 


offices and 5000 total circuits for this study, in order to manage the 


quantity of data that would be collected.  The final study design 


resulted in selection of 7 ISPs in 9 central offices (5 in VA, 2 in MD 


and 2 in Washington, D.C.)  and 4887 circuits.  New circuits within a 


multiline hunt group ordered  by each of the ISPs during the four week 


sample period were automatically added to the sample.


<<P


We selected a four week period (February 25 - March 23) to avoid holidays






and other &quot;events&quot; that might lead to traffic spikes.  Of the 






4887 ISP circuits that we sampled, approximately half involved PRI 


connections, and the other half involved standard analog connections.  


The central offices selected varied in their mix of residential and 


business customers.




In addition to sampling ISP circuits, we also determined that it would be






useful to have a benchmark traffic sample with which to compare results. 






Within the same 9 central offices, and during the same 4 week period, we






selected 16 business/government customers having multline hunt groups 


serving multiple lines.  We sampled a total of 777 lines for these 16 


customers.  As another benchmark, we also collected total usage for the 9






central offices during the 4 week sample period.




The output of the study was number of "hundred call seconds" 


(CCS) on an hourly basis.  (There are 3600 seconds per hour, or 36 CCS if






the line is used continuously during the hour).  Our measuring system 


determined the time for each completed call per line, accumulated that 


usage on an hourly basis and converted the results into hourly CCS.  For






the peak hour of each segment studied, averaged over the four week 


period, the CCS results were as follows:




SAMPLE SEGMENT                 AVERAGE PEAK HOUR CCS* PEAK HOUR FOR


SEGMENT




ISPs on 1MB (measured business)       26 CCS          11:00 PM


ISPs on PRI (primary rate interface)  28 CCS          10:00 PM


Business Customers with MLHG          12 CCS           5:00 PM




Office average (entire central office) 3 CCS           4:00 PM




* Maximum utilization is 36 CCS per hour




In order to provide a more visual display of results from the study, we 






selected Wednesday, March 13, and plotted the hourly CCS results for the






four segments in each of the 9 central offices studied.  The graphs are 






attached to this report.  Names of both the customers and the central 


offices are masked to protect the privacy of our customers.




Another key output from the study was average length of completed calls. 






For the four week study period, the average length of all ISP calls was 






17.7 minutes.  This compares to approximately 4 to 5 minutes as the 


average for all other calls on our network.




The results of the study clearly demonstrate traffic levels for ISPs 


which are significantly above normal customer traffic levels. The results






of our study also showed that the heavy traffic levels shown in the above






table were consistent throughout the study period (See "Composite Graph


For


All 28 Days"), adding to our ability to conclude that this study provides






a fair representation of the traffic levels which are occurring on our 


network attributable to ISP usage.




SECTION THREE - Network Impacts




In this section, we will describe the impact ISP traffic is having on our






network.




Traffic engineers size switches based on an average usage assumption of 






about nine minutes per line during the peak hour (5CCS).  Individual 


business customers may normally be in the range of 7 - 8 CCS.  As our 


study demonstrated, traffic levels from ISPs greatly exceed this level, 






and approach 30 CCS in their peak hour.  At the traffic levels they are 






generating, we estimate that the overall traffic loads on the local 


network would double if only a 15% penetration of households were 


connected to the Internet.  Stated another way,  if just 15% of 


households went on line to the Internet at one time and had a call hold 






time of one hour, it would double the capacity demanded.  The reason is 






that 15% of households on line for an hour has the same effect as 100% of






households making a nine minute call in that same hour (5CCS).  With 


on-line data services, a relatively small user group can stress the 


network in ways which have not previously occurred, and were not 


contemplated in designing the network.




Heavy traffic loads affect certain elements of the network, the primary 






elements being Line Units (LU), Switch Modules (SM), and interoffice 


facilities (IOF).  Subscriber lines terminate into LU's when analog 


services are ordered (e.g. 1MB).  Under normal circumstances (i.e. 


traffic loads in the range of 3 - 5 CCS), each LU can accommodate 


approximately 450 subscriber lines, with a maximum capacity of 512 


lines.  As traffic loads on incoming lines increases, the number of lines






which the LU can accommodate decreases.  If all lines are at the 25 - 30 






CCS level, then the LU can accommodate only about 65 subscriber lines. 


This would mean a 7-fold increase in the number of LUs needed to 


accommodate the same number of lines at an approximate cost of $60,000 


per LU.  Since there are 7 LUs per SM, investments in SMs will also 


increase.  The need to add LUs and SMs exists even if ISP traffic peaks 






at a different time than the central office as a whole, because this 


equipment is dedicated to individual lines and cannot accept overflow 


traffic.




Interoffice facilities (IOF) requirements are also engineered to meet 


peak requirements.  The peak busy hour for IOF trunks will vary 


throughout the network, and is influenced by community of interest 


factors associated with customers served by the central offices.  The 


emergence of Internet traffic has greatly increased the amount of IOF 


required to provide acceptable levels of service to all end users 


(regardless of whether PRI or analog lines are used).  For example, the 






number of interoffice trunks in service for the switches in our traffic 






study increased 44% from June 1995 to June 1996 (a total of 16,585 


trunks).  Traditional growth expectations would have been in the range of






9% (3,397 trunks).  The approximate capital cost of IOF is $1,350 per 


circuit.  Thus an &quot;above normal&quot; investment of $17.8 million 


was required for the additional 13,188 IOF trunks for these 9 central 


offices.




Looking at recent experience in actual central offices helps to validate






the argument that Internet traffic loads are adding to the cost of 


maintaining acceptable levels of telephone service for the public as a


whole.




A 5ESS Office Located in Northern Virginia Serving a Large ISP 






During the second half of 1994, this switch was running slightly above 


capacity due to higher than projected growth.  The dial tone delay 


(percent of customers receiving delayed dial tone) varied between 0.54% 






and 0.78%, and the office experienced a busy hour of approximately 2.5 


CCS between 15:30 and 16:30 hours.  Network Administration rarely 


received trouble reports from field technicians and just an occasional 


call from the repair bureau.  


<<P


Customer complaints about poor service began to increase substantially in






February/March 1995.  Trouble reports from field technicians increased 


from zero to in excess of 25 per month and a number of customer appeal 


cases were received.  The situation had deteriorated to the point that 


one maintenance technician had to be assigned specifically to handle 


trouble reports.  Investigations revealed that the ISP served out of that






office was utilizing all available timeslots in various line equipment. 






This blocked access to the switch by basic residential and business 


customers.  In February of 1995, dial tone delay had increased to 2.22%,






as the ISP's lines in service grew rapidly.  These lines were averaging 






35.5 CCS at their busy hour, and had begun to shift the office busy hour






to much later in the evening.




The decision was made to remedy the situation by dedicating certain 


central office equipment to the one ISP customer.  Implementation of this






strategy required the deployment of 6 switching module controllers, 1 


digital line trunk unit, and 1 integrated digital carrier unit.  This 


solution avoided blocking of normal residential and business calls.  Even






though effective, this solution has proven to be extremely expensive.  


The cost for switching equipment alone was $1.9M, and was five times the






normal cost per line for office equipment.  Labor expenses for 


rearranging lines were in excess of $100,000.  By contrast, the revenue 






from the ISP for the lines affected was approximately $20,000 per 


month.




A 1AESS Office Located in Northern Virginia Serving An ISP




During the 1994/1995 busy season review this office was running well.  


The office overflow was averaging between 95.50 and 98.00 index each 


month.  Busy hour CCS was averaging 3.21 during the busy season which was






consistent with historical data for this office.




In Late December, 1995, the Network Administration Center received a 


trouble report for slow dial tone from the ISP.  Subsequently, it was 


discovered that 50% of the ISPs lines were assigned to 2 line units which






had been recently installed.  These lines were averaging around 35 CCS 


during the evening hours, and 17 CCS over a 24 hour period.  The office 






overflow increased from 14,000 in October 1995 to 73,600 in March 1996. 






The overall impact to the switch was to increase the busy hour CCS to 


3.7, which represents a 15% increase in total switch demand.




Line and equipment transfers were performed to spread the ISP's access 


lines across multiple line units.  All additional lines for the ISP have






to be manually assigned to prevent overload conditions.  An additional 


312 interoffice trunks were also added to alleviate congestion in this 


part of the network.




Another 1AESS Office Located in Northern Virginia Serving An




This office was traditionally one of the most trouble-free wire centers 






in Northern Virginia.  The CCS per network access line was 2.56, and 


there were very few switch related trouble reports.  The Network 


Switching Administration Group noticed a gradual increase in the 


"incoming matching loss" failures (calls incoming via trunks 


which are blocked in the switch) for the office in the fourth quarter 


1995.  Subsequently, "office overflow" failures (which is the 


sum of incoming and outgoing matching loss, as well as trunk overflow) 


increased from 2,445 to 21,000.  This increase in failures was due to a 






substantial increase in call attempts and duration time incoming to the 






switch destined for the ISP multiline hunt group.  The net effect of this






activity was an increase in the average busy hour CCS to 3.11 and a shift






in the busy hour to the early evening.




The fix again involved spreading lines over multiple line switches, which






required transfer of many residential and business customers to provide 






additional switching capacity for the ISP's lines.  The labor and capital






expense to accommodate the ISP's traffic are in effect allocated across 






all other customers.




A 5ESS Remote Located In Maryland Serving An ISP




Remote offices such as this one are hosted from another office located 


nearby.  During the 1994 busy season review, this remote office was 


equipped with 5 line units and 2050 working lines.  The capacity was 


sufficient to handle normal growth expectations.  The dial tone delay was






0.09%, office overflow was 0.18% and the busy hour was 15:30 - 16:30.




In February 1995 numerous customer complaints associated with slow dial 






tone were experienced.  An investigation revealed the cause of the 


problem was traffic destined to the ISP served out of this office.  Heavy






utilization of these access lines had driven CCS up to 3.18, as well as 






increasing dial tone delay to 0.40% and office overflow to 1.09% during 






the normal busy hour of 15:30 - 16:30.  ISP traffic had shifted the busy






hour to 20:00 - 21:00, with CCS exceeding 4.0, with a dial tone delay of






14.3% and office overflow which approached 19%.  Switch capacity was 


exhausted.




The conclusion was reached that remote switching centers could not be 


equipped to support the demand generated by an ISP.  Therefore the 


decision was made to rehome the ISPs lines directly to the host switch. 






Approximately $200,000 in capital was expended to implement this strategy






due to the fact that a new digital loop carrier terminal was required in






the host office, so that the ISP's lines could be spread across multiple






line units, thereby lessening the overall CCS impact to the office.




A Northern Telecom DMS-100 In Maryland Serving An ISP




This wire center contained strictly analog line units with limited spare






floor space which did not allow a building addition.  As the ISP traffic






rapidly increased, it became evident that this office would be unable to






support this growing traffic requirement.  It was determined that the 


only feasible solution was to home all growth lines for the ISP to 


another office via a digital loop carrier system .  The cost of this 


facility was approximately $300K.  Since revenues from this ISP are about






$15,000 per month, it will take at least 20 months to recover just this 






one equipment item, not addressing all remaining equipment and labor 


costs required to serve the customer.






SECTION FOUR - Cost Impacts






Traditional cost models designed for general ratemaking across all 


customer segments do not have the ability to identify and attribute costs






specifically to ISPs.  However,  information derived from our network 


engineers can be used to generally  illustrate the conclusion that 


current revenues derived from local services provided to ISPs do not come






close to recovering the cost of providing service.  Hence a cross 


subsidization is occurring between users of ISP services and all other


users.




The network elements most affected by heavy traffic loads from ISPs are 






line units, switch modules and interoffice trunking.  Per subscriber line






served, these units generally result in a capital cost of approximately 






$245.  This assumes the normal traffic load of 3 to 4 CCS.  However, as 






CCS approaches 30, the capital costs for these units approaches $2400 per






subscriber line, because of the reduced number of lines they can serve 


and the increase in interoffice traffic.    As shown below, this 


translates into an approximate monthly cost per subscriber line of $75, 






compared to the average tariff rate of about $17 per month.  Preliminary






studies show that the comparable monthly cost per subscriber line for PRI






circuits is estimated to be $50.




For illustrative purposes, we have attempted to size this for our entire






network as follows.  We estimate that during 1996, ISP circuits will 


average 40,000 throughout our region. (This is estimated based on input 






from sales organizations, as system databases do not specifically 


identify ISP circuits, and ISPs themselves often do not identify 


themselves as being ISPs.  Since ISP circuits are growing very rapidly, 






the end of year total will significantly exceed the beginning of year 


total.)  Roughly half of the circuits are analog lines (e.g. 1MB) and 


half are PRI.




The $2400 in capital costs for what in effect are traffic sensitive 


investments can be converted to a rough monthly cost by utilizing a 


standard annual cost factor of .37 (source: our network engineering 


organization).  This annual cost factor is a shorthand way to incorporate






overheads, maintenance, depreciation, etc.  This results in approximately






$900 in annual costs, or roughly $75 per month per ISP analog line. 




Utilizing the 40,000 average ISP circuit count (roughly 50% analog and 


50% PRI)  and the estimated monthly cost per circuit ($75 cost per month






per analog line and $50 per month per PRI),  the total estimated cost of






serving ISPs in 1996 is $30 Million.  Assuming average revenue from ISPs






for these lines was $17 per month per line, total revenues from this 


segment in 1996 for public switched network service would be $8.2 


Million.  Thus there is a cross subsidy of approximately $22 Million for






1996.  Assuming an annual growth rate of 40% for illustrative purposes, 






this cross subsidy would grow to approximately $120 Million in five


years.




The above calculations are presented to provide current insight into the






dimensions of the issues raised in this report, and are based on broad 


averages across our region.  We will continue to fine-tune our 


understanding of the cost issues and collect data which result in 


increased accuracy.




SECTION FIVE - ESP Exemption and Incentives for Adopting New Technology




Perhaps the most significant shortcoming of existing service arrangements






provided to ISPs is the flat rate per month.  For about $17 per month, an






ISP can utilize lines from the public switched network that can be 


literally filled to capacity.  Increasingly, ISPs are moving to charge 


their subscribers flat rate prices as competition within their industry 






accelerates.  Flat rate prices, and the nature of on-line communications






has resulted in call characteristics for computer-type "calls"


which vary significantly from traditional voice calls.  The flat rate 


price encourages users to connect, and stay connected throughout the day






(and evening).  Applications such as voice over the Internet can be most






effective if the user's Internet connection stays on all the time.  In 


effect, a circuit-switched architecture has been converted to a private 






line - as a result of the pricing signal we are sending.  Neither end 


users nor ISPs have sufficient incentive to utilize public switched 


network resources efficiently.




Another consequence of today's pricing signal is to retard the adaptation






of more appropriate technologies.  Since data transmissions are generally






more tolerant of minor delays than voice services, a packet technology is






particularly well suited to the transmission of Internet-type 


communications.  Bell Atlantic has introduced a service called Internet 






Protocol Routing Service (IPRS), which utilizes SMDS to transport calls. 






In addition to providing more efficient transport of Internet-type calls,






this service would assist the circuit-switched network used for voice 


calls by alleviating congestion at the central offices that serve ISPs. 






With IPRS, traffic will be collected at many originating end offices and






transported directly to ISP offices.  Other technologies are on the 


drawing board which could both provide better service to end users, and 






help alleviate congestion on the public switched network.




However, as long as current pricing arrangements are in effect (i.e. the






ESP exemption), the time it takes for these technologies to be adopted is






artificially lengthened.  The ISPs in our traffic study generated on 


March 13 (the same day selected for the graphs attached to this report) 






608 minutes of use per line over the 24 hour period.  Based on payment of






$17 per month per line, the ISPs pay 56 cents per day, or $.0009 per 


minute of use.  This contrasts with Bell Atlantic's interstate switched 






access charge of approximately 2 cents per minute.  In effect, ISPs are 






paying 1/22 of the equivalent per minute rate paid by IXCs during a 


business day.  At these levels, ISPs would have little incentive to adopt






voluntarily alternative forms of access.




A usage sensitive price (related to the traffic sensitive costs in our 


local network) is needed to send the appropriate signal to use the public






switched telephone network efficiently.  However, we recognize that this






price must be at a level which does not cause disruption in the 


industry.  As stated at the outset of this report, Bell Atlantic will 


work with the Commission and industry participants to come up with 


pricing options that help to moderate existing cross subsidies, and help






send the type of economic signal that will aid in allowing the faster 


adaptation of technologies which will help alleviate growing congestion 






on the local telephone network, but which will also not lead to undue 


disruption in the industry. 




As an important first step, we applaud the Commission's invitation to 


provide this information which establishes the dimensions of the problems






we are experiencing, and we encourage the Commission to take the next 


step of requesting broader input.  The problems we have experienced 


thusfar, while severe in our view, have not been recognized by the public






at large.  Given the rapid, and almost volatile, growth of traffic 


related to serving ISPs, however, it is not difficult to envision 


scenarios whereby traffic surges might occur which would overwhelm the 


ability of local networks to sustain service.  Service interruptions of 






even a temporary length could affect public safety services such as 911 






service, with unthinkable consequences.   Therefore it is important for 






the Commission to address quickly the issues raised in this report.




Bell Atlantic Calls on FCC to Consider Internet Impact on Phone Network




Telephony Magazine Article - "Superhighway Traffic Taxes


 Current LEC Network"



---


Lyle Evans  evansl () vt edu






============================== ISP Mailing List


==============================


Email ``unsubscribe'' to inet-access-request () earth com to be removed.


Experience varies directly with equipment ruined.






<fontfamily><param>Times New Roman</param><smaller>


</smaller></fontfamily>