Motherboard/Vice are embarking on an admirable plan to build their own, neutral, mesh, internet network connection provider for their neighborhood in Brooklyn.
What they build, how they build it, and what they learn doing the project, will be documented and available to the general public as a DIY "How To Become Your Own ISP" kind of document.
This is FANTASTIC.
But it doesn't actually route around the Net Neutrality problems introduced this week.
The key sentence happens way down near the end of the article:
- "To be clear and to answer a few questions I've gotten: This network will be connected to the real internet and will be backed by fiber from an internet exchange. It will not rely on a traditional ISP."
If you do not buy your internet connection/service from an ISP, then you must provide two things:
1. A connection to the Internet backbone. Last time I looked at this type of connection, they were more than $10k/mo for a connection that worked with a small 20-person business.
2. DNS services. If you are on the backbone, then you MUST both receive and send all the internet traffic that comes to you. This means you must provide DNS services - so the people connected to your local network have an IP Address, so other servers on the internet can talk to them - and you must route traffic to other servers on the internet.
The first problem is paid for by your subscribers, who pay you to provide the connection, and offset your costs for the connection.
The second problem requires you to have a group of computers set up, running highly technical services, and the network admin staff to run, maintain, and keep these systems up to date, protected from hackers, and functioning properly in their role in supporting the larger internet.
Now your little community ISP has three shifts of highly technical and adept staff (say, 3 geeks minimum), working 52 weeks a year, and running multiply redundant systems. AND each of those servers and services you run on those servers need to be backed up by a second, identical, live system, so if a hard drive fails, or software needs updating, or a system gets hacked, the live system fails, and the backup system immediately drops into place.
---
Let's tot this up, shall we?
$10k/mo for the sole internet connection. (Double that if you want a redundant one for backup purposes.)
(approx) $200k/year for geeks. Three network engineers who are making anywhere from, say, $50k-$120k/year, depending on the size/complexity of your customer base and their needs.
(approx) $500k in equipment. Servers, software, hardware, etc., that your three geeks keep up and running. And it all needs to be replaced every 3 years or so.
$??? - Offices. Rent. Support staff. Receptionist to answer the phones. Phone system. Desks, chairs, Mountain Dew, HR department, Accounting Department, Billing, etc.
And what do you get in the end, for all this? AN ISP.
Because now you have a company. That company has bills to pay, salaries to cover, employee benefits, 401k plan, operational expenses, etc. So you need to make money. And sell more internet connections to more people, because people die or move out of the neighborhood, or the big local employer moves out of the area and jobs are scarce, so people drop their fancy NetNeutral ISP and go with the cheaper free city-provided wifi.
But your company is the BEST NetNeutral ISP in the area. So people flock to your services.
Now your services are overloaded. You need more hardware. You need more bandwidth. You need more geeks and more phone support staff.
But man... you could expand to the next city over... if you could just get a couple million dollars together.
So you go public. IPO. You sell shares and become a publicly traded company, and get a big infusion of cash, and rapidly expand your operations state-wide. You have hundreds of thousands of new customers all across the state.
At the big shareholders meeting, you have to report that roll-out state-wide has hit unexpected snags, and so quarterly results are down. Your stock price takes a hit.
All of the people who own stock in NetNeutralISP see the value of their stock portfolio dropping, due to NNISP's poor financial performance this quarter.
The board ousts you - the founder - and puts you into a purely advisory position, with a big paycheck and parachute.
They hire a new CEO who promises to revive the company - and the stock value - through offering high-end services to your most exclusive (rich) customers. Getting them to sign up for a premium service will not only help boost quarterly numbers, it'll also give you the capital infusion you need to finish the state-wide rollout, so you can nail down 1.2m new customer accounts.
And that premium service you offer to businesses and high-net-worth individual clients in gated communities? Priority, high-speed, internet access.
And you have just become the problem you originally set out to address, with your community-focused ISP.
And damn you for making me spend half an hour learning about peering, by the way.
here's a place that agrees with your $10k/month. However, they put $7500 of that as the gear and maintenance. They also list that price for 2010, when transit fees (wat) were $5/Gb, while these guys put transit fees at pennies per GB in 2016.
This is well out of my wheelhouse but I know when I call Comcast business services, I get local guys. Every conglomerate has a local office and that local office isn't a giant ISP; it's a piece of a network. We're talking about building a piece of a network.
Assume that $7500/mo hardware cost hasn't gone down. Assume you're still buying 10GBps. Assume you're promising your clients 50MBps. If you stone-cold guarantee everybody 50 (rather than assuming a duty cycle of 10% or whatever), you have 200 clients. If they pay $37.50 and tax for their connection (we'll assume a mesh network) you break even.
Obviously there's a lot of slush there. But I pay $35 and tax for 30MBps from Comcast and they're profitable. It ain't like they're making it up on scale.
Most of the complaints about starting an ISP relate to the rollout, not the network. If you aren't pulling RG6 or fiber to every house you're hitting, you're in a different place.
Not saying I'm right you're wrong - saying that I think Vice wouldn't go down this road if it were as hard as you think it is. I'm sure we'll find out what they missed soon enough.
Yeah, re-reading my post, it could be summed up as, "Most small businesses fail. And you are trying to set up a small business that has the reliability and low cost of a Utility, so you will most likely fail."
People don't stick around if your internet service goes down for a few minutes.
To fix any internet outage in a few minutes would require on-site tech staff 24/7/365, and duplicate hardware, at a minimum. That alone triples your human resources budget, and doubles your hardware budget, and you haven't even transferred your first bit yet.
And what if a construction company down the block cuts your underground fiber cable that provides your connection? You have a duplicate fiber connection going south, to another network interchange, that you can instantly switch all traffic over to, right?
Or when the couple renovating their apartment on the 3rd floor bust the water pipe and knock out power to the building? You do have a backup diesel generator that can run for 24 hours while the water and electric companies work in the building to restore services, right?
blah. blah. blah.
I used to have an ISP as a marketing client.
Part of what I did is walk through their entire operation, end to end, to see all the systems and backup systems they had in place just to provide basic internet services. These guys are REALLY good, and have been doing it for a LONG time.
They have two internet pipes which run north and south out of their building to two different internet exchanges run by two different companies. That protects against an idiot with a backhoe, but...
... they still buy their connection/bandwidth from CenturyLink and Verizon.
Their have their own building, security, HVAC and power redundancy, etc, etc, etc., and they STILL can't get/don't warrant a direct connection to the backbone. They have to buy their north connection from CenturyLink (at the Westin), and their southern connection from Comcast or Verizon. (Don't remember which one.)
So even this robust, decades-old, reliable data center is still beholden to their internet provider actually providing net neutral internet traffic. If CenturyLink or Comcast decide to throttle Netflix again, there ain't dick the downstream ISPs can do about it.
So there are two problems which need to be solved to move around non-NN ISPs:
1. You need to be an ISP with enough traffic, money, clout, to warrant being connected directly to the internet backbone.
2. Your connection to the backbone needs to be approved by the companies you are competing against - the six Tier 1 providers - who are exactly who you think they are: Comcast, Verizon, NTT, AT&T, etc. They have to agree to interchange traffic with you, or else you get your traffic onto the backbone, and they simply ignore it.
In every regard, the Internet looks exactly like every other Utility in America: a small number of providers, providing an essential service, to homes and businesses, across a proprietary channel (electrical lines, water pipes, internet fiber, etc.), and providing a service that is interchangeable and indistinguishable from a service provided by the government.
It needs Title II protections. Homebrewing solutions just are not going to cut it.
I think you're missing the forest for the trees. Here's the meat:
- So even this robust, decades-old, reliable data center is still beholden to their internet provider actually providing net neutral internet traffic. If CenturyLink or Comcast decide to throttle Netflix again, there ain't dick the downstream ISPs can do about it.
If you have an end-user agreement, that agreement includes Comcast doing whatever they want to your traffic because you aren't negotiating, you're signing or not signing. If you're peering with Comcast as a Tier 1/Tier 2 business agreement, you're negotiating whether or not you're letting them shape your traffic and there's no advantage to Comcast to shape your traffic at that point.
This is a model Comcast knows - it's really easy to become a CATV provider if you've got, say, a nursing home. The price for a single lot of 100 ESPN subscriptions is less than 100 individual ESPN subscriptions. It's pennies, in fact. Comcast your ISP is a different animal than Comcast your peer; hell, Comcast your business internet provider is a different animal than Comcast your home internet provider (example: if I'm not getting 30/12 for my larcenous $135/mo they roll a truck because they fuckin' guaranteed it in writing. Now - they might very well decide to shape my traffic and there's nothing I can do about it... but it isn't in their best interests to do so because just looking at my flow-through (thanks, Ubiquiti!), only 20% of my traffic is video and that's with a family sitting and watching Netflix for 9 hours yesterday. I'm about 35% file transfer and about 20% encrypted data. If they try throttling my Netflix I'm going to drop below my 12 and they're in breach.
The rest of it is tort stuff - if ConEd nukes your fiber it's hardly your problem because you aren't the one on fiber. Vice appears to be meshing it up quite nicely; I can buy 100km microwave transmitters on Amazon Prime and redundancy is really fucking simple on Ubiquiti. And it's SOHO-grade shit.
I dunno. Vice seems to have done a pretty good job of being a successful business. I'm curious to see how easily they become a successful ISP. Remember - the cable companies got into it simply because they had the pipe. The pipe isn't really necessary anymore, at least not for last-mile shit.
We are broadly in agreement. Vice is performing an extremely valuable service, and I am truly excited for their discoveries and instructions.
However ...
- "...If you're peering with Comcast as a Tier 1/Tier 2 business agreement, you're negotiating whether or not you're letting them shape your traffic and there's no advantage to Comcast to shape your traffic at that point..."
You still need the agreement with Comcast, even if you are peering for Tier 1 capabilities. That means your data flows through their pipes and their routers, on the way to the backbone. At any point along that line, Comcast can throttle Netflix, and you are still stuck with taking the data at the pace they decide to deliver it.
If you don't like that? Buy your Tier 1 peering net connection from one of the other 5 providers... who are all in cahoots with Comcast, have been lobbying for the opportunity to charge for exactly these services, and go to lunch with Comcast execs every day. (And, BTW, NTT do not own wires going to your data center, Comcast does. So now you need to run new fiber to your data center from NTT, or whoever, because Comcast owns the ones in your area.)
---
Related Divergence: So we have too few Tier 1 providers, right? Well, we need to have as few as possible, because a single bad actor at Tier 1 or 2, and we get something like we had last week, where a badapple routed legitimate traffic through a Russian hacker site. Tier 1/2 is a trust-based network, and a weak point in the internet design that can be exploited in really really bad ways. So we trust these guys to be good actors, and act in the general benevolent belief in a neutral internet. THAT is the core of why the loss of NN is so painful. We trusted these fuckers with a monumental responsibility, for the benefit of - literally - all mankind, and they broke that trust.
- At any point along that line, Comcast can throttle Netflix, and you are still stuck with taking the data at the pace they decide to deliver it.
Yeah but no. The FCC regulates the arrangement the end user has with Comcast. They decided that the Internet wasn't a utility. But as soon as Comcast decides that they get to interfere with a potential competitor (which is what you are, as a Tier 2 peer), they walk smack dab into anti-trust.
If you don't have any choice but to work with Comcast because of an arrangement Comcast has as a Tier 1 provider, and Comcast doesn't allow you to use that Tier 1 network as a peer, the FTC comes down like a shithammer on Comcast. It's no longer an FCC issue, it's garden variety anti-trust and Comcast wants to not set that precedent so hard that there's no reason they'd ever do it.
The telecoms know the future. They know their time in the sun draws to a close. They're making money while the sun shines and transitioning to content (there's a reason it's Comcast Universal). The whole Net Neutrality thing is a staying action, not a victory and they know it and the longer they can keep the government from making them behave like a utility the more money they make.
But they can't keep doing it forever. And a precedent like fucking with a peering agreement would end the party faster than just about anything else they might try.
Anti-trust. Good point. I hadn't considered that angle.
- The whole Net Neutrality thing is a staying action, not a victory and they know it and the longer they can keep the government from making them behave like a utility the more money they make.
So why spend more than a billion dollars on lobbying and astroturfing just to get Title II restrictions lifted?
The only way I can see for them to make money from that deregulation is by charging higher fees for higher priority traffic. And since they can charge both ends of the transaction - the consumer for the fast connection, and the web site for the fast connection - they stand to make back 100x that billion dollars they spent to get one niggling detail altered in the regulations.
I just don't see where you get a 100x payback on a billion dollar investment, otherwise... and if they aren't in it for the expanded pofits, why go through the colossal expense and effort to repeal Title II?
Comcast's operating income was $26 billion in 2016. Presume for the sake of argument that they paid all the lobbying. $1b represents 4% of their income.
Presume they bump their rates 10% for one year. Their ROI on lobbying is over 100%.
The broadband industry's revenue in 2016 was $140b. That's expected to drop $3b by 2020. A $1b lobbying push, if it so much as closes the gap left by catv subscription attrition, is more than a sound business decision, it's a no-brainer.