I II III the most common complaint about the NSA is that they collect more than they can ever possibly analyze. Based on their charter, they're doing it exactly wrong. But they're not doing it based on their charter. HOW IT'S SUPPOSED TO WORK Igor Isis intends to blow up New York with a surplus Soviet warhead. He texts this to Jerry Jihad on O2 and the NSA siphons up literally everything on O2. Semantic Forest picks up the keywords "Warhead" "New York" and "blow up" and triggers an alarm. Then a human reads the conversation, sets Igor Isis and Jerry Jihad to surveillance, and everything they say is monitored by Jack Bauer and Ethan Hunt. Just as they're about to pick up the warhead, Delta Force swoops in and saves America thanks to those marvelous whiz kids at the NSA. HOW IT ACTUALLY WORKS The NSA siphons up every single data packet O2 has crossing their networks. There is no possible way any computer or human or combination can possibly parse any of it in realtime so Jerry and Igor's conversation goes unnoticed. Three months later, with millions dead and Manhattan a radioactive hole, the FBI finds Igor Isis' Facebook page and notices that he claimed he was going to blow up Manhattan. The FBI subpoenas his phone number and gets his records. They appeal to the NSA to give them everything they have on Igor and Jerry, and the NSA literally has everything. Then the NSA uses voiceprints and IPs and shit to find their way bak to Igor and Jerry's numbers on O2 and leafs back through their records to find the conversation about how they were going to blow up New York. They show this to the FBI, who then grabs the data on everyone they ever talked to. Several dozen people are swept up into CIA black sites, Guantanamo grows, and millions of people are still dead. But the NSA gets to point out that their surveillance was successful. This whole kerfuffle is over the fact that the FBI no longer needs to ask "mother may I" to see the phone records of dead, successful Jihadis. They can run their own search. theoretically the FBI might find something that might save lives or some shit but let's be real: Igor Isis could post a fuckin' Youtube video of himself with an ISIS flag and an AK and a fuckin' warhead in the background and the world wouldn't know about it until it was too late. Your link isn't about intelligence gathering. It's about statistics. Statistically, 6% of terrorists are Islamic. but you don't need to find six percent. You need to find Dzhokar Tsarniev. The NSA's approach will tell you everything you've ever wanted to know about Dzhokar Tsarniev after he's blown up the Boston Marathon, not before.
Ah, maybe my line of thinking is completely incorrect then. Which would be: To find Dzhokar Tsarniev, to find whoever your person of interest is, there has to be a profile, right? (Maybe this is where I'm wrong?). The profile would come from the absurd amount of data being collected from everything, and then parsed into various keywords, time entries, etc. that all create a generic "Dzhokar" or "Timothy McVeigh". For that to be useful or correct enough to work, bias has to be avoided, and everything else outlined in that statistics reading would come into play, otherwise you'll never be any closer to finding something/someone useful.
I am almost positive this has come up before, and I'm equally confident bfv had something useful and insightful to say. Lemme take a crack at it. What is your profile of Dzhokar Tsarniev? Immigrant, loner, angry, access to weapons, says nasty shit about the US to his friends on the phone, has traveled abroad, possibly to scary places? On the one hand, that covers waaaaay too many people to make it useful. On the other hand, if you decide to push everybody that might be in that pocket, you're likely to radicalize more than a few of them, alienate the rest, and show your hand to anybody you might actually be after. I mean, the CIA is adamant that they didn't radicalize Anwar al-Awlaki. But they totally radicalized Anwar al-Awlaki. Assume one person in a hundred actually intends to do violence. Push a hundred of them. How many intend to do violence now? And this is the NSA we're talking about - they don't push. They have no boots-on-ground. They have wiretaps. So back it out the other way. List your favorite 30 terrorists and build a profile that matches all of them. What you discover is that (A) if you build it wide enough to include all of them you've included much too large a sample size (B) if you take your sample size down to something reasonable you've excluded a lot of your hits (C) (and C is the real problem) what makes them terrorists generally isn't the shit they say on the phone. I mean, look. That's a credible threat, method, means and opportunity, from a former Army marksman against a high-profile target. It's also a stand-up comic paraphrasing Fight Club on Facebook. The NYPD took it seriously. The NSA can't because the amount of people suggesting violence in private conversations is orders and orders of magnitude more than the amount of people saying shit in public. It's a real problem: the FBI investigated Omar Mateen for ten fucking months. That's real human beings following a suspect around in unmarked cars'n'shit for longer than a school year and determining the dude wasn't a threat. You're going to pick this guy out via his text messages? It makes more sense when you realize what the NSA is designed for: spying on zaibatsus and nation-states. Your "find me a terrorist" profile goes to shit but your "find me all communications out of Sary-Shagan related to lasers" or "find me every Siemens contractor that has ever mentioned semiconductors" is super easy. Not only that you can run all of those intercepts together and find commonality. You can get common trends and fill in the blanks. You can do all that old-school Cold War intelligence-gathering goodness that built the whole artifice in the first place. The link I'm looking for explains how taking a large statistical analysis and backing it out to individual people will never work because the stochastic variation of individuals thwarts any attempt at statistical profiling. However, a large statistical analysis is exactly what you need to determine the size and shape of a nuclear missile program, or clandestine troop movements, or a KGB black site, or a new Samsung semiconductor. And while the first, second and third examples have dropped in priority since the fall of the Iron Curtain, the fourth example is going strong.Joe Lipari might walk into an Apple store on Fifth Avenue with an Armalite AR-10 gas powered semi-automatic weapon and pump round after round into one of those smug, fruity little concierges.