The FBI has a problem. Never before in its history have its agents had access to more information, and yet never have they found it so difficult to see inside that data to snoop on communications generated by terrorists, international drug dealers, domestic criminals and other outlaws. But privacy advocates say the FBI, and other law enforcement agencies, have plenty of other options when it comes to electronic surveillance.

The issue has come to a head with the bureau’s ongoing legal battle with Apple. The FBI wants a judge to order Apple to create a tool that would let it bypass the passcode on an iPhone belonging to one of the San Bernardino shooters. Apple is resisting, claiming that doing so would undermine iPhone security worldwide.

This is the crux of a crisis the FBI has dubbed the “Going Dark issue,” claiming the widespread availability of advanced encryption and other security technologies in everyday consumer products has led to an inability to track, monitor and stop criminals.

“Encryption of stored data is not new, but it has become increasingly prevalent and sophisticated,” Amy Hess,  executive assistant director at the FBI’s science and technology branch, told the House Oversight and Government Reform Committee last year. “The challenge to law enforcement and national security officials has intensified with the advent of default encryption settings and stronger encryption standards on both devices and networks.”

In the past, outlaws had little choice but to communicate through conventional means like the telephone, which can be wiretapped, or voice conversations, which can be bugged. But now they have access to relatively inexpensive and easy-to-use tools, including the Apple iPhone and encrypted apps like Wickr, Cryptocat and ChatSecure, which offer complete privacy.  

The problem is not entirely new. Since as far back as 2010, the FBI has been using the metaphor “going dark” to describe what it sees as an increasing inability to get at the contents of terrorists’ and criminals’ communications. The bureau requested $9 million in its 2010 budget specifically for a “Going Dark Program” to bolster surveillance capabilities.

But is the FBI really in the dark, or has technology given it unprecedented insights into criminals’, and the public’s, communications? The answer lies somewhere in between. While the bureau may not be able to crack an iPhone, such devices leave an electronic trail with phone companies and other service providers that can be monitored.

Such so-called metadata can tell investigators where a call was made from, the number contacted and the length of the conversation, among other things. Such information can be cross-referenced against databases containing phone numbers and other data linked to known terrorists.

Combine that with the emerging Internet of Things, and investigators have no shortage of electronic clues to follow.

Law enforcement’s battle for access to phone networks stretches back to the 1970s and came to a head initially with the Communications Assistance to Law Enforcement Act (CALEA) in the mid–1990s, which required telephone companies and others to ensure that their networks could be wiretapped with appropriate legal process. CALEA is no longer deemed sufficient, however.

“Currently thousands of companies provide some form of communication service, and most are not required by CALEA to develop lawful intercept capabilities for law enforcement,” the FBI said in a document describing the issues it sees with Going Dark.

The problem is not encryption as such. The FBI even states it “supports strong encryption” and encryption of digital communications has been around since before CALEA was enacted. But until now using encryption was simply too difficult and time-consuming for the vast majority of people to bother with.

However, in recent years companies including Apple, Google and Facebook have all enabled encryption by default on various products, including both encryption of data at rest (the latest versions of iOS and Android encrypt data on smartphones by default) and data in transit (such as the end-to-end encryption employed by WhatsApp and Apple’s iMessage).

FBI Director James Comey has been pushing the “Going Dark” agenda more than anyone else.

James Comey FBI Director James Comey was faced with a barrage of hostile questioning by the House Judiciary Committee on the ongoing legal case between his agency and Apple. Photo: Drew Angerer/Getty Images

Comey, testifying before the Senate Judiciary Committee last July, said ISIS is “recruiting and tasking dozens of troubled Americans to kill people, [using] a process that increasingly takes part through mobile messaging apps that are end-to-end encrypted, communications that may not be intercepted, despite judicial orders under the Fourth Amendment.”

He points to an attack in Garland, Texas, where an attacker opened fire on an event last May in which demonstrators cast aspersions on the Prophet Muhammad. The attacker had sent over 100 messages to “an overseas terrorist” before the attack but Comey told the Senate in December that the FBI didn’t know what he was saying because the messages were encrypted. 

In many cases, the initial recruitment of potential homegrown terrorists is done with open communications channels such as Facebook or Twitter, but radicalization takes place away from public view. “After a potential recruit is identified, groups like ISIS continue to groom and radicalize recruits by shifting to apps that hide the user’s identity, location and communications,” Leo Taddeo, a former FBI special agent, told International Business Times. “This makes it very difficult for the FBI to identify who the most dangerous terrorist recruits are and whether they are planning an attack.”

For all of Comey’s comments he and the FBI have yet to suggest a practical solution, calling instead on private companies like Apple to create an all-encompassing answer that secures citizens’ data without compromising the intelligence services’ ability to access that information when necessary.

Critics say Comey’s claim that terrorists and criminals are acting outside of the FBI’s scope doesn’t tell the whole story. The counterargument to the bureau’s position is that the agency has never had it so good in terms of the data that’s available to it.

Privacy law professor Peter Swire from Georgia Institute of Technology told the same Senate committee that heard Comey’s testimony that “it is more accurate to say that we are in a ‘Golden Age of Surveillance’ than for law enforcement to assert that it is ‘Going Dark.’ ”

A report published last month by the Berkman Center for Internet & Society at Harvard University called “Don’t Panic: Making Progress on the Going Dark Debate” suggests the FBI is using the wrong metaphor. While it concedes that encryption and “provider-opaque services” make surveillance more difficult in certain cases, the landscape is far more varied. “There are and will always be pockets of dimness and some dark spots — communications channels resistant to surveillance — but this does not mean we are completely ‘going dark’,” the report said.

WhatsApp Launches Desktop Apps For Windows And Mac WhatsApp, which is owned by Facebook, is available on all major smartphone operating systems, through web browsers and now on desktop operating systems Photo: Getty Images

The report was compiled by a group that included security and policy experts from academia, civil society and, crucially, the U.S. intelligence community. It concludes: “Are we really headed to a future in which our ability to effectively surveil criminals and bad actors is impossible? We think not.”

The report highlights three key factors. The first is that while encryption is on the rise, encryption in and of itself is not a silver bullet and it does not “prevent intrusions at the end points, which has increasingly become a technique used in law enforcement investigations.” The end points referred to are the smartphones and computers used by terrorists, similar to the one at the center of the FBI’s current court battle. While Apple claims newer iPhones can’t be hacked, even by its own engineers, others, including cybersecurity pioneer John McAfee, have said otherwise in recent days.

“Encryption is not a panacea in all cases for protecting data,” Tim Erlin, director at security company Tripwire, told IBT. Default encryption may be on the rise but an overwhelming percentage of internet users communicate through services — email, instant messages and social networks — that are not end-to-end encrypted.

The reason most communications on the internet are not encrypted is because companies like Google and Facebook are almost completely reliant on using customer data to generate revenue. For these services to work, personal data, which is increasingly stored centrally in the cloud, needs to be accessible to the companies who collect it so that it can help them sell targeted ads. This is unlikely to change in the coming years.

The second reason the FBI’s claim does not stand up, according to the Berkman report, is that the surveillance footprint it has access to is set to expand dramatically. As homes become smarter, everything from fridges to kettles will get connected. Cisco projects that by 2020 there will be almost 50 billion devices hooked to the internet in our homes, offices and shops.

Not only will there be billions more of these devices, protecting and securing them is going to be a virtually impossible task given the lack of standardization in the industry today. According to AT&T’s Cybersecurity Insights Report, published this week, 90 percent of businesses don’t think they can secure IoT devices against hackers.

Imagine a smart television with a built-in microphone and a network connection. It could be used to listen in to a telephone conversation — no matter how encrypted the telephone service itself might be. This is not a dystopian privacy nightmare. Last year, Samsung was criticized when it was revealed that its smart TVs were listening to users’ conversations and sharing them with third parties.

Indeed, the U.S. intelligence apparatus is fully aware of the opportunity that the explosion of connected devices provides. In evidence given before the Senate Select Committee on Intelligence just last month, James Clapper, the director of national intelligence, seemed to highlight the security problems of connected devices. “Innovation is central to our economic prosperity, but it will bring new security vulnerabilities,” he said in oral evidence. “The Internet of Things connects tens of billions of new physical devices that could be exploited.”

However, in written evidence submitted to the committee, Clapper outlined the advantages for the intelligence community. “In the future, intelligence services might use the loT for identification, surveillance, monitoring, location tracking, and targeting for recruitment, or to gain access to networks or user credentials.”

While the Internet of Things may prove to provide a huge source of intelligence in the future, today the FBI is using another device — the StingRay — to monitor suspects’ communications.

The StingRay is a controversial cellular phone surveillance device that was initially developed for the military and national security community. It allows users to monitor an area typically the size of a city block and capture data such as call logs and location information from mobile phones — even if they are not active or have location privacy enabled.

The FBI defends its use of StingRays, claiming they are necessary in terrorism investigations, allowing the agency to scour large geographical areas quickly. The StingRay’s use has become increasingly widespread in recent years, with privacy advocates estimating that at least 50 state and local law enforcement agencies have the technology today.

Last month, Wisconsin Rep. Jim Sensenbrenner called on Comey to reveal just how widespread the use of StingRay devices is today, raising concerns that the technology was being used without the appropriate oversight. “If the technology is so important for national security that it must be kept secret, then its use for routine law enforcement was inappropriate,” the Republican lawmaker said in his letter. “Either the technology should have been kept secret as a vital national security tool or it should have been made public so that it could be used by law enforcement.”

The final point made by the Berkman report is that not all government agencies are made equal, and while the FBI may be in the dark, that is not to say other agencies are in the same boat: “The government is not a monolithic organization, and the encryption debate is not viewed the same way across governmental organizations or among the individuals within these organizations.”

For instance, the resources available to the bureau for defeating encryption may be fewer than those available to the National Security Agency, and as Edward Snowden has shown, the capabilities of the NSA are wide-ranging and powerful.