Click2Gov or Click2Breach? What We Know About the Click2Gov Breach

Here on the Cyber Risk Analytics research team, we have more than our fair share of “glitch in the matrix moments” – you know, that proverbial black cat walking across your screen that makes you think: “Didn’t I just see this breach?” Usually it’s a case of similar circumstances or simply two names that are a lot alike. Other times, it might be something more.

While not yet verified, that “something more” appears to be the case with a utility bill payment processing application known as Click2Gov from the company Superion.

We have been tracking a handful of breaches taking place across the country that on the surface look to be unique events with somewhat similar descriptions. A local city or town discovers their online utility payment portal has been attacked. The service goes dark while the city investigates – along with their trusty vendor that may or may not run the portal – only to learn that payment card details used to pay utility bills online have been compromised. The city takes responsibility for the event and starts posting notices to impacted persons. All and all, there was nothing especially remarkable about the individual reports – until, that is, the name Click2Gov started popping up.

What we know so far

On May 25, 2018, the City of Oxnard, CA was notified by a bank that their online utility bill payment service appeared to have been breached, leading to a number of fraudulent transactions. Transactions taking place between March 26, 2018 and May 29 (yes, 4 days after the city first learned of the issue – more on that later) were exposed. The city identified  Click2Gov as their payment processing application.

On June 6, 2018, the Village of Wellington, FL was notified by Superion that certain vulnerabilities in Click2Gov might have led to a possible breach of their online utility payment installation. Once again, Wellington officials in conjunction with Superion shut down the system to investigate. While a breach has yet to be confirmed, there was sufficient information for the Village to state that payment card data used for online bill payments between July 2017 and February 2018 is considered to be ‘at risk’.

Two events in a row referencing the same application got our attention and sparked our curiosity. Especially so since the City of Oxnard event began one short month after the Village of Wellington event seemingly ended. Our immediate thoughts went to questions like: “Are there more breaches involving Click2Gov? Could it be the same attackers jumping from one vulnerable installation to the next? Is it possible that the source of the issue is attackers inside Superion, picking off data from various clients?” Definitive answers are not yet apparent, but it is clear that the issue is larger than just two breaches.

Looking back in our database, the City of Ormond Beach, FL experienced a similar incident with their Click2Gov system in October 2017. Like Oxnard, it was a credit card issuer that first traced the issue back to Ormond Beach utility payment system, alerting them of the problem on October 11. This, despite the fact that customers had been reporting fraudulent charges they believed to be linked to the City since September 22nd. Ultimately, cards used for payment between approximately mid-September 2017 and October 4, 2017, when the city opted to shut down their system, may have been compromised.

Shortly after, the City of Port Orange, FL launched their own investigation into their Click2Gov system. Their system was down for 5 days but ultimately, they could find no evidence of a breach. Curiously, their statement included a quote that their Click2Gov system had no “potential flaws that could leave the system exposed to a data breach.” One can only wonder if they are equally confident of no flaws now that Superion has notified at least one customer, the Village of Wellington, of “certain vulnerabilities” in the Click2Gov system.

Our research identified more breaches at several other cities that fit the profile of a Click2Gov issue. The vendor wasn’t named in official statements, but in several instances is clear Click2Gov is source:

• City of Goodyear, AZ – May 7, 2018 the City became aware of an issue with their unnamed online payment system. They worked with the vendor and determined transactions between June 13, 2017 and May 5, 2018 had been exposed. Although the city does not come out and name Click2Gov as the vendor, it’s clear from the payment landing page URL that Click2Gov is the service provider: https://click2gov.goodyearaz.gov/Click2GovCX/index.html
• City of Thousand Oaks, CA – February 28, 2018, the city learned of unauthorized access to their online payment system “Click to Gov”, exposing payment card details for transactions between November 21, 2017 and February 26, 2018.
• City of Fond du Lac, WI – Once again, on December 12, 2017, the city got word from a bank that a breach had been traced back to their water payment portal. Payments made between August 2017 and October 2017 were exposed. Yet again, Click2Gov was not named but is clearly they are the provider of payment services: https://click2gov.fdl.wi.gov/Click2GovCX/index.html
• City of Beaumont, TX – On August 24, 2017, the city announced they had received complaints of unauthorized charges after using the online water bill payment system. Payments made between August 1st and August 24, 2017 may have been “jeopardized”. Beaumont did not indicate a vendor was involved, but it’s clear who their service provider is as well: https://beau-egov.aspgov.com/Click2GovCX/index.html
• City of Oceanside, CA – In near lock step with Beaumont, on August 14, 2017 the city received complaints from customers that credit cards used between June 1, 2017 and August 15, 2017 on the now-defunct “utility bill payment” link had been compromised. The link is no longer available so it is unknown whether it was Click2Gov, but the city’s notification letter does state their forensic examiner found “malicious code had infiltrated this vendor supported online payment system.” Perhaps most telling, the letter goes on to state, “the City is exploring alternative online payment solutions that offer improved security processes and systems.” Clearly a wise decision on their part.

As you can imagine, we suspect there are others.

About Click2Gov

Unfortunately, we aren’t intimately familiar with how Click2Gov software works exactly. From how the cities are reporting the events, it appears to be a software package that is downloaded and run independently for each city. After all, the cities seem to be taking responsibility for the breach, hiring the forensic teams to investigate and making statements to the effect of updating their software and making changes to servers in response. But further digging seems to reveal that while it is a software package, there may be some vendors that are hosting it on behalf of their clients and the Click2Gov solution may also provide credit card processing capabilities.

What makes this interesting is that, for each incident that has been reported, the breach is presented as some sort of misconfiguration issue or a problem at the city itself, but it seems that it might be something larger.

Despite indications there were issues with the service dating back to August of 2017, it wasn’t until May 30th of this year in the City of Oxnards’ breach notification that we start to see clear evidence the problem lies with Click2Gov – and it’s not encouraging. Oxnard officials posted the following on their Facebook page:

“Upon discovery, the city immediately reported the issue to the Police Department and the city’s vendor, which engaged a third-party forensic firm to determine what happened and what information may have been affected. The city’s vendor alerted the city to a software vulnerability that had the potential to allow an unauthorized individual to gain access to the computer used to process credit card transactions.”

Keep in mind the City of Oxnard first learned of a possible breach on May 25, 2018. They reached out to Superion, seeking help with the issue. Additionally, Superion most likely knew of potential security problems since the City of Oceanside stopped using their service back in the summer of 2017, and certainly since Beaumont, Texas was breached at approximately the same time. Both facts make this next paragraph from Oxnards’ breach notice all the more concerning:

“Security patches were applied by the city’s vendor on a new server to eliminate the vulnerability with the thought that the issue was resolved. On May 29, 2018, the city’s vendor informed the city of additional security controls that were required to secure the system. The city shut down the system immediately so these security controls could be implemented. Even though the vendor’s investigation could not specifically confirm or verify the exact method by which any credit card data could have been compromised, the city decided to notify customers as a precaution.”

Multiple clients are breached over the course of a year and still it takes two tries to get a fix in place? And is the problem really corrected if they cannot confirm or verify the exact method of compromise? Looking back to the City of Fond du Lac’s breach notification, it seems this is not the first time they stumbled over incident response.

Unfortunately for the Village of Wellington, it seems they too are now caught up in Superion’s questionable patching and incident response practices. But at least this time, it was Superion that reached out to Wellington instead of waiting for a call from a bank fraud department:

“On June, 6 2018, the Village received a call from our vendor, Superion, notifying us of vulnerabilities in their software. The software problem was with the Click2Gov online payments for utility bills. Credit card information may have been taken during transactions.

The Village immediately shut down out payment connection to Superion and begin working with them to determine if our resident’s information was compromised. The forensic analysis is continuing, security patches are being installed and new hardware and software are being installed to eliminate the breach. Even though Superion could not specifically confirm that our customer credit card data has been compromised, the Village decided to notify our customers as a precaution.”

Superion security

There isn’t a lot publicly known about potential security issues with the Click2Gov solution. In taking a more detailed look at Superion’s website for any updates, there were none to be found for the Click2Gov software product. In fact, when looking on their website we were unable to find any links to security notices and when trying to find a dedicated security page (e.g. https://www.superion.com/security) we found nothing existed.

We then decided to reach out to Superion directly and email them at security@superion.com as well as call their general enquiry and sales numbers. Unfortunately, both phone numbers gave the same automated message and then offered us to leave a voicemail.

As for a security@ mailbox, sadly but not unexpectedly, it bounced.

We then forward the message to their Media Inquiries address (media@superion.com) to hopefully get some more information on the situation. If we receive a reply we will update this post.

What comes next?

The issue might affect quite a few more cities than initially expected.  As we were conducting our investigation we attempted to determine how wide is the installation base of Click2Gov. Our results varied widely but what we found was that there appears to be between 600 to 6,000 installations of Click2Gov indexed (and potential thousands more depending how you look at it).  Without spending much time digging, we quickly saw what appeared to be quite old versions of Click2Gov running.

Unfortunately, given what we have seen so far we anticipate seeing more breach reports coming to light thanks to the Click2Gov system. Superion and their clients are clearly struggling to wrap their hands around the problem and lock it down once and for all. In the meantime, any organization that is currently a Superion customer using Click2Gov should be on alert for suspicious activity. They should also consider reaching out to Superion for more information on the vulnerabilities that have been identified in Click2Gov, so that they can investigate whether they are exposed to the issue and implement patches or workarounds to mitigate the issue.

We suspect there will be more to this story and will update this post as we learn more.  If you have any information please contact us!

Superion’s Press Contact replied to our email with the following:

“Thank you for your email.

Protecting our customers and their clients’ data is of the utmost importance to Superion. Last year we reported that a limited number of on-premise clients had identified suspicious activity on their servers that are used to host Superion’s Clock2Gov product. Upon learning of the activity, we proactively notified all Click2Gov customers. Additionally, Superion launched an investigation and engaged a forensic investigator to assess what happened and determine appropriate remediation steps.

Throughout our investigation with the third-party forensic team, we have kept in direct contact with every Click2Gov customer to assist in the resolution of this issue, informing them of our findings via email, phone calls, and one-on-one working sessions. We assisted many customers with analyzing their Click2Gov environment and provided them with best-practice guidance to assist them in securing their servers and networks.

To date, Superion has deployed the necessary patch to our software and a related third-party component, and over 99% of these customers have applied these patches. At this time, we have no evidence showing that it is unsafe to make payments utilizing Click2Gov on hosted or secure on-premise networks with recommended patches and configurations. Superion does not control our customers’ networks, so we recommend citizens contact their municipality or county if they have any questions related to security.

Meanwhile, we continue to work closely with our customers to swiftly resolve and remediate this matter.“

Click2Gov Update

It’s been three months since our original post was published and as feared, breaches of the Click2Gov system continue to be reported. Here is what we’ve learned:

Attackers are exploiting an unpatched vulnerability in Oracle’s WebLogic. Early on, we speculated whether the problem was with the Click2Gov application itself and whether it impacted the cloud-based version of the system. It has since come to light that only local installations are at risk. Attackers are gaining access to application servers due to a known vulnerability in WebLogic and escalating the attack from there.

Few other details about the attack methods have come to light. That said, one intriguing detail has remained consistent –  only one-time payments are at risk. Data for customers with auto-pay enabled has not been exposed. That does make us wonder if there is another weakness in play, perhaps associated with the form or page used to enter payment information.

Nine more incidents involving Click2Gov installations have come to light. The targets include:

• City of Waco, TX – On January 10, 2018 Waco disclosed a lack of encryption which led to a compromise of credit card details after water bills were paid online. This one slipped by our original post, as there was no mention of Click2Gov and the description varied somewhat from the others. After digging more, the URL for the payment portal revealed the service involved: https://c2g.ci.waco.tx.us/Click2GovCX/.

• City of Lake Worth, FL – On June 14, 2018 journalists investigating the breach at the Village of Wellington reported that Lake Worth, located a short drive east of Wellington, had also been compromised. Although the breach only came to light this June, it appears Lake Worth was an early target, with the incident beginning around April 3, 2017. That would make Lake Worth the first organization confirmed to have their Click2Gov installation breached.

• City of Midwest City, OK – At the same time the City of Thousand Oaks was under attack, Midwest City was also being breached. Unfortunately, unlike Thousand Oaks which discovered the compromise in February of this year, the incursion into Midwest City’s installation – which started on December 11, 2017 –  lasted until June 21, 2018.

• City of Midland, TX – In a near carbon copy of the Midwest City breach, Midland  was first compromised on or around December 1, 2017. The incident was discovered concurrently as well, on June 21, 2018.

• City of Bozeman, MT – On July 16, 2018 Bozeman announced that customers using their Click2Gov installation between July 1, 2017 and October 24, 2017 had their payment card information compromised. Curiously, the city first became aware of the problem in the Fall of 2017, when customers started reporting fraudulent charges popping up after paying their utility bills online. At that time Bozeman hired a forensic firm to assist with the investigation but could find no evidence payment card data was taken. On July 3, 2018, Superion reached out to the city informing – or more accurately confirming – their installation had been compromised.

• City of Medford, OR – On July 23, 2018 Medford announced they had been breached, with malware capturing customer payment details from February 18, 2018 to March 14, 2018 and again between March 29th and April 16, 2018. Like so many other cities, the compromise was first identified when fraudulent charges began appearing on customers’ accounts.

• City of Bossier City, LA – On August 16, 2018 with very little fanfare and even less detail, Bossier City announced the system that allowed customers to pay their utility bills online may have been compromised. Although the notice made no mention of Click2Gov, it is clear from the payment portal URL who is behind the service: https://epayments.bossiercity.org/Click2GovCX/index.html. An announcement on the city’s website indicates the service will be unavailable until late October.

• City of San Angelo, TX Water Utilities – On August 17, 2018 – one day after Bossier City’s announcement – the water utility service for San Angelo released an equally quiet statement that the city had temporarily suspended their Click2Gov platform due to reports from customers of suspicious card activity after paying bills online. Little else has been reported about this event.

• City of Tyler, TX – On September 10, 2018, Tyler announced they had become the latest victim in the long string of attacks. Malicious actors gained access to their Click2Gov installation around June 18, 2018 – just 4 days after our original post. Like the others before them, the breach came to light after an external party, this time the Secret Service, reached out to officials informing them of suspicious activity.

In all, the Cyber Risk Analytics research team has linked 18 breaches targeting Superion’s Click2Gov service. With the City of Tyler coming forward just last week, it’s clear the campaign continues and it’s likely we’ll continue to see Click2Gov in the headlines. We’ll monitor the story and keep you updated as more information comes to light.