Security of Self-Hosted Devops Tools

Sentry is an error/log aggregation platform. It offers lots of really great functionality for exploring errors generated by your applications. If you haven’t given it a try yet, I really recommend it. The guys behind Sentry offer a hosted solution or, like myself, you can self-host if you wish.

I recently received an email from a chap named Julian Uphoff. He very kindly notified me that my Sentry installation was open for public registration. This is apparently the default for new self-hosted Sentry installations. He’s written an excellent blog post detailing his methodology.

I quickly disabled public registrations and assessed the damage. Thankfully, Julian himself had been the only registered user, and even if others had registered they couldn’t see my projects or any other application-specific data. They could, however, see my email address and the email addresses of any other registered users, which Julian pointed out would be quite useful to anyone hoping to mount a social engineering attack.

Looking Further

Shortly after receiving the email from Julian, I became concerned that I might have overlooked other, similar self-hosted devops tools that might have public registration enabled out of the box. I checked a few installations of various different tools and thankfully found that only one had a similar concern - Grafana.

Grafana is a metrics visualisation/dashboarding tool. Again, it’s an absolutely fantastic tool that is incredibly flexible and powerful. Recommended, again. However, like Sentry, it does have public registration enabled out of the box.

By default, when new users register they can see any dashboards that exist for the default (main) organisation. This is quite bad, as I imagine a lot of organisations have some sensitive information about their application internals on those dashboards. Aside from infosec sensitive information, this could also include commercially sensitive information like numbers of users, revenue, costs etc. Additionally, malicious users could potentially use Grafana as a proxy to access other systems inside networks or make queries against systems that Grafana is configured to use as a data source and extract sensitive information.

I disabled public registration by uncommenting/adding the following line to /etc/grafana/grafana.ini under the [users] section:

[users]
allow_sign_up = false

Then restarted the grafana-server service.

Survey

I did a quick survey similar to that performed by Julian on Sentry installations. The results were not particularly surprising. Of the 1000 most-recently issued SSL certificates for domains matching grafana.% at crt.sh, over a third had public registration enabled.

292 domains hosted installations that were open for public registration, 462 domains were not (or were not a Grafana installation/were broken at the time of the test). 246 domains tested did not accept a HTTPS connection at the time of the test and therefore were not included in the above chart.

I expect this to be the worst offending group as these were the most recently obtained certificates - obtained between 11th June 2017 and 27th June 2017. Other installations have been around for longer and administrators are more likely to have noticed the public registration option and disabled it by now.

Whilst I’d like to contact everyone and let them know, there are over 13,000 certificates on the list and Grafana doesn’t reveal email addresses of registered users in the same way Sentry does, so getting in touch with each owner could be quite difficult.

Lessons

I’ve reported this as an issue to Grafana on GitHub. A developer has responded and the issue has been milestoned for the 4.4.0 release of Grafana, which is excellent. Kudos to the Grafana team for taking security through usability seriously.

Unfortunately, it looks like efforts to make Sentry Secure by Default are not being met with the same enthusiasm by the Sentry team.

I’m sure there are many more tools like Sentry and Grafana that are open for public registration by default. I’ll certainly be paying more attention to this option when I deploy a new tool for any purpose in future. I think it’s always good to remember that even when an application doesn’t provide anything particularly sensitive to attackers in itself, it can act as a vector to allow attackers to enter and then move laterally with an organisation.

Honestly, if possible, I’d try to avoid making these tools public at all. They’re devops tools - potentially containing all kinds of juicy information about internal systems and processes. They can be gold dust for bad guys. It might be best to make them available on a private intranet only and require users to connect via VPN to use them if possible.