Don’t Give Special people Special Access

Written for Uptycs by Lee Atchison

When building modern applications, an astute observer will quickly see an issue in the development process. How do you give developers and operations engineers access to the application to build, fix, and operate the application, without exposing customer data or violating business processes and systems? After all, if your engineers have access to all the data in the database, they have access to customer private information. And if they can modify data in the database, or create or change privileged communications between services, they have complete control over your business and your business processes.

Security best practices specify that engineers — both developers and IT operations personnel — should have as little access as possible to the production application and its infrastructure. Sometimes, business requirements make these restrictions even more critical. Some industry regulations (such as HIPAA), can even involve legal requirements and restrictions. This security best practice is known as the principle of least privilege.

However, this can present a problem. What happens if the site has a problem during the middle of the night, and the on-call engineer is brought in. This engineer will need access to the application­–access they may not have because of the security access requirements. How can an on-call engineer do the work they need to do without the additional permissions needed during the emergency?

The answer is permission escalation. Permission escalation is a process of giving an on-call engineer temporary increased system access during an emergency.

This increased access typically comes with increased scrutiny. The engineer’s activities while they have the increased access are typically logged, and sometimes they must be reviewed by a second set of eyes to ensure that only the necessary actions are actually being taken. Additionally, an engineer can typically only get escalated permissions during a registered site emergency. All of this ensures that a rogue engineer can’t perform malicious activities on the site — including access customer private data inappropriately — and cause damage to the system or the business. This process maintains the principle of least privilege.

There are many ways to perform permission escalation, including Break the Glass, Logged Escalation, and Two Person Escalation. Another option is to create specialized, restricted, limited scope tools to assist the engineer in performing specific critical operations, such as rebooting a server.

Engineers often don’t like the additional headaches and restrictions these rules require. They’d rather simply have all the permissions they need all the time. They argue that time is of the essence during a crisis and any delay caused by excessive hurdles getting in the way during a crisis is, in their mind, too much of a restriction.

However, site owners need to make sure their systems remain secure, and that means following all security and industry best practices. This includes abiding by all industry regulations. Giving engineers unrestrictive access to everything in a production application — including and especially customer data — is just not safe and is just not good business practice.

For more information, check out the white paper I wrote on the topic, ***Architecting a Cloud Security Strategy***, published by Uptycs.