Do Specifications Lead To More Secure Code
Written by Alex Armstrong   
Tuesday, 02 March 2021

A recent arXiv paper provides insights into whether program specification is a useful tool for security-related tasks. It also reveals that developers often fail to store passwords securely, despite claiming to be doing it right.

Concerned about the safe storage of passwords, researchers at the University of Bristol, set out to investigate:

  1. Does specification writing lead to a measurable improvement in password storage method?
  2. What approaches do developers take when implementing password storage
  3. How do developers justify their implementation approach 

To look into these three questions they asked 138 developers to write code to store a password in whatever language they were most comfortable with or in pseudocode. Half the developers were asked to write down a specification of how they would implement the solution before proceeding to write it and this could be any specification from a simple description through to a formal definition. The others were allowed to write their implementation immediately and later asked if they had made some form of specification or plan before starting their implementation (without being asked to), in which case their results were allocated to the specification group. 

Both groups were asked to provide a rationale for their coding approach in a free text box. They were asked if they considered what threats might attack a stored password, and, whether they referred to any standards for password storage when implementing the code. 

The security of all implementations were scored using a metric for password storage security based on the NIST SP 800-63-3 standard and attributed to Naiskshina et al:


With regard to the main question addressed by this research, namely the effect of writing a specification, the  results of the study were that developers prompted for a specification (n = 61) scored better (µ = 1.03) than those that were unprompted (n = 77, µ = 0.47). However, you'll notice that an average score of just over one point is hardly impressive. In fact only 53 developers (38%) produced outputs that fulfilled at least one part of Naiakshina’s criteria.

Looking into the results to see which of the approaches included in the Naiskshina citeria developers used for password security, the most common was that of hashing data (demonstrated by 38% of participants who scored a point, 14% of overall sample). Just under 20% of the developers who scored a point used a random salt or an appropriate hash length (7% overall); and the remainder of the points in Naiakshina’s criteria were awarded rarely. 

In fact developers in the study seemed to be unaware of Naiskshina's criteria or the standard behind it. According to the researchers:

An ideal specification might have listed the criteria in full as functional requirements, but it might also have been as simple as: “store the password securely, following NIST SP 800-63-3.”; yet none of the developers in our study made reference to any standard in their specifications. Developers seemed to know there was a best practice they ought to be following, yet didn’t appear to go look up what it actually was.

Turning to their third question about how developers justified their implementations, the paper notes:

Our analysis of the reasons why developers implemented password storage in the ways they did reveal two interesting sub-groups. Several answers appear to indicate that the developers thought they had stored the passwords properly ([citing] experience, replication of previous efforts, perceived best practice and taken under advisement codes); whereas others seemed to know that their implementation was limited and that they didn’t know how to do it.

The finding I take away from this research is that, whether prompted on not, 62% of developers failed to hash, salt or add any security mechanism whatsoever when implementing password storage. Commenting on this the paper states:

Explicitly prompting developers to write a specification does help improve the quality of password storage; but developers are still mostly failing at password storage whilst still believing they are getting it right. Giving developers time to reflect helps them realize the limitations of their approach: but until we have developer-centered usable password storage methods, the problem of poorly stored passwords isn’t going away. 


More Information

Do specifications lead to securely stored passwords?

by Joseph Hallett, Nikhil Patnaik, Benjamin Shreeve, Awais Rashid

Related Articles

Hashing - The Greatest Idea In Programming

Fluid Passwords - Never The Same Password

Veracode Reveals Security Flaws

Ever Increasing Need For Secure Programming

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.


Does AI Copy Code - Lawsuit Says No

Are we worried about AI code assistants? Well some of us were worried and offended enough to take GitHub/ Microsoft and Open AI to court over code copying by GitHub Copilot. But the judge came down on [ ... ]

Pg_lakehouse Makes PostgreSQL Quack

Pg_Lakehouse from ParadeDB is an extension that turns PostgreSQL into the analytical engine of DuckDB. Why is that useful? How do you use it?

More News

kotlin book



or email your comment to:

Last Updated ( Tuesday, 02 March 2021 )