360 is Performance, Security and Virtualization

Understanding, Commissioning, & Maximising Value from Penetration Testing

This paper provides the reader with a contemporary, balanced view of Penetration Testing. The information contained herein will prevent you from being bamboozled by jargon and will ensure that you remain in full control of your project while enlisting help from security specialists. It will put you in a better position to know what sort of Security Assessment your organisation needs, how frequently it should be repeated, how to prepare for it, and what to do with the findings when they are delivered.

1. An Introduction To Penetration Testing

1.1 Prologue

This chapter is the first of 3 aimed at helping organisations and individuals achieve their goals and obtain maximum value from Penetration Testing and Security Assessments. Unlike much of the available literature on the subject, it has been written using the minimum of jargon and takes a UK/European viewpoint rather than a US-centric one. It is concordant with the regulatory, legal, and commercial environment at time of publication, though ultimately shares a weakness with all static texts; the document represents a snapshot in time. If you require updated information, or have questions or corrections for the authors, please address them using the contact details at the end of this document. Feedback is most welcome.

1.2 Introduction

In this the first of three chapters, we introduce the reader to some standard terminology for talking about Penetration Testing and briefly cover the questions that most organisations and individuals will have about the activity, not having previously embarked on Security Assessment or Penetration Testing project. We also take a step back and look at where Penetration Testing fits within the wider context of Security Assurance, and discover what it can and cannot deliver to the IT security function of your organisation. In the final part of this chapter, we look at the different philosophies and methods of delivery for these projects, in preparation for our second chapter, Selecting a Penetration Testing Provider.

1.3 The Language & Context Of Penetration Testing

The IT industry struggles with inconsistent language, and the area of IT Security is no exception. Different writers use different words to describe the same concept. Part of the reason for such an explosion of confusing, overlapping terminology in IT security is the different perspectives of the speakers and writers, (attacker-speak versus defender-speak), the technology (server-speak versus network-speak), or even the role of the person communicating (auditor-speak versus system administrator-speak). This proliferation of terminology is made worse by the marketing efforts of security product vendors, as each seeks to put their spin on a niche within Information Security.

The inclusion of a Penetration Testing nomenclature would make this document long, dull, and perishable. While this text is internally consistent, the reader should allow for a certain amount of play between terms used here and language in other works on the subject. The author loosely defines a Penetration Test as any activity where Information Security measures are subjected to aggressive, practical, investigation, in order to identify weaknesses. Advancements in technology and changes in industry practices have a habit of rendering disagreements over terminology irrelevant in the long term. We shall not labour them here.

We consider Penetration Testing the Realpolitik of Information Security, concerned only with practical and material factors and considerations, rather than ideological notions or moralistic or ethical premises. Regardless of what you think your Information Security posture may be, Penetration Testing provides a glimpse of the reality.

This document was written to assists readers in understanding what a Penetration Test can realistically do for their Information Security program, how it fits with activities they may have previously undertaken, and how they can evaluate what prospective supplier’s tell them. This document helps to accelerate early-stage discussions, by equipping the reader with an independent, contemporary, UK-centric explanation of the subject.

When meeting with a client for the first time, 360is consultants frequently spend time establishing a common frame of reference and form of language for Security Assessments. Previously the client may have had a sobering external security audit, their own system administrators might have made a troubling vulnerability analysis, or the organisation could have had a “scan” that highlighted a multitude of seemingly critical weaknesses. Early on in the discussion we establish what triggered their interest in a Penetration Test, what the results were of any prior exercise, what the output was, and what was ultimately done with that output.

1.4 What Is A Penetration Test?

A Penetration Test or Security Assessment, takes the view of an outsider who may or may not have access to initial information that would help them attack and circumvent your Information Security measures. It could be your company domain name, an IP address range, or a specific list of servers and infrastructure. This information is commonly used to limit the scope of a test, in order to focus on a particular organisational division, geography, or group of systems. “Blind” or “Black Box” tests, where no prior information or direction is supplied, are less common today than in the past. Most independent Security Assessments are targeted against a specific, constrained, group of systems or applications.

The practical activity itself involves researching the target, probing defenses and attempting to circumvent security measures. Vulnerabilities are detected and catalogued and may be exploited to uncover further vulnerabilities, or combined with other attacks which ultimately lead to compromise of the target’s security measures. Penetration testers assume nothing about a target, and often take an unconventional approach in order to defeat its security. This is no theoretical proof or paper exercise. No points are awarded for having ticked all the classical security boxes, when a left-of-field attack succeeds in compromising a critical asset.

1.5 The Limitations Of Penetration Testing

Today, most Information Security professionals acknowledge that Penetration Testing has something to contribute to an organisations spectrum of activities. This has not always been the case however. At first, traditional Information Systems auditors didn’t consider active Security Assessments a valuable exercise, this changed once their closed systems were opened up to partners and customers through TCP/IP networking and eventually the public Internet.

In turn, Penetration Testing professionals have become better at expressing the results of their work in terms that are useful, and organisations have become better at using the results to inform medium to long term security strategy rather than just tactical hole-plugging. However, Penetration Testing has its limitations and it is important that all parties understand them.

If Penetration Testing alone were an efficient way of securing networks and applications, then software like Internet Information Server (released mid-1995 and revised many times since) would by now be very secure. It is not; testers still uncover serious security flaws in this application at the rate of 1 or 2 per month. While Penetration Testing deals with reality, it does not on its own result in systems that are secure by design. If your security team repeatedly scrambles to address this quarter’s web vulnerability, or this year’s Windows worm, then you should be investing strategically in infrastructure that is more secure by design . Penetration Testing alone will not transform your security outcome.
Typically, a Security Assessment or Penetration Test results in a carefully prioritised report of vulnerabilities for the target to remedy. Besides facilitating the immediate plugging of holes, a good report will identify wider issues with security process, people, and policy in your organisation.

Sir Ian Andrews“If people apply basic standards of information assurance, basic secure operating procedures, an awful lot of the problems we have would be defeated.”

Sir Ian Andrews, Chairman, Serious Organised Crime Agency

1.6 Why Commission A Penetration Test?

The security industry (along with the help of mainstream media) does itself few favours by mystifying the work of Penetration Testers. The real utility of Security Assessments is far simpler than you may have been lead to believe. Simply put, a good Penetration Test will allow you to identify:

  • Significant un-patched vulnerabilities in software and systems
  • Failure in internal procedures (e.g. use of insecure OS builds/configuration)
  • Compromised systems already present in your organisation
  • Orphaned/unmaintained systems
  • Relics/retired systems that should have been removed long ago
  • Badly configured (e.g. unduly permissive) systems
  • Unintended consequences within complex environments

It may also identify weaknesses in your own IT staff. A typical example of which might be a Windows administrator put in charge of a UNIX estate, or developers also acting as administrators on their own systems. While one must accept that Penetration Testing will not identify every weakness in your systems, it does provide an indication of the vulnerability profile you present to attackers.

360is act frequently as an independent external assessor, however we are also sometimes retained to assemble internal-teams and infrastructure within larger organisations. Those teams then go on to conduct the testing, reporting, and remediation themselves.

1.7 Independent Or In-House Assessment?

When embarking on a Security Assessment, one must consider whether or not to engage an external party to conduct the work, or to task existing staff members with the project.

The choice of in-house versus independent assessor will be influenced by several factors, many of which will be specific to your organisation or situation. Once the drivers for a Penetration Test are fully understood, it is normally clear which route (independent or in-house) makes sense.

Iain Lobban, Director GCHQ“I can attest to attempts to steal British ideas and designs - in the IT, technology, defence, engineering and energy sectors as well as other industries - to gain commercial advantage or to profit from secret knowledge of contractual arrangements”

Iain Lobban, Director GCHQ

1.7.1 Using An Independent Security Company

If your requirement is driven by a broader company audit, industry regulation, or is linked to current or future legal process (confidentiality, product liability, UK Data Protection Act, or the many relevant European Directives) then an independent assessment is advisable. Likewise, if an external entity mandates that you have a Penetration Test as part of the trading relationship, “we audit ourselves” is not going to provide them with quite the level of assurance that using an external specialist would.

Even if your organisation maintains a highly competent, separate internal security team, it may not be prudent to have them assess a colleague’s handiwork. Indeed under certain circumstances legislation allows no flexibility whatever in choosing whether to use an independent provider or an in-house project team (see Payment Card, Online Gaming, Government, and parts of the Financial Services industry).

If there is a chance that serious failures uncovered during a Penetration Test will create friction or if internal organisational politics may stand in the way of a full and frank analysis of the results, then use of a politically neutral external 3rd party is advisable. From a purely practical point of view, in order to successfully operate your own Penetration Testing program, you will need both talent (skill) and resource (people), and depending on the size of your estate you also may require tools (software and hardware infrastructure) to help with automation and processing of the results. If your organisation lacks any of these then an external independent assessment offers the only way to achieve your goal.

One of the most important features of a Penetration Test is that it is an evaluation of your infrastructure against the very latest vulnerabilities and hacker techniques. In order to stay current, an external testing company will send it’s analysts to the right security conferences (BlackHat, DEF CON, t2, and RSA, among others) most of which are not held in the UK. By using an external assessment company, you are offloading the expense of sending your staff to these conferences.

External security assessment providers also monitor underground boards, chatrooms, and unconferences, where zero-day vulnerabilities, exploits, and stolen data are regularly exchanged. Unless specifically exempted, it is probably against your own security and ethics policy for any or your employees to engage in such activities using company time and equipment. By using an external assessment company you obviate the need for your own staff to engage in such difficult and delicate work.

360is have built and equipped security teams at some of the UK’s largest telecommunications providers. Ask our consultants about choosing the right structure, skills and personalities for your team, and how to provide them with appropriate technology and training to do the job. Our blueprint for an in-house penetration testing office is outside the scope of this document; instead we shall look at how such a team if already present, might conduct a Security Assessment.

1.7.2 Using An In-House Security Team

Typically only larger organisations in the UK have a dedicated IT security team consisting of more than just 1 or 2 people. If present, such a team may operate in an environment with thousands or tens of thousands of internal systems under their watchful eye. If your organisation is fortunate enough to have its own specialist IT security team then using them to perform a Penetration Test is an option.

In spite of limitations discussed previously, a Security Assessment or Penetration Test carried out by your internal IT security team can still be a worthwhile exercise. Indeed, an internal team is well placed to know:

  • The relative business importance of a system or application
  • Future developments, which may make findings more or less significant
  • If the presence of a given service/application is anomalous

Penetration Testing is a time consuming process, in particular the element of vulnerability chaining (where 1 or 2 minor weaknesses may be combined to facilitate a serious compromise), the elimination of false positives (which generate unnecessary remediation activity) and ruthless prioritisation (essential if more than a handful of systems are being assessed). While the operating expense of an in-house security team is already funded, consider whether their time would not be better spent on activity that is less easy to outsource, such as remediation work, or strategic improvement projects.

1.7.3 Automated Or Expert-Lead

Any Security Assessment or Penetration Test will have a degree of scripting or automation, if only to make the exploitation of certain vulnerabilties quicker for the tester.
Fully automated scanning services have the advantage of very low cost relative to consultant-based services (a few hundred or a few thousands of pounds per scan depending on the size of your network). They are also by their very nature easily repeated, for example on a monthly basis. However, it is important to understand the limitations of a fully automated approach, be it managed in-house or outsourced as a service to some 3rd party.

  • These automated scans cannot know the relative value of a vulnerable asset
  • Prioritisation is therefore based upon purely technical criteria
  • False positives can lead to a large number of spurious findings
  • It takes time for vulnerabilities to make their way into the scan databases
  • Some hacker exploits are never added
  • No way of knowing how well you are performing versus your peers
  • No ability to discuss findings (tactical or strategic) with an expert
  • May require you commit to a contract with a minimum term
  • Sensitive information about your vulnerabilities is now somewhere “in the cloud”, possibly in another country, without you knowing exactly where or how it is being kept
  • Potential exists for an automated test to disrupt target systems, and for such disruption to go unnoticed (or recurr) since the test is not being driven by an analyst.
    Automated systems don’t phone your help-desk when something breaks.

For very large networks, the capital cost of setting up an automated scanning infrastructure in-house may also be significant as you will require several scanning systems, an enterprise console, and the database infrastructure that supports it. Additional 3rd party software may be required to visualize the results in a way that enables prioritisation and strategic improvement to be made.

Our consultants endorse automated “scanning” (including 3rd party automated services) as part of a comprehensive Penetration Testing program.

1.7.4 Depth Versus Frequency

With the advent of GUI-driven automated scanning (provided as a service, software license, or free download) network administrators may check for vulnerabilities in their infrastructure more frequently than ever before. However, most organisations change their infrastructure relatively infrequently and it is their ability to absorb the output from such scanning exercises and carry out remedial actions that becomes the bottleneck, not the software’s ability to generate findings. Most large companies with dedicated IT Security departments find that a monthly automated scan, and quarterly expert-lead assessment is sufficient. Smaller organisations may have trouble coping with the work that this generates (taking into account the number of false positives), in which case a quarterly automated scan and an annual in-depth test may be more appropriate.

In any case, constantly reacting to scan results is little better than constantly reacting to the latest vendor vulnerability disclosure, escaping from a never-ending cycle of “penetrate and patch” (be it monthly or annually) will require a more strategic approach to your IT security.

1.8 Turning Tactical Findings Into Strategic Advice

Your system administrators and IT security analysts fight a constant battle. Vendor software updates introduce new security bugs (or re-awaken old ones), complex and changing systems present ever more opportunities for misconfiguration, unintended consequences of interactions between components further conspire to leave systems and networks vulnerable to attack. Penetration Testing will catch many of these problems before your systems are compromised, but it remains a tactical activity, initiated as a reaction to the fact that vendors are incapable of producing secure software.

A good independent Penetration Testing exercise will do more than simply indentify individual weaknesses in systems at the point of assessment. It will tell you how you compare to your peers, it can be used to track your improvement since the last assessment, and can provide valuable input to strategic decision making within your Information Systems organisation. The results of a good Security Assessment or Penetration Test are always revealing:

  • Identify missing skills or structural problems within your team
  • Highlight opportunities for strategic sourcing/outsourcing
  • Inform your vendor/component selection process based on security record
  • Measures the effectiveness of your remediation process

If vulnerability scanning is done in-house, an independent interpretation of those results will draw out the strategic conclusions.

360is advocates a practical, mixed approach, comprised of annual or quarterly expert-lead testing with monthly, automated “scans” and an alerting service for new threats. For large organisations, or those whose IT infrastructure is subject to rapid growth or change, 360is recommends quarterly tests rotated through a quartet of suppliers.

1.9 Summary

This chapter, the first of 3, attempted to succinctly describe what a Penetration Test or Security Assessment is, what it can realistically achieve, and the different methods of delivery available including the use of internal or external testers and the role of automation. In the next part we shall equip the reader to assess Penetration Testing providers, their certifications and methods, by balancing the industries marketing messages with practical facts and anecdotal evidence.

Pauline Neville Jones“There is a vast swathe of corporates who have valuable intellectual property, much more valuable than they understand, which is inadequately protected. They don’t even realise it has been stolen.”

Pauline Neville Jones, former Minister of State for Security and Counter-Terrorism, former Chairman Joint Intelligence Committee


2. Selecting A Penetration Testing Company

2 Prologue

This is the second in a trilogy of articles aimed at helping organisations and individuals achieve their goals and obtain the most value from penetration testing or security assessments. Unlike much of the available literature on the subject, it has been written using the minimum of jargon and takes a UK/European viewpoint rather than a US-centric one. It reflects the regulatory, legal, and commercial environment at time of publication, though ultimately shares a weakness with all static texts; the document represents a snapshot in time. If you require updated information, or have questions or corrections for the authors, please address them using the contact details at the end of this document. Feedback is most welcome.

2.1 Introduction

Selecting an independent, external security partner is hard, terminology differs between companies, methodologies differ, the people even look different, they have different qualifications, offer different prices for what appears to be the same project, and have different ways of engaging with you and your organisation. In spite of efforts to introduce standard practices, qualifications, and language, none of these facts are going to change any time soon.

Unlike disciplines with hundreds of years of history (education, law, engineering, medicine) IT security is relatively new and changing all the time, sometimes significantly. A particular development environment may be all the rage one year, yet is suddenly passe the next. New capabilities are added to applications and operating systems with every release; too fast for standards bodies, training courses, and expert consensus to keep up. Consider access methods, we have moved from using dumb serial terminals, client server, fat client, thin client, and now mobile devices all inside 25 years, just 1 of many aspects of Information Technology that has changed completely in a relatively short space of time.

This, the second part of our guide, takes a contrarian view and will likely engender some criticism from our peers. However Penetration Testing teaches pragmatism if nothing else, and this paper is written in that spirit. We know many in the IT security industry agree with the position we take, though they may be unable to publicly endorse it.

While the majority of readers are not experts in Penetration Testing and may have no prior experience of IT security matters, we refute any assertion that it is impossible for such a person to do a good job of selecting and appointing a Penetration Testing provider. Every day people appoint expert help for important, high-value, high-risk projects, whether they are choosing a house builder, a doctor, or financial planner. Choosing a Penetration Testing provider is no different as we hope to prove.

One must be careful to distinguish between reputable, long established, accredited, academic institutions running respected Information Security courses and the increasing number of unaccredited, organisations which we regard as little more than degree mills. Certifications are available from such “Universities” by mail for as little as $250, literally “no questions asked”.

2.2 Certification & Qualifications

Over the last 15 years there has been an explosion in the number of IT security certifications and qualifications that can be obtained through online and in-person training courses and programs. Today there are perhaps 60 mainstream vendor-neutral security industry certifications, another 20 that are specific to a niche (such as the Payment Card Industry), in addition there are several hundred vendor-specific qualifications centred on a particular commercial product or vendor ecosystem. There are still relatively few academic qualifications in Information Security, although the number is growing and at least one highly regarded post-graduate course is offered by a respected UK University.

In addition to individual professional or vendor certifications, some schemes also attempt to certify a company. Normally this means that the firm has paid an annual membership fee to the accreditation body, and that they have agreed to maintain some minimum number of active, certified consultants, and to keep those consultants current with annual paid-for re-certification. If a company fails to pay its fees, or in some way fails to meet the criteria for membership, then it is immediately stripped of certification.

In return for the fees, the accreditation body publishes a list of members and may undertake marketing activities aimed at promoting the certification and driving business to members of the scheme. While such schemes normally start with a handful of founding members promoting a high degree of assurance (and an implied commitment that the pool of members will be small and therefore certification will be perceived as valuable), they tend to grow quickly and eventually become fee-driven. At this point the scheme fragments and an alternate body is established by dissatisfied members of the first. This cycle then repeats every few years.

2.3 Security Clearance & Vetting

The UK Ministry Of Defence and certain parts of central government may require that consultants be “Security Cleared” (SC) in order to work on a project. While this is not generally a requirement for external assessments (where the consultant is not working with protectively marked information) it may be needed if they are working on the LAN handling private systems and sensitive information as part of an internal Security Assessment. If TOP SECRET assets are to be assessed unsupervised, the more stringent “Developed Vetting” (DV) is applied. For large contracts, you should approach a Project Officer in the Defence Procurement Agency (DPA) or Defence Logistics Organisation (DLO) to sponsor your chosen consultant. For consultants in smaller sub-contracted organisations, sponsorship can be provided through the prime contractor. Any clearance granted has a limited lifetime.

While other European countries have different rules regarding Security Clearance and government work, most are broadly similar to the process operated in the UK.

Bruce Schneier“I’ve met too many bad security professionals with certifications and know many excellent security professionals without certifications.”

Bruce Schneier, Cryptographer & Author.

2.4 Ethics

It goes without saying that the individual or organisation you trust with validating your Information Security should exhibit behaviour in accordance with the highest ethical standards, but in a practical sense what does that really mean?
Some Penetration Testing companies also resell security products and services for vendors or are employed by those software and hardware vendors to provide security advice to them in building their products. This can lead to a conflict of interest:

A Security Assessment identifies a need for a new Firewall, the Penetration Testing firm is financially incentivised to supply and configure one from the vendor whose products they resell above any other solution.

A security flaw is detected in an application that is being assessed at the request of the software vendor, as part of an exercise that is subject to Non Disclosure Agreement. Subsequently the same flaw is detected in a Penetration Test for a different client. The agreement with the vendor precludes disclosure of the security bug to the client.

Both of these cases present a conflict of interest. The former calls into question the Penetration Testing firm’s independence in recommending a product they make a business reselling, and for which they likely have a quarterly sales target and volume commitment. When considering the latter, one can imagine how a software vendor supplying a reseller who also performs Penetration Testing, might react to scathing findings in a report prepared for one of the resellers other clients. At the very least, his superiors may put the security consultant conducting the test in an extremely difficult position. In the past there have been instances of security consultants being fired for criticising vendor products.

In our experience Vendor Non Disclosure Agreements, Reseller Contracts, and Vendor Partnership Agreements create a legal and ethical minefield when attempting to perform Penetration Testing services. For this reason 360is does not resell any security products and prefers to remain completely neutral, recommending whatever technology is the best fit for a given client. 360is works in the sole interest of our clients, free of any conflicting agreements, incentives, or vendor relationships.

2.5 Experience

Penetration Testing like any investigation or audit work (financial audit, criminal investigation, information-technology audit), is best conducted with the benefit of extensive practical experience. Experience is important because security vulnerabilities come in such a wide variety, are not always easy to spot, and even relatively small infrastructures can harbour hundreds of them. It takes experience to be able to prioritise hundreds or thousands of vulnerabilities into a smaller, workable, schedule of remedial actions. Previously we have explained that the true value of a Penetration Testing exercise is in the strategic lessons that can be learned from the results. It takes experience to turn a morass of findings into strategic guidance.

The best way for an individual or organisation to demonstrate experience in the field of Security Assessments and Penetration Testing is by describing the many successful projects they have completed over several years, and by offering references from satisfied clients or former employers. Although case studies and prepared materials provide good background information, the truly experienced individual will be able to answer questions face to face in a relaxed, agile, and lucid manner. They will enjoy the opportunity to demonstrate their deep understanding of the subject.

Pauline Neville Jones“I learned I was fired from a press release. When I did eventually speak to the CEO, it was cold and short, and he had nothing to say but, “Your services are no longer required.”

Dan Geer, Former CTO @Stake (Symantec), distinguished security researcher

2.5.1 Vertical Sector Experience

Some industries are subject to specific challenges in the area of Information Security. These challenges may take the form of regulatory compliance, stipulations made by partners or customers, or increasingly, threats to hard-won Intellectual Property obtained through costly research and development. Selecting a Penetration Testing partner with specific experience of your sector will mean they can add relevance to the reported technical findings and strategic recommendations. At the very least such a firm or individual can set findings within the context of your industry peers. It is worth something to know that “For a medium sized Financial Services firm, you are within the top 25% of your peer group for secure deployment”.

2.5.2 Quantifying The Value Of Experience

Common sense dictates that of the factors influencing the outcome of a Penetration Test, the experience-level of the specialist performing the work probably has the greatest impact. A more experienced consultant will require less time to reach the same conclusions as a journeyman analyst, and less time normally means less cost. While professional rates may be higher for a more skilled expert, the range of rates is narrower than the range of consultant’s ability, thanks in part to the fact that some firms have an annual intake of bright but completely inexperienced graduates.

A better-constructed report, of the sort produced by an expert, will normally take less time to absorb and execute than one consisting largely of unrefined raw data. A properly prioritised report, taking into account software dependencies, your organisational specifics, and the intended audience, can shave weeks or months off of the remediation phase, which is normally the most costly part of any Penetration Testing project. Finally, a more experienced testing team generally means more vulnerabilities detected, and so a greater level of assurance. While “level of assurance” is hard to put a price on, it cannot be denied that it has a value.

360is’ story starts in the early 1990s, operating the security function of the UK’s largest Internet Service Provider. Since then our consultants have been involved in detecting, investigating, and neutralising, cases of industrial espionage, professional sabotage, organised crime, and cyber activism for clients around the world.

2.6 Recommendations & References

References are vital whenever someone requires the services of a specialist, without themselves being fully qualified to evaluate that specialist. Consider again the simple case of appointing a builder; strong references (especially from a trusted 3rd party) trump qualifications or a folder of certificates every time.

While most security professionals are rightly cautious about disclosing client names and prior engagements, it is reasonable to expect references, even references that have a relevance to the project at hand, a client in the same vertical sector, or one that faced similar challenges. The very best references come from trusted sources, colloquially known as “the old boy network”. Today, social and business networking tools make it easier to identify common contacts for both parties.

Security is one of the most delicate areas of Information Technology. Successful Information Security practitioners build their career on reputation, relationships and recommendations. While most clients are sensitive about exact details of prior projects, they are normally happy to give references, attest to the quality of the work, user-friendliness of the professional or firm, and whether or not they would recommend them for the project under consideration.

Professor Ross Anderson“Fundamentally, assurance comes down to the question of whether capable, motivated people have beat up on the system enough.”

Professor Ross Anderson, Cryptographer & Author, University of Cambridge

2.7 Summary

This part of our guide to maximising value from Penetration Testing attempts to provide a neutral starting point from which the reader can begin to evaluate candidates for their project. We have introduced the subjects of qualification, certification, ethics, and experience, along with the importance of references. Selecting a Penetration Testing partner is difficult, budgets are limited, some projects are very small, making the time available to choose expert-assistance short. This paper recommends a pragmatic approach. Do as you would when employing a builder. Obtain references, invest time in talking to them about their recent projects, get past the sales and marketing and down to practicalities. Find out how a potential supplier intends to tailor their deliverable to your situation. If the contractor or firm you want is busy, either wait or ask them for details of another supplier they trust. Remember:

  • Threats change over time; formal courses and certifications always lag
  • Finding vulnerabilities is like detecting fraud; it rewards the experienced more than the book-smart. While aspects of vulnerability detection can be taught, there is no substitute for a practicing expert
  • An automaton, while designed by an expert, does not produce output tailored to your particular organisation, nor will a piece of software be accountable for any “professional advice” it gives you
  • Focus on references, face to face discussions, and the supplier’s depth
  • The contractor may be a technical maestro but if they are difficult to work with, your project may not achieve a positive outcome

In the next and final chapter we shall examine what you can do before, during and after your Penetration Test, in order to ensure the process goes smoothly and your organisation gets maximum value from the project.

3. Maximising Value From Penetration Testing

3 Prologue

This is the third and final chapter aimed at helping organizations and individuals achieve their goals and obtain the most value from Penetration Testing or Security Assessments. Unlike much of the available literature on the subject, it has been written using the minimum of jargon and takes a UK/European viewpoint rather than a US-centric one. It reflects the regulatory, legal, and commercial environment at time of publication, though ultimately shares a weakness with all static texts; the document represents a snapshot in time. If you require updated information, or have questions or corrections for the authors, please address them using the contact details at the end of this document. Feedback is most welcome.

3.1 Introduction

Penetration Tests are often commissioned as a reaction to an unforeseen event, a security breach, suspicion of information theft, a concern of the auditor or the regulator. Sometimes the event is more benign, a new IT Director wishing to know what he has inherited from his predecessor. This final section of the guide will help you regain the initiative in these situations by managing the Security Assessment process better. Following the advice here will increase the chances that your project is a success and delivers maximum value to your organisation.

Ensuring success is 2-way street; you must take some responsibility for it too. This part of our guide explains how.

3.2 Before The Penetration Test

Preparing for a Penetration Test does not mean scrambling to patch, lock-down, fix, or hide as many sins as possible before the consultant arrives. A skilled Penetration Testing company will still report some findings, your staff will still wish they had done a better job of implementing secure systems. All this frantic burst of activity will achieve is to obscure the real picture of your organisations Information Security posture, and therefore reduce the value of the Penetration Test to your organisation. It is far better to have a real picture, from which to base real decisions about how you manage Information Security. An accomplished Penetration Tester knows if you have scrambled to patch things up before the start of the project, they have seen it all before. One thing you should do however, is to ensure that a Non Disclosure Agreement is in-place if you are using anything other than an in-house testing team.
Confirm the testing company has everything they need so they can start the project on time. If they have questions then try to answer them promptly. Adhere to the same good practice you would while working with any external contractor, don’t be the bottleneck, especially if your chosen contractor is invoicing you for time-spent rather than a fixed cost engagement.

3.2.1 What To Expect

Your Penetration Testing provider should explain to you the contact procedure if there is any interruption to production systems or if you need something clarified by the consultant during the test. Almost all projects stipulate non-disruptive testing, however if something does go wrong then it is important that either party can contact the other immediately. For large clients or those with complex IT it may be that there is an unconnected problem with their systems during the test. If there is a glitch during the same timeframe as the publicised Penetration Test, the natural inclination of most help desks is to suspect the Penetration Test is somehow a causative factor. Good communication is essential. Genuine disruptions are rare, although the chance of problems increases with the age of the infrastructure. Legacy systems, such as those from the early 90’s were not designed with the threat of today’s Internet in mind. Experienced testers know how to work with these often mission critical legacy systems, in such a way as to complete their work without impacting availability.

360is notifies the client immediately upon detection of what we call a “red ball”, a system that is wide-open and in danger of imminent compromise or worse, an already compromised system. Full details of the vulnerable system and more strategic remediation advice are included in the final report. This instant notification is reserved for discoveries so serious it is vital that they be fixed immediately.

3.3 During The Penetration Test

3.3.1 Logging

Expect increased amounts of data logging by your systems, especially Web Servers and security devices such as Firewalls, Intrusion Detection Systems (IDS) or Security Information Event Management (SIEM) systems they feed. If you know that some of these systems struggle with day-to-day logging, be prepared for dramatically increased traffic during the test.

3.3.2 Alarms

It is very likely that the active part of the Penetration Test will generate some security alarms, if you make use of reflexive/reactive security measures then consider what will happen if they are triggered. Penetration Testing analysts will begin with a light touch and get increasingly probing as they detect an interesting potential vulnerability, this will be reflected in logs and alerts. Consider tuning your IDS/SIEM to screen out the IP address of the tester. These alarms may come out of business hours, as your testing partner may deliberately work on the systems during their quiet-hours, or may be executing an automated process that continues into the night. Ensure first responders to any such alarms know that there is a test in progress, and do not waste energy countering the perceived attack.

3.3.3 Interim Findings

Your Penetration Testing provider may provide interim findings, particularly for long- running projects, although the strategic findings only tend to come out when all or most of the data is analysed.

3.4 Final Report Delivery

The final report contains information that is extremely sensitive. Findings can impact reputation, share price, peoples employment, relationships with technology vendors, outsource partners (if you use any), findings can have legal ramifications, and affect your position with the industry regulator. For these reasons both the subject and the expert performing the assessment should treat it as highly confidential. Your organisation may already have guidance in place for the handling of such sensitive material, guidance on whether or not it can be duplicated, where it should be stored, how it may be distributed, and to what level of executive. Some organisations follow a practice used by financial auditors, using code words to represent business units, operating companies, geographies, and projects. Now that strong encryption is inexpensive and easier to use, this practice has declined. Password protection built in to some document formats does not have a strong security record, and should be counted on only to deter the casual inquisitive employee.

360is distribute final reports in accordance with the client’s wishes. Generally such documents are protected by strong public-key encryption and are further password protected as individual files. Our client Information handling procedures dictate that raw data, interim findings, and final reports are stored encrypted. For the most sensitive engagements and at the request of the client, keywords are used in place of some proper nouns.

3.5 After The Report

Delivery of the final report is far from the end of any Penetration Testing project, it is, merely “the end of the beginning”. Certainly for the organisation that is the subject of the test, it is the point at which the real work starts. For very large networks it may take months or years to implement all the findings, both tactical and strategic. A final report may run from tens to hundreds of pages depending on the number of individual findings and the degree to which the detailed technical information within has been consolidated. At some point most reports are divided up into chunks that are then distributed internally to stakeholders and those responsible for implementing or considering the different recommendations.

3.5.1 Distributing The Findings

The structure of your IT organisation and how many findings the Penetration Testing exercise has uncovered will determine how sophisticated you need to be in assigning remedial actions to those charged with doing the work. There are a myriad of options for dividing up findings:

  • Team leaders want vulnerability counts before deploying staff on remediation
  • Technical team members need a detailed view of individual systems and vulnerabilities
  • Separate Microsoft and UNIX teams, or split engineers into OS and Application groups
  • In small companies, a specific individual may look after groups of systems exclusively
  • If you develop some of your own applications, it may be appropriate to divide findings into in-house versus commercial off the shelf applications
  • Some divide by severity, with the internal security team addressing the highest priority problems, while medium and low priority fixes are left to regular IT staff to fix
  • Putting long-term strategy adjustments into action involves architecture or strategy groups

Your Penetration Testing provider will help you decide how to distribute and implement the recommendations in their report. They will also be able to advise you on tactics that will increase the likelihood that those tasked with taking remedial action actually do their job in a timely manner.

Our reports list vulnerabilities according to a proprietary 4-point system, taking into account both business impact and technical factors. Using this system, any number of findings may be processed into a manageable list of priorities for the remediation team. We are able to provide different views into your vulnerability data, by Geography, Business Unit, Administrative Function, Operating System, or Application.

3.5.2 Closing The Loop

Once remediation work has been completed, or at the very least the high priority problems have been addressed, it is prudent to re-test the infrastructure to ensure that the fixes have had the desired effect. In all but the smallest, simplest cases, it is impossible to eliminate 100% of the vulnerabilities, but a re-test should show significant progress and the elimination of all high priority vulnerabilities.

3.5.3 Learning Strategic Lessons

Fixing individual vulnerabilities by patching, disabling, or reconfiguring Operating Systems and applications is relatively straightforward. Interdependencies conspire to make this harder, but with modern technologies such as Virtual Machines, Containers, and Application Isolation Environments, an accomplished system administrator is rarely prevented from implementing the technical recommendations of a Penetration Test.

Taking action on the strategic recommendations generally requires more effort and greater expenditure than closing off a few (even quite a few) vulnerabilities. Implementing the strategic recommendations may mean structural changes to the IT department, the selective outsourcing (or in-sourcing) of certain functions, changes in procurement, the creation of new processes or teams and possibly the elimination of old ones.

360is recommend that your testing program find a senior sponsor within IT, someone whose word has the power to motivate disparate groups within to work together to carry out any recommendations. Knowing that a re-test will be conducted and the performance of the those tasked with remediation work will be noted, works wonders in getting fixes done before the deadline, as does having a prize for “most improved” department or infrastructure.

3.6 Why Penetration Tests Fail

When we talk about Penetration Tests failing, we do not mean that the technical exercise fails to find every vulnerability (no test will ever innumerate all the flaws in a single non-trivial application, let alone an entire infrastructure). We say a Penetration Testing program has failed when it does not improve the subject’s security outcomes. Unfortunately, Penetration Test failure is a reality for many organisations, whether they use internal or external staff, and automated or expert-lead analysis. While the security industry remains preoccupied with certifications, taxonomy, and technical frameworks, less attention is paid to improving the end- to-end process up to and including security outcomes. It is easy to see how a vicious circle might develop. Weak outcomes erode perceived value in testing, which erodes budgets, this results in less comprehensive testing, and that means weaker outcomes.

If you find yourself in this cycle or wish to avoid it, then you may find the following list of “what goes wrong” helpful. Do you recognise the following?

3.6.1 Why Problems Do Not Get Fixed
  • Too many vulnerabilities, even the prioritised list is overwhelming
  • Vulnerabilities are simply ignored and never fixed. No perceived consequences
  • System administrators are too busy or not allowed to make time
  • Staff don’t understand the importance of what the report is telling them
  • Skills/expertise does not exist to follow the advice given
  • Software vendors do not co-operate or fail to do their part in some way
  • It is impossible to take systems out of service to work on them
  • There is no maintenance window during which to work
  • Staff are terrified of breaking something by patching/changing it
  • Wrong attitude. Staff would rather hide problems from management
3.6.2 Why Fixes Are Often Temporary
  • Lack of process. No established secure new OS/Application build is created
  • Lack of change control. Unauthorised/dangerous changes are subsequently made
  • Lack of skill. Bad decisions are made when administering systems
  • Strategic lessons in the report are simply never learnt or remembered
  • No periodic (e.g. annual) re-testing. Leads to a gradual decline in security
  • Processes are never revised in light of periodic testing results

3.7 Summary

Maximising value from Penetration Testing is your responsibility as the commissioning party with the power to appoint those working on the project (be they an internal security team or an external firm of specialist consultants). Penetration Testing is not a passive activity for the target, although those with a purely technical outlook may assume it to be. At the very least it is important you expend significant effort preparing in order to increase your chances of having a successful project.

Focus on what you will do with the results of your test, and how you will put them into action with particular emphasis on strategic advice. In doing this you will increase your chances of extracting the maximum possible value the project. Finally, hedge against failure, before the work begins ask the question “Do I recognise any of the factors described in the section on Why Penetration Tests Fail?” If you do, take steps to address those weaknesses early.

3.8 Conclusion

This paper attempts to provide you with a contemporary and balanced view of Penetration Testing. Whether using an independent expert or an internal security team, the information continued herein is designed to help prevent you being bamboozled by jargon and ensure that you remain in full control of your project while enlisting help from a qualified security specialist. Having read this document you will be in a better position to know what sort of security assessment your organisation needs, how frequently it should be repeated, how to prepare for the project and what to do with the findings when they are delivered. Using the information here along with your own observations, you should find it easier to justify the time and expenditure required to execute your Penetration Testing project.

Penetration Testing is not a passive activity that one is simply subjected to like an inoculation. It is more like a personal fitness program; it requires engagement and participation in order to achieve the best results, furthermore, in order to stay fit such programs dictate that you make time for regular exercise and potentially eliminate some of your unhealthy habits. Penetration Testing is no different.

Organisations achieving the most value from their Penetration Testing projects, work with those doing the technical analysis to plot their progress over time. These organisations fine-tune their approach to IT security using empirical data based on what the evidence shows them. Organisations accept that it is prudent to have IT security assessed. By maximising the value of this assessment you can create a virtuous cycle that improves your security outcomes.

Major General Jonathan Shaw“We once thought of AIDS as an existential threat, now we live with it. Our reaction to cyber threat today is similarly out of balance…. we’re never going to cure it, we have to live with it… But how much intellectual property will we have left by the time we get it right?”

Major General Jonathan Shaw, commander of UK Cyber Policy at the Ministry of Defence


About The Author

Nick Hutton has been researching Information Security since 1990 and has been a practicing consultant since 1995. He has been a guest speaker on matters relating to Internet Security at the London School Of Economics and E-Commerce Security at the Institute Of Directors, he has been an advisor to the Criminal Intelligence unit of a major UK police force. Nick has been involved in detecting, investigating, and neutralising, cases of industrial espionage, professional sabotage, organised crime, and cyber activism for clients around the world. He is a graduate of the University of Portsmouth where he read Informatics and studied Business Administration at the Tuck School Of Business, Dartmouth College, New Hampshire.

About Three Sixty Information Security Ltd

Three Sixty Information Security Ltd is a boutique professional services company. We provide expertise to the public and private sector. We enable our clients to secure their valuable data, meet compliance standards, and maintain customer confidence. Our consultants can help you reduce your exposure to global threats while capitalising on business opportunities, and protecting employees and assets. Our consultants are experts, having worked previously in senior security roles for the world’s largest and most successful ISP. With an estimated 70% of the Internet’s traffic passing over their network, they deployed and managed thousands of systems securely for blue-chip Times Top 100 and Fortune 500 customers. We are CISSP and BS7799 Lead Auditor certified, and referenced via previous engagements with top 5 Investment Banks, Telcos, Online Gaming and E-Commerce companies.

Back to Top