Paul Gerrard Managing Software Projects With g g j

Assurance with Intelligence
Managing
g g Software Projects
j
With
Intelligence
Paul Gerrard
Gerrard Consulting
1 Old Forge Close
Maidenhead
Berkshire
SL6 2RD UK
e: [email protected]
w: http://gerrardconsulting.com
t: +44 (0)1628 639173
Slide 1
Assurance with Intelligence
Paul Gerrard
Paul is the founder and Principal of Gerrard Consulting, a services company focused on
increasing the success rate of IT-based projects for clients. He has conducted
assignments in all aspects of Software Testing and Quality Assurance
Assurance. Previously,
Previously he has
worked as a developer, designer, project manager and consultant for small and large
developments using all major technologies and is the webmaster of
gerrardconsulting.com and several other websites.
Paul has degrees from the Universities of Oxford and London, is Web Secretary for
the BCS SIG in Software Testing (SIGIST), Founding Chair of the ISEB Tester
Qualification Board and the host/organiser of the UK Test Management Forum
conferences. He is a regular speaker at seminars and conferences in the UK,
continental Europe and the USA and was recently awarded the “Best
Best Presentation of
the Year” prize by the BCS SIGIST.
Paul has written many papers and articles, most of which are on the Gerrard website.
With Neil Thompson, Paul wrote “Risk-Based E-Business Testing” – the standard text
for risk-based testing.
Slide 2
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 1
Assurance with Intelligence
Agenda
„
„
„
„
„
„
„
Why Projects Fail
Introduction to Project Intelligence
Results Chain Analysis
PI Strategy
Test Strategy Completion
Goal, Risk and Requirements Based Testing
Close
Slide 3
Assurance with Intelligence
Why Projects Fail
© 2003-7 Gerrard Consulting Limited
Version 1.0
Page 2
Assurance with Intelligence
Why projects fail
„
„
„
„
„
Won’t bore you with yet another survey of IT projects that
fail - take it for granted, most do
But why is this?
Hundreds of modes of project failure – all very well
documented
But a large proportion of project failures can be traced to
poor decision making:
- Constraints to decision makingg - “you
y are not allowed to decide”
- Decisions already made cannot be changed
- Decisions that are ‘too big’ to make now – it’s simply too late
Sometimes, the worst decisions are those that are never
made at all.
Slide 5
Assurance with Intelligence
Decision-making when things go wrong
„
„
„
„
Imagine a world where the project plan is perfect, and
everything goes well
There are few decisions to make except exit/acceptance
reviews
- These would be easy to make, wouldn’t they?
But plans aren’t perfect and things don’t go to plan
Most decision-making is required when things don’t go to plan
- Late delivery – we are late or expect to be late
- Poor
P
quality
lit or stability
t bilit – deliverables
d li
bl nott meeting
ti requirements,
i
t ffailil tto
stabilise or are defective
- Technical challenges that cannot be met – technology doesn’t work,
can’t demonstrate working systems, cannot achieve objectives with
proposed solution.
Slide 6
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 3
Assurance with Intelligence
Decision-making dependent on good
information
„
When called upon to make a call on delays, quality or solution
viability…
- Stakeholders and project managers usually lack the right information at
the right time
„
„
„
- Most of the information actually required is TEST EVIDENCE
Early and continuous review and testing will highlight
- Items not ready (we can’t test them)
- Poor quality (they are buggy)
- Developers
p
that cannot deliver
I call this evidence Project Intelligence
THE TEST STRATEGY IS A STRATEGY FOR GATHERING
PROJECT INTELLIGENCE.
Slide 7
Assurance with Intelligence
Our project management approach is
critically flawed
The way we plan projects and monitor/manage
progress is flawed
„ Project plans set out a prediction of the future
„ A network of interdependent activities aligned
with a calendar
„ We identify activities,
activities costs and timelines
„ We monitor activities, spend and the calendar
„ We act when progress doesn’t meet the plan.
„
Slide 8
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 4
Assurance with Intelligence
But STOP! Imagine we planned a
journey this way
„
We monitor progress by checking…
- The time – “it’s 12.45pm
p – we must be half way”
y
- The fuel gauge – “we are nearly out of gas – we must
be there!”
„
„
„
„
„
This is nonsense of course
We are monitoring INPUTS, NOT OUTPUTS
We should be monitoring ACHIEVEMENTS, not
ACTIVITY
What are achievements?
How do we monitor them?
Slide 9
Assurance with Intelligence
The measurement of achievement
„
The achievements of most significance are
„
But wait! The REAL measure of achievement is the
acceptance decision, based on evidence
„
Deliverables (on time and of good quality)
Decisions made (on the acceptability of deliverables)
Requirements
Designs
Code
S
Systems
Etc…
Testing provides that evidence.
Slide 10
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 5
Assurance with Intelligence
Pawnbroker model of IT development
• Developers usually complain
that requirements
q
change
g
• Were the users focused on the
Required
Designed
Used!
Delivered
business goal?
• Developers focus on technical
deliverables, as usual
• But who keeps them on the
right track?
• Required, designed, not delivered (cuts through deadline pressure?)
• Not required, designed, delivered (an analyst's bright idea?)
• Required, not designed, delivered (the user bribed a programmer?)
Assurance
Slide
11 with
Intelligence
Slide 11
Four-eyed plans
Increasingly
gy
Inaccurate
IT- focused
Initial Plans
The plan is a model of the project. The real project
consists of people, organisation, goals and risks.
Assurance
Slide
12 with
Intelligence
© 2003-7 Gerrard Consulting Limited
Version 1.0
Slide 12
Page 6
Assurance with Intelligence
Testing provides the “intelligence”
„
Our deliverables are intermediate project goals
„
Testing can answer these questions
„
Are they available? Are they acceptable? Do they meet
requirements??
No tests completed? The deliverable isn’t available
Test successes/failures indicate quality and whether
deliverable meets requirements
We manage product risks and goals, not activities:
-
We analyse
W
l
the
h risks
i k associated
i d with
i h this
hi d
deliverable
li
bl
We plan tests to address uncertainty
We monitor tests planned, passed and failed.
Slide 13
Assurance with Intelligence
Poor contracts result in poor plans and
poor project management
„
Inputs to project plan for delivery
„
„
The Contract
The Requirements (business or technical level)
The requirements may be good or bad
But if the contract is poor or inflexible
-
Things get forgotten, underestimated
Testing, risk management, reporting all suffer
Project Management attention on timescales and costs
(inputs) not on deliverables, quality, benefits (outputs)
Acceptance process and criteria are loose, so plans are
woolly when precision is most important.
Slide 14
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 7
Assurance with Intelligence
Introduction to Project Intelligence
Slide 15
Assurance with Intelligence
Testing throughout the life cycle: the
W model
Write
Requirements
Test the
Requirements
Logical
Design
Install
Test the
Design
Physical
Design
Build
System
Test the
Specs
Code
Slide 16
© 2003-7 Gerrard Consulting Limited
Version 1.0
Build
Software
Acceptance
Test
System
Test
Integration
Test
Unit Test
Assurance with Intelligence
Page 8
Assurance with Intelligence
Testing through the lifecycle
„
„
„
Almost all project activities generate deliverables
Project goals depend on deliverables
Every deliverable has an associated test activity
„
Testing provides the evidence of achievement
„
Static tests: reviews, inspections, static analysis etc.
Dynamic tests: component, link, system, acceptance tests
etc.
Of intermediate goals throughout a project
Of final project goal
(This is standard PRINCE approach, but is
appropriate for all methodologies, by the way).
Slide 17
Assurance with Intelligence
Three types of risk
Internal/Process
Risk
(Logistics,
Planning and
Control)
External/Project Risk
Management Domain
Few, easy to understand, manage
variances in planning and
estimation, shortfalls in staffing,
failure to track progress, lack of
quality assurance and
configuration management
resource constraints, external
interfaces, supplier relationships,
contract restrictions
Not usually under
management direct control
Usually under management control
Product Risk
Deliverables
Practitioner Domain
Many, hard to
understand, manage
The “problem of quality”
Testers are
mainly
concerned with
Product Risk
Slide 18
© 2003-7 Gerrard Consulting Limited
Version 1.0
complexity, design quality, coding
quality, non-functional issues, test
specifications.
Assurance with Intelligence
Page 9
Assurance with Intelligence
What is a test?
Deliverable
Test
Testing doesn’t change
the product in any way.
W do
We
d it
i to gather
h
‘intelligence’.
Slide 19
Deliverable
(unchanged)
intelligence
g
Assurance with Intelligence
Testing generates “project intelligence”
„
„
„
If you don’t know where you are, a map won’t help
If you don
don’tt know the status of the deliverables
deliverables, the
best plan won’t help you manage your project
A project is like driving off-road in the dark
„
„
Testing provides the headlights for the journey
Testing provides data on the status of deliverables
and generates “PROJECT INTELLIGENCE”
Stakeholders and management need intelligence to
make decisions, command and control the project.
Slide 20
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 10
Assurance with Intelligence
Slide 21
Military Intelligence?
Assurance with Intelligence
Project Intelligence starts with goals
What are our project goals?
„ Our
O goals
l d
depend
d on the
h deliverables
d li
bl
„
„
How could our deliverables fail?
„
Which are the key deliverables?
What are the dependencies between them?
Evidence required to be sure they won
won’tt fail?
How can I be sure our goals are met?
-
Evidence required to demonstrate they are met?
Slide 22
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 11
Assurance with Intelligence
Key sources of project intelligence
„
The value of project intelligence depends on how
well we achieve and communicate:
1.
2.
3.
4.
„
„
Coverage – planned and achieved
Product Risks – and the status of those risks
Business Goals – and the status of those goal
Test Status – passes, failures, incidents
Time to deliver intelligence depends on the test
process
Willingness of the organisation to accept and act
upon project intelligence is key.
Slide 23
Assurance with Intelligence
Based on established disciplines
„
The PI approach uses a combination of established
techniques (you might know one or two already!)
-
Results-Based Management
Results Chain Analysis (modelling)
Benefits Realisation
Performance Measurements
Software and System
y
Testingg Best Practices.
Slide 24
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 12
Assurance with Intelligence
PI Techniques
Results Chain Analysis
„ PI Strategy
S
„ Test Strategy Completion
„ (Assessment and Testing)
„ PI Reporting
„
Slide 25
Assurance with Intelligence
PI and the project lifecycle
Results
Chain
Analysis
PI
Strategy
Test
Strategy
Acceptance
Goal Assessment
Project Intelligence Planning
Goal Assessment
Project Planning
Goal Assessment
Project
Initiation
PI Management
Stakeholder Involvement/Project Assurance/Governance
Key:
Project planning and initiation
Project Intelligence Activities
Development activities
Review and Test activities
Slide 26
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 13
Assurance with Intelligence
Results Chain Analysis
Results Chain Analysis
Results
Chain
Analysis
PI
Strategy
Test
Strategy
Acceptance
Goal Assessment
Project Intelligence Planning
Goal Assessment
Project Planning
Goal Assessment
Project
Initiation
RCA defines the
relationship between
ggoals,, activities and risks
PI Management
Stakeholder Involvement/Project Assurance/Governance
Key:
Project planning and initiation
Project Intelligence Activities
Development activities
Review and Test activities
Slide 28
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 14
Assurance with Intelligence
Goal, Benefits and deliverables
Technology
Features
Change in
Work
Practices
Electronic
communication
with suppliers
Web
Closer working
with suppliers
Electronic entry
System
Integration
Automatic reordering
Portal
System accessible
to all
requisitioners
Project
D li
Deliverables
bl
Bar Coding
Etc. etc
Expedition and
progressing
facilities
Electronic
information
service
Supplier use and
performance
evaluation
Improved supplier
understanding
More timely
procurement
More accurate
deliveries
More timely
deliveries
To reduce
production
downtime
Project
G l
Goals
Improved coordination
between functions
Increased time for
buyers to add
value
Optimised
selection of
supplier to meet
given need
To reduce
costs
Improved
negotiating
position
Etc. etc.
Slide 29
Business
Goals
Benefits
Assurance with Intelligence
The Results Chain
Programme/Project Management
Inputs
Activities
Outputs
Outcomes
Impact
Administrative By-products
Business Results
Spending the budget,
budget meeting
deadlines, filling timesheets,
compiling RAG reports, ticking
boxes, endless meetings
Achievement of business goals as a
consequence of the investment in the
programme/project.
Waste Products?
Slide 30
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 15
Assurance with Intelligence
Results Chain Analysis (RCA)
„
„
Takes place as part of the overall project planning
process
Intended to document the relationship between:
„
Activities (initiatives)
Deliverables (outcomes)
Dependencies (contributions)
Risks and assumptions (assumptions)
Intermediate and final project goals (outcomes again)
Used to identify risks early and tie activities to goals
and risks.
Slide 31
Assurance with Intelligence
RCA Diagram elements
Slide 32
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 16
Assurance with Intelligence
Simple Results Chain example
Slide 33
Assurance with Intelligence
Using RCA
„
„
„
„
„
„
RCA models the benefits realisation process
Stakeholders and management
g
can see the road mapp
leading to the business goal
Derived from interviews and workshops
Promotes discussion, consensus and commitment
A shared understanding of the linkages between
business and IT initiatives
Makes implicit thinking explicit,
explicit brings hidden
assumptions to the surface and enables better
communication and decision making.
Slide 34
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 17
Assurance with Intelligence
PI Strategy
PI Strategy
PI Strategy generates the
assessment process
Results
Chain
Analysis
PI
Strategy
Test
Strategy
Acceptance
Goal Assessment
Project Intelligence Planning
Goal Assessment
Project Planning
Goal Assessment
Project
Initiation
PI Management
Stakeholder Involvement/Project Assurance/Governance
Key:
Project planning and initiation
Project Intelligence Activities
Development activities
Review and Test activities
Slide 36
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 18
Assurance with Intelligence
Use goals and risks to define the process
for intelligence gathering
„
„
„
„
„
Test process supports the intelligence gathering
p
process
Cannot rely on fixed, inflexible test processes (but
we keep the best of ‘best practice’)
Test process design driven by the need to report
progress against goals and risk
NB: coverage goals are included as usual
NB2: we will need new forms of assessment beyond
traditional static/dynamic testing.
Slide 37
Assurance with Intelligence
Goals and risks as PI Drivers
„
Strategy for goal-based progress reporting:
„
Intermediate and final business goals
Deliverables: documents, software, infrastructure, training,
resources etc.
Coverage goals: functional/non-functional requirements,
scenarios, regulatory requirements etc.
“To demonstrate that these goals have been met or
identify the problems that block the goal”
Strategy for risk
risk-based
based progress reporting
-
Product risks: “to demonstrate that risks have
materialised, or have been addressed”.
Slide 38
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 19
Assurance with Intelligence
Project Intelligence Strategy
Project Phase
PI Drivers
Ass. Obj.
Reqs
Design
Build
Integ
Systest
UAT
Trial
Prod.
Business goals
Requirements
Objectives for each
test phase are easily
identified
Risks
Assurance
Slide
39 with
Intelligence
Slide 39
Test Strategy Completion
© 2003-7 Gerrard Consulting Limited
Version 1.0
Page 20
Assurance with Intelligence
Test Strategy Completion
Test Strategy Completion
provides the test process
Results
Chain
Analysis
PI
Strategy
Test
Strategy
Acceptance
Goal Assessment
Project Intelligence Planning
Goal Assessment
Project Planning
Goal Assessment
Project
Initiation
PI Management
Stakeholder Involvement/Project Assurance/Governance
Key:
Project planning and initiation
Project Intelligence Activities
Development activities
Review and Test activities
Slide 41
Assurance with Intelligence
Test Strategy Completion
„
E.g. System test…
-
Objectives (from PI strategy)
Environment (technology, mainly)
Responsibility (plan, spec, execution, approval)
Deliverables (plan, spec, incidents, reports)
Techniques (test design, intelligence reporting)
Tools (p
(plan,, spec,
p , execution,, data pprep,
p, reporting
p
g etc.))
Entry/exit criteria (goals met, risks addressed)
Etc. etc.
Slide 42
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 21
Assurance with Intelligence
Test Strategy Completion 2
Coverage policy
„ Timescales/estimates
Ti
l / i
(or
( the
h policy
li ffor
estimation)
„ Regression testing policy
„ Incident management process
„ Reporting
R
ti approach
h
„ Usual stuff…etc. etc.
„
Slide 43
Assurance with Intelligence
PI Process – designed to handle change
Project Risk
Management
PI Reporting
3
Project or
Process
Risks
5
Product Risks
Out of Scope
Risk will not be
tested (no test
objective)
Risk is not product-related,
or is programme-critical
7
Goals Outstanding
g
Risks Outstanding
Coverage Outstanding
12
Goals Achieved
Risks Addressed
Coverage Achieved
All new goals/risks
are “outstanding”
initially
Test Process Management
Goals and Risks
Test Phase
4
Evaluation &
Analysis
2
Classification
Test
objective
defined
6
Allocation to
Test Phase
8
Define/
Design Tests
9
Run
Test
Failed tests must
be repeated
when fixes
received
Tests have
revealed a new
risk or changes
to a risk
New risk identified
Coverage Goals
1
Requirements
Analysis/
coverage
10
Identify
Regression
Test
Change affects
requirements
(and tests)
Change
13
Impact Analysis
Slide 44 Slide 44
© 2003-7 Gerrard Consulting Limited
Version 1.0
Change requires
regression tests
11
Run
Regression
Test
Assurance with Intelligence
Page 22
Assurance with Intelligence
G l Ri
Goal,
Riskk and
dR
Requirements
i
B
Based
d
Testing
Slide 45
Assurance with Intelligence
Requirements Coverage
– Typical System Test
Tests
Requirements
q
1
2
3
Manage Customers
X
X
X
Manage Products
4
5
6
X
X
X
Create Requisition
Authorise Requisition
Raise Sales Order
Allocate Stock
Fulfil Order and Distribute
Raise Invoice
Match Payment
7
8
9
10
X
X
X
X
Large number of tests,
each focusing on a
limited scope
And so on…
Slide 46
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 23
Assurance with Intelligence
Requirements Coverage
– Typical User/End to End Test
Tests
3
4
Manage Customers
X
X
Manage Products
X
X
Requirements
q
1
Create Requisition
Authorise Requisition
2
X
X
X
X
Raise Sales Order
X
X
X
X
Allocate Stock
X
X
X
X
Fulfil Order and Distribute
X
X
X
X
Raise Invoice
X
X
Match Payment
X
X
X
5
6
7
8
9
10
Smaller number
of tests, each
focusing on a
business
process path
And so on…
Slide 47
Assurance with Intelligence
Product risks as potential modes of
failure
„
„
„
Product risks are speculative failures and incidents
The risk impact scale is the SAME as incident severity
Some modes of failure are unacceptable so we select tests
to demonstrate selected failure modes are unlikely
„
„
NB: these tests COULD be in any project phase
Means we focus on the tests that MUST pass
We know defects that cause these failures MUST be corrected
If we can’t correct failures of risk-based tests we can say
these risks have materialised
Critical risks block acceptance
-
Force a change of plan: re-schedule, redesign, re/de-scoping or
abandonment!
Slide 48
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 24
Assurance with Intelligence
Risk-Based Testing complements
requirements-based
Project Phase
Product Risks
Test Obj.
Reqs
Design
Build
Integ
Systest
UAT
Trial
Prod.
High
Objectives for each
test phase are easily
identified
Medium
Low
Assurance with Intelligence
Slide 49
Risk-based reporting
start
today
Planned
end
all risks
‘open’ at
the start
residual risks
of releasingg
TODAY
Progress through the test plan
Assurance
Slide
50 with
Intelligence
© 2003-7 Gerrard Consulting Limited
Version 1.0
Slide 50
Page 25
Assurance with Intelligence
Goal-Based Testing completes the
picture – Project Intelligence
All projects have underpinning goals/benefits
„ Stakeholders
S k h ld
want to know:
k
„
„
The status of these goals
The status of the risks that block those goals
The confidence with which they may make a
decision ((test coverage)
g )
These are the three areas where test evidence
is required: Project Intelligence.
Slide 51
Assurance with Intelligence
Project Intelligence and Coverage
Any test may be designed to
address the needs of one,
two or all three drivers.
The tests in scope cover
requirements, risks and
goals.
Assurance
Slide
52 with
Intelligence
© 2003-7 Gerrard Consulting Limited
Version 1.0
Goal
Coverage
Corollary: a test that doesn’t
address any driver is a
pointless test.
Slide 52
Page 26
Assurance with Intelligence
Project Intelligence Strategy
Project Phase
PI Drivers
Ass. Obj.
Reqs
Design
Build
Integ
Systest
UAT
Trial
Prod.
Business goals
Requirements
Objectives for each
test phase are easily
identified
Risks
Assurance
Slide
53 with
Intelligence
Slide 53
PI reporting
Project Phase
PI Drivers
Ass. Obj.
Reqs
Design
Build
Integ
Systest
UAT
Trial
Prod.
Goals
Requirements
Risks
Progress of achievement is
clearly seen. Outstanding
objectives are highly visible.
Assurance
Slide
54 with
Intelligence
© 2003-7 Gerrard Consulting Limited
Version 1.0
Slide 54
Page 27
Assurance with Intelligence
Objective
Objective
Objective
Benefit
Objective
Benefit
Beenefit
Goal
G
Goal
G
Goal
G
Goal
G
Goal-based test reporting
Open
Risks
R
Closed
Open
Closed
Closed
Open
Open
Closed
Slide 55
Goal achieved,
Benefits available
Assurance with Intelligence
Close
Slide 56
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 28
Assurance with Intelligence
PI Summary
„
„
„
„
„
Historically, coverage (requirements, code etc.) was the
only game in town
Risk-based introduced a second dimension
Goal-based is the third
Testing is transformed from just defect-detection to
intelligence gathering and decision-support
Project Intelligence
„
Extends from business case through to post deployment
assessment
Involves document, code, system, user, service testing
Not necessarily more tests, but more focused tests,
much better control and stakeholder reporting.
Slide 57
Assurance with Intelligence
Finally
„
„
„
„
„
„
Best PM practice: manage goals, not activities
PI ggives management
g
visibilityy of risks, goals
g
and progress
p g
towards them
Project managers and stakeholders are PI customers
The test process is the nervous system of your project,
designed to provide useful PI in a timely manner
What’s new?
- Goal and risk-based testing
- New styles of test reporting throughout the lifecycle
- Appreciation of testing as a key source of Project Intelligence
PI is usable in ALL methodologies.
Slide 58
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 29
Assurance with Intelligence
Managing Software Projects With
Intelligence
Thank-You!
gerrardconsulting.com
d
lti
uktmf.com
Slide 59
© 2003-7 Gerrard Consulting Limited
Version 1.0
Assurance with Intelligence
Page 30