Good Practice Does Not Ensure Success Christopher Comey

Best
Practice
Good Practice Does
Not Ensure Success
Good
Practice
Right
Practice
Christopher Comey
Testing Solutions Group
St. Mary's Court
20, St. Mary at Hill
London EC3R 8EE
Off: +44 (0) 20 7469 1500
Fax: +44 (0) 20 7623 8459
Workshop Overview
• What do we mean by ‘good practice’?
• Some examples of ‘acknowledged’ good
practices – discussion
• What do these good practices ‘require’ in order
to add benefit?
• A scenario to discuss
– Walk through and talk through of a fictional project
• Risk Based Testing Exercise
• Discuss results and Close
What do we mean by ‘Good
Practice’
Can you name some ‘good practices’
Please join in
What do we mean by ‘Good
Practice’
Can we group these?
• Test Activities
• Test Controls
• Practices/Techniques
• Supporting processes
What do these good practices
‘require’ in order to add benefit?
Do you see this happening?
• Entry Criteria – a critical check or do they just
block progress and we always waiver them
anyway?
Reviews – most of the stuff is meaningless, but I
am required to complete it anyway
Formal test process means I don't take risks and therefore I'm not accountable!
Is a risk based approach just an excuse to not
test parts of the system?
• Nice test technique but not applicable here!
Test Scenario for discussion
Technical Architecture
New
Account
Cheque Deposit IN
Cash Deposit (ATM) IN
P
r
o
c
e
s
s
Process Cheque IN
Bank Transfer IN
Debit Card Refund IN
Manage
Direct
Debit
Manage Account
(balance)
d
e
p
o
s
i
t
Account
History
Archive
P
r
o
c
e
s
s
Direct Debit OUT
D
e
b
i
t
Bank Transfer OUT
Cheque Payment OUT
Cash W’Drawal (ATM) OUT
Debit Card purchase OUT
Cash Withdrawal Branch OUT
Account
History
dB
Produce
Statement
Print
Statement
Post
e-mail
Delivery Schedule
Jan
Drop 1
Drop 2
Drop 3
Live
Feb
Mar
Development
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
Test
Development
Test
Development
Test
Jan
Feb
Mar
Release Contents (suggested by Dev)
New
Account
P
r
o
c
e
s
s
Cheque Deposit IN
Cash Deposit (ATM) IN
Process Cheque IN
Debit Card Refund IN
Drop 2
Drop 3
Manage Account
(balance)
d
e
p
o
s
i
t
Bank Transfer IN
Drop 1
Manage
Direct
Debit
Account
History
Archive
P
r
o
c
e
s
s
Direct Debit OUT
D
e
b
i
t
Bank Transfer OUT
Cheque Payment OUT
Cash W’Drawal (ATM) OUT
Debit Card purchase OUT
Cash Withdrawal Branch OUT
Account
History
dB
Produce
Statement
Print
Statement
Post
e-mail
What do we need to consider?
• What is the objective of packaging
releases?
• Interested parties: Dev, Test, the project.
• Timeframes, workload and resources
• A release must be testable:
– Either as a ‘set’of integrated deliverables
– Or accompanied by drivers and Stubs
• Risk is the main driver, but within the
project constraints
New
Account
Delivery Schedule (suggested)
P
r
o
c
e
s
s
Cheque Deposit IN
Cash Deposit (ATM) IN
Cash Deposit Branch IN
Manage
Direct
Debit
Manage Account
(balance)
d
e
p
o
s
it
Bank Transfer IN
Debit Card Refund IN
P
r
o
c
e
s
s
D
e
b
it
Direct Debit OUT
Cheque Payment OUT
Cash W’Drawal (ATM) OUT
Bank Transfer OUT
Debit Card purchase OUT
Cash Withdrawal Branch OUT
Drop 1
Drop 2
Account
History
Archive
Account
History
dB
Produce
Statement
Print
Statement
Drop 3
Post
Is this testable?
e-mail
Risk Factors
• What do we need to consider?
– Business criticality (customer impact)
– Technical criticality (complex means more
defects)
– Test needs (Testability)
Business Critical Functions
New
Account
P
r
o
c
e
s
s
Cheque Deposit IN
Cash Deposit (ATM) IN
Cash Deposit Branch IN
Manage
Direct
Debit
Manage Account
(balance)
d
e
p
o
s
it
Bank Transfer IN
Debit Card Refund IN
Account
History
Archive
P
r
o
c
e
s
s
Direct Debit OUT
D
e
b
it
Bank Transfer OUT
Cheque Payment OUT
Cash W’Drawal (ATM) OUT
Debit Card purchase OUT
Cash Withdrawal Branch OUT
Account
History
dB
Produce
Statement
Print
Statement
Post
e-mail
New
Account
Technically Risky
P
r
o
c
e
s
s
Cheque Deposit IN
Cash Deposit (ATM) IN
Cash Deposit Branch IN
Manage
Direct
Debit
Manage Account
(balance)
d
e
p
o
s
it
Bank Transfer IN
Debit Card Refund IN
Account
History
Archive
P
r
o
c
e
s
s
Direct Debit OUT
D
e
b
it
Bank Transfer OUT
Cheque Payment OUT
Cash W’Drawal (ATM) OUT
Debit Card purchase OUT
Cash Withdrawal Branch OUT
Account
History
dB
Produce
Statement
Print
Statement
Post
e-mail
Compromise
New
Account
P
r
o
c
e
s
s
Cheque Deposit IN
Cash Deposit (ATM) IN
Cash Deposit Branch IN
Debit Card Refund IN
Drop 2
Drop 3
Manage Account
(balance)
d
e
p
o
s
it
Bank Transfer IN
Drop 1
Manage
Direct
Debit
Account
History
Archive
P
r
o
c
e
s
s
Direct Debit OUT
D
e
b
it
Bank Transfer OUT
Cheque Payment OUT
Cash W’Drawal (ATM) OUT
Debit Card purchase OUT
Cash Withdrawal Branch OUT
Account
History
dB
Produce
Statement
Print
Statement
Post
e-mail
Delivery Schedule
Jan
Drop 1
Drop 2
Drop 3
Live
Feb
Mar
Development
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
Test
Development
Test
Development
Test
Jan
Feb
Mar
What about
•
•
•
•
Planned Contingency?
Where is Unit & Link Test?
What about defect fixing effort or builds
No consideration for regression testing of
previous drops
• What about CRs
• Where are the decision points e&x, go/no go
• Non-Functional, OAT, Acceptance/UAT?
Is this more realistic?
J
Drop 1
F
M
Development
A
Link
M
J
J
A
S
O
N
D
J
F
M
A
M
J
J
A
Test
Build 1
Build 2
Build 3
Build 4
Drop 2
Development
Link
Test
Reg
Development
Link
Build 1
Build 2
Build 3
Drop 3
Test
Reg
Build 1
Build 2
Build 3
Build 4
Live
Freeze
S
New
Account
Change
Requests
Cheque Deposit IN
Cash Deposit (ATM) IN
Cash Deposit Branch IN
Bank Transfer IN
Debit Card Refund IN
P
r
o
c
e
s
s
Manage
Direct
Debit
P
r
o
c
e
s
s
Manage Account
(balance)
d
e
p
o
s
it
D
e
b
it
Direct Debit OUT
Cheque Payment OUT
Cash W’Drawal (ATM) OUT
Bank Transfer OUT
Debit Card purchase OUT
Cash Withdrawal Branch OUT
Calculate
Interest
Account
History
dB
Produce
Statement
Print
Statement
Post
Account
History
Archive
e-mail
Key Dates on the delivery schedule
Drop 1
• 30th April – end link test
• 30th August – End System Test
Drop 2
• 30th Sept – end link test
• 31st Dec – End System Test (Merry Christmas)
Drop 3
• 30th Jan – end link test
• 30th May – End System Test
Live
• 30th Sept
What would you do?
On 23rd April, 1 week before delivery of the first drop you
are advised that dev are running late and the release will
not complete link test until 2 weeks after the planned
delivery date.
How do you decide what to do?
What questions would you ask?
The project Manager wants you to start anyway!
Options & Questions
1) Take the untested release and start testing
2) State that the entry criteria have not been met
and refuse to start testing
3) Establish what has been successfully tested
and start testing those areas while link testing
completes
What other questions might you ask?
The Test Execution Phase
• Entry Criteria – Met?
• What do you test first
–
–
–
–
New functionality?
Defects?
CRs?
Regression
Risk Based Testing Exercise
Do you want to see the model answer?
Good Practice Does
Not Ensure Success
Best
Practice
Good
Practice
YOU DO - Thanks
Christopher Comey
Testing Solutions Group
St. Mary's Court
20, St. Mary at Hill
London EC3R 8EE
Off: +44 (0) 20 7469 1500
Fax: +44 (0) 20 7623 8459
Right
Practice
Acknowledged good practices?
Approaches
Formal (e.g. V-Model)
Agile (e.g. Exploratory)
Good
Practice
Risk Based Testing (RBT)
Testing Techniques
Reviews (documentation based)
Structured Testing (Test condition based)
White Box (code based)
Black box (functionality based)
Control Mechanisms
Strategies, Plans, Documentation, test Scripts, Traceability to
requirements, Specified Test Coverage
Entry and Exit Criteria
Test Techniques
•
•
•
•
•
•
•
EQP and BVA
StateTransition
Cause and Effect/Decision Table Testing
Classification Tree
Syntax
Random
Orthogonal Arrays (pairwise)
Critical Processes
•
•
•
•
•
Change Management
Configuration Management
Defect Management
Release Management
Communication Management
– Objectives and Targets
– Meetings
– Reports