Building an IBM Watson Solution IBM Watson Version 2

IBM Watson
Version 2
Building an IBM Watson Solution
IBM Confidential
23 January 2015
IBM Watson
Version 2
Building an IBM Watson Solution
IBM Confidential
23 January 2015
IBM Confidential
Notes
Before using this information and the product it supports, be sure to read the general information under Notices at the end
of this information.
Prerelease information disclaimer: This document contains proprietary information. All information contained herein shall
be kept in confidence. None of this information shall be divulged to persons other than (a) IBM employees authorized by
the nature of their duties to receive such information, or (b) individuals with a need to know in organizations authorized
by IBM to receive this document in accordance with the terms (including confidentiality) of an agreement under which it is
provided.
This edition applies to Version 2 Release 3 of IBM Watson. This information applies to all subsequent releases and
modifications until otherwise indicated in new editions.
This edition replaces Version 2 Release 1.
© Copyright IBM Corporation 2014, 2015.
US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract
with IBM Corp.
IBM Confidential
Contents
Chapter 1. IBM Watson partner
enablement . . . . . . . . . . . . . 1
Chapter 2. Helping to teach Watson . . . 5
Establishing user roles . .
Building a Watson solution
.
.
© Copyright IBM Corp. 2014, 2015
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. 7
. 9
Evaluating questions and content resources.
.
.
. 15
Notices . . . . . . . . . . . . . . 19
Trademarks . . . . . .
Privacy policy considerations
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. 20
. 20
iii
IBM Confidential
iv
IBM Watson Version 2: Building an IBM Watson Solution
IBM Confidential
Chapter 1. IBM Watson partner enablement
Planning a strategy to enable and build an IBM Watson™ solution or integrating
Watson™ into a solution entails a thorough understanding of the set of processes
for designing, developing, and delivering a Watson solution. It includes an
engagement process for working on the high level planning and scoping a Watson
solution, tooling to help provide data that is used to train Watson, and a
methodology for developing applications for users to Ask Watson questions or
other types of requests, such as diagnosis or treatment advice.
IBM Watson is a natural language question answering system. It determines its
answers and associated confidence scores based on the knowledge it has acquired.
Watson solutions provide a combination of the following key characteristics:
v Understand questions through natural language
Watson provides natural language processing to help understand the
complexities of human communication both a means of user interaction and to
extract knowledge from sources of data
v Generate and evaluate hypotheses, answers and supporting evidence
Watson provides hypothesis generation and evaluation by applying advanced
analytics to weigh and evaluate a panel of responses based on only relevant
evidence.
Watson is an adaptable system that can be customized for specific application
domains and questions. After defining use cases for a Watson solution, developers
and domain experts can help train Watson for a domain-specific solution by
supplying the content resources and the training data that is required. User
interface developers can use the Watson SDK to create Watson enabled applications
to support specific kinds of users, contexts, or use cases.
Questions are submitted through a Watson enabled application to a Watson
pipeline. The pipeline initiates the process for analyzing, searching, and returning
information to a user. The information that is returned is based on the data in the
knowledge base, analytics algorithms, and machine learning models. The question
analysis involves searching and generating large amounts of data, and requires
hardware and software resources to support a massively parallel computing
environment.
Watson use cases or solutions can be applied to different kinds of scenarios such as
the following ones:
v Ask - Users can ask Watson direct questions in natural language the same way
they ask friends or colleagues questions.
v Discover - Users can discover new insights or find the rationale for a given
response with Watson. For example, a researcher might be looking for the best
way to treat a disease for a specific group of patients.
v Decide - Users might use Watson to help them decided on the best course of
action. For example, a user might be looking for confidence-based
recommendations for their next action when they have many options to chose
from such as what course of treatment to prescribe to a patient or what
investment choice to make.
© Copyright IBM Corp. 2014, 2015
1
IBM Confidential
Natural language processing
Natural language is the real language that people use to naturally capture and
communicate knowledge. In solving natural language questions, Watson must
evaluate puns, slang, jargon, and acronyms to determine its confidence in
returning an answer.
Whereas a web search engine returns a ranked list of web pages based on
keywords, Watson analyzes the structure and wording of questions being asked,
and quickly formulates an answer that has the highest level of "confidence" is
correct. A search engine becomes less accurate as more information is introduced to
the search, but Watson is able to use additional information to become increasingly
accurate and precise the more data it is given.
Watson includes many algorithms to analyze natural language processing. It also
has algorithms that enable it to learn, so it remembers what it learned to apply that
information to the next analytical task it must conduct (for example, when a
person says "fluid," the person can mean "liquid" as well).
Unstructured data
Unstructured information represents the largest, the most current, and the fastest
growing source of information that is available to businesses and governments. By
using Watson, you can create applications that can analyze large volumes of
unstructured information to discover, organize, and deliver relevant knowledge to
specific types decision makers. Currently, Watson uses textual data only.
Unstructured data must be analyzed to interpret, detect, and locate concepts of
interest that are not explicitly tagged or annotated in the original document.
Documents can include any kind of domain-specific information that you define as
such.
Steps to building Watson solutions and services
At a high level, there are different steps to building components in a Watson
solution or service. For example, the following image shows some of the possible
ways to be part of a Watson ecosystem:
v Define a use case for which users submit questions to Watson for answers,
advice, evidence, or information.
2
IBM Watson Version 2: Building an IBM Watson Solution
IBM Confidential
Have people provide questions that they expect users of the system to ask
Watson. These questions will help drive and identify the content that needs to be
made part of the Watson knowledge source.
v Identify and upload content resources to form the knowledge source for Watson
to answer questions.
v Help train Watson for a specific domain by creating training data.
v Configure the user interface of an Advisor application or an integration for users
to ask Watson questions.
The following image shows some of the possible workflows in a Watson
ecosystem. Each hosted instance of Watson is made available for developers,
subject matter experts, and administrators to perform tasks that can build or train
components of a Watson solution or service.
There are two different modes of operation for a Watson solution or service. Each
of these modes is made available as its own hosted instance of Watson.
v Customization mode is for developing a Watson solution or service.
The Watson core is a domain-independent implementation that developers
customize by adding new algorithms and providing relevant content that
Watson can incorporate into its base of knowledge. Its ability to answer
questions based on this customization can be verified and refined over time as
the information changes or new analytics become available.
Customization environments are where iterative system configuration,
development, and testing of all components occurs. It includes the following
kinds of tasks:
Chapter 1. IBM Watson partner enablement
3
IBM Confidential
– Uploading and ingesting content resources to form the knowledge source to
answer questions.
– Providing training data to help train a system.
– Using the Watson SDK for developers to build Watson enabled applications
for users to submit questions.
UI developers can use the Watson SDK to build web or mobile applications
that enable users to submit questions or can create Watson services and
integrate them into existing applications or portals.
Development tasks might also include customizing a data model, defining
training data, creating and running experiments through questions and answers
to train or test a system, analyzing test results to assess precision and accuracy,
and creating or making improvements to system components.
The goal in Watson system design and development is to deliver solutions that
enable:
– Effective analysis: Understand and process rich natural language questions
over a broad domain of knowledge.
– Precise answers: Determine what is being asked and give precise responses.
– Accurate confidence: Determine likelihood that the answer is correct.
– Consumable justifications: Explain why the answer is right.
– Fast response times: Return answers with precision and confidence quickly.
Actual response times will be different for each solution and based on several
factors.
v Production mode is for enabling users to submit questions and receive answers.
Production environments are for systems that are deployed for users of solutions
or services. The system could be configured for specific governance, security, or
compliance requirements.
The question answering might be for one person or for many concurrent users
and questions. Answer-confidence is based on how the system is designed. For
example, one solution might be for only returning a single answer and greater
than 95% confidence and another solution could let a user specify the number of
top answers with supporting evidence to return based on a confidence level the
user can specify.
4
IBM Watson Version 2: Building an IBM Watson Solution
IBM Confidential
Chapter 2. Helping to teach Watson
Help train and teach Watson by supplying content resources to add to its
knowledge content and creating training data in the form of questions and
answers.
IBM Watson is a cognitive computing solution that relies on users to help train the
system rather than explicitly programming it to run tasks. Watson can be improved
through interactions with users and with the addition of data to the system.
An IBM Watson solution brings together a set of transformational technologies to
drive optimized outcomes:
v Processes large amounts of data quickly.
v Understands natural language.
v Generates answers with a confidence level.
v Returns information with a confidence level that is based on evidence that is
found in content resources.
v Adapts to the responses a user selects.
To analyze and answer questions, Watson relies on:
v Content resources, such as a set of documents, to be uploaded and used to form
a knowledge store that Watson uses to find answers and supporting evidence to
questions.
Content resources are existing documents that contain knowledge about the
domain topic (such as Product guidelines, specifications, or policies). These
resources might be in the form of unstructured (natural language text) or
structured (database) formats. Examples include a product manual or a table
that describes different types of specifications.
Ingestion refers to the processing of content resources such as a corpus of
documents from which Watson finds answers, advice, or evidence for questions.
The ingestion process includes natural language preprocessing of text corpora
and the creation of derived data resources such as search indexes and a
knowledge base.
v Question and answer pairs that form the training data that is used to help train
Watson to answer questions.
Training data is a set of representative question and answer pairs that explicitly
link expected input to expected output. That is, users who understand the target
domain, create questions and link them to correct answers that are found in the
uploaded content resources. Training data represents a sample for the intended
system function. It is used to train and test the system performance.
Watson benefits by improving the quality of the training data or adding more
content. If more content resources are added to the system, then current question
and answer pairs might need to be modified.
Watson Experience Manager is a suite of tools that is provided in a customization
environment to enable each of the following kinds of tasks:
v Creating questions to help determine the target Watson solution.
© Copyright IBM Corp. 2014, 2015
5
IBM Confidential
The Question input tool in IBM Watson Experience Manager is for users to
submit questions that represent the kinds of questions that users of the target
solution might ask.
v Adding content resources to form the knowledge content to answer questions.
Reviewing representative questions can help identify the content resources that
are needed to answer them. Content is added by using the Corpus management
tool in IBM Watson Experience Manager.
v Creating and managing the training data by creating the kinds of question and
answer pairs that are similar to the questions that users would submit to Watson
for answers.
The process to create training data is to submit questions and specify one or
more correct answers for each question. The answers must be available in the
content resources that provide supporting evidence for the answers. IBM Watson
Experience Manager provides an Expert training tool to manage this process.
Questions that are submitted by users of the Question input tool appear in the
Expert training tool and can be used as the initial list of questions to work with.
Additional tasks:
v Administering the user roles for controlling access to the tasks to perform in the
tool to build knowledge content and training data.
For example, user roles specify who can:
– Submit questions
– Add content
– Create and deploy a corpus
– Specify answers
– Approve questions and answers
v Configuring the user interface of the Advisor application for users of a
production system to submit questions.
Training
Watson is trained much the way a human learns. It learns from historical question
and answer pairs to understand how questions are asked and how the answers can
be uncovered in the evidence. Watson then uses sophisticated machine learning
algorithms to "get smarter" over time.
Training data is the collection of questions and answers that are representative of
questions that are asked by users of a Watson solution.
Subject matter or domain experts create these representative questions and answers
by using the Expert training tool in Watson Experience Manager. This training data
will help Watson understand the language characteristics of the domain that
Watson is operating on. This process is similar to what a person would follow if
they were new to an environment, that is, practice by asking and answering some
questions to become familiar with the way concepts are expressed.
Reviewers and approvers of training data for a Watson project might assign a
particular user to author a collection of documents. The collection might cover a
subset of the complete corpus of documents that Watson uses to answer questions
for a domain. As a domain expert, a user might be assigned a target number of
questions that need answers. The goal in creating questions and answers is not to
imagine every possible question and answer that a user might ask. Instead, the
6
IBM Watson Version 2: Building an IBM Watson Solution
IBM Confidential
goal is to build a representative set that enables Watson to understand the
domain's language characteristics.
Learning
Just as people continue to learn after their formal education, Watson continues to
learn from experience after it is deployed. In real life, people learn from feedback
that they receive from others and from new material they read or ingest. Watson is
no different.
When a user confirms that one of the responses from Watson is correct, in
particular, if that answer is one that Watson ranked second or third, and not the
most likely answer, then Watson can use that feedback to learn. The system is able
to capture these questions and correct answers and append them to the training set
that was initially used to train Watson for the specific domain.
It is not efficient for Watson to retrain on one question at a time. Watson batches
new questions, and when the wanted threshold is met, a training process can be
run to improve how Watson handles future questions. This cycle can continue on
an ongoing basis so that Watson continues to perform more accurately.
There are several variations on how to customize the way that Watson learns. For
example:
v Select users can be designated as Approvers.
– Approvers can review the question answer pairs that are captured from users
of the system to validate the correctness of the feedback.
– Any feedback that is provided by these users would automatically be
accepted for more learning. Feedback from all other users can either be
reviewed, or discarded.
– Approvers can either discard question answer pairs that are created through
user feedback that are deemed to be incorrect or provide new answers to
those questions.
v Domain experts can improve training data by vetting questions where no correct
answer was indicated.
The cause for no correct answer might be because Watson did not have sufficient
confidence to provide an answer or the user indicated that no correct answer
was provided (by adding negative feedback and not selecting an alternative
answer). For these kinds of questions, domain experts might identify the
appropriate answer and add the question answer pair to the queue of new pairs.
Questions deemed inappropriate by the reviewers or approvers would be
deleted.
Establishing user roles
There are several different user roles and associated tasks for building a Watson
solution. Start by first understanding and identifying the people who can fill the
roles in the system development workflow.
To apply the correct roles to your users, review the roles that are involved with
Watson.
Chapter 2. Helping to teach Watson
7
IBM Confidential
Subject matter experts
Subject matter experts (SME) help identify and upload content to form the
knowledge content from which Watson finds answers. They also help create the
training data in the form of question and answers.
The subject matter expert has these responsibilities:
v Understands the data resources and requirements for industry-specific and
real-world question analysis and answering.
v Understands the domain and the kinds of questions and answers for an actual
solution.
v Knows the terms and forms that users would typically use to ask questions.
v Helps identify and upload the information that is needed to become part of the
Watson knowledge content. The SME might also identify the types of domain
adaptations (such as more content, training, and functional) for the question and
answer data model.
v Provides input to more resources for system usage and storage. These additional
resources include domain-specific terms, words, and other information and
features in the data, such as abbreviations, synonyms, acronyms, and semantic
types.
SMEs are domain experts who create question and answer pairs that form an
answer key. The answer key is used to help train Watson so it is more familiar
with a specific domain.
After creating a question, the experts specify answers by selecting text from the
knowledge content documents.
The documents for SMEs to select answers from is based on a keyword search of
the documents for terms in the question text. For example, if a SME submits the
question, what is term insurance?, then the documents returned with potential
answers are based on the keywords, in the question, such as term and insurance.
At this stage of building a Watson solution, SMEs are providing training data to
help Watson learn. However, they are not training Watson directly, nor is Watson
answering the questions that are submitted. Rather, at this stage, when a SME
submits a question, they see the results for the text passages searched and found
across the content that was uploaded to this point. From the passage search results,
the SME can specify answers by selecting one document at a time and highlighting
text in it.
Question and answer pairs are submitted by authors and reviewed by approvers.
A subject matter expert might be an author, be an approver, or handle both tasks.
Administrators
Administrators are those people who are responsible for configuring the
application and who have access to reports and metrics in IBM Watson Experience
Manager.
There are two types of administrator roles:
v Customer administrators assign user roles and user permissions and configure
the Advisor application that customers use to ask Watson questions. The user
8
IBM Watson Version 2: Building an IBM Watson Solution
IBM Confidential
role permissions establish what tasks that each user can perform in the
supported tooling for building a solution.
v System administrators configure the connection from user applications to a
Watson pipeline.
Administrators can monitor training progress and review content resources and
generate usage reports to understand what content is being used by Watson to
answer questions.
User interface developers
User interface developers can implement user applications by using the
personalization page to customize the chat window for users, or integrating the
application by using an iFrame. They can embed a Watson application within an
existing application and initialize the chat window such that the right information
is passed to it.
They can also use the Watson Platform Software Development Kit to create
applications by using the Question and Answer REST service and a set of
JavaScript widgets.
End users
End users are the people who ask Watson questions by using the web and mobile
interfaces of the Watson Advisor in a production environment. Users might also
help provide representative questions and answers for training data, test the
applications, and help teach Watson by providing feedback.
Building a Watson solution
Building a Watson solution is enabled by using a collection of supported tools and
processes that a team of users performs to help train and teach Watson to become
a domain-specific solution. IBM Watson Experience Manager is a suite of tools to
help enable the process to build a Watson solution. Experience Manager provides
tools for each of the different user roles to do their job.
Watson Experience Manager is for domain and subject matter experts,
administrators, managers, and user interface developers to train, customize, and
monitor a Watson application.
Watson Experience Manager applications are hosted and managed by the IBM®
Cloud Technology and Support (WCTS) team , and available through a web
interface. Each Watson instance is made available through a URL from WCTS.
Watson Experience Manager includes tools that are visible to users based on their
user roles that are assigned by an administrator:
v Collecting representative questions
Building a Watson Engagement Advisor solution begins with collecting
representative questions. You need to collect representative questions from the
intended users of the solution. The target Watson Engagement Advisor solution
should be able to answer representative questions and thus the questions help
drive the direction of the target Watson solution.
This question set also helps you identify the content resources that need to be
added to the Watson corpus to answer the questions. Subject matter experts
should not create representative questions.
Chapter 2. Helping to teach Watson
9
IBM Confidential
A minimum of 2000 representative questions is required to help train Watson. A
lower number of questions will not be enough to help achieve a high level of
system performance.
There are different methods to collect questions. Users can use the Question
input tool that is part of Watson Experience Manager to submit questions.
Questions that are submitted in the Question input tool are added directly to the
Expert Training tool. Subject matter experts then use this tool to find answers to
these new questions that need an answer. Then, the question answer pairs go
through a review process before they can be approved and added as part of
ground truth for training Watson.
Representative questions can also be gathered from the Adaptive Learning tool
to add questions that have been submitted by users of the Advisor user
interface. Questions can also be collected in a spreadsheet and then imported
into the tool.
Adding more questions can help drive the creation or acquisition of more corpus
content to be able to answer the questions.
v Administering users for access to Watson Experience Manager
System administration is for defining users and assigning user roles to them.
Administrators can also generate user report metrics.
Users see only the tools or options in the tools that they are authorized to view
based on their assigned user roles.
IBM system administrators can also use the tool to specify the connection to a
Watson processing pipeline.
v Creating a corpus
Use the Corpus Management tool to add content to include as part of the
Watson corpus. The corpus forms the answer store, or knowledge content, for
Watson to find answers
First, the corpus content required to answer the representative questions is
identified. The content is then uploaded, preprocessed, indexed, and stored in
the file system before it can be used to find and specify answers to questions in
ground truth.
v Creating or updating ground truth
The Expert training tool is for creating question answer pairs that help teach
Watson about your domain. These question answer pairs form the question
store, and are also referred to as ground truth.
After acquiring representative questions and creating a corpus, subject matter
experts must match questions to similar questions or with answers. To answer
questions, you must search and find, and then select the similar question of the
correct answer. Searching for an answer is based on a keyword or filename
search of the documents added to the Corpus management tool. Answers are
specified by matching a question to a similar question, or to an answer unit. The
possible answers are document segments that are formed by the content
ingestion process.
The Expert training tool provides support for managing a review and approval
process for question answer pairs. Some users might be assigned the role to
create question answer pairs while others are assigned to review and approve or
reject pairs.
v Configuring the user interface of the Advisor application
Configuration is for configuring the look and feel of the Advisor application for
users.
10
IBM Watson Version 2: Building an IBM Watson Solution
IBM Confidential
User interface developers or administrators customize the user interface for the
Watson Advisor for users of the solution. They can configure the chat window
that customers can use to ask Watson questions.
UI developers can also design and implement Watson enabled applications or
integrated services into an existing application for users to submit questions to
Watson.
v Testing the Advisor application
The Test option in Watson Experience Manager is for testing the application that
users see when the application is deployed to a production environment. The
Advisor application is a chat window in which users can submit questions, view
common questions, and view returned answers or responses.
When asking questions, users should be aware that the wording of a question
matters. How a questions is phrased can affect the answering results. For
example, "what is", "what's", and "tell me about" are all ways to ask a question
but Watson might not always return the same answer, nor with the same level of
confidence for these variations, so it is good try variations to receive the best
answer. For example, by asking "What is Watson?" there is little for Watson to
analyze and to base its confidence scores on for a correct answer because "what
is" and "Watson" are not enough domain-specific words to distinguish possible
answers in corpus content with high confidence. Another example, of seeing
different answers and confidence scores would be "How do you train a Watson
solutions," versus "How do I train a Watson solutions," since "you" is
ambiguous, and refers to Watson, not a user.
v Improving accuracy through training and testing
Ground truth is the collection of approved question and answer pairs in the
Expert Training tool. Randomly generated subsets of ground truth are used to
train and test the system.
IBMers work with partners and customers to train and test a Watson
Engagement Advisor solution:
– They run training to train machine learning (ML) models to a specific
domain.
– They use the Watson Workbench to measure accuracy of a system by running
a test experiment and analyzing the experiment results.
This data shows accuracy, precision, and recall metrics and can help provide
insight to what updates to the corpus or ground truth that might improve
these accuracy metrics.
Workflow for building a Watson solution
The process for establishing, developing, and delivering a Watson solution involves
three phases. Many of these steps are completed by working with IBM services
representative and are not required in all instances.
Services consultants and system administrators help configure and train a Watson
solution by using the standard solution methodology that includes a set of best
practices.
Before users can submit questions to Watson from a Watson Advisor or chat
window in a production environment, the system must first be trained and tested.
From a tools and process perspective, training is making Watson more familiar
with a particular domain. At a high level, the tasks to perform include the
following:
v Collect representative questions from users of the target solution.
Chapter 2. Helping to teach Watson
11
IBM Confidential
Collecting representative questions is the important first step drives the steps to
build a Watson Engagement Advisor solution.
Training a Watson solution starts by collecting representative questions that are
submitted by the intended users of the target solution. It is important that the
questions be questions that actual users of a target production system might ask.
v Collect content for a corpus that provides answers to questions.
Answers to questions must be available in the corpus. The answers to questions
that reside in corpus documents must be correctly preprocessed to create
meaningful document segments or answer units that subject matter experts will
then be able to find and select to create the question answer pairs that are used
to train and form the question store. The collection of question and answer pairs
that have been reviewed and approved form the ground truth for a Watson
solution.
v Use the Corpus Management tool to add content that forms the corpus.
The tool preprocesses documents to form answer units, or document segments,
that are made available for mapping to questions by subject matter experts. The
tool also allows corpus managers and administrators to view the answer units to
be sure that documents have been preprocessed correctly. Some documents
might need to be modified or a different document preprocessor might be
needed.
v Use the Expert Training tool to create a ground truth of approved question and
answer pairs.
– Find and match questions to similar questions or to answer units.
– Match a question to similar questions forms a question cluster.
Create question clusters to group questions that are paraphrases with the
same answer. Clusters can help increase the accuracy and confidence of
Watson to answer questions.
– Find and match an answer to a question.
Individual questions that are matched to an answer are known as singletons.
A higher occurrence of singletons in a ground truth can lead to lower
accuracy and confidence than by having question clusters. For example, a
ground truth that includes multiple ways of asking the same question that are
all part of a question cluster leads to better training for Watson.
– Review and approve question and answer pairs.
Only the Approved questions in the Expert Training tool form the ground
truth that is used to train and test Watson.
v Click the Create Corpus button in the Corpus Management tool to create a new
question store and answer store, and to deploy the updates to the test
environment to be able to test the system.
v Test the current system from the Test user interface in Experience Manager, and
then deploy to the production environment.
v Run training to train the machine learning (ML) models to the specific domain
for a solution. This step is currently performed by IBMers.
Training is performed by using a random subset of the question and answer
pairs in ground truth. The set is randomly generated one time and then used for
each training run.
v Run testing and perform accuracy and headroom analysis to measure current
accuracy and identify changes to make to the corpus and ground truth based on
the analysis to improve accuracy.
12
IBM Watson Version 2: Building an IBM Watson Solution
IBM Confidential
Testing is performed by running an experiment with the Watson Workbench.
Testing is performed by using a random subset of the question and answer pairs
in ground truth. The set is randomly generated one time and then used for each
test experiment.
v Continue iterating on this process to improve accuracy.
Once a corpus is deployed to a production pipeline it can be made available for
users of the Advisor interface to ask questions.
At a high level, the process to enable a Watson solution involves the following
phases:
1. Try Watson
Use a sample Watson Advisor instance to ask Watson questions that are based
on an initial set of uploaded documents. In this phase, you work with IBM
Managed Services to complete the following steps:
a. Upload an initial set of documents into a configuration mode instance of
Watson.
b. Assess the content to ensure an appropriate fit with Watson capabilities.
c. Experiment by submitting sample questions by using the Watson Advisor.
d. Optionally perform these tasks by using tools that are included in Watson
Experience Manager:
v Update the corpus by uploading more documents.
v Customize the Watson Advisor user interface for users by adding logos,
themes, and choosing other options.
v View metrics that show ingestion results, question answering results, and
recommendations for improvements to the solution.
v Use the Watson SDK to create Watson applications for users by using the
available APIs and JavaScript widgets.
2. Train Watson
Use a Watson instance to build a domain-specific solution.
a. Configure Watson.
You can migrate the configuration and corpus from the trial instance or
create a new configuration and upload new content and form a new corpus.
b. Help build the Watson solution.
1) Collect representative questions.
Have users of the intended solution to submit representative questions.
Collect these questions and import them from a spreadsheet or have
user enter the questions directly through the question input tool. The
goal is to collect at least 2000 questions that represent questions that
would be asked by users of the target system.
2) Create knowledge content by uploading documents.
Use the corpus management tool in Watson Experience Manager to
update the knowledge content by uploading more documents. Domain
experts or content suppliers can help supply content resources by
uploading them with Watson Experience Manager. The knowledge
content is generated by an ingestion process for one or more sets of
uploaded documents. You can also use the tool to delete content and
view metrics on the ingestion process.
3) Create a ground truth to train Watson.
Subject matter experts understand a specific domain and help train
Watson by matching questions that are representative of what users
Chapter 2. Helping to teach Watson
13
IBM Confidential
might ask Watson to similar questions or to answers. These question
and answer pairs are used to train Watson and establish a baseline
system with initial levels of accuracy and precision. They can use the
Expert Training tool in Watson Experience Manager to define, review,
refine, and approve question and answer pairs.
4) Improve the quantity and quality of answers by refining ground truth or
the corpus.
By reviewing the ground truth to ensure that questions are grouped
correctly and with correct and consistent answers subject matter experts
can achieve higher levels of accuracy and precision. This step is an
iterative process where incremental improvements can be achieved.
5) Configure the user interfaces of the Advisor application for the
appropriate look and feel for the custom user experience of a specific
solution.
Use the configuration options in Watson Experience Manager to
customize the Watson Advisor user interface for users by adding logos,
themes, and choosing other options.
User interface developers can integrate the Advisor into an existing
application. They can also choose to design and implement other user
interfaces or services by using the Watson SDK to add functions
available in the Watson JavaScript toolkit and the Watson REST APIs.
6) Optionally:
v Integrate a Watson Advisor into an existing environment.
For example, you can create a more comprehensive business
application by integrating a Watson application with existing
information sources, process flows, and applications.
v View metrics that show ingestion results, question answering results,
training results, and recommendations for improvements to the
solution.
3. Run a solution to a production environment for users to interact with Watson.
A Watson instance can be provisioned to deploy the solution into a production
environment. The run instance is where users can ask Watson questions
through Advisors that can be made available through mobile and web user
interfaces. Users can submit questions and provide feedback that is used to
improve the Watson solution. Provision the run instance after all of the
customization and testing is completed. The configuration of the train instance
can be migrated to a run instance.
v Deploy solution
Integrate and deploy the solution to complete your business application.
v Monitor use
Monitor metrics and performance measures to understand system and
content usage and performance and to look for opportunities for
improvement.
v Run solution
Continue to update the knowledge content and training data to adapt to new
or changing information and improve accuracy.
14
IBM Watson Version 2: Building an IBM Watson Solution
IBM Confidential
Evaluating questions and content resources
To help ensure successful Watson Engagement Advisor solutions, start by
understanding the kinds of questions it can answer and evaluating sample
questions to see whether they are of a type that can be answered by Watson. Then,
evaluate content resources to see whether they can be used by Watson for
providing answers and evidence.
You must also assure that the number and quality of question answer pairs that
are gathered is adequate for training Watson.
The following kinds of questions are currently supported:
v Descriptive questions - Asking for a definition, explanation, or description.
v Yes or No questions - Asking for a statement that indicates whether something is
true or false.
v Procedural how-to questions - Asking for the series of steps to accomplish a
task.
v Procedural troubleshooting questions - Asking for the series of steps to diagnose
and resolve a problem.
Watson Engagement Advisor cannot answer the following kinds of questions:
v Predictions of the future unless predictions are in the content resources
v Judgments
v Requests for calculations
v Requests for assimilation of information from various sources
For more information:
v “Guidelines for questions”
v “Guidelines for content resources” on page 17
Guidelines for questions
Use the following guidelines to evaluate the suitability of a specific solution.
General
v The answers must be found in available content resources.
v Answers must be explicit in the content. Watson Engagement Advisor
cannot synthesize answers.
v Watson cannot yet combine information from various documents or
create an answer that is a deduction from passages it finds.
v Questions must not require Watson to make judgments.
Descriptive questions
v Questions can be approximately one to three sentences in length. Long
paragraphs are not good questions.
v Questions should require answers of a small paragraph (about three
sentences) or less. Longer answers are allowed if the answer is a list that
exists in the text.
Descriptive question examples
v Good Examples
– Q: How much interest do I get from my Savings Deposit
Program if I am on duty?
Chapter 2. Helping to teach Watson
15
IBM Confidential
A: Although the interest is taxable, the account pays a
guaranteed interest rate of 10 percent during deployment.
– Q: How do I find out about a DTAP briefing:
A: Contact your local Transition Office to find out when a
DTAP briefing is scheduled on your installation. If DTAP
briefings are not available at your installation, the Transition
Office Staff will refer you to other sources where similar
information is available.
v Bad Examples
– Which insurance policy should I pick?
Requires much information about the users and requires
judgment.
– How do I close this account?
There needs to be enough context to know what the user is
asking about. In this example, there needs to be information
provided about which account.
Yes or No questions
v Same question length as Descriptive questions.
v Need to provide enough contextual information for .
Yes or No question examples
v Good Examples
– Q: As a renter, am I covered by my landlord's insurance
policy?
A: Landlords' policies typically cover the dwelling structure,
contents they own and their liability. Their policy does not
cover the goods or liability of a tenant.
– Q: Can my EVP coverage be transferred if I sell my car?
A: You can transfer the Extended Vehicle Protection plan if
you sell your vehicle to a private purchaser.
v Bad Examples
– What is the best homeowner’s policy for me?
Requires significant personalization information, rule
evaluation. Currently, Watson uses limited information about
the user and does not evaluate business rules.
– Did the ABC policy cover flood damage in 1987?
Watson cannot determine what was true at what time with a
high degree of accuracy.
Procedural questions
v Same question length as Descriptive questions.
v Answers are the length of a document section that documents a
procedure.
Procedural – How-to questions
v Good Examples
– Q: How do I change the ring tone on my iPhone?
A: <Set of steps to change the ring tone.>
– Q: How do you configure a new email provider in my Galaxy
IIIS
16
IBM Watson Version 2: Building an IBM Watson Solution
IBM Confidential
A: <Set of steps to configure a new email provider>
v Bad Examples
– How can I make a million dollars?
Must be answerable by specific information in a customer’s
content.
– How do I change my ring tone?
- Watson cannot answer questions related to specific
information about the user, such as what product they own,
unless there is agreement to support specific client profile
information.
- The Watson Advisor dialog capability can address some
missing information, but the support is limited.
Procedural - Troubleshooting questions
v Good Examples
– Q: The email on my iPhone is frozen
A: <Step by step procedure to determine why the email is frozen>
– Q: I have a Galaxy S4 Why is my battery not lasting very
long anymore?
A: <Step by step procedure to determine why the battery is being
drained>
v Bad Examples
– Q: My email is frozen
As for How-to questions, Watson does not know what phone
you are using for email, unless there is integration work to
add that information to the question.
– Q: I was charged too much last month.
Watson does not address billing, sales, or account problems.
Guidelines for content resources
Evaluate content resources by following these guidelines:
v Watson supports the following content types: HTM, HTML, PDF, DOC, DOCX,
MHT, and ZIP.
Within these documents, Watson can interpret Text and Tables.
v
v
v
v
Watson cannot interpret Diagrams, Pictures, and other graphics, which are
embedded video and audio, and Open form mathematical expressions.
All content must be in English. US, UK, Canada, Australia, New Zealand can be
supported if the questions are from the same locale. Content cannot be
multilingual documents.
Individual documents should not mix internal and public information.
Unless the solution is an Agent-assist or internal-only use case.
Internal content means information for inside a company use only.
Documents should provide good context for the information they convey.
Try to provide well-organized document structure with descriptive section
headings.
Watson can use some poorly structured documents, but if more than 25% are
poorly structured, it is difficult to achieve good accuracy and precision.
Watson supports finding answers from tables in documents, but these factors
apply:
Chapter 2. Helping to teach Watson
17
IBM Confidential
– Tables can be complex and they are difficult to extract meaning from.
– If documents are primarily tables, accuracy might be poor compared to other
sources.
– Content resources that include table intensive documents can be used.
v Watson supports documents that are formatted in two columns, but with these
notes:
– Works correctly for HTML resources.
– Some PDF files are difficult to interpret in multicolumn format.
– Content resources that include many multicolumn PDF files can be used.
18
IBM Watson Version 2: Building an IBM Watson Solution
IBM Confidential
Notices
This information was developed for products and services offered in the U.S.A.
IBM may not offer the products, services, or features discussed in this document in
other countries. Consult your local IBM representative for information on the
products and services currently available in your area. Any reference to an IBM
product, program, or service is not intended to state or imply that only that IBM
product, program, or service may be used. Any functionally equivalent product,
program, or service that does not infringe any IBM intellectual property right may
be used instead. However, it is the user's responsibility to evaluate and verify the
operation of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matter
described in this document. The furnishing of this document does not give you
any license to these patents. You can send license inquiries, in writing, to:
IBM Director of Licensing
IBM Corporation
North Castle Drive
Armonk, NY 10504-1785
U.S.A.
For license inquiries regarding double-byte (DBCS) information, contact the IBM
Intellectual Property Department in your country or send inquiries, in writing, to:
Intellectual Property Licensing
Legal and Intellectual Property Law
IBM Japan, Ltd.
1623-14, Shimotsuruma, Yamato-shi
Kanagawa 242-8502 Japan
This information could include technical inaccuracies or typographical errors.
Changes are periodically made to the information herein; these changes will be
incorporated in new editions of the publication. IBM may make improvements
and/or changes in the product(s) and/or the program(s) described in this
publication at any time without notice.
The following paragraph does not apply to the United Kingdom or any other
country where such provisions are inconsistent with local law:
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS
PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER
EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS
FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or
implied warranties in certain transactions, therefore, this statement may not apply
to you.
IBM may use or distribute any of the information you supply in any way it
believes appropriate without incurring any obligation to you.
Licensees of this program who wish to have information about it for the purpose
of enabling: (i) the exchange of information between independently created
© Copyright IBM Corp. 2014, 2015
19
IBM Confidential
programs and other programs (including this one) and (ii) the mutual use of the
information which has been exchanged, should contact:
IBM Corporation
Bldg. 101/4th Floor/Dept. MU5A/Div F6
11301 Burnet Road
Austin, TX 78758
U.S.A.
Such information may be available, subject to appropriate terms and conditions,
including in some cases, payment of a fee.
The licensed program described in this document and all licensed material
available for it are provided by IBM under terms of the IBM Customer Agreement,
IBM International Program License Agreement, or any equivalent agreement
between us.
If you are viewing this information softcopy, the photographs and color
illustrations may not appear.
Trademarks
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of
International Business Machines Corp., registered in many jurisdictions worldwide.
Other product and service names might be trademarks of IBM or other companies.
A current list of IBM trademarks is available on the Web at Copyright and trademark
information at http://www.ibm.com/legal/copytrade.shtml.
Java and all Java-based trademarks and logos are trademarks or registered
trademarks of Oracle and/or its affiliates.
Other company, product, and service names may be trademarks or service marks
of others.
Privacy policy considerations
This IBM Watson Software Offering does not use cookies or other technologies to
collect personally identifiable information. This offering uses cookies only to track
user sessions and to carry authorization tokens.
If additional software that is deployed by you the customer or Business Partner,
along with this Software Offering, provides you the ability to collect personally
identifiable information from end users via cookies and other technologies, you
should seek your own legal advice about any laws applicable to such data
collection, including any requirements for notice and consent.
For more information about the use of various technologies, including cookies, for
these purposes, see IBM’s Privacy Policy at http://www.ibm.com/privacy and the
section titled Cookies, Web Beacons, and Other Technologies in IBM’s Online Privacy
Statement at http://www.ibm.com/privacy/details Also, see the IBM Software
Products and Software-as-a-Service Privacy Statement at http://www.ibm.com/
software/info/product-privacy
20
IBM Watson Version 2: Building an IBM Watson Solution
Product Number:
IBM Confidential
Printed in USA