IBM PureData System for Hadoop Partner Program Jan Shineman, Director, Alliance Management

IBM PureData System for Hadoop
Partner Program
Jan Shineman, Director, Alliance Management
Brian Hess, Director, Advanced Analytics
1
© 2013 IBM Corporation
If this were easy, everyone would already be leveraging big data
“Big Data offers big business gains but hidden costs and complexity
present barriers that most organizations will struggle with”
- The Cost of Big Data, Eric Savitz, Forbes 5/2012
 Open source Apache Hadoop implementation for enterprise usage is
incomplete
 Hadoop skills are in short supply
 Custom built solutions lack integrated cluster management
 Requires integration effort within the existing analytic ecosystem
 Most integrated solutions do not help with archival
2
© 2013 IBM Corporation
Table 1
Top
Main
Worldwide Hadoop-MapReduce Ecosystem Software Revenue, 2010–2016
Revenue ($M)
Growth (%)
2010
38.5
2011
77
2012
209.2
2013
378
2014
539
2015
682.5
NA
100
171.7
80.7
42.6
26.6
2011–2016
2016 CAGR (%)
812.8
60.2
19.1
Note: See Table 2 for top 3 assumptions and Table 3 for key forecast assumptions.
Source: IDC, 2012
Figure 2
Worldwide Hadoop-MapReduce Ecosystem Software Revenue, 2010–2016
Top
Main
“Currently, Hadoop in general and
MapReduce in particular require
special skills and a great deal of
manual effort. This means a great
risk of error and a shortage of
people to do the job. Over the
forecast period, tools will be made
available to build and manage
these solutions, thus amplifying
demand for Hadoop-MapReduce
software overall.”
IDC 2012
3
© 2013 IBM Corporation
Agenda
 IBM PureData for Hadoop – Overview and Launch
 Technology and Partner Integration
 Next Steps
Q & A
4
© 2013 IBM Corporation
Announcing the new PureData System for Hadoop
 Accelerate time to value
System for Hadoop
 Accelerate time to insight
 Simplify big data adoption and consumption
 Extend the value of the data warehouse
 Implement enterprise class big data
 Minimize system setup and administration
5
© 2013 IBM Corporation
IBM PureData System for Hadoop
Accelerate Hadoop analytics with appliance simplicity
System for Hadoop
Speed
 Speed to insight with built-in analytics
 Speed to value with accelerated deployment
Simplicity
 Ready to load data in hours
 Integrated system management
 Appliance approach reduces complexity
 Single point of support
Smart
 Establish a cost efficient online data archive
 Easily leverage data across the big data
platform
 Enterprise security, governance and high
availability
6
© 2013 IBM Corporation
PureData System for Hadoop Enables Key Use Cases
Enrich Your Information
Base
Improve Customer
Interaction with
with Big Data Exploration
Enhanced 360º View
of the Customer
99%
Reduction
In Time Required
For Analysis
1,100
with Operations Analysis
600K
Metered
Customers
in Five States
7
with Security and
Intelligence Extension
42TB
Association
Publishing
Partnerships
Optimize
Infrastructure
and Monetize Data
Prevent Crime
Real-time
Acoustic
Data Analyzed
Gain IT efficiency
and scale with Data
Warehouse
Augmentation
40X
Gain in Analysis
Performance
© 2013 IBM Corporation
Use Case: Big Data Exploration
Use Cases
• Explore new data and previously
untapped sources
• Visualize and gain new insight with easy
to use spreadsheet-style analysis
• Identify useful information that would
add value when integrated
• Used for data profiling to understand
data before moving to other systems
8
© 2013 IBM Corporation
Use Case: DWA Pre-Processing Hub
Use Cases
• Aggregation of data
• Pre-process cleansing
• Compliance requirements
• Simple analytics / exploration
9
© 2013 IBM Corporation
Use Case: DWA Active Archive
PureData
System for Hadoop
PureData
System for Analytics
Use Cases
• Immediate storage alternative of cold
data
• Cost savings for cold data
• Compliance requirements
• Simple analytics / exploration
10
© 2013 IBM Corporation
BDIG: Launch Event
Agenda
Start
End
Title
8:30
9:00
Registration & Breakfast
9:00
9:10
Welcome
Eric Sall, VP IM
Marketing
9:10
9:30
Building Confidence in Big
Data
Bob Picciano,
GM IM
9:30
10:00
Leverage Confidence within
Your Big Data Use Case
Inhi Cho Suh,
VP, Big Data,
Integration &
Governance
10:00
10:30
Applying Integration and
Governance to Big Data
Use Cases
Michele Goetz
(Forrester)
10:30
10:45
Break
10:45
11:20
Innovations in IIG:
Automation, Context, and
Agility
Martin
Wildberger, VP
IM Development
11:20
11:35
Demonstration: Building
Confidence in Big Data
Guenter Sauter,
IM Product
Management
11:35
12:10
Client Panel – How to Build
Confidence in Big Data
Inhi Cho Suh
12:10
12:15
Wrap up
Eric Sall
12:15
1:15
Lunch
1:15
3:00
1:1s with thought leaders &IBM Execs
Event Description
 September 10th, 2013
 590 Madison Executive Briefing Center, New York
 Launch event type:
 Exclusive, by-invitation only, executive level event
 Local Analysts & Influencers
 Target # of attendees 30-40
Building Confidence in Big Data
Confidence in your Data
InfoSphere builds confidence with trusted & protected data
•
•
•
Automated Integration with Data Click
Visual Context via the Governance Dashboard
Agile governance to protect big data with InfoSphere
Guardium
Confidence in Accelerating Value
PureData for Hadoop simplifies Hadoop.
•
•
Appliance simplicity for Hadoop systems,
Get up and running in hours
Confidence in your Skills
Stampede delivers everything needed to start with Big Data
•
•
11
All the resources needed to get value from big data quickly
Software, Expertise, Training
Speaker
© 2013 IBM Corporation
From Getting Starting to Enterprise Deployment:
InfoSphere BigInsights Brings Hadoop to the Enterprise
PureData for Hadoop
Enterprise class
- Appliance simplicity for the enterprise
BigInsights Enterprise
Edition
Sold by # of terabytes managed
BigInsights for
Hadoop QuickStart
Edition
Free download
Apache
Hadoop
12
- Web-based
mgmt console
- Jaql
- Integrated install
- Many enterprise
features
- Tutorials
- Accelerators
- Performance Optimization
- Visualization Capabilities
- Pre-built applications
- Text analytics
- Spreadsheet-style tool
- RDBMS, warehouse connectivity
- Administrative tools, security
- Eclipse development tools
- Enterprise Integration
....
Breadth of capabilities
© 2013 IBM Corporation
IBM BigInsights Leads the way for Hadoop Platforms
Big Data Platforms look at data end to end – no just in Hadoop
“IBM has the deepest
Hadoop platform and
application portfolio. IBM,
an established EDW vendor,
has its own Hadoop
distribution; an extensive
professional services force
working on Hadoop projects;
extensive R&D programs
developing Hadoop
technologies; connections to
Hadoop from its EDW.”
–The Forrester Wave™: Enterprise
Hadoop Solutions, 1Q12
13
© 2013 IBM Corporation
Agenda
 IBM PureData for Hadoop – Overview and Launch
 Technology and Partner Integration
 Next Steps
Q & A
14
© 2013 IBM Corporation
Offering Highlights for IPDH V1
Current issues customer face…
… and what we plan to deliver
IBM InfoSphere BigInsights
High Data Growth
•
•
•
•
•
Unreliable
Hadoop – subprojects suite, Jaql
Analytics – built-in text analytics & tooling
Usability – web console management
Enterprise Class – security, cluster mgmt.
Integration – DB2, Netezza, JDBC DBs
Netezza Appliantization
Support Complexity
• Manageability – appliance, cluster mgmt.
• Appliance level serviceability – replacement,
update, support
Lack of Analytics
IBM System X Hardware
High Support Costs
Silo System
• Reliable, enterpriseready appliance
• Easy-to-use,
manageable system
•
•
•
•
Management Node – x3550 M4
Data Node – x3630 M4
Rack Switch – BNT 8264
Management Switch – BNT 8052
• Built-in analytics
15
15
© 2013 IBM Corporation
Software
 BigInsights Enterprise Edition
–Based on version 2.1
• Hadoop v1.0, Zookeeper, Hive, HBase, Oozie, etc
• BigSQL, Console, JAQL, BigSheets, Accelerators, etc
–Update Plan: Incorporate BigInsights releases within a quarter
 System Management
–New “Hardware” tab to manage hardware
–Component level view (server, disk, fan, CPU, etc)
 High Availability
–Protects critical Hadoop services (NameNode, JobTracker, etc)
–HA Master node via Linux HA
 EasyArchive+
–New “EasyArchive+” tab and archive/retrieve Console apps
–Extract data from Netezza to HDFS and surface via Hive tables
–Return data from HDFS archive back to Netezza
16
© 2013 IBM Corporation
IBM
Open Source
BigInsights Enterprise Edition Components
IBM InfoSphere BigInsights
Development Tools
Visualization &
Discovery
Eclipse Plug-ins
BigSheets
Text Analytics
MapReduce
Jaql Dev’t
Hive Query
Systems Management
Web Admin
Console
Connectors
JDBC
Netezza
Advanced Engines
Text Processing Engine
and Extractor Library
Workload Optimization
Integrated
Installer
Splittable Text
Compression
ZooKeeper
Oozie
Jaql
Lucene
Pig
Hive
Data Store
File System
DB2
Streams
Enhanced
Security
Runtime
System ML
Adaptive
MapReduce
Flexible
Scheduler
R
Flume
BigIndex
Sqoop
MapReduce
HBase
Column Store
HDFS
GPFS (Beta)
© 2013 IBM Corporation
BigInsights Value Above and Beyond Hadoop
 Analytic Accelerators
– Social Media Accelerator
– Machine Data Accelerator
– BigSheets spreadsheet and visualization
– Advanced Text Analytics Accelerator
– JAQL query language
 Performance and Optimization
– Adaptive Map Reduce
– Advanced Scheduler
– BigIndex for large scale indexing
– Fast, splittable compression
 Security
– Role based authorization
 Optim Development Studio
– Eclipse based IDE for Java
 Big Data Integration
– Information Server, InfoSphere Streams,
Netezza, DB2
 Enterprise Enablement
– Big SQL
– GPFS-FCO
IBM’s distribution is based on Apache Hadoop and utilizes many of the capabilities
includes in that distribution, but IBM is focused on making its distribution more of an
enterprise class offering.”
18
© 2013 IBM Corporation
IPDH Architectural Highlights
Master
nodes
Data
nodes
Appliance Management
HA
 Appliance hardware
management
 High-Availability (via
failover) on master nodes
BigInsights
Unified GUI
 Complete self-contained
BigInsights appliance
 Redundant components
for failure resilience
(disks, switches, power
supplies)
 Secure operation (access
only via edge nodes)
 GUI, CLI, and API for
administrative controls
19
19
© 2013 IBM Corporation
Hardware Overview
Full Rack
Data Nodes
Memory per node
96 GB
CPU Cores per node
12
Drives per node
12
Drive Size
20
18
3 TB
Total Raw Storage
648 TB
User Space (w/ replication)
216 TB
© 2013 IBM Corporation
Hardware Specifications
GTM
Master Node
(x2)
Data Node
(x18)
Rack Switch
x3550 M4
•Dual Intel E5 2600 series
•128 GB RAM
•3 x 3 TB 3.5” Drives
•2 x 10Gbe ports
•4 x 40Gbe ports
x3630 M4
•Dual Intel E5 2400 series
•96 GB RAM
•14 x 3 TB 3.5” Drives
•Dual Port 10 Gbe
BNT 8264TR
(x2)
•48 x 10 Gbe
•4 x 40 Gbe
Management Switch
BNT 8052
(x1)
•48 x 1 Gbe
Designed for High Availability
© 2013 IBM Corporation
Application Integration (IPDH version 1.0)
 Secured configuration – Default
– Access to the system via BigInsights REST APIs
– ODBC/JDBC for BigSQL and Hive
– REST API for:
• HDFS read/write
• Console app execution
– Access to Master Node services:
• Oozie, Zookeper, Hive, etc
– No access to Data Nodes, so:
• No HDFS RPC API (or libhdfs)
• No HBase RegionServer access
 Connected Client configuration
– Access to Data Nodes and Master Node
• HDFS NameNode, JobTracker, Zookeper, Hive, HBase Master, etc
• AND HDFS RPC API, HBase RegionServer, etc.
– Should support all Hadoop applications
22
© 2013 IBM Corporation
Agenda
 IBM PureData for Hadoop – Overview and Launch
 Technology and Partner Integration
 Next Steps
Q & A
23
© 2013 IBM Corporation
IPDA Technical Validation Process and Acceptance into
“Ready For PureSystems”
Program
Registration
Obtain IBM ID and Register
your Company in IBM
PartnerWorld
Execute IPDA /IPDH
Attachment in IBM
PartnerWorld
Technical
Validation
Request Access to Hosted
Partner Network or
Download Big Insights 2.1
Use your usual test
methodology for verifying
data source capability
“Ready for”
PureSystems
IBM will confirm your
acceptance into Ready For
Pure Systems Program
Sign the online supplement
that grants authorization to
use the Ready For mark
Contact us with questions
Enter your solution
information into Global
Solution Directory
24
Confirm validation of
specific versions to technical
management
Use the mark on your web
site and other marketing
efforts.
© 2013 IBM Corporation
Next Steps
 IPDH is accessible in the partner lab – [email protected]
 No emulator at this time
 Big Insights 2.1 is downloadable (see next slides)
 Completion qualifies your solution for the PureSystems
Center (ReadyFor IPDH)
25
© 2013 IBM Corporation
Value Package - PartnerWorld Software Benefits Purchase Required
 IBM Value Package
$2K US subscription
$1.8K US renewal
– Software access
– Pre-sales pre-deployment technical support
– You Pass, We Pay
• Designed for Business Partners invested in IBM software
• Critical for attaining/retaining Software Value Plus authorization
• Tiered by membership level
 Software Access Option
– Software
26
© 2013 IBM Corporation
Software Access Option
 Annual subscription $795 US
– Access to IBM Software; usage
includes
• Demonstration & evaluation
•
Development and testing
•
Internal training
•
Run Your Business
www.ibm.com/partnerworld/value
27
© 2013 IBM Corporation
Summary
Value for partners:
 The Hadoop market is forecast to experience a CAGR of over 60% by 2016.
(IDC, 2012)
 The IBM PureData System for Hadoop brings appliance simplicity to customers
and partners for enterprise class Hadoop deployments.
 IBM’s focus and leadership on Big Data can help business partners grow sales
and expand target markets.
28
© 2013 IBM Corporation
Questions?
29
© 2013 IBM Corporation
© International Business Machines Corporation 2012
International Business Machines Corporation New Orchard Road Armonk, NY 10504
IBM, the IBM logo, PureSystems, PureFlex, PureApplication, PureData and ibm.com are trademarks of International Business
Machines Corporation, registered in many jurisdictions worldwide.
A current list of IBM trademarks is available on the Web at www.ibm.com/legal/copytrade.shtml
All rights reserved.
WAP12402-USEN© 2013 IBM Corporation
30 01