Information Server Roadmap

Information Server
Roadmap
Delivering business value via an integrated
platform
Agile Integration
Wherever your integration resides,
integrate it quickly and flexibly
Business Driven Governance
Make decisions with confidence using
trusted data at the point of impact
Sustainable Quality
Ensure information accuracy and
quickly adapt to strategic
business changes
2
Analyst Surveys in the past 12 months
Critical Capabilities: Data Delivery
Styles for Data Integration Tools,
December 2012
http://www.gartner.com/technology/reprints.do?
id=1-1DASS01&ct=121218&st=sb
3
Magic Quadrant for Data
Integration Tools
July 2013
http://www.gartner.com/technology/reprints.do?
id=1-1HYC2VJ&ct=130731&st=sb
Informa:on Server Recent Ac:vity Prior Highlights Opera0ons Console Browser-­‐based project/job monitoring to view and analyze run-­‐:me environment 8.7
self-­‐service data integra:on on-­‐demand through a simple web-­‐based interface for any user 9.1
Informa0on Server & Hadoop Workload Management Provide priori:za:on of mission cri:cal tasks or specific project of job workloads 9.1
4
InfoSphere Data Click Leverage the same designer UI to read/wr:e to Hadoop and automa:cally generate MapReduce 8.7
Informa:on Server Recent Ac:vity Prior Highlights Connector Accelerators •  New connectors available for download on developerWorks •  Plugs into InfoSphere DataStage and QualityStage and operates just like any other stage. •  Includes features to exploit specific data sources o 
o 
o 
o 
o 
o 
o 
o 
o 
Mongo Cassandra Hbase Avro Hive JMS Lotus Notes WebSphere Transforma:on Extender And more … h"ps://www.ibm.com/developerworks/community/files/app?
lang=en#/folder/4645e12a-­‐7bdb-­‐40ed-­‐a103-­‐f1160b707758 5
5
!
Informa:on Server Recent Ac:vity Overview Data Governance 2010 8.5 Data Quality 2011 2012 FP1 FP2 8.7 Data Integra0on Accelera0on -  Advanced transforma:on features (looping/v.pivot) Stronger Governance -  zOS File Stage -  Opera:ons Console -  Integrated Balanced -  Business Glossary Workflow Op:mizer capabili:es -  Blueprint Task Management Robust Enterprise Support -  Metadata Asset Manager -  New Suite Installer Product Integra0on -  Ac:ve/Passive High Availability support -  Leverage Data Valida:on Rules in DataStage Jobs -  Source Code Control - 
Advanced Data Replica:on Integra:on integra:on -  Next Genera:on Netezza Simple Data Quality Connec:vity & Op:miza:on -  Standardiza:on Quality - 
HDFS Integra:on Assessment -  Match Specifica:on Advanced Admin & Produc0vity Report -  Match Designer Updates -  Parallel Debugger -  New Backup/restore tooling -  Maintenance Mode -  Stronger Encryp:on 6
Data Integra:on Infrastructure 2013 2014 FP3 FP2 FP1 9.1 FP1 Agile integra0on -  InfoSphere Data Click -  Enhanced Workload Mgmt -  ODM Integra:on -  Hadoop Balanced Op:miza:on -  HDFS Extensions -  InfoSphere Streams Integra:on Business Driven Governance -  Policy and rules support for informa:on governance -  Web-­‐based blueprints -  Integrated metadata mgmt enhancements Sustainable Quality -  Data Quality Console -  Standardiza:on Rules Designer -  Data Rules Advancements 9.1.2 Business Driven Governance -­‐  IDA 8.5 -­‐  Addi:onal Workflow Roles -­‐  Data Rules Meetadata -­‐  Bulk metadata import Sustainable Quality -­‐  Profiling Big Data -­‐  Excep:on Stage -­‐  New QS standardiza:on rulesets (Thailand , Ireland , India update) Anywhere Integra0on -­‐  Big Data Features * JSON support * JDBC connector -­‐  DB2 on z/OS load op:miza:on -­‐  Data Click new data sources/targets What’s New in Informa:on Server v9.1.2 Overview Data Governance Data Quality Data Integra:on Infrastructure General Notes •  9.1.2 Released August 29. 2013 to Fix Central •  Image in Passport Advantage updated the following week •  Informa:on Server 9.1.2 becomes the primary maintenance branch for the 9.1 version (any subsequent fix pack for 9.1 will be building on this branch) •  This is a true maintenance branch, including about 200 fixes from across the suite •  Fix list can be reviewed here… hep://www-­‐01.ibm.com/support/docview.wss?uid=swg21640382 •  A few things we wanted to include in 9.1.2 shipped later in Q4, and will install as a patch on top of 9.1.2 7
What’s New in Informa:on Server v9.1.2 Overview Data Governance Data Quality Data Integra:on Infrastructure Support for Informa0on Data Architect 8.5 •  Builds on the new metabroker introduced at 9.1 for Informa:on Data Architect which: -­‐ introduced be"er performance at lower resource cost -­‐ removed Windows only dependency •  Cer:fica:on of IDA v 8.5 added •  Tolerance for orphaned and invalid objects (ability to ignore those that don’t impact rest of model) •  Improved error/warning logging 8
What’s New in Informa:on Server v9.1.2 Overview Data Governance Data Quality Data Integra:on Infrastructure Business Glossary Enhancements Data Rules Metadata •  Display data rule asset types (including unpublished rules) from InfoSphere Informa:on Analyzer in the Browse All Assets page, can be searched, assigned to terms, governance rules, business labels and data stewards •  Drill down from a Governance Rule to a Data Rule to the Database column to which its applied 9
What’s New in Informa:on Server v9.1.2 Overview Data Governance Data Quality Data Integra:on Infrastructure Business Glossary Enhancements (con0nued) Workflow Roles • Development Log now captures every history event including crea:on and reviewer comments • Security roles have been changed to provide a higher degree of granularity for exis:ng roles: Author, Published and Reader • Two new workflow roles: • Reviewer: can review changes and make comments • Approver: can approve changes to a new or exis:ng term (but no edit abili:es themselves) • Can now add comments at every step of the workflow process. Export Development Glossary • Can now export either development or published glossary 10
What’s New in Informa:on Server v9.1.2 Overview Data Governance Data Quality Metadata Workbench Enhancements Data Integra:on Infrastructure JDBC Connector support •  Display details for JDBC Connector stage, including URL Defini:on, Schema, Table and SQL statements •  Includes ability to s:tch JDBC into lineage flows XML/JSON Support •  Browse, query and detail display for XML/JSON •  Displays column level informa:on within asset page •  Can be linked via manual binding for lineage 11
What’s New in Informa:on Server v9.1.2 Overview Data Governance Data Quality Data Integra:on Infrastructure Execu0ve Governance Dashboard Business-­‐Driven & Measured Governance Innova0on •  Measurements for policies and KPIs •  Rapid crea:on of tailored dashboards •  Leverages SQL Views across data quality metadata and profiling results Value •  Immediate insight into governance policy status •  Intercep:on of issues when they start, right at the source Usage •  Raises data confidence with visual governance status 12
1000 s
of data points and policies visualized What’s New in Informa:on Server v9.1.2 Overview Data Governance Data Quality Data Integra:on Profiling Big Data in Hadoop •  Informa:on Analyzer has been extended to support assessing data quality of Hive sources via the business analyst interface. •  Includes the following data quality capabili:es for Hive •  Table and column level profiling for detailed inspec:on of values, including frequency distribu:on, cardinality, completeness, etc… •  Advanced analysis and monitoring provides source system profiling and analysis capabili:es to help you classify and assess your data. •  Integrated rules analysis uses data quality rules for greater valida:on, trending and paeern analysis (same rules that are being applied via the Data Rules stage in DataStage and QualityStage) 13
Infrastructure What’s New in Informa:on Server v9.1.2 Overview Data Governance Data Quality Data Integra:on Infrastructure New Standardiza0on Rules • Country specific rule sets for India, Ireland and Thailand • Provide for data standardiza:on of names (individual and organiza:onal), addresses, phone and locality (varies per country) • All rule sets can be used with the Inves:gate and Standardize stages • Delivered as archive files in the QSRules folder of the install directory • Client = ./Informa:onServer/Clients/Classic/QSRules • Server = ./Informa:onServer/Server/PXEngine/QSRules 14
What’s New in Informa:on Server v9.1.2 Overview Data Governance Data Quality Data Integra0on Infrastructure InfoSphere Data Click self-­‐service data integraPon on-­‐demand Overview •  Business users need quick and easy access to informa:on to support their analy:cal projects. •  Organiza:ons need to avoid data sprawl, so governance best prac:ces must be ensured New in this release •  Universal Connec:vity via ODBC to now support DB2, Netezza, Oracle, Teradata, Sybase, SQL Server, Greenplum, and others…. as source or target •  Automa:c filtering of columns with data types not supported by the target data store •  Leverages connector framework enhancement for data sampling via “row limits” •  hep://www.youtube.com/watch?v=hUGGudh2iWI&feature=youtu.be 15
IBM CONFIDENTIAL
What’s New in Informa:on Server v9.1.2 Overview Data Governance JSON Document Support Data Quality •  Derive metadata format automa:cally from sample JSON documents… • Supports hierarchical formats with simple fields, objects and arrays • Schema views •  New Parsing and Composing steps provide for complex hierarchical data in JSON syntax; with value and structure valida:on op:ons •  Mul:ple op:ons for reading/wri:ng data -­‐  files directly from disk -­‐  as part of a long string -­‐  passed in/out as a LOB 16
Data Integra0on Infrastructure What’s New in Informa:on Server v9.1.2 Overview Data Governance Data Quality JDBC Connector •  JDBC Connector provides Informa:on Server products with access to JDBC data sources •  Supports data read and write opera:ons and metadata import opera:ons •  Cer:fied in this release with Apache Derby and IBM Big Insights Big SQL drivers •  Simple setup – introduces isjdbc.config file to track JDBC drivers to be used •  Create in the DSEngine subdirectory •  Include CLASSPATH=semicolon separated driver classpaths •  CLASS_NAMES=semicolon separated driver class names (needed only for JDBC 3.0 drivers and older) 17
Data Integra0on Infrastructure What’s New in Informa:on Server v9.1.2 Overview Data Governance DB2 Z Bulk Load Op0miza0on Data Quality Data Integra0on Infrastructure Huge Performance Gains For Load • Moved from a single load stream to parallel streaming via Z pipes. • Mul:ple LOAD u:li:es targe:ng separate par::ons which performs faster than a single LOAD u:lity targe:ng all par::ons. • 9.1.2 is 80 to 160% faster than 9.1 (depending on number of par::ons) • Performance scales almost linearly as you increase the number of par::ons, regardless of load method. • Internal tes:ng loading almost 1TB per hour using 16-­‐way load Huge Performance Gains For Read • connector determines the number of par::ons in the table and dynamically configures the number of DataStage nodes to match the number of par::ons • Parallel read using the 9.1.2 DB2 connector is 40% faster than the 9.1.2 DB2Z stage, regardless of the number of par::ons. Resilience • When Retry on connec:on failure is set to Yes the connector will try to establish an FTP connec:on again when the ini:al aeempt to connect fails. 18
What’s New in Informa:on Server v9.1.2 Overview Data Governance Data Quality Data Integra:on Infrastructure Metadata Import Performance Op0miza0on (IMAM) •  Performance benefits of BI Simplifica:on in 9.1 -­‐ 46% reducPon in execuPon Pme of Express Import •  Performance benefits of physical model import (Erwin) -­‐ 44-­‐60% reducPon in execuPon Pme of Express Import (Erwin) •  IMAM Express Import in 9.1 is 7 -­‐ 1200% faster than in 8.7 for the following workloads •  IDA: -­‐ Small workload (55K assets): +1200% (Throughput: 450 objects/s) -­‐ Large workload (119K – 430K): significantly lower resource requirements •  Erwin (124K assets): +50% (Throughput: 351 objects/s) •  BO import (175K assets): +18% (Throughput: 318 objects/s) •  DB2 PDR (108K assets): +7% (Throughput: 149 objects/s) •  Cognos (141K assets): -­‐20% (Throughput:120 objects/s) •  MITI in 9.1 extracts more metadata (+45% reports &+27x more models, etc) than MITI in 8.7 Note: Performance results may vary in other environments 19
What’s New in Informa:on Server v9.1.2 Overview Data Governance Data Quality Data Integra:on Infrastructure Connector Enhancements Limit number of returned rows • New property to support database sampling(required for feature of Data Click) • Applies to the following Connectors: ODBC, DB2, Netezza, Oracle, Teradata and JDBC Name LimitRows Label Limit number of returned rows. Descrip0on Select Yes to limit the number of rows that are returned by the connector. Default value False (No) Limit Limit Enter the maximum number of rows 1000 that w
ill b
e r
eturned b
y t
he c
onnector. ODBC Connector expanded binary support • The ODBC Connector now supports automa:cally generated 'CREATE TABLE' statements for types Binary, VarBinary or LongVarBinary 20
What’s New in Informa:on Server v9.1.2 Overview Data Governance Big Data Connec0vity for Hive Data Quality Data Integra:on •  High-­‐performance and throughput with support for Hive2 and concurrent connec:ons •  Improved authen:ca:on for increased data security •  Full driver metadata •  Support for parameter arrays, processing the arrays as a series of execu:ons, one execu:on for each row in the array •  Support for standard SQL func:onality, including Create Index, Create Table, Create View, Drop Index, Drop Table, Drop View •  Support for a wide range of data types: Int, TinyInt, SmallInt, BigInt, String, Double, Binary, Boolean, Float, and Timestamp 21
Infrastructure Thank you