Nessun titolo diapositiva

GENIUS: a Web Portal for DataGRID
Roberto Barbera(*)
(*)work
Prague,
12.12.2002
CHEP 2000,
10.02.2000
in collaboration with A. Falzone and A. Rodolico
1 Barbera
Roberto
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Outline
The Grid Vision: the concept of Collaboratory
Grid activities in Europe: the DataGrid Project
The main “actors” of the DataGrid Project
“Easy” ed “ubiquitous” access to the Grid: the
GENIUS web portal
The Future Grid Projects
Conclusions and perspectives
Prague, 12.12.2002
2
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
A new way to Co(l)laborate…
A Co(l)laboratory is a place where scientists and
researchers work together to solve complex interdisciplinary problems, despite geographic and
organizational boundaries.
Co(l)laboratories provide “uniform” and “democratic”
access to computational resources, services and/or
applications.
Co(l)laboratories expand the resources available to
researchers, enable multidisciplinary collaborations and
problem solving, increase the efficiency of research , and
accelerate the dissemination of knowledge.
Computational grids can be the natural solution for the
computing problems of a Co(l)laboratory.
Prague, 12.12.2002
3
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Prague, 12.12.2002
4
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
DataGRID Project
The Grid metaphor
(courtesy of F. Gagliardi)
Mobile Access
G
R
I
D
Workstation
M
I
D
D
L
E
W
A
R
E
Supercomputer, PC-Cluster
Data-storage, Sensors, Experiments
Visualising
Internet, networks
Prague, 12.12.2002
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
DataGRID Project
-CPU
-storage
Mobile Access
- memory
- I/O
G
R
I
D
Workstation
01011010110
M
I
D
D
L
E
W
A
R
E
Visualising
Prague, 12.12.2002
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
DataGRID Project
Mobile Access
virtual
services
virtual
services
Workstation
Visualising
Prague, 12.12.2002
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Present Grid investments in EU and USA
Million
6
French ACI GRID
38
Italian Funding (MIUR+CNR+INFN)
51
EU IST Funding
UK Government’s Office of
Science and Technology
196,1
60,3
Distributed Terascale Facility
(USA)
0,0
20,0
40,0
60,0
Future figures:
US Cyber Infrastructure:
Japan (A-P) Grid:
Prague, 12.12.2002
80,0
100,0
120,0
140,0
1020 M$
~500 M$
160,0
180,0
200,0
8
Roberto Barbera
GRID Projects – EU IST (~37 M€)
An integrated approach
Applications
EGSO
CROSSGRID
Middleware GRIP EUROGRID
& Tools
DAMIEN
Underlying
Infrastructures
Industry / business
Prague, 12.12.2002
GRIDSTART
GRIA
GRIDLAB
DATAGRID
DATATAG
iVDGL
Science
9
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
DataGRID main contractors
CERN – International (Switzerland/France)
CNRS - France
ESA/ESRIN – International (Italy)
INFN - Italy
NIKHEF – The Netherlands
PPARC - UK
Prague, 12.12.2002
10
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Other DataGRID partners
Industrial Partners
•Datamat (Italy)
•IBM-UK (UK)
•CS-SI (France)
Research and Academic Institutes
•CESNET (Czech Republic)
•Commissariat à l'énergie atomique (CEA) – France
•Computer and Automation Research Institute,
Hungarian Academy of Sciences (MTA SZTAKI)
•Consiglio Nazionale delle Ricerche (Italy)
•Helsinki Institute of Physics – Finland
•Institut de Fisica d'Altes Energies (IFAE) - Spain
•Istituto Trentino di Cultura (IRST) – Italy
•Konrad-Zuse-Zentrum für Informationstechnik Berlin - Germany
•Royal Netherlands Meteorological Institute (KNMI)
•Ruprecht-Karls-Universität Heidelberg - Germany
•Stichting Academisch Rekencentrum Amsterdam (SARA) – Netherlands
•Swedish Research Council - Sweden
Prague, 12.12.2002
11
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
EDG overview : work packages
EDG Project is structured in 12 Work Packages:












}
WP1: Work Load Management System
WP2: Data Management
WP3: Grid Monitoring / Grid Information Systems
Middleware
WP4: Fabric Management
WP5: Mass Storage Management
WP6: Testbed and demonstrators
WP7: Network Monitoring
WP8: High Energy Physics Applications
Applications
WP9: Earth Observation
WP10: Biology
WP11: Dissemination
WP12: Management
Prague, 12.12.2002
}
12
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Computational biology
Prague, 12.12.2002
13
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Medical Diagnostic Imaging
Prague, 12.12.2002
14
Roberto Barbera
Earth Observation Community
GRID interactive scenario
Common access to EO
missions catalogues
Acquisition plan, order, delivery
On demand high
level products
generation
Parametric data fusion
and models integration
Collaborative
publishing of
results
Prague, 12.12.2002
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Earth Observation
ENVISAT (launched 01.03.2002 !)
• 3500 MEuro programme cost
•
•
•
•
•
10 instruments on board
200 Mbps data rate to ground
400 Tbytes data archived/year
~100 “standard” products
10+ dedicated facilities in Europe
• ~700 approved science user projects
16 Barbera
Roberto
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
High Energy Physics
http://www.cern.ch
LHC
~9 km
SPS
CERN
Prague, 12.12.2002
17
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
High Energy Physics
Prague, 12.12.2002
18
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
High Energy Physics
ATLAS
CMS
~6-8 PetaBytes / year
~O(108) events/year
~O(103) batch and interactive users
LHCb
Prague, 12.12.2002
19
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Prague, 12.12.2002
The Little Bang
The Big Bang
The Quark Gluon Plasma
Pb+Pb @ LHC (5.5 A TeV)
20
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Total weight
External diameter
Total length
Magnetic Field
10.000t
16,00m
25m
0.2-0-5Tesla
ALICE On-line System
multi-level trigger to filter
background and reduce
the amount of data
ALICE Collaboration already
includes 1223 peoples from
85 institutes in 27 countries
Prague, 12.12.2002
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
1/100 of a Pb+Pb @ LHC !
Simulation and reconstruction of a “full” (central) Pb+Pb collision
at LHC (about 84000 primary tracks!) takes about 24 hours of a
top-PC and produces an output bigger than 2 GB.
Prague, 12.12.2002
22
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
6-8 Petabytes
˜10.000.000 CD-ROM
Prague, 12.12.2002
5 times the
Eiffel Tower
˜1500 m
23
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
ALICE “grid” sites
OSU/OSC
LBL/NERSC
Birmingham
Nantes
Dubna
NIKHEF
Saclay
GSI
CERN
Merida
Lyon
Torino
Padova
Bologna
IRB
Bari
~O(108) events/year
~O(103) batch and interactive users
in 2006
Cagliari
Yerevan
Catania
Kolkata, India
Capetown, ZA
Prague, 12.12.2002
24
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
The LHC “web” in the world
Europe:
267 institutes, 4603 users
Elsewhere: 208 institutes, 1632 users
Prague, 12.12.2002
25
Roberto Barbera
Prague, 12.12.2002
26 Barbera
Roberto
Job Submission work-flow
UI
JDL
Replica
Catalogue
Input “sandbox”
DataSets info
Information
Service
Output “sandbox”
Prague, 12.12.2002
Storage
Element
Globus RSL
Job Status
Logging &
Book-keeping
Publish
Job Query
Job Submit Event
Author.
&Authen.
Expanded JDL
Resource
Broker
Job Status
Job Submission
Service
Compute
Element
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
EDG m/w has been released but…
EDG software (Globus, UI, JDL, WP2, WP3, etc.) contains tens of
commands/switches which also have their own logical sequences (“B”
after “A”, “C” before “D” and so on).
Browsing Grid VO “directories” (users, RC’s, DB’s, etc.) requires LDAP
“speaking” and tomorrow could require SQL “speaking”.
“User gridification” is a tough task for a “rookie”  this does not fit with the
claim that “grids” are for everybody and that grid computing will be as
easy as surfing the Internet ?
Furthermore, all this holds for DataGrid. What will happen when other
grids’ software (especially UI’s) will come up (PPDG, iVDGL, etc.) ? Will
users have to learn tens of “grid dialects” ?
Today “grid computing” is a rather complicated experience only possible
at selected machines (UI’s)  this does not fit with the claim that one
could do “grid computing” even from a PDA ?
Is there any way to set-up a “user-friendly” grid ?
Prague, 12.12.2002
28
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
A web portal: why and how ?
It can be accessed from everywhere and by
“everything” (desktop, laptop, PDA, WAP phone).
It can keep the same user interface to several
back-ends (grid “dialects”  command-line UI’s).
It must be redundantly “secure” at all levels: 1)
secure for web transactions, 2) secure for user
credentials, 3) secure for user authentication, 4)
secure at VO level.
All available grid services must be incorporated in
a logic way, just “one mouse click away”.
Its layout must be easily understandable and user
friendly.
Prague, 12.12.2002
29
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
GENIUS®
(Grid Enabled web eNvironment for
site Independent User job Submission)
[https://genius.ct.infn.it]
GENIUS web portal
Applications’
specific layer
DataGRID
architecture
GLOBU
S
toolkit
ALICE
ATLAS
CMS
LHCb
Other apps
High level GRID middleware
Basic Services
OS & Net services
Prague, 12.12.2002
30
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
GENIUS:
ALICE Collaboration
how it works
https+java/xml+rfb
WEB Browser
GENIUS
EnginFrame
3-tier model
Loc
al
WS
Apache
ED
G
UI
EDG+GSI
the Grid
31
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
GENIUS
ALICE Collaboration
show: the main page
32
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
GENIUS
ALICE Collaboration
show: the authentication
33
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
GENIUS
ALICE Collaboration
show: file services
34
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
GENIUS
ALICE Collaboration
show: the authorization
35
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
GENIUS
ALICE Collaboration
show: job submission
36
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
GENIUS
ALICE Collaboration
show: job queue
37
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
GENIUS
ALICE Collaboration
show: job output
38
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
GENIUS
ALICE Collaboration
show: job data
39
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
GENIUS
ALICE Collaboration
show: personal spooler
40
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
GENIUS show: interactive analysis
41
Roberto Barbera
42
43
44
45
46
47
48
49
50
51
52
53
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Present status and perspectives
Current implementation of GENIUS already includes:
 secure web transactions, user authentication and
authorization;
 remote interaction with the user’s file system;
 interfaces for job submission/control, to VO servers
(users’ and catalogues), and to monitoring systems;
 persistent (user’s) book-keeping and spooler system;
 interactive analysis !
 rpm available !
Todo:
 multi-jobs (parallel and sequential);
 interface to data management and other grid services;
 more application-specific customizations;
 web-guided creation of a work flow system.
54
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
GENIUS vs. other grid portals
It is not a toolkit. It is a complete production-ready
environment which combines the concepts of “user portal”
and “science portal”.
No client software needs to be installed apart from the
web browser. GENIUS can be accessed from everywhere.
No security delegation (“à la MyProxy”) is needed. Access
passwords are securely “streamed” only when needed.
Interactive analysis (via VNC) and web access to personal
spooling areas are possible.
User file system is not limited to input and output files.
EnginFrame modularity makes different customizations
easy to implement. Already available for EDG m/w,
GLOBUS, and LSF. Under definition for CONDOR
(hungarian grids).
It is compatible with the Tomcat open source java servlet
55
container available from SUN.
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
The LHC Computing Grid
Prague, 12.12.2002
56
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Tier-1’s for LHC Experiments
Prague, 12.12.2002
57
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
LHCC
Reports
The LHC Computing Grid
Project Structure
Reviews
The LHC Computing Grid Project
Common
Computing
RRB
Resource
Matters
Project Overview Board
Other
Computing
Grid
Projects
Other HEP
Grid
Projects
Project Manager
Requirements,
Monitoring
Project
Execution
Board
Software and
Computing
Committee
(SC2)
GDB
EU
DataGrid
Project
Prague, 12.12.2002
implementation teams
RTAG
Other
Labs
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
LCG Applications Area Projects
Software Process and Infrastructure (operating)

Librarian, QA, testing, developer tools, documentation, training, …
Persistency Framework (operating)

POOL hybrid ROOT/relational data store
Mathematical libraries (operating)

Math and statistics libraries; GSL etc. as NAGC replacement
Core Tools and Services (just launched)

Foundation and utility libraries, basic framework services, system services,
object dictionary and whiteboard, grid enabled services
Physics Interfaces (being initiated)

Interfaces and tools by which physicists directly use the software.
Interactive (distributed) analysis, visualization, grid portals
Simulation (coming soon)

Geant4, FLUKA, virtual simulation, geometry description & model, …
Generator Services (coming soon)

Generator librarian, support, tool development
Prague, 12.12.2002
59
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
LCG Grid Deployment DRAFT
~April 03 - LCG-1 Global Grid Service






deploy a sustained 24 X 7 service
based on “converged” toolkit emerging from GLUE
and delivered by Datagrid and iVDGL
~10 sites – including sites in Europe, Asia, North America
> 2 times CERN prototype capacity (= ~1,500 processors, ~1,500
disks)
permanent service for all experiments
~Oct 03 –


reliability & performance targets
wider deployment
This Grid services evolves slowly through 2005



new middleware – functionality, performance
grows with more sites and users
provides continuous service for the LHC experiments
Prague, 12.12.2002
60
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
EoI for EU FP6 2002: EGEE
Enabling Grids for E-Science and industry in Europe
Creation and support
E-Science centres
European
Infrastructure
Modulable
Testbeds
R&D Agenda
Semantic GRID
Database
Security
Deployment with
IT Industry
S/W Hardening
GLOBUS
EuroGrid, Gridlab etc.
Prague, 12.12.2002
National
eScience
Centres
Integrated Project
ENABLING GRIDS
ESCIENCE EUROPE
EGEE
Science
Outreach
Consulting
Prototyping
Deployment
Industry
Applications
Industry Outreach
Consulting
Training Courses
Dissemination
Forum
SMEs developing
Grid-enabled Applications
Tools and
Service Development
Applications in
Other Sciences
EIROforum
61
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
EGEE key objectives
Integrate Grid technological developments and expertise from
across Europe
Establish a production quality Europe-wide Grid infrastructure
Enable the creation of e-Science applications from across the
science and industry spectrum
Ensure the timely delivery if its programme of work guided by
the needs of its users
See http://www.cern.ch/egee for more
Participants from Czech Republic:

Faculty of Informatics of Masaryk University, Brno


CESNET z.s.p.o., Prague

Prague, 12.12.2002
Contact: Ludek Matyska
Contact: Jan Gruntorad
62
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Important news from EU
Date: Wed, 30 Oct 2002 12:01:49 +0100
From: [email protected]
Subject: RE: Request of information
To: [email protected]
Dear Domenico,
I know, it is even for us in the Commission hard to follow the story...
The current status is the following:
 There will be an invitation for work on Grids in the context of the
Research Infrastructures (RI) 1st Call (expected to be announced on 17
December this year, budget 50 MEuro).
 Important in this case:
 the RI instruments have differences compared to the IST
instruments since the first are targeted to support the building of RI.
 In IST: work on Grids will be invited under two mainly action-lines,
 the IST-RI and
 the Complex problem solving one.
 Both above action lines are expected to open in mid June 2003.
 I do not have safe information on the budget in the case of IST.
Kyriakos
63
Roberto Barbera
Indicative roadmap of calls
1. Budget from Structuring the ERA Programme (€200m)
Year 2003
Year 2004
€ 50m
Year 2005
€100m
Year 2006
€ 50m
2. Budget from IST (€100m)
Year 2003
€ ?m
Year 2004
Year 2005
Year 2006
€ ?m
64
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Conclusions
Computational grids could represent the “natural”
environment for next generation high energy physics
experiments, computational bio-medecine, Earth
observation and many other inter-disciplinary
applications.
“Grid” could be the Internet “new age” where users can
seamlessly and ubiquitously access not only
information but also huge computing resources and
mass storage systems distributed worldwide with their
own applications.
However, in order to turn dreams into reality, grid
access must be easy and intuitive especially for the
vast majority of non-expert users and this is the
mandate of the GENIUS team.
Prague, 12.12.2002
65
Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Prague, 12.12.2002
66
Roberto Barbera