Beijing, Asia 4, 11.04.2012

The EPIKH Project
(Exchange Programme to advance e-Infrastructure Know-How)
GRID Science Gateway
Riccardo Rotondo ([email protected])
Consortium GARR
Tutorial for Grid Application Porting on Grid
Science Gateway
Beijing, 11.04.2012
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
1
1
Outline
• Grid Usage
• Science Gateway
• The e-Collaboration environment
– The old paradigm
– Science Gateway architecture
– The new “brick” approach
• Use Cases:
– The DECIDE Science Gateway
– The INDICATE e-Culture Science Gateway
– The CHAIN Science Gateway
– The Gilda t-Infrastructure Portal
– The RICeVI Portal
• Conclusions
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
2
2
GRID at world scale
>340 sites in 57 countries
~337,000 CPU cores
>220 PB of storage
~1 million jobs/day
~20,000 users in ~280 VOs
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
3
3
But…using Grids is not straightforward 
JobType = "MPICH";
CpuNumber = 16;
MPIGranularity = 4;
$ voms-proxy-info --all
subject : /C=IT/O=GILDA/OU=Robots/L=INFN
Catania/CN=MrBayes/CN=proxy/CN=proxy
issuer : /C=IT/O=GILDA/OU=Robots/L=INFN
Catania/CN=MrBayes/CN=proxy
identity : /C=IT/O=GILDA/OU=Robots/L=INFN
Catania/CN=MrBayes/CN=proxy
type
: proxy
strength : 1024 bits
path
: /tmp/x509up_u512
timeleft : 500:38:44
=== VO gilda extension information ===
VO
: gilda
subject : /C=IT/O=GILDA/OU=Robots/L=INFN Catania/CN=MrBayes
issuer : /C=IT/O=INFN/OU=Host/L=Catania/CN=voms.ct.infn.it
attribute : /gilda/Role=NULL/Capability=NULL
attribute : /gilda/generic-users/Role=NULL/Capability=NULL
attribute : /gilda/idl/Role=NULL/Capability=NULL
timeleft : 23:59:33
uri
: voms.ct.infn.it:15001
Scripts
Type = "Job";
MPIType = "MVAPICH2_PGI706";
echo Staging Input Data \(Courtesy of European Space
Agency\);
#edg-rm --vo=gilda copyFile lfn:$1.N1 file://$PWD/$1.N1;
lcg-cp --vo=gilda lfn:$1.N1 file://$PWD/$1.N1;
echo Staging Application;
gunzip beam20.tar.gz;
tar xvf beam20.tar;
cd beam-2.0/bin;
echo Starting Application;
./pds2jpg-ASAR-run.sh $1;
mv $1-b*.jpg ../..
cd ../..
rm -fr beam-2.0;
rm -fr $PWD/$1.N1;
rm -fr $PWD/beam20.tar;
echo Input ENVISAT Product courtesy of European Space
Agency
touch
ENVISAT_Product_courtesy_of_European_Space_Agency
echo No Output Packaging;
echo Done!;
JDL
Executable = "flash2";
StdOutput = "mpi.out";
StdError = "mpi.err";
InputSandbox =
{"watchdog.sh","mpi.pre.sh","mpi.post.sh","flash.par","summers_den_
1e0.rates","flash2"};
CLI
$ glite-wms-job-status https://gilda-lb-01.ct.infn.it:9000/1o4BVjqg2tJ4rNXTAIGAg
*************************************************************
BOOKKEEPING INFORMATION:
Status info for the Job : https://gilda-lb-01.ct.infn.it:9000/1o4BVjqg2tJ4rNXTAIGAg
Current Status: Done (Success)
Logged Reason(s):
- Job terminated successfully
Exit code:
0
Status Reason:
Job terminated successfully
Destination:
grid010.ct.infn.it:2119/jobmanager-lcgpbs-gilda
Submitted:
Tue Jun 29 15:34:40 2010 CEST
OutputSandbox =
{"mpi.err","mpi.out","watchdog.out","flash_bubble.log","amr_log"};
Requirements = (other.GlueCEUniqueId == "unipa-ce01.pa.pi2s2.it:2119/jobmanager-lcglsf-hpc");
RetryCount = 0;
*************************************************************
GSI
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
4
4
Grid Interface Evolution
• The way users access Grid resources has continuously
evolved towards simplicity and transparency:
• Command Line
– Globus and gLite CLI
– Used by the enthusiastic and early adopter scientists
• GUI applications
– gEclipse, Grid2Win
“robot”
certificate
– Good to expand the communities but difficultsto maintain
on “e-tokens”
• Web Interface
– GENIUS, P-GRADE
– Easier for new users but monolithic
• Science Gateways
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
5
5
Science Gateways
A framework of tools that allows scientists to run applications with
little concern for where the computation actually takes place. This is
similar to cloud computing in which applications run as Web
services on remote resources in a manner that is not visible to the
end user. However, a science gateway is usually more than a
collection of applications. Gateways often let users store, manage,
catalogue, and share large data collections or rapidly evolving novel
applications they cannot find anywhere else. Training and education
are also a significant part of some Science Gateways
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
6
6
Reference Model
Embedded Applications
Appl 2
Appl N
Science
Gateway
Appl 1
Administrator
Power User
Basic User
Grid Services
Users from different
organisations having
different roles and
privileges
Other Middleware
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
7
7
Federated Grid User
Science
Gateway
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
8
8
Identity provided federated
{ idp1, idp2, …
idPN }
{ idp1, idp2, …
idpN }
{ idp1, idp2, …
idpN }
{ idp1, idp2, …
idpN }
Beijing, Asia 4, 11.04.2012
{ idp1, idp2, …
idpN }
{ idp1, idp2, …
idpN }
{ idp1, idp2, …
idpN }
Riccardo Rotondo
{ idp1, idp2, …
idpN }
9
9
Federated Grid User
Science
Gateway
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
10
10
AuthN & AuthZ Schema
Science
Gateway
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
12
12
Identity Provider Federated
{ idp1, idp2, …
idPN }
{ idp1, idp2, …
idpN }
{ idp1, idp2, …
idpN }
{ idp1, idp2, …
idpN }
Beijing, Asia 4, 11.04.2012
{ idp1, idp2, …
idpN }
{ idp1, idp2, …
idpN }
{ idp1, idp2, …
idpN }
Riccardo Rotondo
{ idp1, idp2, …
idpN }
13
13
eduGAIN
(www.edugain.org)
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
14
14 14
Federation Supported
• 7 Federations supported:
•
•
•
•
•
•
•
GrIDP (“catch-all”, Italy);
GARR-IDEM (Italy);
CARSI (Chinese);
GRNET-AAI (Greece);
RCTSaai (Portugal);
SIR (Spain);
eduGAIN (Europe).
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
15
15
Federation supported by DECIDE
Science Gateway
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
16
16
Other federation supported (1/2)
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
17
17
Other federation supported (2/2)
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
18
18
The GrIDP Identity Provider (1/2)
• 4 Identity Providers are available in GrIDP:
•
•
•
•
A “catch-all” IdP created at Catania;
The maat-G (enterprise) IdP;
INFN-AAI IdP (all INFN researchers and associates);
An idp that enables Social Networks credentials.
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
19
19
The GrIDP Identity Provider (2/2)
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
20
20
Liferay (www.liferay.com)
• Highly-configurable, scalable, open source portal
framework;
• Compatible with JSR 168/286 standards
and based on modern web 2.0
technologies;
• Liferay services planned to be used:
– Portal;
– CMS & WCM;
– Collaboration and “social” software
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
21
21
Grid Access enable: Portlets as bricks
• Portlets can interact with the Grid
Infrastructure
e-
• Different approaches are available:
– Execute the Command Line behind the portal;
– Using API where available:
• Must be in Java or other languages supported by Liferay;
– Call REST services from Javascript code in the browser;
• Additional layers between Liferay and the Grid can
be necessary for some services;
• Each portlet can follow its own communication
method.
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
22
22
Usage Workflow
eTokenServer
1. Sign in
GRID
2. Grid
Request
5.
Tracking
6. Getting
Results
Beijing, Asia 4, 11.04.2012
5. Grid
Interactions
User
Tracking DB
Riccardo Rotondo
23
23 23
Functionalities
Riccardo Rotondo
24
Beijing, Asia 4, 11.0
24
Science Gateway Developed
•
•
•
•
•
DECIDE Science Gateway
EUMDEDGRID SUPPORT Science Gateway
INDICATE Science Gateway
VIRALGRID Science Gateway
GISELA Science Gateway
• CHAIN Science Gateway
• The Gilda t-Infrastructure Portal
• The RICeVI Portal
Riccardo Rotondo
25
Beijing, Asia 4, 11.0
25
Liferay AT INFN-CT
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
26
26
GRID Science Gateway
GRID
SCIENCE
GATEWAY
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
27
27
The DECIDE Science Gateway (1/5)
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
28
28
The DECIDE Science Gateway (2/5)
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
29
29
The DECIDE Science Gateway (3/5)
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
30
30
The EUMEDGRID-Support Science Gateway (1/2)
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
31
31
The EUMEDGRID-Support Science Gateway (2/2)
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
32
32
The INDICATE e-Culture Science Gateway (5/6)
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
33
33
The INDICATE e-Culture Science Gateway (6/6)
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
34
34
CMS Science Gateway
CMS
SCIENCE
GATEWAY
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
35
35
The CHAIN Science Gateway
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
36
36
The CHAIN Science Gateway
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
37
37
The CHAIN Science Gateway
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
38
38
The Gilda t-Infrastructure Portal
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
39
39
E-learning Science Gateway
E-learning
SCIENCE
GATEWAY
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
40
40
RICeVI Portal
Collaboration
Sharing
Dissemination
Training
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
41
41
Current Status
• 16 Liferay-based Science Gateways (hosted in 2 servers) are
currently powered by Shibboleth at INFN Catania;
• 7 Federations supported;
• 4 instances are registered as official IDEM Service Provider;
• 4 Identity Providers are available in GrIDP.
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
42
42
Credits & Acknowledgments
Credits
•
•
•
•
•
•
•
•
•
•
•
Acknowledgments
Valeria Ardizzone (GARR);
Roberto Barbera (UNICT & INFN)
Riccardo Bruno (COMETA);
Antonio Calanducci (COMETA);
Marco Fargetta (COMETA)
Elisa Ingrà (GARR);
Giuseppe La Rocca (INFN)
Salvatore Monforte (INFN);
Fabrizio Pistagna (INFN);
Rita Ricceri (INFN);
Diego Scardaci (INFN);
Beijing, Asia 4, 11.04.2012
• Vincenzo Ciaschini (INFN);
• Enrico Fasanelli (INFN);
• Maria Laura Mantovani
(GARR);
• Barbara Monticini (GARR);
• Simona Venuti (GARR)
Riccardo Rotondo
43
43
References
• Science Gateway Url:
–http://applications.eu-decide.eu/
–http://applications.eumedgrid.eu/
–http://gilda.ct.infn.it
–http://www.chain-project.it
–http://indicate-gw.consorzio-cometa.it
–https://gridp.ct.infn.it/
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
44
44
Questions?
Thank you for your attention
Beijing, Asia 4, 11.04.2012
Riccardo Rotondo
45
45