Presentazione standard di PowerPoint

Stato degli Science Gateway di Catania
Roberto Barbera ([email protected])
Univ. of Catania and INFN
Riunione tecnica sul portale general purpose e sugli science
gateway di Catania – Catania, 7 Marzo 2012
Summary of standards adopted

The framework for Science Gateways developed at Catania is
fully web-based and adopts official worldwide standards and
protocols, through their most common implementations

These are:

The JSR 168 and JSR 286 standards (also known as "portlet 1.0" and "portlet 2.0"
standards)

The OASIS Security Assertion Markup Language (SAML) standard and its
Shibboleth and SimpleSAMLphp implementations

The Lightweight Direct Access Protocol, and its OpenLDAP implementation

The Cryptographic Token Interface Standard (PKCS#11) standard and its
Cryptoki implementation

The Open Grid Forum (OGF) Simple API for Grid Applications (SAGA) standard
and its JSAGA implementation
2
Our reference model
Embedded Applications
App. 2
....... App. N
Science
Gateway
App. 1
Administrator
Power User
Basic User
Standard-based (SAGA)
middleware-independent
Grid Engine
Users from
different
organisations
having different
roles and
privileges
3
3
AuthN & AuthZ Schema
Science Gateway
Authorisation
1. Register to
a Service
GrIDP
(“catch-all”)
LDAP
2. Sign in
(“catchall”)
Social
Networks’
Bridge IdP
IDP_y
.........
IDPCT
The Grid IDentity Pool (GrIDP)
(http://gridp.ct.infn.it)
eduGAIN
(www.edugain.org)
6
Catania Grid Engine
Liferay Portlets
Science
GW 1
Science
GW 2
Science
GW 3
eToken
Server
Science GW Interface
Data Engine
Job Engine
Grid Engine
Users
Track &
Monit.
SAGA/JSAGA API
Users
Tracking
DB
Grid MWs
7DONE
DONE
DONE
By mid April
By end of April
7
Job
Submission
8
Worker Threads for
Status Checking
MONITORING
MODULE
Job
Check Status/
Get Output
USER
TRACKING
DB
WT
WT
WT
WT
WT
WT
WT
WT
WT
WT
Job Queue
Worker Threads for Job Submission
GRID INFRASTRUCTURE(S)
Job Engine - Architecture
Job Engine - Features

The Job Engine has been designed with the following features in mind:
Feature
Description
Status
Middleware
Independent
Capacity to submit job to resources running different
middleware
DONE
Easiness
Create code to run applications on the grid in a very
short time
DONE
Scalability
Manage a huge number of parallel job submissions
DONE
fully exploiting the HW of the machine where the Job
Engine is installed
Performance
Have a good response time
Accounting &
Auditing
Register every grid operation performed by the users DONE
Fault Tolerance
Hide middleware failure to end users
ALMOST
DONE
Workflow
Providing a way to easily create and run workflows
IN PROGRESS
DONE
9
Job Engine – Scalability
Job submission time (h)


Submission time
scales linearly with
number of jobs
>10,000 jobs a
hour
Time to submit 10,000 jobs (h)
40,000 jobs
submitted
in parallel !
10
10
Job Engine – Performances
JobEngine vs. WS-Pgrade (10,000 Jobs)
100000
10000
Tempo in
minuti per
sottomettere
10000 Jobs
1000
100
10
1
JobEngine
11
WS-PGRADE
Job Engine – Middleware interoperability
• Both sequential and MPI-enabled
jobs successfully executed
• Tests with Globus planned
12
Job Engine –
Accounting & Auditing



A powerful accounting & auditing system is included in the Job
Engine
It is fully compliant with EGI VO Portal Policy and EGI Grid
Security Traceability and Logging Policy
The following values are stored in the DB for each job submitted:








13
User ID
Job Submission timestamp
Job Done timestamp
Application name
Job ID
Robot certificate ID
VO name
Execution site (name, latitude, longitude)
Catania Science Gateways in numbers
Registered users
180
160
140
120
100
80
60
40
20
0
09/2011
10/2011
11/2011
12/2011
01/2012
02/2012
Overall usage (arb. units)
350
300
250
200
150
100
50
0
14
Dec
Jan
Feb
Data Engine – Requirements

A file browser shows Grid files in a tree

File system exposed by the Science Gateway is
virtual

Easy transfers from/to Grid (through the SG at
the moment) are done in a few clicks

Users do not need to care about how and where
their files are really located
15
Data Engine –
Usage Workflow
3. Proxy
request
4. Proxy
transfer
eTokenServer
1.
Sign
in
2. Upload
request
7. Upload
on Grid
5. File
Upload
16
7.
Tracking
6. Update
DB
User
Track.
DB
DOGS
DB
GRID
16
DOGS: Data On Grid Services –
Back-end implementation




JSAGA API used to transfer data from/to
storage elements
Hibernate to manage the VFS collecting
information on files stored on Grid; any
changes/actions in the user view affect the VFS
MySQL as underlying RDBMS
An additional component has been developed
in order to keep track of each
transaction in the users tracking DB
17
DOGS: Data On Grid Services –
Front-end implementation

A portlet has been created wit access provided only
to federated users with given roles and privileges

The portlet view component includes elFinder, a
web-based file manager developed in Javascript using
jQuery UI for a dynamic and user friendly interface

18
http://elrte.org/elfinder
Data Engine in action (1/2)
19
Data Engine in action (2/2)
«Share» to be
added soon
20

Middleware-dependent

Middleware-independent

Does not support all m/w in Italy

Supports all m/w in Italy

Data management missing

Data management (almost) included

Non-customisable (by definition)

Highly & quickly customisable

Personal certificates needed (until
online CA becomes operational)

No personal certificates needed
(online CA can be used in future)

Support for workflows included

Support for workflows in progress

Job submission quite complex
(for non expert users)

Job submission straightforward
(details hidden to end users)

VO selection procedure exposed to
end users

VO selection transparent to end
users and managed via LDAP roles

Non user-driven development

User-driven development

User base unclear

Rapidly adopted and widely used
21
Catania Science Gateway
General-purpose portal
General-purpose portal vs.
Catania Science Gateway
Proposta di collaborazione (1/2)
(durata: 1 anno, rinnovabile)

Parallelamente al portale general purpose, IGI adotta il modello
di Science Gateway di Catania e crea una Task Force allocando a
Catania un congruo numero di FTE sul bilancio 2012

In cambio, il gruppo di Catania:
22
Proposta di collaborazione (2/2)
(durata: 1 anno, rinnovabile)

Condividerà con il portale general purpose tutte quelle componenti ritenute di
reciproco interesse/importanza

Opererà per tutti gli utenti di IGI i servizi della federazione «catch-all» GrIDP nelle
more che questi si registrino a IDEM (n.b.: GrIDP include già l’IdP dell’INFN)

Creerà uno Science Gateway specifico per IGI registrandolo sia in IDEM che in
eduGAIN e gestirà sia l’autenticazione che l’autorizzazione a tale portale per conto
di IGI

S’incaricherà della produzione del materiale per promuovere l’adozione dello Science
Gateway e permettere alle varie comunità di proporre nuove applicazioni

S’incaricherà, in parallelo, dello sviluppo delle portlet per le nuove applicazioni da
integrare nello Science Gateway e della formazione di nuovi sviluppatori di portlet,
mettendo a disposizione di IGI tutto il materiale di training già creato

Farà da tramite con gli sviluppatori di JSAGA e si farà carico dello sviluppo degli
eventuali nuovi adaptor che si rendessero necessari

Includerà negli Science Gateway già esistenti l’infrastruttura IGI in modo da
incrementarne l’utilizzo

Migrerà ad IGI le comunità nazionali già supportate (agricoltura, beni culturali, salute,
23
scienze della terra, ecc.) aumentandone rapidamente il numero di utenti