EGI Task Force Membership

Il MW di Grid da EGEE-II a EGI
Evoluzione del Processo del Middleware dallo Sviluppo
alla Integrazione, Certificazione e Deployment
A.Ghiselli
Riunione con i referee di infn-grid
21 aprile 2008
Mware - INFN
EGEE
– Security
• VOMS/VOMS Admin/VOMS-SAML (OMII-Europe)
• AuthZ service (like GPbox)
–
–
–
–
CREAM/CREAM-BES (OMII-Europe)
BLAH
WMS/EIS
DGAS/DGAS-RUS-UR(OMII-Europe)
INFN-Grid
–
–
–
–
StoRM
GridICE
OGF GLUE-schema (OMII-Europe)
GLUEMAN (OMII-Europe)
ETICS
– ETICS-tools per building, testing
2
MW in EGEE-II
processo di pre-certificazione, integrazione e packaging
• Nuovo TAG
– JRA1 consegna il tag CVS per il componente middleware da
certificare ad SA3:
– in particolare crea una patch in Savannah contenente la lista
degli .rpm e relative versioni per il componente da certificare,
insieme alle istruzioni per installare e configurare correttamente
la componente.
• Integrazione e packaging
– SA3 prepara un repository di certificazione con la lista completa
degli .rpm che servono per installare la componente e con le
configurazioni YAIM
• Pre-certificazione
– Una volta pronto il repository e yaim, un partner esterno inizia a
certificare la componente
3
MW in EGEE-II
processo di pre-certificazione, integrazione e packaging
PROBLEMI (esempio del WMS)
• per quanto riguarda il passo 2), sia la preparazione del
repository di certificazione che la creazione degli script
yaim per configurare correttamente il wms, hanno subito
ritardi, spesso dovuti a tempi di attesa di comunicazione
tra SA3-CERN e sviluppatori
- per quanto riguarda il passo 3), forse la scelta del
partner esterno (Imperial College) non e' stata delle
migliori, si sono riscontrati tempi molto lunghi anche solo
per installare e configurare un WMS su cui eseguire i
tests di certificazione. Poi talvolta, dopo i primi tests si
scopriva che la macchina utilizzata per la certificazione
non aveva i requisiti hardware adatti.
 Manca un responsabile del processo della componente
MW
4
Dal DoW di EGEE-III
attivita’ di SA3
• TSA3.1. Integration and packaging
• The integration of the gLite distribution will be performed by a core
team located in one place. This team will be led by a release
manager who drives the component release process and ensures
that the associated documentation is of acceptable quality and
uniformity
• TSA3.2. Testing and certification
• Testing and certification are the most important tasks ensuring the
released gLite distribution provides the required functionality,
performance, scalability, and dependability. We distinguish between
testing and certification whereby testing leads to production
readiness of a component and will be carried out by SA3 teams that
are closely linked with JRA1 teams and certification ensures that
after modifications the components still work inside the stack and
fulfill the functional and performance requirements, thus it includes
regression tests, and will be carried out by the SA3 certification
team.
5
TSA3.1: Integration and
packaging
• The effort required for this task is 186 PM, provided by:
– CERN 138 PMs: Coordination of the activity, development and
tracking of the process for integration and release management,
maintenance, evolution and operation of integration tools,
configuration tools, repositories, process tracking tools.
Integration and packaging of the overall distribution. Interaction
with SA1. Collection and maintenance of documentation
– INFN 24 PM: Integration and packaging work related to gLite
WMS, CE components, VOMS/VOMS –Admin, DGAS,
authorisation and prioritisation frameworks.
– TCD 4 PM: Integration and packaging of security infrastructure
middleware, tools for interoperation
– STFC 6 PM: Integration and packaging of service discovery and
information system APIs
– CESNET 6 PM: Integration and packaging for logging and book
keeping components and Job Provenance services
6
TSA3.2: Testing and certification
TSA3.2: Testing and certification
• This task will test and certify the gLite middleware stack, develop the
necessary test suites, and operate the distributed test and
certification testbeds. Apart from standard functional and
performance tests, interoperation, security, and vulnerability testing
will be included. Pilot services will be set up for large scale tests on
the production infrastructure if necessary.
• The effort required for this task is 319 PM, provided by:
– CERN 146 PM: Coordination of the activity, development and tracking of
the process for testing and certification, maintenance, evolution and
operation of testing tools, virtualized testbeds, operation of a large scale
testbed (120+ nodes), coordination and tracking of partners test
activities, coordination of testes with SA1 for PPS and pilot services,
coordination and participation in patch certification, driving the
regression tests.
– INFN 48 PM: Testing up to production readiness of the components
integrated by INFN. This includes participation in patch verification and
test case development.
7
Differenza fra egee-ii ed egee-iii
• Seguendo il nuovo concetto di "Cluster of Competence" (come
descritto nel Description of Work), ogni componente middleware
sara' seguita nel ciclo completo dallo sviluppo all'integrazione fino al
deployment in certificazione da un team localizzato vicino agli
sviluppatori JRA1 della componente. Questo significa che partner
SA3 vicino e in stretta collaborazione con gli sviluppatori
JRA1 eseguiranno il lavoro di integrazione, packaging e testing fino
a che la componente abbia raggiunto un livello accettabile per poter
essere usata in produzione.
• La differenza fondamentale con EGEEII e che dovrebbe ottimizzare
il processo e' che tutto il lavoro di integrazione, packaging e testing
di pre-certificazione verra' eseguito da un team composto
da persone SA3 + JRA1 nello stesso posto in cui la
componente viene sviluppata. Questo dovrebbe ridurre/eliminare i
ritardi di comunicazione tra team distribuiti come in EGEEII e
velocizzare i tempi di rilascio per ogni componente.
8
necessita' di modifica per
ottimizzare il processo
• Perche’ Il man power e’ quasi tutto centralizzato al
CERN, quindi il concetto di ‘cluster of competence’ non
e’ sufficientemente supportato.
• RICHIESTA: Per avere una garanzia di ottimizzazione
del processo e’ necessario che la certificazione di SA3 in
EGEE III sia sotto la supervisione del responsabile di
Jra1 del componente certificato che rimane l'unico
responsabile della conclusione del processo con
successo. Il componente certificato che passa poi a SA3
per la certificazione finale deve aver passato tutti i
normali tests di accettanza a un livello di scala tale di
poter essere usato cosi' com'e' in EGEE.
9
Il Middleware in EGI
Scenarios for EGI
• “Market” based
– EGI model is focused on the static operation of well
consolidated services
– Assume that some external provider will:
• Produce the set of coordinated services needed for the
robust and resilient operation of the e-Infrastructure
• Introduce the required standards for interoperability
• Introduce the new functionalities required by new
applications or EGI operational needs
• Incorporated development
– The MWare development cycle is built in in the EGI
model with the shared goal of arriving to the required
consolidated interoperable services after a series of
development cycles
– Unified management of the development cycle has
proven for gLite and the other EU stacks to be
essential to achieve stable operations of evolving
services
11
Main Grid Middleware Stacks
Available Today
• From Europe
– gLite universe
– UNICORE
– ARC
– OMII-UK
• From US: Globus, Condor/VDT
• From Asia: NAREGI, CrownGrid
Europe plays a key role in middleware
development
12
The 3 EU Mware stacks
• ARC (Advanced Resource Connector)
– developed by the NorduGrid collaboration since 2001 and
supported by EU funds
– deployed in more than 60 sites, with over 20,000 CPUs.
– adopted by the Nordic DataGrid Facility (NDGF)
• gLite
– developed by EU supported projects (EDG, EGEEs) since 2001
– deployed in 250 sites comprising more than 50,000 CPUs and
very large (25 PB (Petabytes)) storage systems.
• UNICORE (Uniform Interface to Computing Resources)
– middleware for the European distributed HPC Grid infrastructure
DEISA as well as in the starting PRACE initiative for European
PetaFlop/s Supercomputers . EU support several projects…
• Services of the 3 stacks are often complementary,
though not yet fully interoperable
• Essential commonalities, e.g. the security framework,
very similar resource and task description, and some
common underlying protocols
13
The Role of EGI for Grid
Middleware
• Evolve current middleware stacks towards:
– Achieving standard-based interoperability
• systems/organizations able to provide/accept
services from other systems/organizations and to
use the services exchanged to enable them to
operate effectively together
– Adding new functionalities based on
requirements from:
• supported VOs
• potentially new VOs
• operations
14
Tasks for MW TF
• Specify the set of key service components (incl.
interfaces) required for the European Grid
infrastructure
• Describe transition period (as roadmap) from
today’s stack-based approach to a candidate
implementation of the EGI service components
specification
• Provide cost estimations (contributions, service
charges, project grants) for both tasks
15
Remarks from MB to TF
• Deadline: Monday, 5 May 2008
• Relation to Globus and other international
MW providers should be taken care of in
terms of a collaboration agreement
• Cloud technology should not be excluded
• Interaction with Industry through EGEE
Industry Forum (Action Bob: Identify
person)
• Identify possibilities for enlargement of
user base for general services
16
TF “Middleware” - Members
•
•
•
•
•
•
•
•
•
•
Alistair Dunlop (UK)
Achim Streit (DE)
Farid Ould-Saada (NO)
Mirco Mazzucato (IT)
Ludek Matyska (CZ)
Ian Bird (CERN)
Christoph Witzig (CH)
Sergio Andreozzi (IT)
Ignacio Martin Llorente (Spain)
Uwe Schwiegelshohn (Germany)
17
The gLite
Consortium
An opportunity for the EU
Companies and Research
Institutions
18
Why a gLite Consortium
• Need to preserve and evolve the EGEE
middleware in the scenario of the
European Grid Initiative (EGI) and National
Grid Initiatives
• EGI will support more stacks and foster
convergence: gLite, Unicore, ARC…
• gLite need to speak with one voice and
continue to evolve and support users
• Globus, Unicore and ARC have already
19
created their Consortia
The likely membership
• The core of the gLite Open Consortium will be
constituted by the same institutions that are
involved with the gLite developments, i.e.: ElsagDatamat (IT), CESNET (CZ), INFN (IT),
SWITCH (CH), FOM (NL), UH.HIP, CERN,
STFC, UNIMAN.
• The second level will be constituted by the
Institution who operate or use the gLite services.
• Additional industrial or business partners should
be included in order to create the conditions for
an expansion of the Consortium’s activities
towards the commercial markets.
20
Funding model
• Open Source software development and offering of related
maintenance and support services by middleware Consortia has a
less straightforward economic sustainability model than service
activities ( e.g. network bandwidth offering by the Dante/NRENs
organization).
• The Consortium should aim at extending and generalizing the offer
of services to non-skilled communities (new VOs, industries,
government, extra-network research institutions, etc) to reach
sustainability. This will take time and the continuation of the
development efforts to generalize and standardize the offered
services
• It is an absolute requirement for EGI that the current European cofunding will continue after the EGI transition and the funds
necessary to maintain and evolve the current stack will be made
available through “reserved” projects a’ la Géant subscribed by the
Consortium members. Additional R&D activities aiming at service
generalization and standardization, again co-funded by the
European Commission, will create the base for a general uptake and
for an increased sustainability
21
. Relationship with EGEE-III and EGI
(and how they could work together)
• The gLite Open Consortium in the initial period should coexist with
the EGEE-III JRA1 activities, preparing to act directly as the
dedicated entity for the gLite development and support activities at
the end of the EGEE III project. The gLite middleware JRA1 activity
remains during the initial “EGEE-III phase” a privileged middleware
provider, while the gLite Open Consortium will be the legal entity
which will progressively maintain coordinated all the gLite
development activities (2008-20010) taking them over at the end of
EGEE III.
• In the “EGI phase” (from 2010), the gLite Open Consortium will
coordinate all the former gLite JRA1 activities (and relative
personnel) and will be the middleware provider in the European
arena (where UNICORE and ARC are other providers ).
• EGI should strongly support a strategy that keeps the European
leadership in grid middleware technology by continuing to
guarantee leading edge services to the European infrastructure that
is supposed to manage.
22