Slides

Development of portlets for special jobs: parametric,
collections, workflows
Mario Torrisi ([email protected])
Istituto Nazionale di Fisica Nucleare – Sezione di Catania
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
Outline
●
Job Engine
–
●
Introduction on Special Jobs
–
–
–
●
–
–
2
Grid Engine API to execute Special Job on DCIs
Portlet template
–
●
Job Collection
Parametric Job
Workflow N-1
Manage Special Jobs
–
●
A short recap on the Job Engine seen in the last section
Get code from SVN
Compile and test
Customize
Summary and conclusion
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
Catania Grid & Cloud Engine
3
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
Job Engine
●
●
The Job Engine is made of a set of libraries to
develop applications able to submit and
manage jobs on a grid infrastructure;
It is compliant with the OGF SAGA standard;
–
●
●
4
JSAGA is the SAGA implementation adopted
It is optimized to be used in a Web Portal
running an application server (e.g. Glassfish,
Tomcat,…) based on J2EE;
It can be used also in stand-alone mode;
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
Job Engine
Easiness
●
Allows to develop applications able to submit
jobs on a DCI in a very short time;
●
Exposes a set of very intuitive APIs;
●
Provides support for MPI applications;
●
The developer has only to submit the job:
– The Job Engine periodically check the job
status;
– When the Job is done, the job output is
automatically downloaded and (if set) an
email is sent to notify the the user.
5
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
Job Engine
Features
●
●
●
●
●
●
●
Middleware independent
Easiness
Scalability
Performance
Accounting
Fault tolerance
Workflow
●
6
Provides easy mechanisms to support parallel applications
●
JobCollection
●
Parametric Job
●
Workflow N-1
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
Introduction on Special Jobs
Job Collection
●
●
Job Collection: is a set of independent
jobs that run in parallel
When all jobs are successfully completed
the whole collection becomes DONE
Job Collection
Job 1
7
Job 2
...
Job N
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
Introduction on Special Jobs
Parametric Job
●
●
Parametric Job: is a set of jobs that run in
parallel having the same executable, but with
different arguments (e.g., input parameters)
When all jobs are successfully completed the
whole parametric job becomes DONE
Parametric Job
Job 1
8
Job 2
...
Job N
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
Introduction on Special Jobs
Workflow N-1
●
●
Workflow N-1 is a Special Job consisting of two different levels of
jobs:
–
First level jobs is a set of independent jobs that run in parallel
–
Final (Collector) job collects all first level jobs' outputs and
generates a file output
When all first level jobs are successfully completed the final job
is submitted and when it becomes DONE the whole workflow has
considered completed
Workflow N-1
Job 1
Job 2
...
Job N
Final Job
9
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
Manage Special Job
●
10
Parametrics job and Workflows N-1 are
designed as generalization of Job
Collection
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
JobCollection.java
●
●
This class handles the
behaviour of the Job
Collection.
It is responsible for:
–
–
–
11
creation;
status updating;
closing of the collection
when all jobs are DONE.
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
JobParametric.java
●
●
●
12
This class handles the
behavior of the
Parametric Job.
Inherits all attributes
and methods defined in
its parent class.
Adds a new attribute
that specify the single
executable.
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
WorkflowN1.java
●
●
●
●
13
This class handles the
behavior of the Workflow
N-1.
Inherits all attributes
and methods defined in
its parent class
Adds 3 new attributes:
Overrides some
JobCollection methods
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
Example
1. Define Infrastructures
InfrastructureInfo infrastructures[] = new InfrastructureInfo[2];
String wmsList[]={"wms://wmsdecide.dir.garr.it:7443/glite_wms_wmproxy_server"};
String wmsList2[]={"wms://wms.magrid.ma:7443/glite_wms_wmproxy_server"};
infrastructures[0] = new InfrastructureInfo("gridit",
"ldap://gridit-bdii-01.cnaf.infn.it:2170", wmsList,
"etokenserver.ct.infn.it", "8082", "332576f78a4fe70a52048043e90cd11f",
"gridit", "gridit");
infrastructures[1] = new InfrastructureInfo("EUMED","ldap://bdii.eumedgrid.eu:2170", wmsList2,
"etokenserver.ct.infn.it", "8082", "bc681e2bd4c3ace2a4c54907ea0c379b",
"eumed", "eumed");
14
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
Example
2. Define Descriptions
ArrayList<GEJobDescription> descriptions = new ArrayList<GEJobDescription>();
for (int i = 0; i < 2; i++) {
GEJobDescription description = new GEJobDescription();
description.setExecutable("/bin/sh");
switch (i) {
case 0:
description.setArguments("hostname.sh");
description.setInputFiles("./hostname.sh");
break;
case 1:
description.setArguments("ls.sh");
description.setInputFiles("./ls.sh");
break;
}
description.setOutputPath("/tmp");
description.setOutput("myOutput-" + i + ".txt");
description.setError("myError-" + i + ".txt");
descriptions.add(description);
}
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
15
Example
Job Collection submission
3. Create JobCollection object
JobCollection collection = new JobCollection("mtorrisi",
"Collection - TEST - ", "/tmp", descriptions);
4. Submit Job Collection
JobCollectionSubmission tmpJobCollectionSubmission = new
JobCollectionSubmission("jdbc:mysql://localhost/userstracking",
"tracking_user", "usertracking", collection);
tmpJobCollectionSubmission.submitJobCollection(infrastructures,
"193.206.208.183:8162", 10);
16
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
Example
Parametric Job Submission
3. Create JobParametric object
JobCollection p = new JobParametric("test",
"Parametric Job - mtorrisi", "[email protected]",
"/tmp", descriptions, "hostname.sh");
4. Submit Parametric Job
JobCollectionSubmission tmpJobCollectionSubmission = new
JobCollectionSubmission(
"jdbc:mysql://localhost/userstracking", "tracking_user",
"usertracking", p);
tmpJobCollectionSubmission.submitJobCollection(infrastructures,
"193.206.208.183:8162", 10);
*The first two steps are common to the "Job Collection" submission
17
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
Example
Workflow N-1 Submission (1/2)
3. Create final job
description
GEJobDescription finalJobDescription = new GEJobDescription();
finalJobDescription.setExecutable("ls.sh");
finalJobDescription.setArguments("./ls.sh");
String tmp = "";
for (int i = 0; i < descriptions.size(); i++) {
if (tmp.equals(""))
tmp = descriptions.get(i).getOutput();
else
tmp += "," + descriptions.get(i).getOutput();
}
finalJobDescription.setInputFiles(tmp + ",./ls.sh");
finalJobDescription.setOutputPath("/tmp");
finalJobDescription.setOutput("myOutput-FinalJob.txt");
finalJobDescription.setError("myError-FinalJob.txt");
*The first two steps are common to the "Job Collection" submission
18
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
Example
Workflow N-1 Submission (2/2)
4. Create WorkFlowN1 object
JobCollection wf = new WorkflowN1("mtorrisi", "Workflow N-1", "/tmp",
descriptions, finalJobDescription);
5. Submit Workflow N-1
JobCollectionSubmission tmpJobCollectionSubmission = new
JobCollectionSubmission(
"jdbc:mysql://localhost/userstracking", "tracking_user",
"usertracking", wf);
tmpJobCollectionSubmission.submitJobCollection(infrastructures,
"193.206.208.183:8162", 10);
19
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
Special Job Portlet
20
Outline
●
●
●
●
Introductory consideration on miparallel-app-portlet
Preliminary operations
Get from SVN and deploy
Test
– Special job execution
– Log extraction
– MyJobs
●
21
Customize
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
Preliminary operations
●
Before testing mi-parallel-app-portlet
–
–
–
–
Install MyJobs portlet
Check the eTokenServer service is reachable
Check the system date is aligned with
current time
Check that Grid CA certificates are updated
• Update CA certificates
–
Follow instructions to fix GridEngine 1.5.1
●
22
Refer to presentation seen in the last webinar to
perform the above mentioned preliminary
operations
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
Preliminary operations
GridOperations table
●
23
Insert a new line into the GridOperations table:
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
mi-parallel-app-portlet
Get from SVN and deploy
●
Get code form SVN
svn checkout svn://svn.code.sf.net/p/ctsciencegtwys/liferay/trunk/gilda/mi-parallel-app-portlet miparallel-app-portlet
●
Run deploy command
ant deploy
●
Watch server.log until see something like:
[AutoDeploy] Successfully autodeployed : /home/mario/liferay-portal-6.1.1-ce-ga2/glassfish3.1.2/domains/domain1/autodeploy/mi-parallel-app-portlet.
Install portlet in the portal
through Liferay menu:
●
24
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
mi-parallel-app-portlet
Test
1. Select collection type
3. Click “OK” button
2. Insert task number
4. Fill input fields
5. Insert a brief
collection description
25
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
6. Submit collection
Special job execution
Log extraction(1/2)
●
Check GridEnigne.log file under /logs folder
– Jobs Submission
2013-07-26 22:35:57,270 [submitJob] INFO JSagaJobSubmission Job Submitted: [wms://wmsdecide.dir.garr.it:7443/glite_wms_wmproxy_server][https://wmsdecide.dir.garr.it:9000/n_tn5247FcS3Cas4v3gfxA]
[…]
2013-07-26 22:36:01,046 [submitJob] INFO JSagaJobSubmission Job Submitted: [wms://wms024.cnaf.infn.it:7443/glite_wms_wmproxy_server][https://wms024.cnaf.infn.it:9000/fnu6HvRxj71vmfyufqaslQ]
[…]
2013-07-26 22:36:04,531 [submitJob] INFO JSagaJobSubmission Job Submitted: [wms://wms024.cnaf.infn.it:7443/glite_wms_wmproxy_server][https://wms024.cnaf.infn.it:9000/7lXUSv5n1em0CcrZlvQS1A]
– Check job status thread: updates jobs status
2013-07-26 22:37:04,205 [checkJobStatus] INFO UsersTrackingDBInterface - Status for job
[wms://wmsdecide.dir.garr.it:7443/glite_wms_wmproxy_server][https://wmsdecide.dir.garr.it:9000/n_tn5247FcS3Cas4v3gfxA] is RUNNING
[...]
2013-07-26 22:37:04,934 [checkJobStatus] INFO UsersTrackingDBInterface - Status for job
[wms://wms024.cnaf.infn.it:7443/glite_wms_wmproxy_server][https://wms024.cnaf.infn.it:9000/7lXUSv5n1em0CcrZlvQS1A] is RUNNING
[...]
2013-07-26 22:37:05,060 [checkJobStatus] INFO UsersTrackingDBInterface - Status for job
[wms://wms024.cnaf.infn.it:7443/glite_wms_wmproxy_server][https://wms024.cnaf.infn.it:9000/fnu6HvRxj71vmfyufqa
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
slQ]26
is RUNNING
Special job execution
Log extraction(2/2)
– Check job status thread: updates collection status
2013-07-26 22:37:05,697 [pool-31-thread-1] DEBUG UsersTrackingDBInterface - Running collections: 1
2013-07-26 22:37:05,697 [pool-31-thread-1] DEBUG UsersTrackingDBInterface - Running collection Status:
RUNNING
2013-07-26 22:37:05,719 [pool-31-thread-1] DEBUG UsersTrackingDBInterface - NEW Running collection Status:
RUNNING
– Complete collection
2013-07-26 22:40:04,258 [pool-31-thread-1] DEBUG UsersTrackingDBInterface - Running collection Status:
RUNNING
2013-07-26 22:40:04,300 [pool-31-thread-1] DEBUG UsersTrackingDBInterface - NEW Running collection Status:
DONE
2013-07-26 22:40:04,300 [pool-31-thread-1] DEBUG UsersTrackingDBInterface - Closing Done Collection:
JobCollection [id=67, description=Demo Collection: 27/7/2013 - 0:35:33, commonName=test, taskCounter=3]
2013-07-26 22:40:04,301 [pool-31-thread-1] INFO JobCollection - creating a tgz archive containing output files...
[...]
2013-07-26 22:40:04,301 [pool-31-thread-1] INFO JobCollection - tar czvf
/tmp/jobOutput/DemoCollection277201303533_67.tgz --directory=/tmp/jobOutput/ DemoCollection277201303533_67
27
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
MyJobs
Collection status
Jobs current status
Collection DONE
28
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
Customize
●
●
Customize.sh is a BASH script to clone mi-parallel-app-portlet
Make a copy of mi-parallel-app-portlet in
<your_portlet_name>_portlet
– cp -R mi-parallel-app-portlet <your_portlet_name>_portlet
●
Move into <your_portlet_name>_portlet folder and edit
customize.sh
– AUTH_EMAIL=<your@email>
– AUTH_NAME='<your name>'
– AUTH_INSTITUTE='<your_institute>'
– APP_OPERATIONID=<your_app-id>
– APP-NAME=<your_portlet_name>
●
Run customize script
– ./cusotmize.sh
29
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
Summary and conclusion
●
●
●
●
●
30
Catania Grid & Cloud Engine provides powerful APIs to
port applications on e-Infrastructure easily
Developers do not need to care of the differences
between the middleware details
Developers have to only prepare the job(s)
description(s) and then to submit it(them)
The JobEngine:
–
Periodically checks the status
–
Automatically downloads the outputs and notify the
user
Job Engine provides also powerful mechanisms to
guarantee jobs submission (or re-submission), if they
fail due infrastructure issues
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
References
●
Catania Science Gateway Framework
http://www.catania-science-gateways.it/
●
Grid & Cloud Engine Javadoc
http://grid.ct.infn.it/webinar-liferay/
●
Simple API for Grid Applications (SAGA)
http://www.ogf.org/documents/GFD.90.pdf
●
JSaga: Java implementation of SAGA
specification
http://grid.in2p3.fr/jsaga/
●
Training material
https://gilda.ct.infn.it/wikimain
31
Corso per sviluppatori di applicazioni per lo Science Gateway del progetto VESPA
Thank you!
32