Slides

Developing Applications for
the Science Gateway
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Outline
 The Catania Science Gateway Framework
 Use Case
 Portlet Develop
 Hostname porltlet
 Catania Grid Engine API
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Some barrier
limit grid
(*)
adoption
The eResearch2020 report
http://www.eresearch2020.eu/
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Reference Model
Scientific
Application
E-Collaboration
Science
Gateway
Social
Application
Standard Services
GRID
Users of different
Institutions
members of GARR
and/or
international
CLOUD
partners involved
Local Cluster
in European
Projects of the
Riccardo Rotondo
Consortium
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Reference Model
Scientific
Application
E-Collaboration
Science
Gateway
Social
Application
Standard-based (SAGA)
middleware-independent
Grid Engine
Users of different
Institutions
members of GARR
and/or
international
partners involved
in European
Projects of the
Riccardo Rotondo
Consortium
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Federated User
Science
Gateway
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Social User
Science
Gateway
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Integrated Services
JSR 168/268
JSR 168/268
JSR 168/268
JSR 168/268
Catania Science Gateway Framework
CLOUD
GRID
Riccardo Rotondo
Africa Arabia
ROCCluster
(South) All-Hands Meeting, Pretoria, 27.03.2013
Local
Catania Science Gateway Framework
Catania Science Gateway Framework
Grid Engine
Data On Grid
Services
Cloud Services
JSAGA
Adaptors
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Usage
Workflow
eTokenServer
1. Sign in
GRID
2. Grid
Request
5.
Tracking
6. Getting
Results
5. Grid
Interactions
User
Tracking DB
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
The GARR Science Gateway
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Access
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Applications accessing grid services
 12 applications developed among 5 different
countries and 3 continents (Europe, Latin
America and Asia);
 4 scientific domains:




Life Science;
Mathematic & Computer Science;
High Energy Physics;
Cultural Heritage.
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Job Submission
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Job Submission
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
My Workspace – Active Job List
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
My Workspace - Done Job List
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
My Workspace – MyJobsMap
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
My Data
Sharing features
among users will
soon be added
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
My Cloud
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Facebook Integration
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Facebook Integration (2)
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Several Science Gateways …
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Integrated Services
JSR 168/268
JSR 168/268
JSR 168/268
JSR 168/268
Catania Science Gateway Framework
CLOUD
GRID
Riccardo Rotondo
Africa Arabia
ROCCluster
(South) All-Hands Meeting, Pretoria, 27.03.2013
Local
Web Service
 Application Server (Glassfish)
 Web application
working as Portlet Container
JSR 168/268
JSR 168/268
(Liferay)
JSR
168/268
JSR 168/268
 Portlet (a custom application designed under
users needs)
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
What is Porltet ?
 Defined in the Java Specification Request (JSR
168/286)
 A porlet is a web component able to set a
frame portion of a web page.
 Differently from Servlet it needs a Porltlet
Container to allow the user to interact with it.
 Standard Porltet can be deployed in any Porlet
Container in case they will use standard libray.
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Porlet Life Cycle
YES
NO
End of
Services?
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Porltlet Interface Method
Class GenericPortlets {
init (PortletConfig);
processAction (ActionRequest,
ActionResponse);
render (RenderRequest, RenderResponse);
destroy();
doView (Request, Response);
doEdit (Reuqest, Response);
doHelp (Request, Response);
}
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Portlet Mode
 A porltet works in three
different states.
 View Mode: generate a
markup ( a portion of a
web page);
 Edit Mode: allow portlet
customisation and
setting preferences.
 Help Mode: explains
portlet functionalities.
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Action Request
Action
Request
view.jsp
processAction()
Action
Request
edit.jsp
Render
Request
doView()
edit.jsp
doEdit()
view.jsp
Render
Request
processAction()
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Render Request
Render
Request
view.jsp
doView()
view2.jsp
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Gilda Liferay Virtual Machine
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Portlets
 Standard JSR168/286
 Interaction between jsp pages and java code
Application Server (Glassfish)
JSP page
<jsp:useBean
<form … >
Java code
Class <you portlet>
Extends
GenericPortlet …
<input … />
</form>
 JSP uses <form… statements to send parameters to Java code
 Java code extends class GenericPortlet overriding methods:


Init, doView, doHelp, doEdit, processAction
Java code uses 2 Classes to exchange data from/to JSPs
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Portlet files (root)
Used by ‘ant’ to compile and
deploy the portlet
build.xml
docroot/
WEB_INF/
css/
images/
js/
jsp_page_1.jsp
…
jsp_page_n.jsp
Define here JSP’s styles
JSP image files
Portlet JSP pages
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Portlet files
WEB_INF/
Used to define database and other
resources settings
glassfish-web.xml
web.xml
Jar Library directory (local scope)
lib/
Job files (Custom dir)
Job/
liferay-display.xml
liferay-plugin-package.properties
Portlet
configuration files
liferay-portlet.xml
portlet.xml
src/
Java source files
<package_name>/code.java
tld/
XML structure file
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Parameters exchange (JSP->Java)
<form action=action="<portlet:actionURL
portletMode="view">
<portlet:param
name="param_name_1" value="param value 1"/>
…
<portlet:param
name="param_name_n" value="param value n"/>
…
<input … />
<input … />
<input type="submit" … />
</form>
 Portlet modes:
 VIEW – Userd for normal portlet operations -> calls processAction()
 EDIT – Userd to configure portlet parameters -> calls doEdit()
 HELP – Userd to show help information -> calls doHelp()
•
•
processAction(ActionRequest request,…
doView/doHelp/doEdit(RenderRequest
request,…
// To obtain the parameter just set …
String param_i = request.getParameter("param_name_i");
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Parameters exchange (Java->JSP)
doView/doHelp/doEdit(RenderRequest request,…
// To send the parameter just call …
request.setAttribute("param_name_j", param_name_j);
<%
// To load variables from Portlet Class …
%>
<jsp:useBean id="param_name_k" class="<variable type k>" scope="request"/>
<%
// To reference a param value
%>
<%=param_name_k%>
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
hostname-portlet example
input.jsp
 Portlet to send a test job into a
distributed infrastructure
 Input:
 Input file or a text
 A human readable job identifier
 Ouput
 Hostname of running machine
 File containing information about the
running machine
 Portlet statuses
submit.jsp
 INPUT
(Shows the input fields)
 SUBMIT
(Shows the job submission feedback)
 Jsp/Java variable ‘PortletStatus’
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Hostname Action/View enums
 Actions
Default Action
 ACTION_INPUT
 ACTION_SUBMIT
• Simply forward portlet status to
the doView() in:
RenderParameter
 Views (Possible portlet views)
 VIEW_INPUT
 VIEW_SUBMIT
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Hostname Action/View
 Actions
 ACTION_INPUT
 ACTION_SUBMIT
 Views
 VIEW_INPUT
 VIEW_SUBMIT
View
• Select the INPUT form view
input.jsp (default view mode)
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Hostname Action/View
 Actions
 ACTION_INPUT
 ACTION_SUBMIT
This action will submit the
hostname job into the distributed
infrastructure
 Views
 VIEW_INPUT
 VIEW_SUBMIT
<form enctype="multipart/form-data" action="<portlet:actionURL
portletMode="view">
<portlet:param name="PortletStatus" value="ACTION_SUBMIT"/>
</portlet:actionURL>"
method="post”>
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Hostname Action/View
 Actions
 ACTION_INPUT
 ACTION_SUBMIT
 Views
Select the proper view showing
the
Information page that shows the
job has been submitted
 VIEW_INPUT
 VIEW_SUBMIT
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Portlet Initialization
 portlet.xlm contains portlet initialization parameters
<init-param>
<name>init parameter name</name>
<value>init parameter value</value>
</init-param>
public void init() throws PortletException {
// Load default values from portlet.xml
init_PortletVersion = getInitParameter(“init parameter name");
…
 The init() method uses <init-param>s to load Portlet’
default parameter values
 Some of the default parameters values will be used to load
portlet default preferences
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
hostname’ <init-param>
 To make parameter management easier all values are threated as
String type











init_PortletVersion - Portlet version number
init_bdiiHost - Information system service (ldap://...:2170)
init_wmsHost - Infrastructure resource manager (wms://...:7443/glite_wms_wmproxy_server)
init_pxServerHost - Hostname of Robot proxy server (myproxy.ct.infn.it not used yet)
init_pxRobotId - Id of the robot certificate
init_pxRobotVO - Infrastructure virtual organization
init_pxRobotRole - Role of generated proxy
init_pxUserProxy - Standard proxy file (instead of Robots)
init_pxRobotRenewalFlag - Allows automatic job renewal mechanism
init_SciGwyAppId - Science Gateway application identifier '9' for GILDA VM tester application
GridEngine' User tracking database (UTDB) access credentials






init_SciGwyUserTrackingDB_Hostname
init_SciGwyUserTrackingDB_Username
init_SciGwyUserTrackingDB_Password
init_SciGwyUserTrackingDB_Database – Users should not use Direct DB access to
UTDB
init_JobRequirements - Application job requirements
init_pilotScript - Application pilot script
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Hostname Portlet preferences

private void getPreferences( ActionRequest actionRequest
,RenderRequest renderRequest) {
PortletPreferences prefs=null;
if(null!=actionRequest)
prefs = actionRequest.getPreferences();
getPreferences
else if(null != renderRequest)
 doView
processAction
prefsand
= renderRequest.getPreferences();
method called by both:
if (null != prefs) {
// The second parameter assigns a default value
pref_value=prefs.getValue("<pref_name>", "<init_name>");
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Job Submission (1/4)
// Job details
String executable="/bin/sh";
executable
String arguments="pilot_script.sh";
arguments
String outputPath="/tmp/";
String outputFile="application.out";
String errorFile="application.err";
// Application
// executable'
// Output Path
// Application stdout
// Application stderr
// InputSandbox (string with comma separated list of file
names)
String inputSandbox= appServerPath
+"/WEB-INF/job/pilot_script.sh”
+","+<other files>;
// OutputSandbox (string with comma separated list of file
names)
String outputSandbox="generated_output_file";
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Job Submission (2/4)
// Software tags
String jdlRequirements[] =
pref_JobRequirements.split(";");
int numRequirements=0;
for(int i=0; i<jdlRequirements.length; i++)
if(!jdlRequirements[i].equals("")) {
jdlRequirements[numRequirements] =
"JDLRequirements=("+jdlRequirements[i]+")";
numRequirements++;
}
// Instanciate the JSAGA object
JSagaJobSubmission tmpJSaga = new
JSagaJobSubmission();
tmpJSaga.setBDII(bdiiHost); // Infrastructure
top BDII
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Job Submission (3/4)
// Proxy renewal flag
if((pref_pxRobotRenewalFlag.toLowerCase()).equals("true"))
pxRobotRenewalFlag=true;
else pxRobotRenewalFlag=false;
// Associate a valid proxy (Robot/UserProxy)
// Specifying a path for local proxy it will be used
instead
if(pxUserProxy==null || pxUserProxy.equals("")) {
tmpJSaga.useRobotProxy( pxRobotId
,pxRobotVO
,pxRobotRole
,pxRobotRenewalFlag
);
else tmpJSaga.setUserProxy(pxUserProxy);
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Job Submission (4/4)
tmpJSaga.setExecutable(executable);
tmpJSaga.setArguments(arguments);
tmpJSaga.setOutputPath(outputPath);
tmpJSaga.setInputFiles(inputSandbox);
tmpJSaga.setOutputFiles(outputSandbox);
tmpJSaga.setJobOutput(outputFile);
tmpJSaga.setJobError(errorFile);
// Specify the executeable
// Specify the application' arguments
// Specify the output directory
// Setup input files (InputSandbox)
// Setup output files (OutputSandbox)
// Specify the std-outputr file
// Specify the std-error file
if(numRequirements>0)
tmpJSaga.setJDLRequirements(jdlRequirements); // Assign requirements
// Submit the job
// If a WMS is specified the call to Job submission changes
if(wmsHost!=null && !wmsHost.equals(""))
tmpJSaga.submitJobAsync(username
,hostUTDB
,applicationId
,wmsHost
,jobIdentifier
);
else
tmpJSaga.submitJobAsync( username
,hostUTDB
,applicationId
,jobIdentifier
);
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
From hostname to your portlet
https://gilda.ct.infn.it/wikimain/-/wiki/Main/hostname-portlet
 Extract hostname portlet form svn
svn checkout svn://svn.code.sf.net/p/ctsciencegtwys/liferay/trunk/gilda/mi-hostname-portlet
 ‘cd’ inside the hostname-portlet directory and try to compile it with
the command: ant deploy
 If the compilation is successful create a physical copy of the portlet
directory source code
 cp –r hostname-portlet <yourname>-portlet
 Customize your portlet settings configuring several XML files:
 docroot/WEB-INF/portlet.xml
<portlet-name><yourname>-portlet</portlet-name>
<display-name><yourname>-portlet</display-name>
<portlet-class><your.package.path.<yourname>_portlet</portlet-class>
…
<init-param>
<name>init_PortletVersion</name>
<value>Specify your Portlet' version number</value>
</init-param>
… other init parameters
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Init parameters
Your portlet version
init_PortletVersion
init_bdiiHost
Infrastructure settings …
init_wmsHost
init_pxServerHost
Robot proxy settings
init_pxRobotId
Robot Id, Virtual Organization,
Proxy Role, Renewal Flag
init_pxRobotVO
pxUserProxy (mutually exclusive to Robots)
init_pxRobotRole
Allows the use of a user proxy
init_pxUserProxy
Just provide a valid path to a
proxy file
init_pxRobotRenewalFlag
Each application has a unique Id
init_SciGwyAppId
(GridEngine)
init_SciGwyUserTrackingDB_Hostname
init_SciGwyUserTrackingDB_Username GridEngine connection credentials (Unused)
init_SciGwyUserTrackingDB_Password
init_SciGwyUserTrackingDB_Database
Grid Job requirements
init_JobRequirements
init_pilotScript
Job’ pilot script (bash)
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Parameter in portlet preferences
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
From hostname to your portlet
 docroot/WEB-INF/portlet.xml (still)
<portlet-info>
<title>Title of yor porltlet
<short-title>Short title
<keywords>Portlet' keywords
 docroot/WEB-INF/liferay-display.xml
<category name="<place your category name here>" (your institution ‘GILDA’)
<portlet id="<place your portlet identifier>" (name displayed into the mangment
interface)
 docroot/WEB-INF/lib
Overrides or extends the portlet java code global library repository
(/opt/glassfish3/glassfish/domains/liferay/lib/)
 docroot/WEB-INF/liferay-portlet.xml
<portlet-name>
<css-class-wrapper>
 docroot/WEB-INF/glassfish-web.xml
<context-root> (Information managed by App server Glassfish)
 docroot/images/
Each image used by the jsp pages should be located here
 docroot/icon.png
Place here a 16x16 image file with your portlet icon
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
From hostname to your portlet
 /docroot/WEB-INF/src/it/infn/ct/hostname_portlet.java
 Make a copy of this source file into directory:

<your.package.name>/<yourname_portlet>.java>
WARNING: Do not forget to assign to the class the same name of the java
file
 Start to develop the interface modifying jsp files and change java code
Enums with correct Actions and Views modes using human readable
identifiers.
For simple Appliocation user interfaces there will be no need to add
other JSPs or action/view modes to the existing Enum
 Edit the java code in order to manage your application specific interface
and prepare the correct sandboxes to submit properly the job
 Prepare the pilot script and use it as parameter.
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Conclusion
 A unique web portal is able to access to Grid
and Cloud services in few clicks with singlesign-on in an easy, intuitive interface
 Users are able to share their experiences even
with Social Networks support
 The activities of dissemination contributed to
the spread of these technologies increasing the
number or applications available
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
References




IDPOPEN https://idpopen.garr.it
List of SP: https://idpopen.garr.it/providers
Gilda Portal: http://gilda.ct.infn.it/
Our sourceforge project:
http://sourceforge.net/projects/ctsciencegtwys/
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013
Riccardo Rotondo
Africa Arabia ROC (South) All-Hands Meeting, Pretoria, 27.03.2013