A simple client server python tool to run independent tasks on the GRID.

Author: Riccardo di Meo

This is an about minimal example of system with client-server architecture that does something useful.

It is more an exercise, than a full fledged program, and it's point is to show how simple building a client/server system, with all it's advantages, in python.

- Download and unpack:

 http://www.escience-lab.org/software/dimeo/cost_school/simple_client_server.tar.gz

and enter simple_client_server directory.

Two Python programs are present in the directory, generic_client.py and generic_server.py: the first is supposed to run on the WNs and the second one on a resolved host with inbound connectivity (the UI will nicely do).

The server program opens a connection's end for listening on the UI and waits for a client to contact it. The client can run on multiple WNs at once and contact the server, then the server feed the client with some task to do.

- Both programs are heavily commented: read generic_server.py: step trough the lines and try to figure out what the code does (don't stick to the details, try to get the idea), then read generic_client.py and do the same thing.

- Download

 http://www.escience-lab.org/software/dimeo/cost_school/small_qe_job.tar.gz

and upload it to a suitable location on the grid.

The file tasks.txt contains the operations that will be assigned to the client(s) on the grid: read and understand them, replacing each remote gsiftp location appropriately. Don't forget to read the comments.

- Start the server: since only one server at a time may listen to a specific port, you will likely have to modify the line 17 of generic_server.py to specify a different port than the default one.

Pick a number between [ 20000 , 25000 [, replace the value at line 17 with it and start the server as:

 ./generic_server.py

if the result is something like this:

 $ ./generic_server.py
 Traceback (most recent call last):
   File "./generic_server.py", line 60, in ?
     server=SimpleXMLRPCServer.SimpleXMLRPCServer(("0.0.0.0",PORT))
   File "/usr/lib/python2.3/SimpleXMLRPCServer.py", line 474, in __init__
     SocketServer.TCPServer.__init__(self, addr, requestHandler)
   File "/usr/lib/python2.3/SocketServer.py", line 330, in __init__
     self.server_bind()
   File "/usr/lib/python2.3/SocketServer.py", line 341, in server_bind
     self.socket.bind(self.server_address)
   File "<string>", line 1, in bind
 socket.error: (98, 'Address already in use')

then the port was already chosen, and you will need to select another port, replace line 17 and retry; do that until the server starts silently (hanging the terminal):

 $ ./generic_server.py

leave the server there without interrupting: though apparently doing nothing, the server is really waiting for incoming connections; from now on, we will refer to the output in this terminal as the "server's terminal".

- Open another terminal on ui-1

- Modify line 18 in generic_client.py to match the same port you wrote in generic_server.py: in this way, the client will be able to contact the server at the right address.

- Submit the file generic_client.jdl 4 times using glite-wms-job-submit, make it save the IDs in a file called jobid.txt in the local directory. Each job will execute one and only one line in the list of tasks.

- Keep the server's window open and visible and watch it: as soon as one of your job will be executed on the grid, lines like those 2 will appear:

 $ ./generic_server.py
 Task 1 assigned!
 ce-3wn2.grid.seed - - [23/Oct/2008 02:13:47] "POST /RPC2 HTTP/1.0" 200 -

meaning that a job started on a WN, contacted the server and received a task.

- If you don't see any activity: check the status of your jobs.

If their status is Scheduled, then you have simply to wait for the queues on the CEs to let some of your jobs to run, otherwise if the status is Done, it may be that you have done something wrong in one of the previous steps.

- Wait until all tasks will be executed (resubmit if some of them are aborted): at that point the server will exit, giving you back the control of the server's window.

The output should be similar to this:

 $ ./generic_server.py
 Task 1 assigned!
 ce-3wn2.grid.seed - - [23/Oct/2008 02:13:47] "POST /RPC2 HTTP/1.0" 200 -
 Task 2 assigned!
 ce-2wn2.grid.seed - - [23/Oct/2008 02:18:46] "POST /RPC2 HTTP/1.0" 200 -
 Task 3 assigned!
 ce-3wn2.grid.seed - - [23/Oct/2008 02:18:47] "POST /RPC2 HTTP/1.0" 200 -
 Task 4 assigned!
 ce-1wn2.grid.seed - - [23/Oct/2008 02:19:00] "POST /RPC2 HTTP/1.0" 200 -
 All tasks served!

$

- Retrieve the output for the 3 last commands from the SE locations you specified.

Examine the output: keep in mind, when doing this, that all the output has been produced on the Grid's WNs (each one probably on a different node).

- Rename the file tasks.txt to tasks.txt_old and create a new file tasks.txt.

Create a new task that downloads from a SE location and executes a script you created.

The script (at least 10 lines) will have to perform some simple tasks (e.g. gather info. about the host) and then save it's output on the grid.

- Execute your task and retrieve the output.