{info} Task Manager provides the possibility of scheduling special long lasting tasks for the following web services.
DB Web Services
This documentation describes installation, administration and usage of Task Manager.
Task Manager application is available as part of a microservices system or in standalond web application mode.
In microservices system mode the Task Manager runs together with Config service, Discovery service, Gateway service. These three services are mandatory and optionally other services can also be part of the system. All configuration must be done in the Config module.
DB Web Service must also run according to the tasks planned to be created, executed.
The default configuration applies to the microservices system mode.
The web application runs on host <server-host> and listens on port <gateway-server-port>.
In standalone web application mode the Task Manager and DB Web Service run alone, without the Config service, Discovery service and Gateway service.
The default configuration must be changed according to the standalone web application mode; set
eureka.client.enabled=false in application.properties file in Task Manager and DB Web Service as well.
DB Web service address should be set in com.chemaxon.taskmanager.service.jwsdb property in <server-host>:<server-port> format in Task Manager application.properties file. Example: http://localhost:8062
DB Web service also need access to Task Manager service. com.chemaxon.taskmanager.service property in DB Web Service application.properties file need to be set in <server-host>:<server-port> format. Example: http://localhost:8068
See here.
See here
See here.
Module is installed into folder: jws/jws-taskmanager
See here.
" JChem Microservices DB " license is needed.
See here.
Default configuration:
| application.properties |
|---|
| server.port=8068 |
| logging.file.name=../logs/jws-taskmanager.log |
| eureka.client.enabled=true |
| bootstrap.properties |
|---|
| spring.cloud.config.failFast=true |
| spring.cloud.config.uri=${CONFIG_SERVER_URI:http\://localhost\:8888/} |
| spring.cloud.config.retry.initialInterval=3000 |
| spring.cloud.config.retry.multiplier=1.2 |
| spring.cloud.config.retry.maxInterval=60000 |
| spring.cloud.config.retry.maxAttempts=100 |
For more settings possibilities see spring documentation page.
Task manager service has own database to store added structures and task statuses. H2 and PostgreSQL databases are supported.
Default configuration:
| application.properties | description |
|---|---|
| ${CXN_TASK_JDBC_URL:jdbc:h2:file:./data/task_db} | |
| spring.datasource.driverClassName=${CXN_TASK_DRIVER:org.h2.Driver} | Value is org.postgresql.Driver in PostgreSQL case |
| spring.datasource.username=${CXN_TASK_JDBC_USER:user} | |
| spring.datasource.password=${CXN_TASK_JDBC_PASSWORD:password} | |
| spring.jpa.database-platform=${CXN_TASK_DIALECT:org.hibernate.dialect.H2Dialect} | Value is org.hibernate.dialect.PostgreSQLDialect in PostgreSQL case |
Task manager can import structure from tables which is exported from DB Web Services. It can be uploaded from file or S3 bucket.
AWS credential configuration details can be found here.
Import setup can be configured with below properties.
| application.properties | description |
|---|---|
| com.chemaxon.webservices.db.import.dbImportStrategy=FILE | Specifies whether use FILE based import or S3 based. Default is: FILE |
| com.chemaxon.webservices.db.import_export.dir=./data/export | Import file path in FILE based import strategy case |
| com.chemaxon.webservices.db.import.s3BuckeBasetUrl:s3://export-bucket/ | Import S3 bucket in S3 strategy case. |
Task manager is integrated with DB Web Services. Below configuration can be added to DB Web Services application.properties file regarding task manager communication.
| application.properties | description |
|---|---|
| com.chemaxon.taskmanager.scheduler.enabled=false | When task manager starts a job in DB Web Service only one DB service instance starts processing regardless of running instances. If this attribute is true all DB service instances check running task and join processing (e.g. molecule import). If DB Web Service instance is restarted during task is in progress then process continues if this attribute is configured as true. Default value is false. |
| com.chemaxon.taskmanager.scheduler.frequency=60000 | DB Web service tries to join running task with this regularity if com.chemaxon.taskmanager.scheduler.enabled=true. It is in milliseconds. |
| com.chemaxon.taskmanager.service=http://${TASKMANAGER_SERVICE_URL} | When DB Web service runs in standalone mode it can connect to task manager via this URL. |
| com.chemaxon.taskmanager.db.batchSize:5000 | DB Web service request structure for processing from task manager with this size. |
Task manager has a retry mechanism implemented in it when it communicates with DB Web Service. This is adjustable with these two properties in Task Manager application.properties file
| application.properties | description |
|---|---|
| com.chemaxon.retrytemplate.backOffPeriod=10000 | It is the time in milliseconds the service waits between two attempts to send the request to the other service again if no response was given. |
| com.chemaxon.retrytemplate.maxAttempts=20 | The maximum number of attempts. Note, that it also contains the very first call. |
With the above written default values the service tries to get a response for about 3 minutes.
When DB Web service is restarted during import structure can get stuck in progress state. Task manager reprocess these structures according to below configuration.
| application.properties | description |
|---|---|
| com.chemaxon.taskmanager.scheduler.frequency=60000 | Regularity of task manager job which identifies stuck sructures. It is in milliseconds. |
| com.chemaxon.taskmanager.scheduler.timelimit=600000 | Time limit before task manager tries to reprocess stuck structure. |
| com.chemaxon.taskmanager.scheduler.retrylimit=3 | Number of structure process retry |
Running more instances of the task manager service ensures HA and load balancing.
HA mode requires PostgreSQL database. It is not supported with H2 database.
Prerequisites in case of microservices system mode:
1.Config service is running
2.Discovery service is running
3.Gateway service is running
Run the service in command line in folder jws/jws-taskmanager:
jws-taskmanager-service start (on Windows)
jws-taskmanager-service start (on Linux)
or
run-jws-taskmanager.exe (on Windows)
run-jws-tarkmanager (on Linux)
Find and try out the API on the Swagger UI.
| Mode | URL of Swagger UI | Default URL of Swagger UI |
|---|---|---|
| microservices system | <serverhost>:<gateway-port>/jws-taskmanager/API/ | localhost:8080/jws-taskmanager/API/ |
| standalone web application mode | <serverhost>:<server-port>/API/ | localhost:8068/API/ |
The guidelines on the Swagger UI API documentation of your installed module display the methods and syntax implemented for reaching the functionalities of the Task Manager toolkit.
Task manager can be applied for importing structures into an already existing table. The table must be created at the endpoint /rest-v1/db/additional/createTable/{tableName} of DB Web Services.
The structures can be added via rest endpoint or can be uploaded from file or S3 bucket.
The structures will have ID data in the table. If the structures to be imported have their own IDproperty, this IDproperty can be set as ID in the table. Add structure endpoint (PUT - /rest-v1/work-queue/task/{taskId}) has id attribute which is also set ID in DB Web service side. It is recommended to provide ID property.
Example - Import structure
PUT - /rest-v1/work-queue/task
{
"params": {
"isDuplicateFiltering": "true",
"tableName": "mytable"
},
"service": "jwsdb",
"task": "batchInsert"
}PUT - /rest-v1/work-queue/task/{taskId}
{
"inputFormat": "smiles",
"structures": [
{
"id": 1,
"structure": "O=C1CCCC=C1"
},
{
"id": 2,
"structure": "C1CCCCC1"
}
]
}Note: Provided ID will be used as ID in DB Web Services import.
POST - /rest-v1/work-queue/task/{taskId}/manage
{
"action": "START"
}
GET - /rest-v1/work-queue/task/{taskId}
When task is in progress the response contains the processing percentage.
{
"status": "IN_PROGRESS",
"processedPercentage": 12
}
If import is done task is placed in Ready status.
{
"status": "READY"
}
If task has any issue the status is updated to ERROR and response contains number of processed and failed.
{
"status": "ERROR",
"processedSuccess": 0,
"processedError": 2
}
GET - /rest-v1/work-queue/task/{taskId}/results?resultIdType=ALL
When task has finished the result can be retrieved.
Example result in success import case.
{
"failedIds": [],
"successfulIds": [1, 2],
"duplicatedIds": [],
"idType": "requestedId"
}
Example result in error case.
{
"failedIds": [{
"id": 1,
"errorMessage": "Could not save data to db: com.chemaxon.webservices.db.exception.TableDoesNotExistsException: there is no table registered for name: mytable"
}, {
"id": 2,
"errorMessage": "Could not save data to db: com.chemaxon.webservices.db.exception.TableDoesNotExistsException: there is no table registered for name: mytable"
}
],
"successfulIds": [],
"duplicatedIds": [],
"idType": "requestedId"
}
Note: idType is requestedId, if ID was provided in the Add structures with ID step.