misc doc and logging updates
Checkin to change README from txt to rst format. Jenkins seems to hijack logging so i've removed all of the logging specific bindings that were added previously. I also added "----" to begining of this plugin's logging messages so i could easily keep track of them. Change-Id: Ibd8c56af5b9ad18152bcb0d3ff0c41168a6d2fd1
This commit is contained in:
parent
174d783623
commit
c461e204f2
|
@ -0,0 +1,47 @@
|
||||||
|
---------------------------------------------------------------------
|
||||||
|
Copyright 2013 Hewlett-Packard Development Company, L.P.
|
||||||
|
|
||||||
|
Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
you may not use this file except in compliance with the License.
|
||||||
|
You may obtain a copy of the License at
|
||||||
|
|
||||||
|
http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
|
||||||
|
Unless required by applicable law or agreed to in writing, software
|
||||||
|
distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
See the License for the specific language governing permissions and
|
||||||
|
limitations under the License.
|
||||||
|
---------------------------------------------------------------------
|
||||||
|
|
||||||
|
Installation
|
||||||
|
============
|
||||||
|
|
||||||
|
The plugin can be installed like any other Jenkins plugin. Please refer to the Jenkins documentation on installing plugins.
|
||||||
|
|
||||||
|
Configuration
|
||||||
|
=============
|
||||||
|
|
||||||
|
Configuration of the plugin can be done on the Jenkins configure page (i.e. http://localhost:8080/configure). Click on the help if needed.
|
||||||
|
|
||||||
|
Developers
|
||||||
|
==========
|
||||||
|
Project site:
|
||||||
|
|
||||||
|
* https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin
|
||||||
|
|
||||||
|
Bug report:
|
||||||
|
|
||||||
|
* https://bugs.launchpad.net/~gearman-plugin-core
|
||||||
|
|
||||||
|
Cloning:
|
||||||
|
|
||||||
|
* https://github.com/openstack-infra/gearman-plugin.git
|
||||||
|
|
||||||
|
Patches are submitted via Gerrit at:
|
||||||
|
|
||||||
|
* https://review.openstack.org/
|
||||||
|
|
||||||
|
More details on how you can contribute is available on our wiki at:
|
||||||
|
|
||||||
|
* http://wiki.openstack.org/HowToContribute
|
61
README.txt
61
README.txt
|
@ -1,61 +0,0 @@
|
||||||
---------------------------------------------------------------------
|
|
||||||
Copyright 2013 Hewlett-Packard Development Company, L.P.
|
|
||||||
|
|
||||||
Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
you may not use this file except in compliance with the License.
|
|
||||||
You may obtain a copy of the License at
|
|
||||||
|
|
||||||
http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
|
|
||||||
Unless required by applicable law or agreed to in writing, software
|
|
||||||
distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
See the License for the specific language governing permissions and
|
|
||||||
limitations under the License.
|
|
||||||
---------------------------------------------------------------------
|
|
||||||
|
|
||||||
Jenkins does not support multiple masters. You can setup multiple Jenkins
|
|
||||||
masters but there is no coordination between them.
|
|
||||||
|
|
||||||
One problem with scheduling builds on Jenkins master (“MasterA”) server
|
|
||||||
is that MasterA only knows about its connected slaves. If all slaves on
|
|
||||||
MasterA are busy then MasterA will just put the next scheduled build on
|
|
||||||
its queue. Now MasterA needs to wait for an available slave to run
|
|
||||||
the build. This will be very in-efficient if your builds take a long
|
|
||||||
time to run. So..what if there is another Jenkins master (“MasterB”)
|
|
||||||
that has free slaves to service the next scheduled build? Your probably
|
|
||||||
saying “Then slaves on MasterB should run the build”. However MasterB
|
|
||||||
will never service the builds on MasterA's queue. The client that schedules
|
|
||||||
the builds must know about MasterB and then schedule builds on MasterB.
|
|
||||||
This is what we mean by lack of coordination between masters.
|
|
||||||
The gearman-plugin attempts to fill this gap.
|
|
||||||
|
|
||||||
This plugin integrates Gearman with Jenkins and will make it so that
|
|
||||||
any Jenkins slave on any Jenkins master can service a job in the queue.
|
|
||||||
It will essentially replace the Jenkins build queue with the Gearman
|
|
||||||
job queue. The job should stay in the gearman queue until there is a
|
|
||||||
Jenkins node that can run that job.
|
|
||||||
|
|
||||||
|
|
||||||
This is the typical workflow:
|
|
||||||
|
|
||||||
1) On a 'Launch Workers', we spawn a Gearman worker for each Jenkins
|
|
||||||
executor. We'll call these "executor worker threads".
|
|
||||||
|
|
||||||
Each executor worker thread is associated 1:1 with a Jenkins node (slave or master)
|
|
||||||
|
|
||||||
2) Now we register jobs for each Gearman executor depending on
|
|
||||||
projects and nodes. View the image to see the mapping.
|
|
||||||
|
|
||||||
3) On a 'Launch Workers', we spawn one more thread to be a Gearman
|
|
||||||
worker to handle job management for this Jenkins master. We'll call
|
|
||||||
it the "management worker thread" and register the following function:
|
|
||||||
|
|
||||||
stop:$hostname
|
|
||||||
|
|
||||||
4) Any Gearman client can connect to the Gearman server and send a
|
|
||||||
request to build a Jenkins project or cancel a project. View the
|
|
||||||
examples to see how this can be done.
|
|
||||||
|
|
||||||
5) The Gearman workers will service any client request that come
|
|
||||||
through to start/cancel a Jenkins build.
|
|
22
pom.xml
22
pom.xml
|
@ -135,8 +135,6 @@
|
||||||
<taglist-maven-plugin.version>2.4</taglist-maven-plugin.version>
|
<taglist-maven-plugin.version>2.4</taglist-maven-plugin.version>
|
||||||
<versions-maven-plugin.version>1.2</versions-maven-plugin.version>
|
<versions-maven-plugin.version>1.2</versions-maven-plugin.version>
|
||||||
<xml-maven-plugin.version>1.0-beta-3</xml-maven-plugin.version>
|
<xml-maven-plugin.version>1.0-beta-3</xml-maven-plugin.version>
|
||||||
<slf4j.version>1.7.2</slf4j.version>
|
|
||||||
<logback.version>1.0.9</logback.version>
|
|
||||||
<gson.version>2.2.2</gson.version>
|
<gson.version>2.2.2</gson.version>
|
||||||
<gearman.version>0.6-SNAPSHOT</gearman.version>
|
<gearman.version>0.6-SNAPSHOT</gearman.version>
|
||||||
</properties>
|
</properties>
|
||||||
|
@ -333,26 +331,6 @@
|
||||||
</plugins>
|
</plugins>
|
||||||
</reporting>
|
</reporting>
|
||||||
<dependencies>
|
<dependencies>
|
||||||
<dependency>
|
|
||||||
<groupId>org.slf4j</groupId>
|
|
||||||
<artifactId>slf4j-api</artifactId>
|
|
||||||
<version>${slf4j.version}</version>
|
|
||||||
</dependency>
|
|
||||||
<dependency>
|
|
||||||
<groupId>org.slf4j</groupId>
|
|
||||||
<artifactId>slf4j-simple</artifactId>
|
|
||||||
<version>${slf4j.version}</version>
|
|
||||||
</dependency>
|
|
||||||
<dependency>
|
|
||||||
<groupId>ch.qos.logback</groupId>
|
|
||||||
<artifactId>logback-classic</artifactId>
|
|
||||||
<version>${logback.version}</version>
|
|
||||||
</dependency>
|
|
||||||
<dependency>
|
|
||||||
<groupId>ch.qos.logback</groupId>
|
|
||||||
<artifactId>logback-core</artifactId>
|
|
||||||
<version>${logback.version}</version>
|
|
||||||
</dependency>
|
|
||||||
<dependency>
|
<dependency>
|
||||||
<groupId>org.gearman</groupId>
|
<groupId>org.gearman</groupId>
|
||||||
<artifactId>gearman-java</artifactId>
|
<artifactId>gearman-java</artifactId>
|
||||||
|
|
|
@ -101,7 +101,7 @@ public abstract class AbstractWorkerThread implements Runnable {
|
||||||
*/
|
*/
|
||||||
public void registerJobs() {
|
public void registerJobs() {
|
||||||
|
|
||||||
logger.info("----- AbstractorWorker registerJobs function ----");
|
logger.info("---- AbstractorWorker registerJobs function ----");
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -122,10 +122,10 @@ public abstract class AbstractWorkerThread implements Runnable {
|
||||||
|
|
||||||
if (worker.isRunning()) {
|
if (worker.isRunning()) {
|
||||||
try {
|
try {
|
||||||
logger.info("Stopping " + getName() + ":" + getId().toString() +
|
logger.info("---- Stopping " + getName() + ":" + getId().toString() +
|
||||||
" (" + new Date().toString() + ")");
|
" (" + new Date().toString() + ")");
|
||||||
worker.unregisterAll();
|
worker.unregisterAll();
|
||||||
worker.stop();
|
worker.shutdown();
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
e.printStackTrace();
|
e.printStackTrace();
|
||||||
}
|
}
|
||||||
|
@ -154,7 +154,7 @@ public abstract class AbstractWorkerThread implements Runnable {
|
||||||
public void run() {
|
public void run() {
|
||||||
|
|
||||||
if (!worker.isRunning()) {
|
if (!worker.isRunning()) {
|
||||||
logger.info("Starting Worker "+ getName() + ":" + getId().toString() +
|
logger.info("---- Starting Worker "+ getName() + ":" + getId().toString() +
|
||||||
" ("+new Date().toString()+")");
|
" ("+new Date().toString()+")");
|
||||||
worker.setWorkerID(name);
|
worker.setWorkerID(name);
|
||||||
worker.setJobUniqueIdRequired(true);
|
worker.setJobUniqueIdRequired(true);
|
||||||
|
|
|
@ -135,7 +135,7 @@ public class ExecutorWorkerThread extends AbstractWorkerThread{
|
||||||
@Override
|
@Override
|
||||||
public void registerJobs() {
|
public void registerJobs() {
|
||||||
|
|
||||||
logger.info("----- Registering executor jobs on " + name + " ----");
|
logger.info("---- Registering executor jobs on " + name + " ----");
|
||||||
|
|
||||||
/*
|
/*
|
||||||
* We start with an empty worker.
|
* We start with an empty worker.
|
||||||
|
@ -161,7 +161,7 @@ public class ExecutorWorkerThread extends AbstractWorkerThread{
|
||||||
if (label == null) { // project has no label -> so register
|
if (label == null) { // project has no label -> so register
|
||||||
// "build:projectName" on all nodes
|
// "build:projectName" on all nodes
|
||||||
String jobFunctionName = "build:" + projectName;
|
String jobFunctionName = "build:" + projectName;
|
||||||
logger.info("Registering job " + jobFunctionName + " on "
|
logger.info("---- Registering job " + jobFunctionName + " on "
|
||||||
+ name);
|
+ name);
|
||||||
worker.registerFunctionFactory(new CustomGearmanFunctionFactory(
|
worker.registerFunctionFactory(new CustomGearmanFunctionFactory(
|
||||||
jobFunctionName, StartJobWorker.class.getName(),
|
jobFunctionName, StartJobWorker.class.getName(),
|
||||||
|
@ -180,7 +180,7 @@ public class ExecutorWorkerThread extends AbstractWorkerThread{
|
||||||
if (projectLabelNodes.contains(this.node)) {
|
if (projectLabelNodes.contains(this.node)) {
|
||||||
String jobFunctionName = "build:" + projectName
|
String jobFunctionName = "build:" + projectName
|
||||||
+ ":" + projectLabel;
|
+ ":" + projectLabel;
|
||||||
logger.info("Registering job " + jobFunctionName
|
logger.info("---- Registering job " + jobFunctionName
|
||||||
+ " on " + this.node.getNodeName());
|
+ " on " + this.node.getNodeName());
|
||||||
// register with label (i.e. "build:$projectName:$projectLabel")
|
// register with label (i.e. "build:$projectName:$projectLabel")
|
||||||
worker.registerFunctionFactory(new CustomGearmanFunctionFactory(
|
worker.registerFunctionFactory(new CustomGearmanFunctionFactory(
|
||||||
|
|
|
@ -53,7 +53,7 @@ public class GearmanPluginConfig extends GlobalConfiguration {
|
||||||
* Constructor.
|
* Constructor.
|
||||||
*/
|
*/
|
||||||
public GearmanPluginConfig() {
|
public GearmanPluginConfig() {
|
||||||
logger.info("--- GearmanPluginConfig Constructor ---");
|
logger.info("---- GearmanPluginConfig Constructor ---");
|
||||||
|
|
||||||
gearmanProxy = new GearmanProxy();
|
gearmanProxy = new GearmanProxy();
|
||||||
load();
|
load();
|
||||||
|
@ -96,7 +96,7 @@ public class GearmanPluginConfig extends GlobalConfiguration {
|
||||||
if (launchWorker) {
|
if (launchWorker) {
|
||||||
|
|
||||||
// check for a valid connection to gearman server
|
// check for a valid connection to gearman server
|
||||||
logger.info("--- Check connection to Gearman Server " + host + ":"
|
logger.info("---- Check connection to Gearman Server " + host + ":"
|
||||||
+ port);
|
+ port);
|
||||||
if (!GearmanPluginUtil.connectionIsAvailable(host, port, 5000)) {
|
if (!GearmanPluginUtil.connectionIsAvailable(host, port, 5000)) {
|
||||||
launchWorker = false;
|
launchWorker = false;
|
||||||
|
|
|
@ -61,10 +61,10 @@ public class GearmanPluginUtil {
|
||||||
} else {
|
} else {
|
||||||
try {
|
try {
|
||||||
socket.connect(endPoint, timeout);
|
socket.connect(endPoint, timeout);
|
||||||
logger.info("Connection Success: " + endPoint);
|
logger.info("---- Connection Success: " + endPoint);
|
||||||
return true;
|
return true;
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
logger.info("Connection Failure: " + endPoint + " message: "
|
logger.info("---- Connection Failure: " + endPoint + " message: "
|
||||||
+ e.getClass().getSimpleName() + " - " + e.getMessage());
|
+ e.getClass().getSimpleName() + " - " + e.getMessage());
|
||||||
} finally {
|
} finally {
|
||||||
if (socket != null) {
|
if (socket != null) {
|
||||||
|
|
|
@ -50,7 +50,7 @@ public class GearmanProxy {
|
||||||
|
|
||||||
// constructor
|
// constructor
|
||||||
public GearmanProxy() {
|
public GearmanProxy() {
|
||||||
logger.info("--- GearmanProxy Constructor ---");
|
logger.info("---- GearmanProxy Constructor ---");
|
||||||
|
|
||||||
gewtHandles = new ArrayList<AbstractWorkerThread>();
|
gewtHandles = new ArrayList<AbstractWorkerThread>();
|
||||||
gmwtHandles = new ArrayList<AbstractWorkerThread>();
|
gmwtHandles = new ArrayList<AbstractWorkerThread>();
|
||||||
|
@ -94,9 +94,9 @@ public class GearmanProxy {
|
||||||
try {
|
try {
|
||||||
masterNode = Computer.currentComputer().getNode();
|
masterNode = Computer.currentComputer().getNode();
|
||||||
} catch (NullPointerException npe) {
|
} catch (NullPointerException npe) {
|
||||||
logger.info("--- Master is offline");
|
logger.info("---- Master is offline");
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
logger.info("--- Can't get Master");
|
logger.info("---- Can't get Master");
|
||||||
e.printStackTrace();
|
e.printStackTrace();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -140,7 +140,7 @@ public class GearmanProxy {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
logger.info("--- Num of executors running = " + getNumExecutors());
|
logger.info("---- Num of executors running = " + getNumExecutors());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@ -160,7 +160,7 @@ public class GearmanProxy {
|
||||||
gmwtHandles.clear();
|
gmwtHandles.clear();
|
||||||
numWorkerNodes = 0;
|
numWorkerNodes = 0;
|
||||||
|
|
||||||
logger.info("--- Num of executors running = " + getNumExecutors());
|
logger.info("---- Num of executors running = " + getNumExecutors());
|
||||||
}
|
}
|
||||||
|
|
||||||
/*
|
/*
|
||||||
|
|
|
@ -75,7 +75,7 @@ public class StartJobWorker extends AbstractGearmanFunction {
|
||||||
*/
|
*/
|
||||||
@Override
|
@Override
|
||||||
public GearmanJobResult executeFunction() {
|
public GearmanJobResult executeFunction() {
|
||||||
logger.info("----- Running executeFunction in " + name + " ----");
|
logger.info("---- Running executeFunction in " + name + " ----");
|
||||||
|
|
||||||
// decode the uniqueId from the client
|
// decode the uniqueId from the client
|
||||||
String decodedUniqueId = null;
|
String decodedUniqueId = null;
|
||||||
|
@ -125,7 +125,7 @@ public class StartJobWorker extends AbstractGearmanFunction {
|
||||||
Action [] actions = {runNode, params};
|
Action [] actions = {runNode, params};
|
||||||
|
|
||||||
// schedule jenkins to build project
|
// schedule jenkins to build project
|
||||||
logger.info("Scheduling "+project.getName()+" build #" +
|
logger.info("---- Scheduling "+project.getName()+" build #" +
|
||||||
project.getNextBuildNumber()+" on " + runNodeName
|
project.getNextBuildNumber()+" on " + runNodeName
|
||||||
+ " with UUID " + decodedUniqueId + " and build params " + buildParams);
|
+ " with UUID " + decodedUniqueId + " and build params " + buildParams);
|
||||||
Future<?> future = project.scheduleBuild2(0, new Cause.UserIdCause(), actions);
|
Future<?> future = project.scheduleBuild2(0, new Cause.UserIdCause(), actions);
|
||||||
|
|
|
@ -71,7 +71,7 @@ public class StopJobWorker extends AbstractGearmanFunction {
|
||||||
String jobResultMsg = "";
|
String jobResultMsg = "";
|
||||||
|
|
||||||
if (decodedUniqueId.isEmpty() || decodedUniqueId == null){
|
if (decodedUniqueId.isEmpty() || decodedUniqueId == null){
|
||||||
logger.info("Client passed in an invalid UUID");
|
logger.info("---- Client passed in an invalid UUID");
|
||||||
jobFailureMsg = "I need the job Id please";
|
jobFailureMsg = "I need the job Id please";
|
||||||
jobResult = false;
|
jobResult = false;
|
||||||
} else {
|
} else {
|
||||||
|
@ -131,7 +131,7 @@ public class StopJobWorker extends AbstractGearmanFunction {
|
||||||
|
|
||||||
if (param.getUuid().equals(uuid)) {
|
if (param.getUuid().equals(uuid)) {
|
||||||
|
|
||||||
logger.info("Aborting build : "+buildNum+": "+buildId+" on " + runNodeName
|
logger.info("---- Aborting build : "+buildNum+": "+buildId+" on " + runNodeName
|
||||||
+" with UUID " + uuid + " and build params " + buildParams);
|
+" with UUID " + uuid + " and build params " + buildParams);
|
||||||
|
|
||||||
// abort the running jenkins build
|
// abort the running jenkins build
|
||||||
|
@ -174,7 +174,7 @@ public class StopJobWorker extends AbstractGearmanFunction {
|
||||||
|
|
||||||
if (param.getUuid().equals(uuid)) {
|
if (param.getUuid().equals(uuid)) {
|
||||||
|
|
||||||
logger.info("Aborting build : "+buildNum+": "+buildId+" on " + runNodeName
|
logger.info("---- Aborting build : "+buildNum+": "+buildId+" on " + runNodeName
|
||||||
+" with UUID " + uuid + " and build params " + buildParams);
|
+" with UUID " + uuid + " and build params " + buildParams);
|
||||||
|
|
||||||
// abort the running jenkins build
|
// abort the running jenkins build
|
||||||
|
|
|
@ -1,19 +0,0 @@
|
||||||
<?xml version="1.0" encoding="UTF-8"?>
|
|
||||||
<configuration>
|
|
||||||
<appender name="fileAppender" class="ch.qos.logback.core.FileAppender">
|
|
||||||
<file>GearmanPlugin.log</file>
|
|
||||||
<append>true</append>
|
|
||||||
<encoder>
|
|
||||||
<pattern>%d [%thread] %-5level %logger{35} - %msg%n</pattern>
|
|
||||||
</encoder>
|
|
||||||
</appender>
|
|
||||||
<appender name="consoleAppender" class="ch.qos.logback.core.ConsoleAppender">
|
|
||||||
<encoder>
|
|
||||||
<pattern>%d [%thread] %-5level %logger{35} - %msg%n</pattern>
|
|
||||||
</encoder>
|
|
||||||
</appender>
|
|
||||||
<root level="TRACE">
|
|
||||||
<appender-ref ref="fileAppender" />
|
|
||||||
<appender-ref ref="consoleAppender" />
|
|
||||||
</root>
|
|
||||||
</configuration>
|
|
Loading…
Reference in New Issue