Docker images as Jenkins agents failing, "/agent.log Permission Denied"

[Zach LaCelle]

Hello,

I'm trying to set up a Jenkins master to use another server as a Jenkins
agent host, running Docker containers on the host for different
environments.

My master is running jenkinsci/blueocean. The agent host server is running
Docker with tcp://0.0.0.0:2376 open.

TL;DR Question: Why isn't my ubuntu:16.04-based Docker image launching
correctly, and how should I set up this environment?

I can run my test pipeline on the host with the default jenkinsci/slave
image, which is based on the openjdk-8:jre image.

I've created my own image which is based on ubuntu:16.04, in order to
create a Ubuntu build environment. I've copied in most of the
jenkinsci/slave sections, to do things like get slave.jar in the right
place, set up users/groups, etc. My Dockerfile is located at the bottom of
this post.

The issue is that, when the Master tries to provision an agent to build, I
get the following error:

INFO: Trying to run container for node ubuntu-1604-0001undhvwt76 from
image: test:0.1
Oct 19, 2018 6:10:16 PM com.nirima.jenkins.plugins.docker.DockerTemplate
doProvisionNode
INFO: Started container ID
dab1ac24bbad9ca00855f97306ebbcf938e7c6ce028a9c35a137989cf784d3f6 for node
ubuntu-1604-0001undhvwt76 from image: test:0.1
Exception in thread "main"
java.io.FileNotFoundException: /agent.log (Permission denied)

at java.io.FileOutputStream.open0(Native Method)

at java.io.FileOutputStream.open(FileOutputStream.java:270)

at java.io.FileOutputStream.<init>(FileOutputStream.java:213)


at java.io.FileOutputStream.<init>(FileOutputStream.java:162)


at
org.jenkinsci.remoting.engine.WorkDirManager.legacyCreateTeeStream(WorkDirManager.java:319)


at
org.jenkinsci.remoting.engine.WorkDirManager.setupLogging(WorkDirManager.java:288)


at hudson.remoting.Launcher.run(Launcher.java:304)


at hudson.remoting.Launcher.main(Launcher.java:283)

When jenkinsci/slave runs, it gets the following output:

INFO: Trying to run container for node test-0001uqx3pd659 from image:
jenkinsci/slave
Oct 19, 2018 6:14:54 PM com.nirima.jenkins.plugins.docker.DockerTemplate
doProvisionNode
INFO: Started container ID
4b9d599a4a516ab0f4583bce79e6c6c681b370882ea5307fd5b8a867b773cec3 for node
test-0001uqx3pd659 from image: jenkinsci/slave
Oct 19, 2018 6:14:56 PM hudson.slaves.NodeProvisioner$2 run
INFO: Image of jenkinsci/slave provisioning successfully completed. We have
now 6 computer(s)
channel started

Clearly my Dockerfile is missing something, but I have no idea what. I can
run it with the "docker run -i --rm --name agent --init test:0.1 java -jar
/usr/share/jenkins/slave.jar" and get the correct REMOTE line.

Dockerfile:
# Start with Docker Hub Ubuntu image
FROM ubuntu:16.04

## Install ubuntu-server
USER root
# These configurations are copied so that dpkg-reconfigure doesn't hang
COPY --chown=root:root etc_default_keyboard /etc/default/keyboard
COPY --chown=root:root etc_default_console-setup /etc/default/console-setup
#
RUN apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y ubuntu-server

## Install wget
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y wget

## Install Java 8 and sshd
USER root
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y
openjdk-8-jre-headless ssh

## Setup Jenkins environment
ARG user=jenkins
ARG group=jenkins
ARG uid=10000
ARG gid=10000
ENV HOME /home/${user}
RUN groupadd -g ${gid} ${group}
RUN useradd -c "Jenkins user" -d $HOME -u ${uid} -g ${gid} -m ${user}
ARG VERSION=3.26
ARG AGENT_WORKDIR=/home/${user}/agent
RUN curl --create-dirs -sSLo /usr/share/jenkins/slave.jar
https://repo.jenkins-ci.org/public/org/jenkins-ci/main/remoting/${VERSION}/remoting-${VERSION}.jar
\
&& chmod 755 /usr/share/jenkins \
&& chmod 644 /usr/share/jenkins/slave.jar
USER ${user}
ENV AGENT_WORKDIR=${AGENT_WORKDIR}
RUN mkdir /home/${user}/.jenkins && mkdir -p ${AGENT_WORKDIR}
VOLUME /home/${user}/.jenkins
VOLUME ${AGENT_WORKDIR}
WORKDIR /home/${user}

## Start the Jenkins slave process
#CMD java -jar /usr/share/jenkins/slave.jar -workDir ${AGENT_WORKDIR}

[Zach LaCelle]

OK, I'm one step closer, but still hope to get some direction.

On both types of images (my test image and the jenkinsci/slave image),
/usr/share/jenkins/slave.jar is present with identical permissions.

On the correct image (jenkinsci/slave), the files "remoting-3.25.jar" and
then "agent.log" get created in /home/jenkins.

However, on my test image, "remoting-3.25.jar" gets created in the root
directory, and then agent.log fails to be created (I assume because of
permissions errors, per the previous post).

I'm unable to figure out which environment variable I've failed to set. The
environments look identical. HOME and AGENT_WORKDIR are both set (although
it looks like HOME is where these files are getting created). The commands
for running the agent are not changing: I'm simply running it from the
Jenkins master and switching which image the Docker Cloud is using.

1. What creates remoting-3.25.jar? I assume it's copied from
/usr/share/jenkins/slave.jar, but I'm not sure.
2. What variable, setting, etc is controlling where this file is saved?

Post by Zach LaCelle
Hello,
I'm trying to set up a Jenkins master to use another server as a Jenkins
agent host, running Docker containers on the host for different
environments.
My master is running jenkinsci/blueocean. The agent host server is running
Docker with tcp://0.0.0.0:2376 open.
TL;DR Question: Why isn't my ubuntu:16.04-based Docker image launching
correctly, and how should I set up this environment?
I can run my test pipeline on the host with the default jenkinsci/slave
image, which is based on the openjdk-8:jre image.
I've created my own image which is based on ubuntu:16.04, in order to
create a Ubuntu build environment. I've copied in most of the
jenkinsci/slave sections, to do things like get slave.jar in the right
place, set up users/groups, etc. My Dockerfile is located at the bottom of
this post.
The issue is that, when the Master tries to provision an agent to build, I
INFO: Trying to run container for node ubuntu-1604-0001undhvwt76 from
image: test:0.1
Oct 19, 2018 6:10:16 PM com.nirima.jenkins.plugins.docker.DockerTemplate
doProvisionNode
INFO: Started container ID
dab1ac24bbad9ca00855f97306ebbcf938e7c6ce028a9c35a137989cf784d3f6 for node
ubuntu-1604-0001undhvwt76 from image: test:0.1
Exception in thread "main"
java.io.FileNotFoundException: /agent.log (Permission denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at
org.jenkinsci.remoting.engine.WorkDirManager.legacyCreateTeeStream(WorkDirManager.java:319)
at
org.jenkinsci.remoting.engine.WorkDirManager.setupLogging(WorkDirManager.java:288)
at hudson.remoting.Launcher.run(Launcher.java:304)
at hudson.remoting.Launcher.main(Launcher.java:283)
jenkinsci/slave
Oct 19, 2018 6:14:54 PM com.nirima.jenkins.plugins.docker.DockerTemplate
doProvisionNode
INFO: Started container ID
4b9d599a4a516ab0f4583bce79e6c6c681b370882ea5307fd5b8a867b773cec3 for node
test-0001uqx3pd659 from image: jenkinsci/slave
Oct 19, 2018 6:14:56 PM hudson.slaves.NodeProvisioner$2 run
INFO: Image of jenkinsci/slave provisioning successfully completed. We
have now 6 computer(s)
channel started
Clearly my Dockerfile is missing something, but I have no idea what. I can
run it with the "docker run -i --rm --name agent --init test:0.1 java -jar
/usr/share/jenkins/slave.jar" and get the correct REMOTE line.
# Start with Docker Hub Ubuntu image
FROM ubuntu:16.04
## Install ubuntu-server
USER root
# These configurations are copied so that dpkg-reconfigure doesn't hang
COPY --chown=root:root etc_default_keyboard /etc/default/keyboard
COPY --chown=root:root etc_default_console-setup /etc/default/console-setup
#
RUN apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y ubuntu-server
## Install wget
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y wget
## Install Java 8 and sshd
USER root
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y
openjdk-8-jre-headless ssh
## Setup Jenkins environment
ARG user=jenkins
ARG group=jenkins
ARG uid=10000
ARG gid=10000
ENV HOME /home/${user}
RUN groupadd -g ${gid} ${group}
RUN useradd -c "Jenkins user" -d $HOME -u ${uid} -g ${gid} -m ${user}
ARG VERSION=3.26
ARG AGENT_WORKDIR=/home/${user}/agent
RUN curl --create-dirs -sSLo /usr/share/jenkins/slave.jar
https://repo.jenkins-ci.org/public/org/jenkins-ci/main/remoting/${VERSION}/remoting-${VERSION}.jar
\
&& chmod 755 /usr/share/jenkins \
&& chmod 644 /usr/share/jenkins/slave.jar
USER ${user}
ENV AGENT_WORKDIR=${AGENT_WORKDIR}
RUN mkdir /home/${user}/.jenkins && mkdir -p ${AGENT_WORKDIR}
VOLUME /home/${user}/.jenkins
VOLUME ${AGENT_WORKDIR}
WORKDIR /home/${user}
## Start the Jenkins slave process
#CMD java -jar /usr/share/jenkins/slave.jar -workDir ${AGENT_WORKDIR}

[Zach LaCelle]

Solved. In case anybody makes the same mistake I did:

The docker-plugin does the remoting-3.25.jar copy, with the
injectRemotingJar() function in DockerComputerAttachConnector.java. This
uses the remoteFs variable which gets set way back in DockerTemplate.java's
provisionNode() function. The value is gotten directly from the Docker
Image using getContainerConfig().getWorkingDir().

I had made a silly mistake where, for the Jenkins Docker Cloud Template I
was testing with, setting "Remote Root Filesystem" to "/". Must have been a
fat finger.

But for those following this in the future, make sure you set WORKDIR in
your custom Docker image to /home/jenkins (or wherever your Jenkins home
is).

Post by Zach LaCelle
OK, I'm one step closer, but still hope to get some direction.
On both types of images (my test image and the jenkinsci/slave image),
/usr/share/jenkins/slave.jar is present with identical permissions.
On the correct image (jenkinsci/slave), the files "remoting-3.25.jar" and
then "agent.log" get created in /home/jenkins.
However, on my test image, "remoting-3.25.jar" gets created in the root
directory, and then agent.log fails to be created (I assume because of
permissions errors, per the previous post).
I'm unable to figure out which environment variable I've failed to set.
The environments look identical. HOME and AGENT_WORKDIR are both set
(although it looks like HOME is where these files are getting created). The
commands for running the agent are not changing: I'm simply running it from
the Jenkins master and switching which image the Docker Cloud is using.
1. What creates remoting-3.25.jar? I assume it's copied from
/usr/share/jenkins/slave.jar, but I'm not sure.
2. What variable, setting, etc is controlling where this file is saved?

Post by Zach LaCelle
Hello,
I'm trying to set up a Jenkins master to use another server as a Jenkins
agent host, running Docker containers on the host for different
environments.
My master is running jenkinsci/blueocean. The agent host server is
running Docker with tcp://0.0.0.0:2376 open.
TL;DR Question: Why isn't my ubuntu:16.04-based Docker image launching
correctly, and how should I set up this environment?
I can run my test pipeline on the host with the default jenkinsci/slave
image, which is based on the openjdk-8:jre image.
I've created my own image which is based on ubuntu:16.04, in order to
create a Ubuntu build environment. I've copied in most of the
jenkinsci/slave sections, to do things like get slave.jar in the right
place, set up users/groups, etc. My Dockerfile is located at the bottom of
this post.
The issue is that, when the Master tries to provision an agent to build,
INFO: Trying to run container for node ubuntu-1604-0001undhvwt76 from
image: test:0.1
Oct 19, 2018 6:10:16 PM com.nirima.jenkins.plugins.docker.DockerTemplate
doProvisionNode
INFO: Started container ID
dab1ac24bbad9ca00855f97306ebbcf938e7c6ce028a9c35a137989cf784d3f6 for node
ubuntu-1604-0001undhvwt76 from image: test:0.1
Exception in thread "main"
java.io.FileNotFoundException: /agent.log (Permission denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at
org.jenkinsci.remoting.engine.WorkDirManager.legacyCreateTeeStream(WorkDirManager.java:319)
at
org.jenkinsci.remoting.engine.WorkDirManager.setupLogging(WorkDirManager.java:288)
at hudson.remoting.Launcher.run(Launcher.java:304)
at hudson.remoting.Launcher.main(Launcher.java:283)
jenkinsci/slave
Oct 19, 2018 6:14:54 PM com.nirima.jenkins.plugins.docker.DockerTemplate
doProvisionNode
INFO: Started container ID
4b9d599a4a516ab0f4583bce79e6c6c681b370882ea5307fd5b8a867b773cec3 for node
test-0001uqx3pd659 from image: jenkinsci/slave
Oct 19, 2018 6:14:56 PM hudson.slaves.NodeProvisioner$2 run
INFO: Image of jenkinsci/slave provisioning successfully completed. We
have now 6 computer(s)
channel started
Clearly my Dockerfile is missing something, but I have no idea what. I
can run it with the "docker run -i --rm --name agent --init test:0.1 java
-jar /usr/share/jenkins/slave.jar" and get the correct REMOTE line.
# Start with Docker Hub Ubuntu image
FROM ubuntu:16.04
## Install ubuntu-server
USER root
# These configurations are copied so that dpkg-reconfigure doesn't hang
COPY --chown=root:root etc_default_keyboard /etc/default/keyboard
COPY --chown=root:root etc_default_console-setup
/etc/default/console-setup
#
RUN apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y ubuntu-server
## Install wget
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y wget
## Install Java 8 and sshd
USER root
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y
openjdk-8-jre-headless ssh
## Setup Jenkins environment
ARG user=jenkins
ARG group=jenkins
ARG uid=10000
ARG gid=10000
ENV HOME /home/${user}
RUN groupadd -g ${gid} ${group}
RUN useradd -c "Jenkins user" -d $HOME -u ${uid} -g ${gid} -m ${user}
ARG VERSION=3.26
ARG AGENT_WORKDIR=/home/${user}/agent
RUN curl --create-dirs -sSLo /usr/share/jenkins/slave.jar
https://repo.jenkins-ci.org/public/org/jenkins-ci/main/remoting/${VERSION}/remoting-${VERSION}.jar
\
&& chmod 755 /usr/share/jenkins \
&& chmod 644 /usr/share/jenkins/slave.jar
USER ${user}
ENV AGENT_WORKDIR=${AGENT_WORKDIR}
RUN mkdir /home/${user}/.jenkins && mkdir -p ${AGENT_WORKDIR}
VOLUME /home/${user}/.jenkins
VOLUME ${AGENT_WORKDIR}
WORKDIR /home/${user}
## Start the Jenkins slave process
#CMD java -jar /usr/share/jenkins/slave.jar -workDir ${AGENT_WORKDIR}