Job Failed with "java.lang.OutOfMemoryError: unable to create new native thread" On Kuberentes cluster..

Discussion:

p***@gmail.com

2018-11-11 09:12:43 UTC

Hi All,

While running a job on Kuberentes Cluster

[EnvInject] - Loading node environment variables.
Building remotely on k8s-prod-cl1-pod1-v1v5j <http://hudsonci.oraclecorp.com/jenkins/fmwsyseng_paas_central/computer/k8s-prod-cl1-pod1-v1v5j> (K8S_PROD_Cl1_lab1) in workspace /workspace/LayerManifestBuilder*java.lang.OutOfMemoryError: unable to create new native thread*
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at hudson.remoting.AtmostOneThreadExecutor.execute(AtmostOneThreadExecutor.java:96)

Caused: java.io.IOException: Remote call on JNLP4-connect connection from slc13qka.us.oracle.com/10.247.120.202:35679 failed
at hudson.remoting.Channel.call(Channel.java:913)
at hudson.FilePath.act(FilePath.java:987)
Caused: java.io.IOException: remote file operation failed: /scratch/syseng/workspace/LayerManifestBuilder at ***@12301016:JNLP4-connect connection from slc13qka.us.oracle.com/10.247.120.202:35679

Need to find the reason for the failure.

Regards,

Basanta

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/8f49b9a0-af12-475f-8ae6-afb0436c927d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jan Monterrubio

2018-11-11 21:02:21 UTC

Permalink

Cn you check the ulimit for your pod?

https://plumbr.io/outofmemoryerror/unable-to-create-new-native-thread

Post by p***@gmail.com
Hi All,
While running a job on Kuberentes Cluster
[EnvInject] - Loading node environment variables.
Building remotely on k8s-prod-cl1-pod1-v1v5j <http://hudsonci.oraclecorp.com/jenkins/fmwsyseng_paas_central/computer/k8s-prod-cl1-pod1-v1v5j> (K8S_PROD_Cl1_lab1) in workspace /workspace/LayerManifestBuilder*java.lang.OutOfMemoryError: unable to create new native thread*
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at hudson.remoting.AtmostOneThreadExecutor.execute(AtmostOneThreadExecutor.java:96)
Caused: java.io.IOException: Remote call on JNLP4-connect connection from slc13qka.us.oracle.com/10.247.120.202:35679 failed
at hudson.remoting.Channel.call(Channel.java:913)
at hudson.FilePath.act(FilePath.java:987)
Need to find the reason for the failure.
Regards,
Basanta
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/8f49b9a0-af12-475f-8ae6-afb0436c927d%40googlegroups.com
<https://groups.google.com/d/msgid/jenkinsci-users/8f49b9a0-af12-475f-8ae6-afb0436c927d%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CADgiF9LFYfDLW7pszHcPgLa2CEn-YwCi3%2BtRLD7Y7H6_QaHEkA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

p***@gmail.com

2018-11-12 05:34:24 UTC

Permalink

Hi ,
Thanks for your response.

on Ulimit -aH , I can see one difference on container and host

Container :
*open files (-n) 65536*

Slave Host :
*open files (-n) 327679*

Will that Causeing the OOM Failure .
Regards,
Basanta

Post by Jan Monterrubio
Cn you check the ulimit for your pod?
https://plumbr.io/outofmemoryerror/unable-to-create-new-native-thread

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1c3952a2-6c52-4b21-992f-ca14100c9d63%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

p***@gmail.com

2018-11-13 04:31:02 UTC

Permalink

The same job on rerun after some time passed on the same salve node..

Post by p***@gmail.com
Hi ,
Thanks for your response.
on Ulimit -aH , I can see one difference on container and host
*open files (-n) 65536*
*open files (-n) 327679*
Will that Causeing the OOM Failure .
Regards,
Basanta

Post by Jan Monterrubio
Cn you check the ulimit for your pod?
https://plumbr.io/outofmemoryerror/unable-to-create-new-native-thread

Post by p***@gmail.com
Hi All,
While running a job on Kuberentes Cluster
[EnvInject] - Loading node environment variables.
Building remotely on k8s-prod-cl1-pod1-v1v5j <http://hudsonci.oraclecorp.com/jenkins/fmwsyseng_paas_central/computer/k8s-prod-cl1-pod1-v1v5j> (K8S_PROD_Cl1_lab1) in workspace /workspace/LayerManifestBuilder*java.lang.OutOfMemoryError: unable to create new native thread*
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at hudson.remoting.AtmostOneThreadExecutor.execute(AtmostOneThreadExecutor.java:96)
Caused: java.io.IOException: Remote call on JNLP4-connect connection from slc13qka.us.oracle.com/10.247.120.202:35679 failed
at hudson.remoting.Channel.call(Channel.java:913)
at hudson.FilePath.act(FilePath.java:987)
Need to find the reason for the failure.
Regards,
Basanta
--
You received this message because you are subscribed to the Google
Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/8f49b9a0-af12-475f-8ae6-afb0436c927d%40googlegroups.com
<https://groups.google.com/d/msgid/jenkinsci-users/8f49b9a0-af12-475f-8ae6-afb0436c927d%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a50d0643-ca2e-43e4-9bdf-7267d83bd7db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

p***@gmail.com

2018-11-13 04:31:57 UTC

Permalink

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/7e469ceb-5a0a-4bbc-8c7c-13f1280a4465%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.