changing the way core files are stored in SmartOS VMs so that we can actually use them #492

misterdjules · 2016-09-15T19:05:54Z

This is related to nodejs/node#7649, where a test running on SmartOS made a node process abort, and thus made the system generate a core file.

That core file could have helped us root-causing the problem, but unfortunately it wasn't available anymore because by default core files are stored in the global zone of the server on which the VM runs, and they're deleted after one week.

Another problem with the default setup for core file storage with SmartOS test VMs is that only Triton cloud's operators can access core files stored in the server's global zone.

What we could do is set the configuration of every SmartOS test VM so that:

core files are stored somewhere that build WG members can access
core files are cleaned up after some time (e.g 1 week) by a cron job to avoid filling up space

This way, when a test failure happens due to a node process aborting on a SmartOS test VM, the person who ran the CI tests job can ask a member of the build WG to get the core file and e.g upload it to manta so that it can be inspected with mdb_v8.

Regardless of whether the file needs to be inspected, it would be deleted after a week, and not fill up space on test VMs.

Does that sound like a useful thing to do? If so I can help set it up, just let me know.

jbergstroem · 2016-09-15T19:09:11Z

How about configuring core dump directory to /home/iojs/dump or smth together with a cron job then? (find -mtime -delete)

jbergstroem · 2016-09-15T19:21:24Z

I'm not very well versed on coreadm, but If you have a suggested route wrt dumping to another directory I can update playbooks.

misterdjules · 2016-09-15T20:46:11Z

How about configuring core dump directory to /home/iojs/dump or smth together with a cron job then? (find -mtime -delete)

That's exactly what I'm suggesting, with the nit that I would choose a different name for this directory: /home/iojs/cores instead of /home/iojs/dump, but I don't have a strong opinion about it.

I'm not very well versed on coreadm, but If you have a suggested route wrt dumping to another directory I can update playbooks.

Running coreadm -g /home/iojs/cores/core.%f.%p -e global will enable global core files and will store them in /home/iojs/cores. It will also persist across reboots.

We should also make sure that ulimit -c outputs unlimited for the user under which the tests run.

jbergstroem · 2016-09-15T23:14:08Z

Cool. I'm actually in the process of setting up smartos15 and 16 hosts. I'll look at incorporating this in the playbook.

jbergstroem · 2016-09-29T01:46:27Z

Sorry for the delay here -- i'm a bit swamped at the moment. If anyone wants to chip in with improving playbooks for smartos14..16 that would be appreciated. I can spin up test machines if required.

Also, protip from @chorrell -- we can disable the SmartLogin (solving shared key access boundaries) by doing something in style with:

remove/comment PubKeyPlugin libsmartsshd.so in /etc/sshd/ssd_config
restart sshd (svcadm restart sshd)

jbergstroem · 2016-10-03T20:21:17Z

So, I spent some time on smartos today. Looks like we're running into issues with your openjdk8:

# /opt/local/java/openjdk8/bin/java -Xmx128m -jar slave.jar -jnlpUrl https://ci.nodejs.org/computer/test-joyent-smartos15-x64-1/slave-agent.jnlp -secret foo
Exception in thread "main" java.lang.Error: Error during hash calculation
        at sun.security.ssl.HandshakeHash.getFinishedHash(HandshakeHash.java:249)
        at sun.security.ssl.HandshakeMessage$Finished.getFinished(HandshakeMessage.java:1952)
        at sun.security.ssl.HandshakeMessage$Finished.<init>(HandshakeMessage.java:1899)
        at sun.security.ssl.ClientHandshaker.sendChangeCipherAndFinish(ClientHandshaker.java:1214)
        at sun.security.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:1134)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:348)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:914)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
        at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
        at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:269)
        at hudson.remoting.Launcher.run(Launcher.java:219)
        at hudson.remoting.Launcher.main(Launcher.java:192)
Caused by: java.lang.RuntimeException: Could not clone digest
        at sun.security.ssl.HandshakeHash.cloneDigest(HandshakeHash.java:194)
        at sun.security.ssl.HandshakeHash.getFinishedHash(HandshakeHash.java:247)
        ... 17 more
Caused by: java.lang.CloneNotSupportedException: SHA-384
        at sun.security.pkcs11.P11Digest.clone(P11Digest.java:316)
        at java.security.MessageDigest$Delegate.clone(MessageDigest.java:560)
        at sun.security.ssl.HandshakeHash.cloneDigest(HandshakeHash.java:191)
        ... 18 more
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_STATE_UNSAVEABLE
        at sun.security.pkcs11.wrapper.PKCS11.C_GetOperationState(Native Method)
        at sun.security.pkcs11.P11Digest.clone(P11Digest.java:311)
        ... 20 more

I don't have more time to look into that, but an up to date playbook is available in my repo.

jbergstroem · 2016-10-03T20:30:20Z

..same happens for smartos16.

chorrell · 2016-10-03T20:40:43Z

That's odd. I use a base-64-lts 15.4.0 with jenkins for Joyent image builds and openjdk8 works fine on that node. Is there more than one version of Java install, like a sun-jre package?

jbergstroem · 2016-10-03T20:52:10Z

@chorrell no, just openjdk8

chorrell · 2016-10-03T20:58:05Z

This might be relevant: https://www.illumos.org/issues/7227

chorrell · 2016-10-03T21:01:06Z

So maybe:

sun.security.pkcs11.enable-solaris=false

jbergstroem · 2016-10-03T21:57:25Z

@chorrell: "-Dsun.security.pkcs11.enable-solaris=false" is not a valid option

jbergstroem · 2016-10-03T22:00:07Z

@chorrell sorry, it does work -- I just messed up ordering.

jbergstroem · 2016-10-03T22:12:26Z

First run with 15,16 here: https://ci.nodejs.org/job/node-test-commit-smartos/4584/

jbergstroem · 2016-11-08T11:25:41Z

This has been implemented on all hosts and are available in the playbooks in my refactor.

misterdjules · 2017-02-02T19:52:20Z

@jbergstroem Thank you very much for your work! Should this issue be closed?

jbergstroem · 2017-02-02T20:01:35Z

I guess we could, but seeing how we still need to land my PR it would be slightly misleading?

misterdjules · 2017-02-02T20:18:21Z

seeing how we still need to land my PR it would be slightly misleading

What PR are you referring to?

gibfahn · 2017-02-02T20:21:40Z

I assume this one: #606

misterdjules · 2017-02-02T20:34:31Z

@gibfahn Thanks for the context!

Let's not close this issue until that PR is merged then.

misterdjules · 2017-05-25T22:58:54Z

#606 was merged a while ago, so closing. Thank you very much @jbergstroem!

misterdjules changed the title ~~changing the way core files are stored in SmartOS VMs so that we can use actually use them~~ changing the way core files are stored in SmartOS VMs so that we can actually use them Sep 16, 2016

jbergstroem mentioned this issue Oct 3, 2016

SmartOS cannot build on master nodejs/v8#1

Closed

misterdjules mentioned this issue Feb 2, 2017

test: add --abort-on-timeout option to test.py nodejs/node#11086

Closed

2 tasks

misterdjules closed this as completed May 25, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

changing the way core files are stored in SmartOS VMs so that we can actually use them #492

changing the way core files are stored in SmartOS VMs so that we can actually use them #492

misterdjules commented Sep 15, 2016

jbergstroem commented Sep 15, 2016

jbergstroem commented Sep 15, 2016

misterdjules commented Sep 15, 2016

jbergstroem commented Sep 15, 2016

jbergstroem commented Sep 29, 2016

jbergstroem commented Oct 3, 2016

jbergstroem commented Oct 3, 2016

chorrell commented Oct 3, 2016 •

edited

Loading

jbergstroem commented Oct 3, 2016

chorrell commented Oct 3, 2016

chorrell commented Oct 3, 2016

jbergstroem commented Oct 3, 2016

jbergstroem commented Oct 3, 2016

jbergstroem commented Oct 3, 2016

jbergstroem commented Nov 8, 2016

misterdjules commented Feb 2, 2017

jbergstroem commented Feb 2, 2017

misterdjules commented Feb 2, 2017

gibfahn commented Feb 2, 2017

misterdjules commented Feb 2, 2017

misterdjules commented May 25, 2017

changing the way core files are stored in SmartOS VMs so that we can actually use them #492

changing the way core files are stored in SmartOS VMs so that we can actually use them #492

Comments

misterdjules commented Sep 15, 2016

jbergstroem commented Sep 15, 2016

jbergstroem commented Sep 15, 2016

misterdjules commented Sep 15, 2016

jbergstroem commented Sep 15, 2016

jbergstroem commented Sep 29, 2016

jbergstroem commented Oct 3, 2016

jbergstroem commented Oct 3, 2016

chorrell commented Oct 3, 2016 • edited Loading

jbergstroem commented Oct 3, 2016

chorrell commented Oct 3, 2016

chorrell commented Oct 3, 2016

jbergstroem commented Oct 3, 2016

jbergstroem commented Oct 3, 2016

jbergstroem commented Oct 3, 2016

jbergstroem commented Nov 8, 2016

misterdjules commented Feb 2, 2017

jbergstroem commented Feb 2, 2017

misterdjules commented Feb 2, 2017

gibfahn commented Feb 2, 2017

misterdjules commented Feb 2, 2017

misterdjules commented May 25, 2017

chorrell commented Oct 3, 2016 •

edited

Loading