[JEP-210] Log handling rewrite #15

jglick · 2016-09-23T22:49:27Z

Downstream of jenkinsci/workflow-api-plugin#17. Subsumes #75.

basics
fix rendering of console output from Pipeline Steps
~~way to efficiently get all output from a branch~~
retain compatibility if only this plugin is updated and not workflow-job
optimize AnnotatedLogAction.getLogText
optimize AnnotatedLogAction.strip

jglick · 2016-09-23T22:53:22Z

Not going to bother deploying upstream snapshots at this stage, so CI failures are expected.

… to be usable from LogAction. (Core-defined notes do use Object, explicitly or implicitly via rawtypes.)

…al LogActionImpl. Includes further encapsulation than that started in jenkinsci#10. Also applying ConsoleLogFilter in new mode.

…epContext to AnnotatedLogAction.

…tations.

Using jitpack for jenkinsci/workflow-api-plugin#17, and picking up jenkinsci/maven-hpi-plugin#46.

…p a deployed snapshot.

Also using jitpack and linking to jenkinsci/maven-hpi-plugin#46 & jenkinsci/workflow-api-plugin#17 & jenkinsci/workflow-support-plugin#15.

…able-task-plugin#29 & jenkinsci/workflow-api-plugin#17 & jenkinsci/workflow-support-plugin#15 & jenkinsci/workflow-job-plugin#27

svanoort · 2018-08-08T15:28:12Z

@jglick Does this fix the issue we discussed months ago where it requires reading the entire logfile to retrieve the fragment for a single step?

jglick · 2018-08-08T16:18:28Z

reading the entire logfile to retrieve the fragment for a single step

That has since been refactored into jenkinsci/workflow-api-plugin#17 where it now lives in StreamLogStorage.stepLog. Since streaming a single file is relatively cheap for I/O, and the filtering is simple, that is just left as is to keep things simple. Of course if you are using, e.g., CloudWatch Logs there is a completely different implementation (in that case using server-side filtering).

Rendering speed is anyway not a priority for JEP-210, compared to optimizing writing during the build. (discussion)

jglick · 2018-08-08T17:46:38Z

src/main/java/org/jenkinsci/plugins/workflow/support/actions/AnnotatedLogAction.java

+ * A marker for a node which had some log text.
+ */
+@Restricted(NoExternalUse.class) // for use from DefaultStepContext only
+public class AnnotatedLogAction extends LogAction implements FlowNodeAction, PersistentAction {


This should be renamed.

… had used LogAction, it must continue to do so (now teeing to the overall log).

src/main/java/org/jenkinsci/plugins/workflow/support/actions/LogStorageAction.java

jglick · 2018-08-22T10:39:34Z

src/main/java/org/jenkinsci/plugins/workflow/support/actions/LogStorageAction.java

+    public static @Nonnull TaskListener listenerFor(@Nonnull FlowNode node, @CheckForNull ConsoleLogFilter filter) throws IOException, InterruptedException {
+        FlowExecutionOwner owner = node.getExecution().getOwner();
+        if (LogActionImpl.isOld(owner) || node.getAction(LogActionImpl.class) != null) {
+            return LogActionImpl.stream(node, filter);


Probably the !FlowNode.active closing logic from LogActionImpl should be moved here—for some implementations it may be significant (though not for FileLogStorage). Also rather than PrintStream.close it should check for TaskListener & AutoCloseable. And the expected close behavior for both the overall and step logs should be documented in LogStorage.

…support LogStorageAction, and handling TaskListener & AutoCloseable.

src/main/java/org/jenkinsci/plugins/workflow/support/actions/LogStorageAction.java

svanoort · 2018-09-12T18:19:44Z

src/main/java/org/jenkinsci/plugins/workflow/support/DefaultStepContext.java

+                    @Override public void onNewHead(FlowNode newNode) {
+                        if (!node.isActive()) {
+                            node.getExecution().removeListener(this);
+                            LOGGER.log(Level.FINE, "closing log for {0}", node.getDisplayFunctionName());


For a good time, try to disentangle what all the xxFunctionName/xxDisplayName function for a FlowNode do and which to use when.

And by a "good time" I mean "if you're a masochist."

svanoort · 2018-09-12T18:25:50Z

src/main/java/org/jenkinsci/plugins/workflow/support/DefaultStepContext.java

@@ -97,6 +99,33 @@
        }
    }

+    private synchronized TaskListener getListener() throws IOException, InterruptedException {


Synchronizing is /probably/ the right move here but makes me nervous we're going to hit threading issues (as has happened with analogous changes in WorkflowRun with bottlenecks and deadlocks). Nothing I can point to explicitly wrong here, but something to keep a close eye on.

…e is expected, send it to the overall build log not the step log. This allows us to call context.get(EnvVars.class) even on a FlowNode which has not yet been attached to the graph.

…e is expected, send it to the overall build log not the step log. This allows us to call context.get(EnvVars.class) even on a FlowNode which has not yet been attached to the graph. (cherry picked from commit d5d1f46)

…S-38381

jglick · 2019-01-18T18:46:10Z

src/main/java/org/jenkinsci/plugins/workflow/support/DefaultStepContext.java

@@ -62,17 +67,14 @@
        if (key == EnvVars.class) {
            Run<?,?> run = get(Run.class);
            EnvironmentAction a = run == null ? null : run.getAction(EnvironmentAction.class);
-            EnvVars customEnvironment = a != null ? a.getEnvironment() : run.getEnvironment(get(TaskListener.class));
+            EnvVars customEnvironment = a != null ? a.getEnvironment() : run.getEnvironment(getExecution().getOwner().getListener());


Amended in #62 FTR.

[JENKINS-38381] Prototype of log handling rewrite.

3e85272

jglick added the work-in-progress label Sep 23, 2016

jglick mentioned this pull request Sep 23, 2016

[JEP-210] Log handling rewrite jenkinsci/workflow-job-plugin#27

Merged

11 tasks

jglick added 10 commits September 26, 2016 15:05

Pick up jenkinsci/maven-hpi-plugin#25.

ffac76d

Merge branch 'master' into logs-JENKINS-38381

b70ada1

In practice it seems the type parameter of ConsoleNote must be Object…

87bba53

… to be usable from LogAction. (Core-defined notes do use Object, explicitly or implicitly via rawtypes.)

If we are using an old WorkflowRun with copyLogs, fall back to origin…

fed6abd

…al LogActionImpl. Includes further encapsulation than that started in jenkinsci#10. Also applying ConsoleLogFilter in new mode.

Lighter-weight way of marking which node printed a line.

aef8545

Some Javadoc, and tightening up access control.

710bb87

Rewrote AnnotatedLogAction.decorate to handle remotable listeners.

a8b1f2d

FindBugs

596c2f8

2.16 parent

f7a1743

Seems cleaner to move LogActionImpl compatibility code from DefaultSt…

dfd971d

…epContext to AnnotatedLogAction.

recampbell assigned oleg-nenashev and unassigned oleg-nenashev Oct 5, 2016

jglick added 3 commits October 5, 2016 18:21

LessAbstractTaskListener

97d30f9

API to expose the origin of node text in the HTML log.

158619f

Working around a bug in ConsoleLogFilter.

3a03cf9

jglick mentioned this pull request Oct 11, 2016

[JENKINS-38370] Start defining APIs that are for the master JVM only jenkinsci/jenkins#2556

Merged

jglick added 2 commits October 14, 2016 14:02

LessAbstractTaskListener is likely needed by PipelineLogFile implemen…

5a86013

…tations.

Should also offer a Closeable implementation.

a348381

jglick mentioned this pull request Oct 18, 2016

No messages printed from block-scoped steps after they invoked their bodies #19

Merged

jglick added 4 commits October 19, 2016 11:16

Merge branch 'master' into logs-JENKINS-38381

549b446

Merge branch 'master' into logs-JENKINS-38381

32acc6f

Using jitpack for jenkinsci/workflow-api-plugin#17, and picking up jenkinsci/maven-hpi-plugin#46.

Simpler configuration; no need to use pluginRepository just to pick u…

cf6f11f

…p a deployed snapshot.

Added TODO comment.

2138c6b

jglick added a commit to jglick/workflow-job-plugin that referenced this pull request Dec 7, 2016

Merge branch 'master' into logs-JENKINS-38381

211bd14

Also using jitpack and linking to jenkinsci/maven-hpi-plugin#46 & jenkinsci/workflow-api-plugin#17 & jenkinsci/workflow-support-plugin#15.

hpi-plugin.version=1.121

30b52c3

jglick changed the title ~~[JENKINS-38381] Log handling rewrite~~ [JEP-210] Log handling rewrite Aug 6, 2018

Merge branch 'master' into logs-JENKINS-38381

27a34dc

jglick commented Aug 8, 2018

View reviewed changes

jglick added 4 commits August 13, 2018 18:12

Bump.

628846b

Renamed AnnotatedLogAction to LogStorageAction.

338afdd

Forgot to rename Stapler view of AnnotatedLogAction → LogStorageAction.

8f47806

If during an upgrade we are resuming a build with a running step that…

a3d0646

… had used LogAction, it must continue to do so (now teeing to the overall log).

dwnusbaum reviewed Aug 15, 2018

View reviewed changes

src/main/java/org/jenkinsci/plugins/workflow/support/actions/LogStorageAction.java Show resolved Hide resolved

dwnusbaum self-requested a review August 15, 2018 21:09

jglick commented Aug 22, 2018

View reviewed changes

jglick added 3 commits August 23, 2018 14:48

Centralizing support for closing step logs in DefaultStepContext, to …

b589b8f

…support LogStorageAction, and handling TaskListener & AutoCloseable.

Merge branch 'master' into logs-JENKINS-38381

07c85e5

Bump.

b249afd

This was referenced Sep 4, 2018

updateProperties failed to honor the updateNonincremental flag jenkinsci/incrementals-tools#10

Merged

[JENKINS-45693] Calling TaskListenerDecorator #72

Merged

dwnusbaum reviewed Sep 6, 2018

View reviewed changes

src/main/java/org/jenkinsci/plugins/workflow/support/actions/LogStorageAction.java Show resolved Hide resolved

svanoort reviewed Sep 12, 2018

View reviewed changes

dwnusbaum approved these changes Sep 24, 2018

View reviewed changes

svanoort approved these changes Sep 26, 2018

View reviewed changes

jglick added 2 commits October 3, 2018 14:11

If there is any output from Run.getEnvironment(TaskListener), and non…

d5d1f46

…e is expected, send it to the overall build log not the step log. This allows us to call context.get(EnvVars.class) even on a FlowNode which has not yet been attached to the graph.

jglick mentioned this pull request Oct 4, 2018

Send any output from Run.getEnvironment(TaskListener) to the overall build, not step, log #75

Merged

Merge branch 'DefaultStepContext+Run.getEnvironment' into logs-JENKIN…

93b9b13

…S-38381

jglick merged commit 0dfa35c into jenkinsci:master Oct 4, 2018

jglick deleted the logs-JENKINS-38381 branch October 4, 2018 20:54

jglick mentioned this pull request Oct 8, 2018

[JENKINS-54078] JEP-210 compatibility jenkinsci/workflow-basic-steps-plugin#74

Merged

jglick commented Jan 18, 2019

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[JEP-210] Log handling rewrite #15

[JEP-210] Log handling rewrite #15

jglick commented Sep 23, 2016 •

edited

Loading

jglick commented Sep 23, 2016

svanoort commented Aug 8, 2018

jglick commented Aug 8, 2018 •

edited

Loading

jglick Aug 8, 2018

jglick Aug 22, 2018

jglick Aug 23, 2018

svanoort Sep 12, 2018

svanoort Sep 12, 2018

jglick Jan 18, 2019

[JEP-210] Log handling rewrite #15

[JEP-210] Log handling rewrite #15

Conversation

jglick commented Sep 23, 2016 • edited Loading

jglick commented Sep 23, 2016

svanoort commented Aug 8, 2018

jglick commented Aug 8, 2018 • edited Loading

jglick Aug 8, 2018

Choose a reason for hiding this comment

jglick Aug 22, 2018

Choose a reason for hiding this comment

jglick Aug 23, 2018

Choose a reason for hiding this comment

svanoort Sep 12, 2018

Choose a reason for hiding this comment

svanoort Sep 12, 2018

Choose a reason for hiding this comment

jglick Jan 18, 2019

Choose a reason for hiding this comment

jglick commented Sep 23, 2016 •

edited

Loading

jglick commented Aug 8, 2018 •

edited

Loading