{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":509968990,"defaultBranch":"master","name":"bookkeeper","ownerLogin":"HQebupt","currentUserCanPush":false,"isFork":true,"isEmpty":false,"createdAt":"2022-07-03T08:47:57.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/4970972?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1722430250.0","currentOid":""},"activityList":{"items":[{"before":"be59f209763d549ce4b7acd7b4246c5e97035923","after":"0371d753cef73c0d3efd6757af27acbdaae2fbac","ref":"refs/heads/readOnly","pushedAt":"2024-07-31T12:53:25.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"HQebupt","name":"Qiang Huang","path":"/HQebupt","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4970972?s=80&v=4"},"commit":{"message":"read code","shortMessageHtmlLink":"read code"}},{"before":"868563c6bd2429d16c065b62e15a6a61d40a5415","after":"be59f209763d549ce4b7acd7b4246c5e97035923","ref":"refs/heads/readOnly","pushedAt":"2024-07-31T12:52:14.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"HQebupt","name":"Qiang Huang","path":"/HQebupt","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4970972?s=80&v=4"},"commit":{"message":"read code","shortMessageHtmlLink":"read code"}},{"before":"c9f86b2b9a5872ede076fb5663b0c78f0b489a35","after":"868563c6bd2429d16c065b62e15a6a61d40a5415","ref":"refs/heads/readOnly","pushedAt":"2024-07-31T12:51:20.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"HQebupt","name":"Qiang Huang","path":"/HQebupt","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4970972?s=80&v=4"},"commit":{"message":"read code","shortMessageHtmlLink":"read code"}},{"before":null,"after":"c9f86b2b9a5872ede076fb5663b0c78f0b489a35","ref":"refs/heads/readOnly","pushedAt":"2024-07-31T12:50:50.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"HQebupt","name":"Qiang Huang","path":"/HQebupt","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4970972?s=80&v=4"},"commit":{"message":"read code","shortMessageHtmlLink":"read code"}},{"before":"7c0e16b160eba2a4d6a39c50c67790e6a2588377","after":"999cd0f2ab14404be4d6c24e388456dbe56bb1a8","ref":"refs/heads/master","pushedAt":"2024-07-31T07:39:49.000Z","pushType":"push","commitsCount":79,"pusher":{"login":"HQebupt","name":"Qiang Huang","path":"/HQebupt","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4970972?s=80&v=4"},"commit":{"message":"Fix ReadOnlyLedgerHandle leak issue when checkAllLedgers. (#4468)\n\nWhen the Auditor checkAllledgers, it will open the ledger with NoRecovery mode, it will register the listeners to the AbstractZkLedgerManager#listeners. \r\n\r\nThe listener won't be removed if we don't close the ReadOnlyLedgerHandle, so there will be lots of listeners in the heap memory.\r\n\r\n\r\nThe heap dump:\r\n\r\n<img width=\"1572\" alt=\"image\" src=\"https://github.com/user-attachments/assets/8b4cd6d3-b7bf-4ef2-9c47-2e076bdbbba0\">","shortMessageHtmlLink":"Fix ReadOnlyLedgerHandle leak issue when checkAllLedgers. (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2425137774\" data-permission-text=\"Title is private\" data-url=\"https://github.com/apache/bookkeeper/issues/4468\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/apache/bookkeeper/pull/4468/hovercard\" href=\"https://github.com/apache/bookkeeper/pull/4468\">apache#4468</a>)"}},{"before":"01c91493b4d3425d5d0b28b40040948d343b5256","after":"7c0e16b160eba2a4d6a39c50c67790e6a2588377","ref":"refs/heads/master","pushedAt":"2024-05-13T12:35:57.000Z","pushType":"push","commitsCount":42,"pusher":{"login":"HQebupt","name":"Qiang Huang","path":"/HQebupt","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4970972?s=80&v=4"},"commit":{"message":"[site][bugfix] BK website version 4.17.0, missing sidebar (#4363)","shortMessageHtmlLink":"[site][bugfix] BK website version 4.17.0, missing sidebar (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2290943168\" data-permission-text=\"Title is private\" data-url=\"https://github.com/apache/bookkeeper/issues/4363\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/apache/bookkeeper/pull/4363/hovercard\" href=\"https://github.com/apache/bookkeeper/pull/4363\">apache#4363</a>)"}},{"before":"808b909edf597254aea212c287fe17fb6d808198","after":"01c91493b4d3425d5d0b28b40040948d343b5256","ref":"refs/heads/master","pushedAt":"2024-04-24T03:27:44.000Z","pushType":"push","commitsCount":185,"pusher":{"login":"HQebupt","name":"Qiang Huang","path":"/HQebupt","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4970972?s=80&v=4"},"commit":{"message":"chore: fix some typos in package org.apache.bookkeeper.bookie.storage (#4313)","shortMessageHtmlLink":"chore: fix some typos in package org.apache.bookkeeper.bookie.storage (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2258765874\" data-permission-text=\"Title is private\" data-url=\"https://github.com/apache/bookkeeper/issues/4313\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/apache/bookkeeper/pull/4313/hovercard\" href=\"https://github.com/apache/bookkeeper/pull/4313\">…</a>"}},{"before":null,"after":"89c24b8b2bd07783a4ea2c66e9bc07181c6ffeab","ref":"refs/heads/addReadRateLimterInCompaction","pushedAt":"2023-05-31T08:09:56.404Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"HQebupt","name":"Qiang Huang","path":"/HQebupt","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4970972?s=80&v=4"},"commit":{"message":"[improve] add read rate limter for compaction","shortMessageHtmlLink":"[improve] add read rate limter for compaction"}},{"before":"ac7e9092aebe9fc7e7e16832ae3fc1d2bbd91fc4","after":"808b909edf597254aea212c287fe17fb6d808198","ref":"refs/heads/master","pushedAt":"2023-05-31T07:51:53.159Z","pushType":"push","commitsCount":172,"pusher":{"login":"HQebupt","name":"Qiang Huang","path":"/HQebupt","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4970972?s=80&v=4"},"commit":{"message":"clear channel when channelInactive (#3966)\n\n### Motivation\r\n\r\nI am stopping a pulsar broker, and I observed a lot of the following logs in the broker log:\r\n\r\n```\r\n17:48:35.894 [shutdown-thread-43-1] INFO  org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 10.184.xx.xx:3181\r\n17:48:35.894 [shutdown-thread-43-1] ERROR io.netty.util.concurrent.DefaultPromise.rejectedExecution - Failed to submit a listener notification task. Event loop shut down?\r\njava.util.concurrent.RejectedExecutionException: event executor terminated\r\n        at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:923) ~[io.netty-netty-common-4.1.66.Final.jar:4.1.66.Final]\r\n        at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:350) ~[io.netty-netty-common-4.1.66.Final.jar:4.1.66.Final]\r\n        at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:343) ~[io.netty-netty-common-4.1.66.Final.jar:4.1.66.Final]\r\n        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:825) ~[io.netty-netty-common-4.1.66.Final.jar:4.1.66.Final]\r\n        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:815) ~[io.netty-netty-common-4.1.66.Final.jar:4.1.66.Final]\r\n        at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:842) ~[io.netty-netty-common-4.1.66.Final.jar:4.1.66.Final]\r\n        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:499) ~[io.netty-netty-common-4.1.66.Final.jar:4.1.66.Final]\r\n        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184) ~[io.netty-netty-common-4.1.66.Final.jar:4.1.66.Final]\r\n        at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95) ~[io.netty-netty-transport-4.1.66.Final.jar:4.1.66.Final]\r\n        at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30) ~[io.netty-netty-transport-4.1.66.Final.jar:4.1.66.Final]\r\n        at org.apache.bookkeeper.proto.PerChannelBookieClient.closeChannel(PerChannelBookieClient.java:1090) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]\r\n        at org.apache.bookkeeper.proto.PerChannelBookieClient.closeInternal(PerChannelBookieClient.java:1079) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]\r\n        at org.apache.bookkeeper.proto.PerChannelBookieClient.close(PerChannelBookieClient.java:1063) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]\r\n        at org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.close(DefaultPerChannelBookieClientPool.java:157) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]\r\n        at org.apache.bookkeeper.proto.BookieClientImpl.close(BookieClientImpl.java:587) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]\r\n        at org.apache.bookkeeper.client.BookKeeper.close(BookKeeper.java:1435) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]\r\n        at org.apache.pulsar.broker.ManagedLedgerClientFactory.close(ManagedLedgerClientFactory.java:142) ~[org.apache.pulsar-pulsar-broker-2.8.1.jar:2.8.1]\r\n        at org.apache.pulsar.broker.PulsarService.closeAsync(PulsarService.java:417) ~[org.apache.pulsar-pulsar-broker-2.8.1.jar:2.8.1]\r\n        at org.apache.pulsar.broker.MessagingServiceShutdownHook.lambda$run$1(MessagingServiceShutdownHook.java:62) ~[org.apache.pulsar-pulsar-broker-2.8.1.jar:2.8.1]\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]\r\n        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.66.Final.jar:4.1.66.Final]\r\n        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]\r\n```\r\n\r\nWhen I stopped the broker, there were more than 30,000 lines in such a log, which seriously polluted my normal log of the broker.\r\n<img width=\"1180\" alt=\"image\" src=\"https://github.com/apache/bookkeeper/assets/35599757/0ec3c423-14f1-41c4-85c3-d978ece15044\">\r\n\r\nAccording to my next investigation,\r\n1. When the broker stops, it will close the ManagedLedgerClientFactory. It will close all bookkeeper connections\r\n\r\nhttps://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/DefaultPerChannelBookieClientPool.java#L154\r\n```java\r\n    @Override\r\n    public void close(boolean wait) {\r\n        for (int i = 0; i < clients.length; i++) {\r\n            clients[i].close(wait);\r\n            if (clients != clientsV3Enforced) {\r\n                clientsV3Enforced[i].close(wait);\r\n            }\r\n        }\r\n    }\r\n\r\n    private void closeInternal(boolean permanent, boolean wait) {\r\n        Channel toClose = null;\r\n        synchronized (this) {\r\n            if (permanent) {\r\n                state = ConnectionState.CLOSED;\r\n            } else if (state != ConnectionState.CLOSED) {\r\n                state = ConnectionState.DISCONNECTED;\r\n            }\r\n            toClose = channel;\r\n            channel = null;\r\n            makeWritable();\r\n        }\r\n        if (toClose != null) {   <== a. toClose not null\r\n            ChannelFuture cf = closeChannel(toClose);\r\n            if (wait) {\r\n                cf.awaitUninterruptibly();\r\n            }\r\n        }\r\n    }\r\n\r\n    private ChannelFuture closeChannel(Channel c) {\r\n        if (LOG.isDebugEnabled()) {\r\n            LOG.debug(\"Closing channel {}\", c);\r\n        }\r\n        return c.close().addListener(x -> makeWritable());  <==  b. here if channel is already inactive, add the listener will throw exception\r\n    }\r\n```\r\n\r\n\r\n2. When the bookie server first disconnects the client, the client will trigger channelInactive\r\n```java\r\n    public void channelInactive(ChannelHandlerContext ctx) throws Exception {\r\n        LOG.info(\"Disconnected from bookie channel {}\", ctx.channel());\r\n        if (ctx.channel() != null) {\r\n            closeChannel(ctx.channel());  <== c. here channel is closed\r\n            if (ctx.channel().pipeline().get(SslHandler.class) != null) {\r\n                activeTlsChannelCounter.dec();\r\n            } else {\r\n                activeNonTlsChannelCounter.dec();\r\n            }\r\n        }\r\n\r\n        errorOutOutstandingEntries(BKException.Code.BookieHandleNotAvailableException);\r\n        errorOutPendingOps(BKException.Code.BookieHandleNotAvailableException);\r\n\r\n        synchronized (this) {\r\n            if (this.channel == ctx.channel()\r\n                && state != ConnectionState.CLOSED) {\r\n                state = ConnectionState.DISCONNECTED;\r\n            }\r\n        }\r\n\r\n        // we don't want to reconnect right away. If someone sends a request to\r\n        // this address, we will reconnect.\r\n    }\r\n```\r\n\r\nIf step c happens before b, then b throws the above exception.\r\n\r\nstep c\r\n<img width=\"1400\" alt=\"image\" src=\"https://github.com/apache/bookkeeper/assets/35599757/4a33855c-5682-4812-ae0e-1a02972ef502\">\r\n<img width=\"1407\" alt=\"image\" src=\"https://github.com/apache/bookkeeper/assets/35599757/1c99bccb-b2bd-4cfb-ac4b-ee10b2d236b0\">\r\n\r\n\r\nstep a and b\r\n<img width=\"1403\" alt=\"image\" src=\"https://github.com/apache/bookkeeper/assets/35599757/fec372c5-f97b-47e7-b5d7-ebbaa6edf682\">\r\n\r\nSo when channelInactive, we should reset the channel to null after closed the channel.\r\nHere, when the channel is null, we will reconnect after getting the client, which is consistent with the current behavior.\r\n![image](https://github.com/apache/bookkeeper/assets/35599757/d1178697-27bd-44a5-acd1-33ccec99e7a4)","shortMessageHtmlLink":"clear channel when channelInactive (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1720003459\" data-permission-text=\"Title is private\" data-url=\"https://github.com/apache/bookkeeper/issues/3966\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/apache/bookkeeper/pull/3966/hovercard\" href=\"https://github.com/apache/bookkeeper/pull/3966\">apache#3966</a>)"}}],"hasNextPage":false,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"Y3Vyc29yOnYyOpK7MjAyNC0wNy0zMVQxMjo1MzoyNS4wMDAwMDBazwAAAASOD5kv","startCursor":"Y3Vyc29yOnYyOpK7MjAyNC0wNy0zMVQxMjo1MzoyNS4wMDAwMDBazwAAAASOD5kv","endCursor":"Y3Vyc29yOnYyOpK7MjAyMy0wNS0zMVQwNzo1MTo1My4xNTk4MzJazwAAAAM3mMbh"}},"title":"Activity · HQebupt/bookkeeper"}