Disable rate_limit check on GitHub Enterprise completely

[djdefi]

GitHub Enterprise prior to 2.10 does not have rate limiting, requests to https://hostname/api/v3/rate_limit will return a 404 Not Found status code. in 2.10+ rate limiting can be enabled, but is disabled by default.

Based on the change in #78 there is a check for rate_limit, and then the rate limit is set to an arbitrary high number. On very busy systems this results in tens of thousands of requests per hour to the GitHub Enterprise appliance. Although returning a 404 is a relatively simple operation, there are other things such as logging that are impacted by the volume of requests.

A better method would be to look for /api/v3/ within the configured API URL, and then simply skip the rate limit check altogether. /api/v3/ is unique to GitHub Enterprise, so this will not have any impact on the github.com rate_limit endpoint of https://api.github.com/rate_limit

[KostyaSha]

The better way would be ask gh to make better way of differentiation. imho

Sent from my iPad

On Apr 19, 2016, at 3:06 AM, Ryan Trauntvein notifications@github.com wrote:

As GitHub Enterprise does not enforce rate limiting, requests to https://hostname/api/v3/rate_limit will return a 404 Not Found status code.

Based on the change in #78 there is a check for rate_limit, and then the rate limit is set to an arbitrary high number. On very busy systems this results in tens of thousands of requests per hour to the GitHub Enterprise appliance. Although returning a 404 is a relatively simple operation, there are other things such as logging that are impacted by the volume of requests.

A better method would be to look for /api/v3/ within the configured API URL, and then simply skip the rate limit check altogether. /api/v3/ is unique to GitHub Enterprise, so this will not have any impact on the github.com rate_limit endpoint of https://api.github.com/rate_limit

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub

[djdefi]

/api/v3/ in the URL is the differentiator here. Or the URL not being api.github.com

[recena]

@KostyaSha There are things it can be verified or checked in order to know if you are in front of a github.com or an instance of GitHub Enterprise.

To me, the best way to verify it is asking by the server header. If this header is unique for github.com.

[kohsuke]

I'm not sure what's reported as a problem.

Nothing in this library is doing thrate limit check, so I'm not sure what you mean by "simply skip the rate limit check altogether." Perhaps the request is that the getRateLimit method does not fake a result but pass through an error, but if so I'm not sure what good it serves.

If the goal is that this library provides a method to help you differentiate github.com vs GHE, then I agree with @recena that there are better ways to do that.

[kohsuke]

No further input from the reporter.

[djdefi]

The check I am referring to is here:

https://github.com/kohsuke/github-api/blob/a9fb4546e19e453766a46256c43232ed2ea35e6a/src/main/java/org/kohsuke/github/GitHub.java#L258-L274

Each return retrieve().to("/rate_limit", JsonRateLimit.class).rate; is a request to the GitHub Enterprise instance, which returns 404. This works sure, but on busy Jenkins systems this accounts for a huge number of extra requests, which still needs to be processed (consuming CPU), and fills up logs with noise, etc.

Let me know if I can provide any further information.

[djdefi]

@kohsuke

No further input from the reporter.

Not sure if you saw my feedback in the previous comment https://github.com/kohsuke/github-api/issues/273#issuecomment-238005230☝️

Please let me know if I can provide any further information.

[v1v]

I'm afraid I've seen this behaviour too in one of my jenkins instances:

• Jenkins version: 1.651.1
• GitHub API plugin: 1.71 (although I've just updated to 1.77)
• Github Pull Request plugin: 1.29

Sep 12, 2016 11:37:24 AM org.jenkinsci.plugins.ghprb.GhprbRepository initGhRepository
SEVERE: Error while accessing rate limit API
org.kohsuke.github.HttpException: Server returned HTTP response code: 401, message: 'Unauthorized' for URL: https://github./api/v3/rate_limit
    at org.kohsuke.github.Requester.parse(Requester.java:540)
    at org.kohsuke.github.Requester._to(Requester.java:251)
    at org.kohsuke.github.Requester.to(Requester.java:213)
    at org.kohsuke.github.GitHub.getRateLimit(GitHub.java:263)
    at org.jenkinsci.plugins.ghprb.GhprbRepository.initGhRepository(GhprbRepository.java:71)
    at org.jenkinsci.plugins.ghprb.GhprbRepository.init(GhprbRepository.java:47)
    at org.jenkinsci.plugins.ghprb.GhprbRepository.getGitHubRepo(GhprbRepository.java:334)

± |jira_es_14127 ✓| → curl https://github/api/v3/rate_limit -v
*   Trying X.Y.Z.D...
* Connected to github (X.Y.Z.D) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.sto
* Server certificate: DigiCert SHA2 Secure Server CA
* Server certificate: DigiCert Global Root CA
> GET /api/v3/rate_limit HTTP/1.1
> Host: github
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Server: GitHub
< Date: Mon, 12 Sep 2016 09:15:29 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 122
< Status: 404 Not Found
< Cache-Control: no-cache
< X-GitHub-Media-Type: github.v3
< Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
< Access-Control-Allow-Origin: *
< X-GitHub-Request-Id: e5214e1e-5de9-45ab-ab5d-da2fa930d306
< Content-Security-Policy: default-src 'none'
< Strict-Transport-Security: max-age=31536000; includeSubdomains
< X-Content-Type-Options: nosniff
< X-Frame-Options: deny
< X-XSS-Protection: 1; mode=block
< 
{
  "message": "Rate limiting is not enabled.",
  "documentation_url": "https://developer.github.com/enterprise/2.6/v3"
}
* Connection #0 to host github left intact

I've just fixed by upgrading:

•  github                    => '1.21.1',
  

•  github_api              => '1.77',
  

•  ghprb                     => '1.33.1',
  

The above http 401/404 errors are not prompting when restarting Jenkins often but every 1/2 minutes

Sep 12, 2016 12:13:00 PM org.jenkinsci.plugins.ghprb.GhprbRepository initGhRepository
SEVERE: Error while accessing rate limit API
org.kohsuke.github.HttpException: Server returned HTTP response code: 401, message: 'Unauthorized' for URL: https://github/api/v3/rate_limit
    at org.kohsuke.github.Requester.parse(Requester.java:540)
    at org.kohsuke.github.Requester._to(Requester.java:251)
    at org.kohsuke.github.Requester.to(Requester.java:213)
    at org.kohsuke.github.GitHub.getRateLimit(GitHub.java:263)
    at org.jenkinsci.plugins.ghprb.GhprbRepository.initGhRepository(GhprbRepository.java:98)
    at org.jenkinsci.plugins.ghprb.GhprbRepository.check(GhprbRepository.java:130)
    at org.jenkinsci.plugins.ghprb.GhprbTrigger.run(GhprbTrigger.java:294)
    at hudson.triggers.Trigger.checkTriggers(Trigger.java:272)
    at hudson.triggers.Trigger$Cron.doRun(Trigger.java:221)
    at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Server returned HTTP response code: 401 for URL: https://github/api/v3/rate_limit
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1675)
    at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1673)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1671)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1244)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
    at org.kohsuke.github.Requester.parse(Requester.java:524)
    ... 16 more
Caused by: java.io.IOException: Server returned HTTP response code: 401 for URL: https://github/api/v3/rate_limit
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1626)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
    at org.kohsuke.github.Requester.parse(Requester.java:514)
    ... 16 more

[djdefi]

https://issues.jenkins.io/browse/JENKINS-63078 implemented this.