If this is causing you a problem, it will cause your putative ecosystem of developers a problem (e.g. when they try to develop an alternative UI). If you are really eating your own dog food, make the API (and the rate limiting) work for your application. Here are some suggestions:
Do not rate limit by IP address. Rather, rate limit by something associated with the user, e.g. their user ID. Apply the rate limit at the authentication stage.
Design your API so that users do not need to call it continuously (e.g. give a list call that returns many results, rather than a repeated call that returns one item each time)
Design your web app with the same constraints you expect your developer ecosystem to have, i.e. ensure you can design it within reasonable throttling rates.
Ensure your back end is scalable (horizontally preferably) so you don't need to impose throttling at levels so low it actually causes a problem to a UI.
Ensure your throttling has the ability to cope with bursts, as well as limiting longer term abuse.
Ensure your throttling performs sensible actions tailored to the abuse you are seeking to remove. For instance, consider queuing or delaying mild abusers rather than refusing the connection. Most web front ends will only open four simultaneous connections at once. If you delay an attempt to open a fifth you'll only hit the case where they are using a CLI at the same time as the web client (ot two web clients). If you delay the n-th API call without a gap rather than failing it, the end user will see things slow down rather than break. If you combine this with only queuing N API calls at once, you will only hit people who are parallelising large numbers of API calls, which is probably not the behaviour you want - e.g. 100 simultaneous API calls then a gap for an hour is normally far worse than 100 sequential API calls over an hour.
Did this not answer your question? Well, if you really need to do what you are asking, rate-limit at the authentication stage and apply a different rate limit based on the group your user fits into. If you are using one set of credentials (used by your devs and QA team), you get a higher rate limit. But you can immediately see why this will inevitably lead you to your ecosystem seeing issues that your dev and QA team do not see.
My company has developed a rate-limited API. Our goal is twofold:
- A: Create a strong developer ecosystem around our product.
- B: Demonstrate the power of our API by using it to drive our own application.
Clarification: Why rate-limit at all?
We rate limit our API, because we sell it as an addition to our product. Anonymous access to our API has a very low threshold for API calls per hour, whereas our paid customers are permitted upwards of 1000 calls per hour or more.
So the question is:
How do you secure an api so that rate-limiting can be removed where in the process in removing such rate-limiting can't be easily spoofed?
Explored Solutions (and why they didn't work)
Verify the referrer against the host header. -- Flawed because the referrer is easily faked.
Proxy the request and sign the request in the proxy -- Still flawed, as the proxy itself exposes the API.
I am looking to the brilliant minds on Stack Overflow to present alternate solutions. How would you solve this problem?
Rate limiting an API with a special exception
Since a webapp cannot hide anything to its users, I don't think there is a way to ensure that the application talking to your API is yours.
Everything you can do is security through obscurity, which is by definition unsecure.
I would have a solution, still not secure, but that would be an idea.
The first thing is to have two authentication system.
- Third party authentication (API token)
- Web app authentication (API token + Web token)
Then it's a bit like your idea. You generate a token for each HTTP session. The difference would be that web app (calls authenticate as such) would have a lower limit. (eg. 10k for 3rd party and 1k for web app). The reason why you should decrease the number of calls authorized is because a single HTTP session (1 person behind it) should generates less API calls than a third party app. Is it your case?
Of course the third party application could easily authenticate itself as a web app, but then it would have to renew its session more quickly, so you can monitor the number of sessions per IP. Just be aware that universities and companies are often a lot behind a single IP.