Devin AI rate limiter test: Full Cycle's Wesley Willians built one in Go with PRs, reviews and 429 responses. See what held up.
Devin AI rate limiter: what the test actually proved
Devin AI rate limiter work is feasible today, but only under human supervision. In a session published on 14 February 2025, Full Cycle founder Wesley Willians asked Devin AI to build a Go rate limiter as an HTTP middleware, and the agent produced a repository, documentation, tests and pull requests inside the recording.
The session ran about 62 minutes. Willians supplied a written specification covering storage adapters, an in-memory default, Docker Compose, slog logging and a mandatory README. Devin then implemented it and opened PRs without Willians writing application code himself.
Devin AI is an autonomous software engineering agent sold by Cognition. It runs in a cloud workspace, reads a linked GitHub repository, executes shell commands and opens pull requests rather than editing the developer's local machine.
Why the specification mattered more than the model
The specification, not the model, decided whether the output was usable. Willians wrote the requirements in Notion pasted them into the Devin chat, defining the middleware boundary, the persistence adapter interface, default limits, the HTTP header and the documentation rule.
He stated plainly in the video that he had run other tests where the specification was weaker, and in those cases Devin behaved like a junior developer. He also called his own demonstration the happy path and warned against deploying an unsupervised agent into an existing codebase.
That framing matters because the interesting failure later in the video was not a hallucinated API. It was a defensible but unwanted design decision about how concurrency and request counting interact, which is exactly the kind of choice a written spec has to settle in advance.
The practical requirements the spec pinned down were:
- Rate limiter exposed only as HTTP middleware, decoupled from any specific server framework
- Modular architecture so future extensions and adapters can be added
- IP address as the rate-limiting key, with a configurable window and limit
- Pluggable storage, starting with in-memory and later Redis
- Go 1.23 as the container image version, Docker for local execution
- All documentation in a README, one pull request per unit of work
What Devin AI produced in the first pass
Devin AI delivered a working first pass that included source, tests, documentation and an example server. Willians reported that the agent had initialised the Go module, created the Dockerfile and docker-compose file, implemented a storage interface with an in-memory backend, wired the middleware and generated the README.
One instructive correction happened early. Devin used Go 1.21 in the Dockerfile, and Willians told it the spec said 1.23. The agent acknowledged the mistake and changed the file. That is a low-stakes example of the review loop the whole workflow depends on.
Devin also maintains a knowledge base and offered to record a rule that every future change must update the README. Willians accepted. The rule then applied to subsequent tasks, which is how project conventions persist across separate chat sessions.
The pull request description included implementation details, setup instructions and load-test results. Willians merged the first PR before validating the code locally, a decision he discusses on camera.
The concurrency bug that exposed the limits
Under concurrency, the rate limiter blocked requests it should have allowed, and Devin defended the behaviour. Willians ran a load test with 50 concurrent workers and 150 requests, and observed 20 responses with status 200 and 130 blocked with status 429, even though the configured limit was 100 requests per minute.
He asked Devin why requests beyond the cap were blocked rather than only the excess. Devin answered that once a client exceeds 100 requests per minute, all subsequent requests are blocked immediately, not only those over the limit, so that clients receive a clear back-off signal.
Willians then repeated the test sequentially, one request at a time. That run produced exactly 100 successful responses and 50 blocked ones, which confirmed the limit logic was correct and the problem was the concurrency handling, not the counter.
His diagnosis was that a burst of simultaneous requests crossed the threshold before the earlier ones finished processing, so the limiter rejected everything in flight. He asked Devin to make the operation atomic so the first 100 requests are processed and only the rest are rejected, and to OpenAI new pull request because the first had already been merged.
How the fix landed: atomics and a read-write mutex
Devin AI fixed the concurrency defect by replacing a global mutex and making the start-time and counter updates atomic. Willians asked for a proper read-write lock so the guard covers the whole operation rather than a single mutex section.
Devin created a notes file listing the race condition, the required changes and the test strategy, then changed the start time to an atomic value and reworked the storage implementation. The second pull request removed the global mutex, moved to atomic operations, changed the blocking logic and added high-concurrency test cases.
The reported result after the fix was 100 processed requests and the remainder blocked, which is the behaviour Willians had asked for. He merged the second pull request as well.
A third task followed: adding GitHub Actions CI so tests must pass on every pull request and a load test runs as part of the pipeline. The first CI run failed during container setup, likely because the Alpine-based image lacked the tooling the load test needed, and Devin adjusted the Dockerfile and updated the pull request.
Redis as a second storage adapter
Devin AI added a Redis storage backend after the in-memory version passed its tests. Willians asked for environment variables to configure the connection, a Redis service in Docker Compose for local runs, testcontainers for automated tests, and an example server under an examples directory using the new backend.
Devin's implementation used Redis increment operations for atomicity and a time-to-live for the counting window, and documented the required environment variables in the README. The knowledge base entry it proposed required every new storage implementation to ship an example backend and environment-variable configuration.
This stage produced a genuine misalignment. Devin relied on testcontainers and did not add a Redis service to the Compose file, so Willians could not run the example locally as documented. He asked for the service to be added and a new pull request to be opened.
Devin AI vs GitHub Copilot: different jobs
Devin AI and GitHub Copilot operate at different levels of the workflow. GitHub Copilot completes code inside the editor and leaves the branch, the tests, the documentation and the pull request to the developer. Devin takes a written task, works in a cloud workspace with shell and repository access, and returns a pull request for review.
That difference explains why the Full Cycle session produced so much peripheral material. The rate limiter source code is a small part of the output; the README, the example server, the load-test scripts and the CI workflow account for most of what Devin delivered.
Neither tool removes the review step. Willians merged two pull requests he had not validated locally, and the concurrency defect only surfaced because he ran his own load test afterwards.
What a real Devin AI deployment requires
Devin AI works inside a supervised loop, and the failing step in a real deployment is usually the specification. Willians summarised the pattern: the agent acts like a developer of any level, and if you do not explain the workflow and the rules, it will pick the simplest implementation path it can find.
Treat the following as the minimum operating discipline before pointing an agent at a production repository:
- Write the requirements, interfaces and defaults before starting the task, including how over-limit behaviour should work.
- Link repositories with least privilege and keep a human owner for every merge.
- Require one pull request per unit of work so review stays granular.
- Run your own load or end-to-end test rather than trusting the agent's reported results.
- Check that generated environment configuration matches the documented run steps, as the missing Redis service showed.
The autonomy claim in Devin's marketing should be read against this session. Willians compared each Devin chat to a separate developer and noted you could open many at once, but every one of them needed a human to write the spec, catch the concurrency defect and decide what counted as correct.
No compliance or security guarantee follows from this workflow. Running an agent with repository and shell access is a privileged operation, and the controls around secrets, network access and code provenance belong to the application and the organisation, not to the tool.
Frequently asked questions
- Is Devin AI free to try? No. In the 14 February 2025 Full Cycle video, Willians states that Devin has no trial and starts at $500 per month. Check Devin's pricing page for the current figure, because it has changed since the recording.
- What programming language was used in the Devin AI test? The project used Go, with the container pinned to Go 1.23. Devin also wrote a Dockerfile, a docker-compose file and an example HTTP server in the same language.
- Did Devin AI write the pull requests itself? Yes. Devin opened the pull requests from its own workspace, including descriptions with setup steps and load-test results, and Willians reviewed and merged them from GitHub.
- Why did the rate limiter block too many requests? A burst of concurrent requests crossed the 100-requests-per-minute threshold before earlier ones finished processing, so the limiter rejected everything in flight rather than only the excess.
- How was the concurrency bug fixed? Devin removed the global mutex, made the counter and window start atomic, and added a read-write lock so the whole operation is guarded. High-concurrency test cases were added to the pull request.
- Does Devin AI replace developers? The session shows it replacing specific tasks under supervision, not the role. Someone still had to write the specification, review the pull requests, run independent tests and catch the defect Devin initially defended.
- What was the Nubank claim in the video? Willians references a Devin case study about Nubank rewriting an eight-year-old monolithic system, reporting 12x engineering-hours efficiency and over 20x development cost savings. Those figures are vendor-reported, and the method behind them is not shown.
- Can I run the project locally? Yes. The README documents docker compose up for the in-memory example and a Redis-backed example, plus a load-test command, though the missing Redis service had to be added to Compose first.
- What still needs human review? Concurrency semantics, environment configuration, CI behaviour and any claim the agent makes about its own test results. Willians did not inspect the code line by line, and he says so on camera.
Turning agent sessions into written knowledge
The lesson from this session is not that an agent wrote a Go rate limiter. It is that a written specification drove every useful outcome, and a missed concurrency detail cost a full review cycle. If you already record that kind of knowledge on video, the hard part is turning a long session into something a reader can search and cite.
Skala Blog takes a YouTube URL, transcribes the video and generates a structured article from it.