If you have a ChatGPT subscription, I found that you can get some really good insights from having o1 models (including o1 pro) review your code. To make it faster I've created a rake command that automatically aggregates context on the codebase (including full database schema and routes), and provides the git diff, and custom instructions to get some feedback. The command then copies to the clipboard the output and you're ready to paste everything into ChatGPT.
Scroll down to the end of the article if you want to copy the command right away. Follow along for a quick explanation of the prompt we're building and what output to expect, including with o1 pro.
How the rake task works
The task is basically split into 3 main parts: Context on the codebase, changes, and instructions.
For the context, ideally you would just provide the entire codebase (and I'm sure we'll get there at one point), but that's not always possible or cost-effective. What I found best is to provide the routes (excluding everything generated by Rails itself) and the full schema (without migration IDs at the end). I then add a few other data points that are less important.
For the context, ideally you would just provide the entire codebase (and I'm sure we'll get there at one point), but that's not always possible or cost-effective. What I found best is to provide the routes (excluding everything generated by Rails itself) and the full schema (without migration IDs at the end). I then add a few other data points that are less important.
rails_version = Rails::VERSION::STRING environment = Rails.env current_branch = `git rev-parse --abbrev-ref HEAD`.strip last_5_commits = `git log -5 --pretty=format:'%h %s (%ci)'` routes_output = `bin/rails routes | grep -v "^.*\\(/rails\\|/applications\\).*$"` schema_output = `sed '/INSERT INTO "schema_migrations"/q' db/structure.sql` migrations_status = `bin/rails db:migrate:status:primary | tail -n 5` lang-ruby
Getting the changes depends on your use case. I currently work mostly alone so I'm interested in everything that has been staged but not yet committed. You could change the command to for example get the diff between the current branch and master.
The only notable change is that I request 100 lines around each line changed. This usually provides the entire file which gives essential context to o1.
diff_output = `git diff HEAD -U100` lang-ruby
Finally, I give a few instructions to guide the llm.:
Please carefully review the provided code changes. Look for: - Potential bugs or regressions - Code that may cause performance issues - Unclear naming, logic, or structure that could be improved - Missing test coverage or scenarios not handled - Areas where Ruby on Rails best practices are not followed - Security vulnerabilities, SQL injections, or unsafe handling of params - Any other subtle issues that the developer might have overlooked Provide comprehensive feedback addressing both high-level design and detailed implementation issues. Do not hesitate to provide code examples on how to fix the issues you raise. Order your feedback by severity, starting with the most severe issues. lang-markdown
What results to expect
Overall, considering it takes 10 seconds to ask for feedback (assuming you do not wait around for o1 to be finished thinking), the output is really good. You still have to review your code, and a review from a senior developer will further help, but it's a valuable, cost-effective, and time-efficient first step.
Here are some screenshots of the thinking process and examples of output:
Here are some screenshots of the thinking process and examples of output:
I've tried the command with both o1-preview, o1, and o1-pro. The difference between o1 and o1 pro is quite slim for most requests, although the pro version did solve a few problems o1 missed. Even though you do get an increased context window with the Pro subscription, I'm not sure the extra cost is worth it.
The rake command
# Add the command to /lib/tasks/llm_review.rake in your in a Ruby on Rails application.
# Run with: rake llm:review
# Otherwise, ask ChatGPT to convert the language of your current codebase ;)
namespace :llm do
desc "Prepare a code review snippet by including relevant Rails context, last 5 commits, and the current git diff"
task review: :environment do
rails_version = Rails::VERSION::STRING
environment = Rails.env
current_branch = `git rev-parse --abbrev-ref HEAD`.strip
last_5_commits = `git log -5 --pretty=format:'%h %s (%ci)'`
routes_output = `bin/rails routes | grep -v "^.*\\(/rails\\|/applications\\).*$"`
schema_output = `sed '/INSERT INTO "schema_migrations"/q' db/structure.sql`
migrations_status = `bin/rails db:migrate:status:primary | tail -n 5`
diff_output = `git diff HEAD -U100`
final_content = <<~TEXT
Please carefully review the provided code changes. Look for:
- Potential bugs or regressions
- Security vulnerabilities, SQL injections, or unsafe handling of params
- Code that may cause performance issues
- Unclear naming, logic, or structure that could be improved
- Areas where Ruby on Rails best practices are not followed
- Potential for DRYing up the code or other great refactoring opportunities
- Essential missing test coverage or scenarios not handled
- Any other subtle issues that the developer might have overlooked
Provide comprehensive feedback addressing both high-level design
and detailed implementation issues. Do not hesitate to provide code examples on
how to fix the issues you raise.
Order your feedback by severity, starting with the most severe issues.
### Application Context
- Rails Version: #{rails_version}
- Environment: #{environment}
- Current Branch: #{current_branch}
### Last 5 Commits
#{last_5_commits}
### Routes
#{routes_output}
### Schema
#{schema_output}
### Migrations Status
#{migrations_status}
### Git Diff (HEAD)
#{diff_output}
TEXT
IO.popen("pbcopy", "w") { |io| io.print final_content }
puts "Code review content with suggested prompt copied to clipboard!"
end
end
lang-ruby
💬 Comments