Category: AI

3 weeks of agentic coding

use gh and tell me how many PRs I have authored the last 3 weeks. from these PRs how many are merged? what was the total of added and removed lines?

Your PR Statistics (Last 3 Weeks)
 - Total PRs authored: 31
 - Merged PRs: 27 (87% merge rate)
 - Total lines added: 4,095
 - Total lines removed: 2,758
 - Total lines changed: 6,853
You've been quite productive! Most of your PRs from the last 3 weeks are merged, with a solid mix of refactoring and feature work.

Up until now the main way of using LLMs was through chat. Every time I wanted to do a deep dive on a technical issue I was starting a conversation through which I was getting a better understanding both on the code and the way it should be used.

I had played with the agent tab of copilot in android studio but never given it too much time. That was the reason I believed that it is not that good!

So, I decided to give it a real shot and use it foul time, on production code, in a project that is important. The goal was to write as little code as possible.

The workflow I ended up having

  1. Understand the task at hand and create a mental model of the solution.
  2. Figure out the steps I need to make in order to implement the solution. If the steps are too many I break them into groups.
  3. Start writing these [group of] steps in a prompt where I ask the agent to provide me a plan with the intended changes.
  4. Review the plan, ask the agent to make adjustments (repeat this step as many times as needed).
  5. Ask the agent to save the plan in a markdown file.
  6. Ask the agent to execute the plan.
  7. Review the changes.
  8. If something trivial needs to be changed I do it myself, if the change cascades through many files I tell the agent to do it.
  9. In the second case I also request an update to the plan.
  10. Final review, commit and push.

The prompts must not be too detailed but also not too general. For example:

Take a look at <file #1> and <file #2> and give me a plan with all needed changes in order to:
   1. Start <component #1> as disabled
   2. Enable it every time the user selects an address (<component #2>) or
   3. Enable it every time the user is typing a new zipcode (<component #3>)

Mistakes

It goes without saying that to end up in the above workflow I did many mistakes. Here are the big ones.

Provide the outcome

At first my prompts were a simple description of the outcome I wanted. I thought it will figure things out, make the necessary connections and write exactly what we need. Nope. The agent knows what you allow it to and when it can’t find something it simply creates random solutions.

Straight to the execution

My interaction with the agent was starting by asking it to do something. No plan at all. In simple cases this might be fine but when having a change that touches many components, a simple adjustment, after the agent’s work, might end up in updating a lot of code or in more adjustments.

Getting greedy, asking too much

After making some progress and saw how effective I was I got greedy. I started asking too much from the start and ended up with massive PRs that included changes often unrelated to each other.

Tips

Always have a plan first

  • For me having a plan gives me ease. I am more certain that things will be done as intended because they will be done they way I want to!
  • Through the process of making the plan there will be times that you will understand better the code at hand and figure out missing cases.
  • Especially for repetitive tasks the plan speeds things tremendously:
    I had to migrate a few screens from one pattern to another. I did the first migration using the agent (through a plan etc) and when finished I asked it to change the plan in such a way that will accept “parameters”. After that I just fed the updated plan with the next screen to the agent.
  • It is a memory that can be fed in any agent, in a clean context window, at any time.

Use the agent to figure things out

Some times in order to build the mental model for the solution you need to understand the code better. Use the agent to do that. See how it articulates things and then ask it to save its findings in a file. That file can be part of the plan:

see how component A works by reading file <name>

Always review the code

Perhaps the most important tip of all. Don’t add code to the project that you don’t know what it does. Always review what the agent did. Make sure that it follows the project’s conventions and standards. The fact that it was written by an agent does not mean that it is not your code. You are responsible for it. It is your solution, you just used a different medium to implement it.

Explore more, it is fast now

The benefit of having a tool that implements your thoughts way faster than you is that you can explore multiple solutions! Use git to make different branches/checkpoints and try every approach you thought of.

Keep things small

You can use an agent to implement an entire task but if you break it and do groups of changes then your reviews will be easier and quicker which means that your understanding of the changes will be better.

Bonus

I keep a repo with the Gilded Rose kata. Every now and then I create a new branch and practice on the kata.

This time the practice required to use only an agent. You can see the branch here and the prompts I used here (i asked the agent to save them to a file).

Using Ollama and Kotlin to migrate multiple files into a new library

At work, there is a need to migrate our project from using LoganSquare to kotlinx.serialization.

Part of the work involves replacing the annotations the first library is using with the ones from the second. Unfortunately some cases are not as simple as replacing foo with boo. For example, a property must be annotated with @JsonField(name = ["a_name_here"]) in LoganSquare and @SerialName("a_name_here") is kotlinx.

So, I had to decide:

  1. Do I spend 2-3 hours and migrate 100+ files manually, one by one?
  2. Do I cut the hours in half by using the search and replace tool and then fix anything the tool couldn’t manage?
  3. Do I start a journey of unknown number of hours to figure out how to perform the migration using a local LLM?

Ollama

Yeap, I chose to go with number three! And to do that I started by installing Ollama. Ollama is a tool that allows you to download open source LLMs and start playing with them locally. All prompts are handled in your device and there is no network activity.

You can either download it from its site or, if you are on macOS, use brew: brew install ollama.

After that you can run one of the models it provides, ex: ollama run llama3.2
or fire up the server it comes with and start playing with its API: ollama serve

Kotlin

The flow is simple:

  • Load in memory, one by one, the contents of the files that must be migrated
  • Provide each content along side with a prompt to an LLM
  • Store the LLM’s result to the file
  • (optional) Start dancing for building your first LLM based workflow

Reading the contents and writing them back to the files is easy with Kotlin. Communicating with the ollama server is also easy when using OkHttp and kotlinx.serialization. Believe it or not the most time consuming part was figuring out the prompt!

After a lot of attempts the one prompt that managed to produced the best result was the one where I listed the steps that I would have done manually:

We have a file written in Kotlin and we need to migrate it from LoganSquare to KotlinX Serialization.

To do that we have to replace:
- "import com.bluelinelabs.logansquare.annotation.JsonField" with "import kotlinx.serialization.SerialName"
- "import com.bluelinelabs.logansquare.annotation.JsonObject" with "import kotlinx.serialization.Serializable"
- "@JsonObject\ninternal class <class name>" with "@Serializable\ninternal class <class name>"
- "@JsonObject\nclass <class name>" with "@Serializable\nclass <class name>"
- "@JsonField(name = ["<property name>"])" with "@SerialName("<property name>")"

Everything else in the file should be copied without any changes.

Please migrate the following file:
$contents

We just want the file. Don't comment on the result.

and even then, small details did matter a lot.

For example, at the beginning of the prompt I refer to a file but later in the text I was saying Please migrate the following class. That alone was resulting in various weird migrations where a class was either missing completely or had only half of its initial code. Same results when I wasn’t using \n after the annotations.

The code

import gr.le0nidas.kotlin.ollama.OllamaClient
import gr.le0nidas.kotlin.ollama.request.GenerateRequest
import gr.le0nidas.kotlin.ollama.request.parameter.Model
import gr.le0nidas.kotlin.ollama.request.parameter.Prompt
import java.io.File
fun main(args: Array<String>) {
val ollamaClient = OllamaClient()
val requestBuilder = GenerateRequest.Builder(Model("llama3.2"))
val files = getAllFilePaths(args[0])
files.forEach { file ->
println("- Migrating file $file…")
val content = getFileContents(file)
val request = requestBuilder.build(prompt = createPrompt(content))
val response = ollamaClient.generate(request)
response
.onSuccess { saveFileContents(file, it.value) }
.onFailure { println(it.message) }
}
}
fun createPrompt(contents: String) = Prompt(
"""
We have a file written in Kotlin and we need to migrate it from LoganSquare to KotlinX Serialization.
To do that we have to replace:
– "import com.bluelinelabs.logansquare.annotation.JsonField" with "import kotlinx.serialization.SerialName"
– "import com.bluelinelabs.logansquare.annotation.JsonObject" with "import kotlinx.serialization.Serializable"
– "@JsonObject\ninternal class <class name>" with "@Serializable\ninternal class <class name>"
– "@JsonObject\nclass <class name>" with "@Serializable\nclass <class name>"
– "@JsonField(name = ["<property name>"])" with "@SerialName("<property name>")"
Everything else in the file should be copied without any changes.
Please migrate the following file:
$contents
We just want the file. Don't comment on the result.
""".trimIndent()
)
fun getAllFilePaths(directoryPath: String): List<String> {
val directory = File(directoryPath)
if (!directory.exists() || !directory.isDirectory) {
println("Directory does not exist or is not a directory")
return emptyList()
}
return directory.listFiles()
?.filter { it.isFile && it.name.endsWith(".kt") }
?.map { it.path }
?.toList()
?: emptyList()
}
fun getFileContents(filePath: String): String {
return try {
File(filePath).readText()
} catch (e: Exception) {
println("Error reading file: ${e.message}")
""
}
}
fun saveFileContents(filePath: String, contents: String) {
val file = File(filePath)
file.writeText(contents)
}
view raw ollama-kotlin-playground.kt hosted with ❤ by GitHub

Conclusion

Was I faster than choice number two? Didn’t try this choice but I guess no. Too many things to learn, figure out and write.
Do I regret it? No! I now have a new tool in my belt and I’m pretty sure it will pay off, time wise, in the future.

ollama-kotlin-playground

One more thing that came out of this endeavour is ollama-kotlin-playground. A very simple library that does only one thing: generate a completion without even supporting all possible parameters. It is my way of not copying code from one tool/experiment to another.