Road potholes are more than just an inconvenience.

They can damage vehicles, make journeys uncomfortable, and create safety risks for people using the road every day. Yet identifying a pothole is only one part of understanding the problem.

What happens after a pothole is detected?

Where exactly is it located? Which road segment does it belong to? Which authority might be responsible for maintaining that road? Is there a related road contract or tender? And how could this information help people understand road conditions before choosing a route?

These questions led me to start building PotholeNet.

The Idea Behind PotholeNet

PotholeNet is a road-infrastructure intelligence project that begins with computer vision and aims to connect detections with useful geographic and infrastructure information.

The initial goal is straightforward: build a model that can identify potholes in road images.

But I don't want the project to stop at drawing bounding boxes around damaged road surfaces.

I want to explore how those detections could become structured information about roads, maintenance responsibilities, and potential route risks.

The broader vision looks something like this:

Pothole Detection → Geographic Location → Road Segment → Responsible Authority → Contract or Tender Information → Route Risk

Each step introduces a different engineering challenge.

Detecting a pothole is a computer vision problem. Connecting it to a road segment requires geospatial data. Identifying a potentially responsible authority requires reliable infrastructure records. Matching road damage to contracts or tenders introduces another layer of data integration.

Not every pothole will have enough available information to support every step. Part of the challenge is determining what can be established reliably and where uncertainty remains.

That is what makes this project interesting to me.

Starting With Computer Vision

The first major component of PotholeNet is its machine learning service.

I'm using a fine-tuned YOLO11n object detection model to identify potholes in road images.

The model produces detections containing bounding boxes and confidence scores. These outputs provide the initial signal that the rest of the system can eventually use.

The current ML workflow includes:

• Preparing and validating the dataset.
• Fine-tuning a pretrained detection model.
• Evaluating model performance.
• Inspecting prediction errors.
• Exporting the trained model to ONNX.
• Serving predictions through a FastAPI service.

I've also been learning that training a model is only one part of building a useful ML system.

A model can produce promising evaluation metrics and still struggle with unfamiliar road conditions. Understanding those failures, improving the data, and testing the inference pipeline are just as important as the training process itself.

For me, this is where the project starts becoming more than a model-training exercise.

Building the ML Service

I wanted the detection model to remain separate from the main application backend.

The ML service uses Python, FastAPI, and ONNX Runtime for inference. The main application is being developed with Node.js and Express, with React on the frontend.

The intended architecture allows the backend to send an image to the ML service and receive structured detection results in return.

This separation makes it easier to develop and improve the model without tightly coupling machine learning code to the rest of the application.

It also creates room to experiment with model formats and inference performance independently.

The model is trained locally using an NVIDIA GeForce RTX 5060. The deployed inference service, however, runs on CPU-based infrastructure using ONNX Runtime.

That distinction has been a useful reminder that training a model efficiently and serving a model efficiently are two different engineering problems.

The Challenges Go Beyond Detection

Even if the detector becomes reliable, several important questions remain.

1. Where is the pothole?

An image can show road damage, but the system also needs geographic context to understand where that damage occurred.

This is where geospatial tools and road-network data become important.

2. Which road does it belong to?

Coordinates alone aren't enough. A useful system needs to associate a detection with the appropriate road segment, while accounting for the limitations of location data and map matching.

3. Who is responsible for maintaining that road?

Road ownership and maintenance responsibilities can be fragmented across different authorities.

PotholeNet will need to investigate available public data and develop evidence-based associations rather than assume every detection can be mapped automatically to a single responsible authority.

4. Can the information help people make better decisions?

The longer-term goal is to explore how road-condition information could support route-risk insights and a dashboard that makes the collected information easier to understand.

These capabilities are part of the broader vision, not claims that the entire system is already implemented.

I want to build toward them incrementally, beginning with a reliable detection pipeline.

Why I'm Building It This Way

One of my goals with PotholeNet is to work across several areas of software engineering in one coherent project:

• Machine learning: object detection, fine-tuning, evaluation, and error analysis.
• Backend engineering: APIs, service boundaries, and data processing.
• Geospatial systems: coordinates, road networks, and geographic queries.
• Data engineering: dataset validation and integration of external data sources.
• Full-stack development: building an interface that makes the results useful.

Rather than treating these as isolated technologies, I want to understand how they fit together in a system that addresses a real-world problem.

I'm also trying to keep the development process honest.

A feature isn't complete just because a model produces a prediction. A prediction isn't automatically reliable just because it has a confidence score. And a planned architecture isn't the same as a working implementation.

I want PotholeNet to grow through small, testable steps, with clear distinctions between what works, what needs improvement, and what remains to be built.

What's Next?

My immediate focus is improving the pothole detector and making the inference pipeline more reliable.

After that, the next milestones include integrating the ML service with the main backend, representing detections as structured events, and exploring how those events can be associated with road segments.

From there, I want to investigate authority mapping, public contract and tender data, and the possibility of deriving useful route-risk information.

These are substantial challenges, particularly where public infrastructure data is incomplete or inconsistent. I'll approach them one step at a time rather than assume the answers are already available.

This Is Just the Beginning

PotholeNet started with a simple question:

Can I build a system that detects potholes?

Now, the question is becoming more ambitious:

Can I turn those detections into reliable, useful information about our roads?

I don't have every part figured out yet, and that's part of the journey.

I'll be documenting the process as I build, experiment, encounter problems, and improve the system — from computer vision and model deployment to geospatial data and backend integration.

For now, the focus is on getting the foundations right.

One model. One service. One step at a time.

Welcome to the PotholeNet journey. 🚧