Beyond the Black Box: Building AI Tools That Actually Help Investigators and Analysts

Custom AI Tools | Local AI | Investigative Dashboards | Analyst Productivity

I've spent a good portion of my career helping investigators and analysts make sense of information. From my early work in military intelligence and supporting intelligence agencies to teaching open-source intelligence (OSINT) to thousands of law enforcement professionals, I've seen the same problem over and over again: we have more information available than ever before, but not necessarily more time or better ways to make sense of it.

Now we have artificial intelligence entering the picture, and the possibilities are pretty exciting. But I'm also seeing organizations make some of the same mistakes they made with earlier technologies. They're buying tools before they've figured out what they actually need those tools to accomplish.

At Plessas Experts Network, Inc., we've been exploring a different approach. Instead of asking investigators and analysts to adapt their work to whatever an AI product happens to offer, we're building tools around the work they already do.

Sometimes that means developing a sophisticated local AI assistant. Other times, it's a simple HTML dashboard or a small utility that takes a repetitive task off someone's plate.

The goal isn't to make AI do the investigation. It's to give the investigator more time to actually investigate.

AI Doesn't Replace Good Tradecraft

One of the things I emphasize in my OSINT classes is that finding information and understanding information are two very different skills.

You can have access to the best databases, search tools, social media platforms, and analytical software in the world, and still arrive at the wrong conclusion if you don't understand what you're looking at.

AI doesn't magically fix that.

In fact, it can make the problem worse because it can produce a very convincing explanation that is completely wrong. Anyone who has spent time working with large language models has probably experienced this. The AI confidently provides an answer, sometimes complete with references or explanations, and then you discover that part of it was invented.

For investigators, that's not a minor inconvenience. A mistaken association, an inaccurate timeline, or an unsupported conclusion can affect an entire investigation.

That's why I look at AI as an assistant, not an analyst.

I want it to help organize information, identify potential relationships, find inconsistencies, suggest investigative leads, and highlight what might be missing. I don't want it deciding what the evidence means without someone checking its work.

And I certainly don't want it inventing evidence.

The analyst needs to be able to trace conclusions back to their sources, question the AI's suggestions, and disregard anything that doesn't hold up to scrutiny.

That principle guides the AI tools we're developing.

Why I'm So Interested in Local AI

Most people encounter AI through a cloud service. They type something into ChatGPT or another platform, upload documents, and receive an answer.

For everyday work, that's often perfectly reasonable.

But what if you're handling confidential investigative material, sensitive business records, attorney work product, or information that you're not authorized to share with an external provider?

That's where local AI becomes particularly interesting.

Tools such as Ollama and LM Studio allow us to run language models directly on our computers. Depending on the model and the hardware, we can perform some remarkably useful tasks without sending our information to an outside AI service.

I've been experimenting extensively with these systems, including building local tools on Apple Silicon hardware and developing interfaces that make the models easier to use.

What interests me most isn't simply that the AI runs locally. It's that we can customize how it works.

We can create specialized assistants for particular investigative tasks, connect them to locally stored documents, and build interfaces that reflect how an investigator actually approaches a problem.

We can also make these tools available to people who have no interest in learning command-line programming.

Of course, installing Ollama doesn't automatically make a computer secure. We still need to consider encryption, access controls, backups, software dependencies, and whether any part of a workflow communicates with the internet. A genuinely local-first tool needs to be designed and tested with those requirements in mind.

But having the option to keep sensitive processing on a device we control is a significant advantage.

And perhaps more importantly, it gives smaller organizations the opportunity to develop capabilities that previously seemed reserved for companies with enormous technology budgets.

Custom AI Should Start With the Problem, Not the Product

When someone asks me about using AI, I generally want to know what they're trying to accomplish before we start talking about models or software.

What's taking too much time? Where are people duplicating effort? What information is difficult to locate? Which parts of a process require human judgment, and which are mostly mechanical?

Those questions usually tell us a lot more than a list of AI features.

For example, an investigator might spend hours turning case notes into a report. A corporate security team might have information scattered across spreadsheets, internal records, websites, and different monitoring platforms. An analyst might be manually comparing dozens of sources to see whether a particular name, location, or event appears in more than one place.

Not all of these problems require an elaborate AI system.

Sometimes a searchable dashboard, an automated comparison tool, or a better way to organize the existing data delivers more value than a large language model.

One of the advantages of the development tools available today is that we can build small, highly specialized applications without investing months in traditional software development.

I also encourage investigators to experiment with what people are calling vibe coding—using AI to help write software based on natural-language instructions.

You still need to test the code, understand what it is doing, and pay attention to security. But you no longer have to be a professional programmer to start creating useful investigative utilities.

That's an important change for our field.

Making Investigative Data Easier to Work With

A good dashboard doesn't need to look like the control room of a spaceship.

It needs to answer useful questions.

If I'm investigating a network of online accounts, I might want to see which usernames overlap, what platforms they've appeared on, and which associations are supported by evidence.

If I'm reviewing a fraud investigation, I might need to compare dates, transactions, communications, and locations to identify inconsistencies or gaps.

If I'm managing a large collection of OSINT resources, I want to find the right source quickly without digging through hundreds of bookmarks.

We've been developing tools around these types of needs.

Some are designed to organize and filter investigative resources. Others help generate username variations, launch searches across multiple platforms, or arrange information in a way that makes it easier for the investigator to see relationships.

These smaller tools can be especially useful because they're portable. In some cases, we can build a complete application that runs from a single HTML file on a computer.

No expensive subscription. No complicated deployment. No requirement that the user become a software developer.

And when the task genuinely benefits from AI, we can add a local model to assist with classification, summarization, document comparison, or investigative gap analysis.

The important thing is that the technology serves the question we're trying to answer.

The Part I Find Most Interesting: Identifying What We Don't Know

One of the areas we're currently exploring is using AI to help identify intelligence gaps.

In investigations, it's easy to get buried in the information we already have. We develop a theory, collect supporting evidence, and continue following the leads that appear most promising.

But what are we overlooking?

What information would challenge our theory? Are we missing a time period in a timeline? Does a transaction appear without a corresponding record? Are two people actually connected, or do they simply share a common characteristic?

These are the kinds of questions a good analyst should constantly be asking.

We're developing a local-first tool called Gap Glue to help with that process.

The concept is straightforward: an investigator provides evidence, notes, documents, or an investigative question, and the tool helps identify information gaps, discrepancies, potentially missing sources, and competing explanations.

Depending on the investigation, that could involve dates, locations, people, transactions, communications, equipment, digital artifacts, cryptocurrency records, or even missing imagery and audio.

Importantly, the tool must distinguish between something that is actually supported by evidence and something that merely seems plausible.

If it finds a possible connection, I want to know why it suggested that connection and what evidence would be needed to validate it.

If the record doesn't support an answer, I want it to say so.

We're designing this with explicit checks against unsupported claims and hallucinations. Those safeguards still require testing, of course. No AI system can guarantee that every output will be correct.

But the ability to systematically challenge our own assumptions is one of the more promising applications of AI in investigative work.

Local AI Isn't Just for Investigations

Another thing I've discovered through building these systems is how much of our work involves repetitive tasks that have very little to do with the actual analytical problem.

Report formatting. Organizing notes. Finding the correct file. Editing training material. Reviewing documents. Moving information from one system into another.

These tasks take time, and they often require switching between several different applications.

AI can help, but again, the workflow matters.

For example, we're working on ways to help investigators turn structured case notes into professionally formatted reports while preserving their individual writing styles and allowing them to review and edit the output.

We're also experimenting with local AI workspaces that bring several capabilities into a single interface: document assistance, video processing, content organization, coding tools, and other specialized functions.

The idea isn't to build one enormous AI that does everything.

I prefer smaller tools that perform specific jobs well, with a shared interface when that makes sense.

That makes it easier to test them, fix problems, and replace components when better technology becomes available.

Choosing the Right AI Tools

There isn't one perfect AI model for every job, and I'm skeptical of anyone who tells you otherwise.

Some models are better at writing. Others are more useful for working with code, extracting structured information, or interpreting particular kinds of documents.

Hardware matters, too. A model that works beautifully on a powerful workstation may be painfully slow on an ordinary office laptop.

When helping an organization choose an approach, I look at several practical considerations:

• The task: What does the AI actually need to do, and would a simpler tool accomplish the same thing?

• The information: Is it public, internal, confidential, or subject to special handling requirements?

• The hardware: What can reasonably run on the computers the team already owns?

• The users: Can investigators use the tool without becoming programmers or AI specialists?

• The evidence: Can the user inspect the source information behind the output?

• The maintenance: Who will update, troubleshoot, and validate the tool over time?

I'm also a big believer in building a small working prototype before investing in a much larger system.

You learn an enormous amount once a real user starts clicking buttons and asking why something doesn't work.

Often, the most valuable feedback has nothing to do with the underlying AI model. It's about confusing instructions, missing functionality, or a workflow that doesn't quite match what the investigator needs.

That's exactly the kind of feedback you want early.

Productivity Means More Than Producing More Reports

There's a tendency to measure AI productivity by how many pages it can generate or how quickly it can summarize a document.

Those are useful measurements, but they don't tell the whole story.

A twenty-page report generated in thirty seconds isn't helpful if an investigator needs three hours to find and correct the errors.

A dashboard that displays fifty impressive charts isn't useful if none of them help someone make a decision.

I think the better questions are:

Did the tool save meaningful time? Did it help the analyst find something they might otherwise have missed? Did it reduce repetitive work? Did it improve documentation? Can the results be checked and defended?

And, perhaps most importantly, did it allow the investigator to spend more time doing the things that require experience, judgment, and curiosity?

That's where I believe AI can make a real difference.

Bringing Real Investigative Experience Into AI Development

At Plessas Experts Network, our approach to building AI tools comes directly from our work in intelligence, investigations, and training.

I started my career as an Army intelligence linguist and went on to support intelligence work at DIA and CIA, participate in the early development of a federal OSINT innovation center, and serve as an OSINT technical expert supporting the Department of Homeland Security.

Since founding Plessas Experts Network in 2008, I've worked extensively with law enforcement, government agencies, private investigators, corporate teams, and other professionals who need to find, evaluate, and use information responsibly.

We've also helped train more than 25,000 law enforcement professionals through federal partnerships and other programs.

That experience shapes how we approach AI.

I know that investigators often work with incomplete information, limited budgets, aging systems, sensitive records, and deadlines that don't leave much room for experimentation.

I also know that no two organizations work exactly the same way.

So rather than offering everyone the same generic AI solution, we're interested in understanding the problem and developing something that makes sense for the people who will actually use it.

That might involve local AI, a custom dashboard, better data organization, a standalone investigative utility, or training an internal team to build and maintain its own tools.

It doesn't always require a large project.

Sometimes the most effective solution is surprisingly simple.

A Short Story From the Field: Building Tools Around Real Investigators

One of the projects that has influenced my thinking recently involves a private investigator who handles complex cases, including work connected to major law firms.

She's an experienced investigator. Her problem wasn't that she needed AI to tell her how to investigate. It was that too much of her time was being consumed by the administrative work surrounding her investigations.

She was already using case-management software, maintaining notes, and producing reports for clients. She had established methods and a writing style that worked for her.

The challenge was finding a way to reduce the repetitive work without disrupting those methods or putting confidential information into an external AI service.

So we began developing a local AI reporting assistant tailored to her workflow.

The system is designed to take structured case notes and assist with turning them into reports. We've been working on preserving report templates, learning from examples of her writing, and allowing her to review and edit the results before exporting a finished Word document.

We're also exploring how information from her case-management system can eventually support other administrative functions, such as expense handling.

This hasn't been a matter of simply downloading an AI model and declaring the problem solved. The development process has involved extensive testing, fixing formatting problems, adjusting the interface, checking local model performance, and making sure different functions remain separate and predictable.

Some integrations still require testing in the client's actual environment.

And that's an important point: building useful AI tools is rarely about getting an impressive demonstration to work once. It's about making something reliable enough for an actual professional to incorporate into daily work.

At the same time, I've been building my own local AI workspace to experiment with specialized analytical tools. That work includes the Gap Glue concept for finding intelligence gaps, browser-based OSINT utilities, searchable investigative resource dashboards, and tools that help process and organize training videos.

These are different applications, but they're all based on the same approach: identify a specific problem, build something useful, test it, and improve it.

I'm particularly interested in eventually making more of these tools available to investigators who don't have dedicated development teams or expensive enterprise software budgets.

Not every agency needs a multimillion-dollar AI initiative.

Sometimes they need a better way to organize evidence, identify missing information, search across platforms, or produce a report without spending half the day reformatting it.

And sometimes we can build exactly that.

Let's Talk About What You Actually Need

If you've been considering AI for your investigative team, agency, or business, I'd encourage you to start with your most frustrating workflow rather than a list of AI products.

What takes too long? What keeps getting overlooked? What information do you wish you could see in one place? What tasks are your experienced people doing that really shouldn't require their level of expertise?

Those are the questions that lead to useful solutions.

At Plessas Experts Network, Inc., we help organizations evaluate AI opportunities, build custom local tools and dashboards, and train teams to use these technologies without abandoning sound investigative methodology.

We can help you develop a practical solution, or help your own team learn how to build one.

Visit Plessas.net to discuss a project or schedule a discovery call.

And if you're interested in experimenting with investigative tools, sharing ideas, and helping us develop new capabilities, I invite you to join our Analyst's Annex community.

Some of our most useful ideas start with an investigator asking a very simple question:

"Why isn't there a tool that does this?"

These days, there's a pretty good chance we can build one.