Some Thoughts on Generative AI

Well, I finally gave in to AI... sort of.

My engagement with generative AI

Since earlier in the summer I've been trying to understand how I can usefully incorporate generative AI into my various responsibilities. This is independent of my various environmental and ethical concerns about generative AI. My position on this is that there are a lot of arguments for and against the use of generative AI; for many of the arguments against (for example the environmental impact, the disruption to the work force, and the big elephant in the room in my profession) one can simply learn about the impact without first hand experience with generative AI. But for a complete and balanced view, and to responsibly form an opinion on whether I should or should not use generative AI tools, I need to engage with the argument for the utility of generative AI. And on this front I need at least a working knowledge on how these tools work from a user's perspective.

Which tools have I tried

For the most part I am very skeptical of the idea of giving my data to companies like Anthropic, OpenAI, or Google. Thus I want to compartmentalize as much as I can. For "low risk" uses I've mostly relied on the access to Gemini included as part of my institution's Google Education subscription, though I have also used Elicit for literature search, and occasionally Microsoft Copilot which is part of my institution's Microsoft License (mostly because the built-in search to the Web client for Outlook royally sucks, and copilot often (but not always) do better to find that one hard-to-track-down e-mail from 3 years ago).

As part of my quest to understand how one can potentially use generative AI tools, I've also taken the opportunity to participate in the "Agentic AI" Hot Topics Workshop run through the AIDMM-NRT program co-hosted by my department. And through this workshop I've also had the chance to try out Anthropic Claude (more on this later).

For "higher risk" uses I've largely defaulted to running large language models locally, through Ollama and various user interfaces, including:

• Local deep research
• AnythingLLM
• OpenCode

Summary of some of my experiences

Information Retrieval

One of my first attempts to leverage generative AI is in its supposed ability to process information and return useful summaries. As the serving co-director of undergraduate studies in my department, I often need to dig up University policy and program / course information in the University Academic Catalog, and cross correlate information from different sections.

Obviously this is a task that can be automated by AI... right?

At first I tried using local AI tools and downloaded versions of the catalog. After that always returning heavy amounts of misinformation, I decided to also try it with Google Gemini. This lead to the "obvious" discovery that AI is only as good as the data you feed into it.

As it turns out, Michigan State University's online version of its academic catalog was produced using a tool that prioritized presentation over semantic correctness. The HTML code that was generated use tables for layouts liberally, instead of semantically corrected (nested) ordered lists. This means that a lot of the document structure can not be correctly processed by AI ingestion. (This also is an accessibility no-no! So I am glad that the registrar's office is working on fixing this.) This I am certain contributed to the heavy hallucination when I tried to query information from the academic catalog.

I may return to this later; but it was a bit disappointing that the first AI use case that I can see being useful was in fact not possible, even though the issue stems not from the capabilities of the AI tools but from the poor formatting of the primary sources.

Research Support

I've tried to use AI to support my research in a few ways:

• It is extremely useful as a tool to conduct the preliminary search in a field that is faraway from my main expertise. I think the main benefit is that I can ask questions using the terminology that I am familiar with and provide a plain language description of my ideas and the LLm is able to relate these to field-specific keywords that I don't know about. In many ways, this replaces situations where I would otherwise have to cold call an area expert just to learn that is the correct jargon to search for.
  • In fact, I only started doing this because I sent an e-mail to two experts with one such question. One was kind enough to respond. The other said: "Have you asked AI?" and attached an AI generated answer. In terms of the "leads" generated the two responses are comparable. The human-generated response has the advantage that it was tailored to who I am (my research background and qualifications) because the expert knows me.
• Generative AI is also good for carrying out "routine computations". For example, I may know that a given ODE can be converted to a standard form and be solved with explicit special functions with a bunch of transformations of dependent and independent variables. It may be interesting to me to see the final substitution as a way to get a quick sanity check on intuitions (for example, about dependencies on the coefficients), but the computation itself will not be informative to my understanding of the overall research problem. In cases like this I can give generative AI plain-text descriptions of what computations I want to see carried out. Previously I may sometimes call on a graduate student or an undergraduate research assistant to carry this out, the AI-produced results are typically of similar quality.
  • This is also very useful for choosing coefficients for exam problems.

But I have not had much success in actually getting generative AI to solve any problem that I am interested in (mostly because I have not been willing to send these questions to third party servers that provide the frontier models). (Of course, being a part-time fluid dynamicist, obviously we have seen recent successes by other folks with frontier models solving problems that I have at least a tangential interest in.)

Also, when it comes to generation of mathematical manuscripts, I have been less than impressed with the style of writing even from frontier models. One of the interesting developments at the end of the summer is that a problem I posed 10 years ago was finally solved. Coincidentally an undergraduate student has planned to complete an independent study on the problem this semester, so we pivoted toward verifying the AI-generated proof. We now have high confidence that the proof given is correct (without relying on the Lean certificate), but I do feel that had a human expert written the paper I would've been able to digest the argument in half the time. AI idiosyncrasies made it harder to pick out the main ideas.

Teaching

I happen to be developing / revamping a course this semester. I wasn't happy with the textbook that the course has been using, and have a particular vision for what the course should look like. With that in mind, one of the things I tried is to provide a description of the topic areas that I wish to see covered and used AI tools to find suitable textbooks.

I was able to settle on the textbook I am using for the course through this process... but only after some human intervention. My first attempts at this in the summer suffered from significant amount of AI hallucination: none of the textbooks suggested actually exist. That said, with the understanding that AI hallucinations are often grounded in some kernel of truth, I was able to track down what was probably the intended recommendations using some old-fashioned Google search based on the not-fully-correct information that AI provided for me.

Enters Vibe Coding

Up until last week, most of my engagement with AI resulted in either disappointments, or in minor wins that are not earth-shattering. Some also have drawbacks: for example, the time-saving from using AI tools to do literature search in unfamiliar topics can also mean a missed opportunity to network with experts in other subject areas. Sure, they provide some conveniences, but none of them are transformative.

This changed during the Agentic AI workshop.

The Workshop was intended to provide a hands-on-look on how to leverage "Coding Agents" and incorporate them into our research workflow. We were each invited to come up with a way in which the tools we learned about can be useful to our own personal research project.

There I was stumped.

I saw a lot of applications for applied mathematics, but I cannot see any way to apply agentic AI to my own research problems. I refuse categorically to just ask my research question at the AI and wait for an answer. This is entirely against what mathematics research is about. (This is not to say that agentic AI is not useful. I can see how to leverage them to do some preliminary sanity checks by computing explicit examples, and use this to develop intuition; but none of my projects are currently in that stage and it feels like a waste of time to ask AI to recompute examples that humans have already computed.)

So how should I use the Claude credit that I got "for free"?

Then it occurred to me: one way that AI can help my research is to help generate time for me to do research, by freeing up my time spent on other tasks. For example: what if AI can help me prepare course material for teaching?

I am not thinking about simple stuff like working out examples or preparing worksheets; for the most part I don't anticipate any time-savings there. But I do know that creation of visualizations, especially interactive ones, often take much more time than I really should be devoting to them. Coming back to the course that I am revamping: one of the learning objectives is for students to develop "geometric intuition", and this is really helped by having good visualizations and manipulatives. I've already used my 3D printer to create some physical objects to help with visualizations, what if I use AI to create interactive webpages? As it happens, I've looked for a way to interactively visualize the geometric operation of "parallel projection", and Google did not provide me with anything that I can use.

Result: the parallel projection visualizer.

How it was made

Creating this interactive widget took 90 minutes one evening. While I am comfortable with reading JavaScript, when writing it I constantly would have to refer to references. So this is an instance where the AI coding agent saved me a significant amount of time (based on previous coding projects, this would have taken me 8 or 9 hours at least).

To build this widget, I used Claude Code.

I set up a new project directory, ran Claude in the directory. I provide it with some preliminary specs:

• A plain text description of the overarching goals. For this I relied on the fact that LLMs do know what parallel projection is and so I don't have to describe the mathematics in details to it.
• A plain text description of how I imagined the UI to look.
• Description on how I envisioned this to be deployed (static website with Javascript, all computations done client-side).

Then I asked Claude to generate a plan for this project.

Claude came back to me with a list of clarifying questions on the specs, and I answered them in natural language. Based on this Claude produced a detailed plan / specs for the project. After I reviewed the plan, I provided more feedback, based on which Claude made additional modifications.

Then I let Claude start building.

Claude automatically split the development into stages. After each stage it checked with me for feedback. It really felt like working with an experienced coder. As the app started taking shape, I would discover additional features that I would like, as well as items which I previously specified which in hindsight was unnecessary. And I can just ask the coding agent to change things.

Very helpfully, at each milestone checkpoint, Claude also suggests some main items to review. Many of these have to do with user interface issues, which is as expected.

After 7 rounds of feedback and review, the code for the website was done.

Some caveats

I did this development on my Chromebook, inside Crostini. This turned out to be very lucky for me.

• The Crostini environment does not have a built-in web-browser, so Claude prompted me heavily for decisions on UI. I believe otherwise Claude may have launched browser instances to make some of these decisions itself.
• I also do not have npm installed: this is important, as in automatic mode (as opposed to manual mode where permission is sought for every operation), it appears that Claude would have installed npm packages. Had I asked for it to develop something in python, I'm sure it would have tried (and succeeded) to install python modules, and I am paranoid about this somewhat (in part with the expectation that it will clobber my personal process on maintaining these dependencies).