tuc as text based data structure for online research

I do a lot of research online. And I try to use AI to assist me.

I also use nushell a lot.

After experimenting a lot I’ve found a data structure that is both simple and sufficient for the kind of research I do online and the way I use it with nushell and AI.

I call this datastructure tuc:

╭───┬─────────╮
│ 0 │ title   │
│ 1 │ url     │
│ 2 │ content │
╰───┴─────────╯

In Nushell this corresponds to a record with 3 keys:

record<title: string, url: string, content: string>

With some content this could look like so:

╭─────────┬────────────────────────────────────────────────╮
│ title   │ About 50% Of Jobs Will Be Displaced By AI      │
│         │ Within 3 Years - YouTube                       │
│ url     │ https://www.youtube.com/watch?v=zZs447dgMjg    │
│ content │ hu and I have been talking backstage and we    │
│         │ were deciding should we give you guys some     │
│         │ sweet dreams tonight or some nightmares and we │
│         │  decided maybe a little bit of both uh we have │
│         │  a lot to discuss what the future looks like   │

With Nushell I can easily convert this to json

$record | to json

{
    "title": "About 50% Of Jobs Will Be Displaced By AI Within 3 Years - YouTube",
    "url": "https://www.youtube.com/watch?v=zZs447dgMjg",
    "content": "hu and I have been talking backstage and we were deciding should we give you guys some sweet dreams tonight or some nightmares and we decided maybe ... "
}

$record | to yaml

title: About 50% Of Jobs Will Be Displaced By AI Within 3 Years - YouTube
url: https://www.youtube.com/watch?v=zZs447dgMjg
content: hu and I have been talking backstage and we were deciding should we give you guys some sweet dreams tonight or some nightmares and we decided maybe a little bit of both uh we have a lot to discuss ...

and so on.

Let’s say I have more than one record.

$secondrecord

╭─────────┬──────────────────────────────────────────────────────────────────────╮
│ title   │ Sådan modstår du mediers manipulation                                │
│ url     │ https://www.folkets.dk/node/4939                                     │
│ content │ _Af [Lennart](https://www.folkets.dk/brugere/lennart-kiil "Vis       │
│         │ brugerprofil.")_                                                     │
│         │ _— Onsdag den 28. feb 2024, 09:17_                                   │

I can now append this record to the previous one:

$record | append $secondrecord

╭───┬────────────────────────────────────────────────────────────────────┬─────────────────────────────────────────────┬───────────────────────────────────────────╮
│ # │                               title                                │                     url                     │                  content                  │
├───┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────┼───────────────────────────────────────────┤
│ 0 │ About 50% Of Jobs Will Be Displaced By AI Within 3 Years - YouTube │ https://www.youtube.com/watch?v=zZs447dgMjg │ hu and I have been talking backstage and  │
│   │                                                                    │                                             │ we were deciding should we give you guys  │

...

│ 1 │ Sådan modstår du mediers manipulation                              │ https://www.folkets.dk/node/4939            │ _Af [Lennart](https://www.folkets.dk/brug │
│   │                                                                    │                                             │ ere/lennart-kiil "Vis brugerprofil.")_    │
│   │                                                                    │                                             │ _—                                        │
│   │                                                                    │                                             │ Onsdag den 28. feb 2024, 09:17_           │

When Nushell is given a list of records it turns them into a table as seen above.

I could also save the table as a new variable like so:

let research = $record | append $secondrecord

Then when my tuc-table includes all the research I need, often between 5 and 25 records, I can feed the tuc as JSON to an LLM to analyze it or to give it some other form.

$research | to json | llm "interpret the JSON and create an article with markdown links to sources and plenty of direct quotes

Remember the tuc: title url content.