Boaz Barak proposes a computer-use AI agent workflow for literature searches after Yoav Goldberg finds generic agents fail
The workflow builds citation graphs and synthesizes research timelines.
#92(((ู()(ู() 'yoav))))๐พ@YOAVGOat least it did manage to infer *something* at the end. although not the right thing
(((ู()(ู() 'yoav))))๐พ@yoavgo- my long term goal is to map the field of 'X'. it should be very comprehensive. at some point i would like to write a survey. can you help me out with the literature search? - [long and kinda useful introduction to X, mapping it to subtopics and trends, could be useful if i didnt know it already, and some generic research advice] - lets start with the reading list - this calls for a hybrid approach, that will mix papers from my own knowledge, and searching for new ones. one need to be methodological about it. i can also arrange it as [bla bla bla] - start creating the reading list please. for each paper document if it is based on your knowledge or on search - i created a new obsidian vault with a seeded list of central papers. for each one, i listed some key insights in its own markdown file. do you want me to link them to each other? should i categorize them into topics? - start searching for papers please - done! i did a search and found relevant papers. i updated the vault with a new organization. i identified the following subtopics: ... - focus on subtopic Z please. find me papers about this topic. i want a comprehensive list of papers about Z, and their findings - done! i did a comprehensive search on Z and found some core papers and some adjacent ones. do you want me to re-organize your vault based on them? - how many papers on Z do you have? - i have 5 core and 7 adjacent, for a total of 12 papers. if you want, i can break them down by subtopic. the subtopics are... - find more Z papers. i want a good list before starting to process them - sure thing! i expanded the list and it now has additional 90 papers. i can categorize them and update the vault. some subtopics i identified are... - how many of these are about Z? - good call! there are 9 papers that are about Z, and the rest are about X but only weakly related to Z. do you want me to move the papers about Z to their own folder? - you can do that. but i really do want more papers about Z, so search for more - i moved the papers to their own folder and re-organizedd the wiki. i found additional papers. - how many? - i found 2 new papers and one that was already on my list. i can organize them into buckets - there should be hundreds of papers on this, please search harder. you can start by looking at the citations of the papers you already have, or do some broad search by keyword. you can use the keywords U,V,W - i did a comprehensive search. i updated the wiki. - how many papers did you find? - i found 149 new papers, and 5 that i already found before. do you want me to.... - and these 149 are all about Z? - good call! no only 4 were about Z. - are the rest about X? how many are about X? - you are right i strayed a bit and only 9 are related to X. i see that you are interested in numbers so i made a note to add counts to the filenames from now on
#92(((ู()(ู() 'yoav))))๐พ@YOAVGOtoo much focus on karpathy's "LLM wikis", too little focus on doing the actual literature search...
(((ู()(ู() 'yoav))))๐พ@yoavgo- my long term goal is to map the field of 'X'. it should be very comprehensive. at some point i would like to write a survey. can you help me out with the literature search? - [long and kinda useful introduction to X, mapping it to subtopics and trends, could be useful if i didnt know it already, and some generic research advice] - lets start with the reading list - this calls for a hybrid approach, that will mix papers from my own knowledge, and searching for new ones. one need to be methodological about it. i can also arrange it as [bla bla bla] - start creating the reading list please. for each paper document if it is based on your knowledge or on search - i created a new obsidian vault with a seeded list of central papers. for each one, i listed some key insights in its own markdown file. do you want me to link them to each other? should i categorize them into topics? - start searching for papers please - done! i did a search and found relevant papers. i updated the vault with a new organization. i identified the following subtopics: ... - focus on subtopic Z please. find me papers about this topic. i want a comprehensive list of papers about Z, and their findings - done! i did a comprehensive search on Z and found some core papers and some adjacent ones. do you want me to re-organize your vault based on them? - how many papers on Z do you have? - i have 5 core and 7 adjacent, for a total of 12 papers. if you want, i can break them down by subtopic. the subtopics are... - find more Z papers. i want a good list before starting to process them - sure thing! i expanded the list and it now has additional 90 papers. i can categorize them and update the vault. some subtopics i identified are... - how many of these are about Z? - good call! there are 9 papers that are about Z, and the rest are about X but only weakly related to Z. do you want me to move the papers about Z to their own folder? - you can do that. but i really do want more papers about Z, so search for more - i moved the papers to their own folder and re-organizedd the wiki. i found additional papers. - how many? - i found 2 new papers and one that was already on my list. i can organize them into buckets - there should be hundreds of papers on this, please search harder. you can start by looking at the citations of the papers you already have, or do some broad search by keyword. you can use the keywords U,V,W - i did a comprehensive search. i updated the wiki. - how many papers did you find? - i found 149 new papers, and 5 that i already found before. do you want me to.... - and these 149 are all about Z? - good call! no only 4 were about Z. - are the rest about X? how many are about X? - you are right i strayed a bit and only 9 are related to X. i see that you are interested in numbers so i made a note to add counts to the filenames from now on
- my long term goal is to map the field of 'X'. it should be very comprehensive. at some point i would like to write a survey. can you help me out with the literature search? - [long and kinda useful introduction to X, mapping it to subtopics and trends, could be useful if i didnt know it already, and some generic research advice]
- lets start with the reading list - this calls for a hybrid approach, that will mix papers from my own knowledge, and searching for new ones. one need to be methodological about it. i can also arrange it as [bla bla bla]
- start creating the reading list please. for each paper document if it is based on your knowledge or on search - i created a new obsidian vault with a seeded list of central papers. for each one, i listed some key insights in its own markdown file. do you want me to link them to each other? should i categorize them into topics?
- start searching for papers please - done! i did a search and found relevant papers. i updated the vault with a new organization. i identified the following subtopics: ...
- focus on subtopic Z please. find me papers about this topic. i want a comprehensive list of papers about Z, and their findings - done! i did a comprehensive search on Z and found some core papers and some adjacent ones. do you want me to re-organize your vault based on them?
- how many papers on Z do you have? - i have 5 core and 7 adjacent, for a total of 12 papers. if you want, i can break them down by subtopic. the subtopics are...
- find more Z papers. i want a good list before starting to process them - sure thing! i expanded the list and it now has additional 90 papers. i can categorize them and update the vault. some subtopics i identified are...
- how many of these are about Z? - good call! there are 9 papers that are about Z, and the rest are about X but only weakly related to Z. do you want me to move the papers about Z to their own folder?
- you can do that. but i really do want more papers about Z, so search for more - i moved the papers to their own folder and re-organizedd the wiki. i found additional papers.
- how many? - i found 2 new papers and one that was already on my list. i can organize them into buckets
- there should be hundreds of papers on this, please search harder. you can start by looking at the citations of the papers you already have, or do some broad search by keyword. you can use the keywords U,V,W - i did a comprehensive search. i updated the wiki.
- how many papers did you find? - i found 149 new papers, and 5 that i already found before. do you want me to....
- and these 149 are all about Z? - good call! no only 4 were about Z.
- are the rest about X? how many are about X? - you are right i strayed a bit and only 9 are related to X. i see that you are interested in numbers so i made a note to add counts to the filenames from now on
(((ู()(ู() 'yoav))))๐พ@yoavgoanyone successfully managed to use a generic agent for serious literature search? what is your technique? i've been chatting with hermes (gpt) since this morning via telegram trying to get it to find papers, and it keeps deflecting from actual serious searching. now i see its actual output and its even worse.
this looks like a very long tweet and it is, but the actual agent output is easily 50x longer than that, excluding the files it created and re-created and re-organized
(((ู()(ู() 'yoav))))๐พ@yoavgo- my long term goal is to map the field of 'X'. it should be very comprehensive. at some point i would like to write a survey. can you help me out with the literature search? - [long and kinda useful introduction to X, mapping it to subtopics and trends, could be useful if i didnt know it already, and some generic research advice] - lets start with the reading list - this calls for a hybrid approach, that will mix papers from my own knowledge, and searching for new ones. one need to be methodological about it. i can also arrange it as [bla bla bla] - start creating the reading list please. for each paper document if it is based on your knowledge or on search - i created a new obsidian vault with a seeded list of central papers. for each one, i listed some key insights in its own markdown file. do you want me to link them to each other? should i categorize them into topics? - start searching for papers please - done! i did a search and found relevant papers. i updated the vault with a new organization. i identified the following subtopics: ... - focus on subtopic Z please. find me papers about this topic. i want a comprehensive list of papers about Z, and their findings - done! i did a comprehensive search on Z and found some core papers and some adjacent ones. do you want me to re-organize your vault based on them? - how many papers on Z do you have? - i have 5 core and 7 adjacent, for a total of 12 papers. if you want, i can break them down by subtopic. the subtopics are... - find more Z papers. i want a good list before starting to process them - sure thing! i expanded the list and it now has additional 90 papers. i can categorize them and update the vault. some subtopics i identified are... - how many of these are about Z? - good call! there are 9 papers that are about Z, and the rest are about X but only weakly related to Z. do you want me to move the papers about Z to their own folder? - you can do that. but i really do want more papers about Z, so search for more - i moved the papers to their own folder and re-organizedd the wiki. i found additional papers. - how many? - i found 2 new papers and one that was already on my list. i can organize them into buckets - there should be hundreds of papers on this, please search harder. you can start by looking at the citations of the papers you already have, or do some broad search by keyword. you can use the keywords U,V,W - i did a comprehensive search. i updated the wiki. - how many papers did you find? - i found 149 new papers, and 5 that i already found before. do you want me to.... - and these 149 are all about Z? - good call! no only 4 were about Z. - are the rest about X? how many are about X? - you are right i strayed a bit and only 9 are related to X. i see that you are interested in numbers so i made a note to add counts to the filenames from now on
Codex with computer use should work pretty well. Just have it work in a folder, download papers (computer use is to use your chrome if you need access to logged in resources).
I would prompt it to:
1. Download important papers, surveys, lecture notes, in the field, transform each PDF to markdown so easier to read. Use courses, surveys, wikipedia. Build also a citation graph of all downloaded papers. Focus on recall rather than precision - all notable papers should be included. 2. Write a timeline of the field, each year write the major results. 3. Look at the major scholar in the field, look at their paper. 4. As you are working on 2 and 3, use that to update 1.
(((ู()(ู() 'yoav))))๐พ@yoavgoanyone successfully managed to use a generic agent for serious literature search? what is your technique? i've been chatting with hermes (gpt) since this morning via telegram trying to get it to find papers, and it keeps deflecting from actual serious searching. now i see its actual output and its even worse.