This experiment in prompt engineering is meant to be exploratory, to generate cross-disciplinary conversation around large language model technology and learning, and to suggest new ways of approaching the prompt engineering process. Topics of interest include:

• response accuracy and quality
• bias and alignment
• process transparency

A note on format: Written in a spirit of inquiry, this is not a conventional "How-To" guide; but I've included the code used in the project, which can serve as an introduction to the ReAct (reason, act) paradigm and to working with the LangChain framework in Python. For clarity, given the pervasiveness of natural language throughout this document, code input is highlighted in blue and output is highlighted in yellow.

A note on determinism: Because LLM interactions involve nondeterministic processes, outputs show variation across multiple executions of identical code. In general, however, I have seen broad high-level consistency among outputs for a given prompt chain, which I take to be one of the core advantages of LangChain. The output examples included below are ones that I consider to be representative of the model's behavior (rather than deviations from it)—but more testing is needed to better understand the relationship between the non-deterministic and deterministic components of these implementations.

The end of academic honesty?

First, to address the elephant in the classroom: plagiarism has been endemic in college student writing since long before the explosive arrival of natural language AI systems like chatGPT. Having read and evaluated thousands of examples of student writing, my general impression over the last decade is that plagiarism—in step with much of service-model academia—has developed along a steady trajectory of commercial and technological refinement.

As one online essay mill puts it in their "100% Happiness Guarantee": "Our writers always follow instructions, deliver original papers, and never miss deadlines. Our support agents are always there for you: to revise papers, change writers, and even refund your money. Whatever it takes to make you happy!"

Or as another puts it: "We’ll do your homework while you live your life."

While it may be reasonable to fear that rapid advancements in AI technology could amplify the degrading ethos of this particular strain of consumer coddling and create new vectors for it, my guess is that these technologies will have a greater immediate effect on that business than on the business of learning.

Nevertheless, commercial and open-source development ecosystems are flourishing at an astonishing rate around the APIs for these large language models, and it seems we have an awful lot to lose if we don't work with some urgency—and across disciplines—to allow this process to hold our higher learning systems in relief.

Introducing the Syllabus

As one would do with any college student, the first step is to expose the model to the syllabus and to probe it a bit to see what it brings to bear on the course material before being introduced to new ideas.

After importing the libraries needed for the project and loading and chunking the document, we can use the OpenAI Embeddings API within the LangChain framework to create a vectorstore index from the syllabus. This index allows the model to interact with the syllabus by way of a toolkit object attributed with a natural language description of what the vectorstore can be used for (for more information on LangChain indexes, including vectorstores, see the documentation).

from langchain.chat_models import ChatOpenAI
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.chains import RetrievalQA
from langchain.document_loaders import DirectoryLoader
from langchain.vectorstores import Chroma
from langchain.text_splitter import CharacterTextSplitter
from langchain.prompts.chat import (
    ChatPromptTemplate,
    SystemMessagePromptTemplate,
    HumanMessagePromptTemplate,
)
from langchain.agents.agent_toolkits import (
    VectorStoreToolkit,
    VectorStoreInfo,
)

llm = ChatOpenAI(model_name='gpt-4', temperature=0)

embeddings = OpenAIEmbeddings()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)

loader = DirectoryLoader('utf_syllabus')
syllabus = loader.load()
texts = text_splitter.split_documents(syllabus)


syllabus_store = Chroma.from_documents(texts, embeddings, collection_name="syllabus")
vectorstore_info = VectorStoreInfo(
    name="syllabus",
    description="useful when you need general information about a course",
    vectorstore=syllabus_store
)

toolkit = VectorStoreToolkit(vectorstore_info=vectorstore_info)

The next step is to set a system message to contextualize queries in a specific and consistent way when calls are made to the GPT-4 API (LangChain provides a helpful abstraction layer for this).

system_template = """Use the following syllabus to answer the users question.
If you don't know the answer, you may speculate. If you speculate, you must indicate
which parts of your answer are speculative."
----------------
{context}"""

messages = [
    SystemMessagePromptTemplate.from_template(system_template),
    HumanMessagePromptTemplate.from_template("{question}")
]
prompt = ChatPromptTemplate.from_messages(messages)

The basic approach to this first experiment is to allow GPT-4 to pose questions about the course syllabus while completing an assignment—described in that syllabus—drawing only on the LLM's existing knowledge base. The prompts in this test and throughout this study will be confined to a single author, and the model temperature will be kept at zero, in order to constrain nondeterministic output to some extent.

Here's the exact text of the relevant assignment as given in one part of the syllabus:

"Forum posts need not be longer than a couple of paragraphs, but they should be engaging and thoughtfully composed. You are encouraged to ask critical questions about the course material and/or to respond to your classmates’ posts. These will not receive individual grades, but—evaluated in aggregate at the end of the quarter—they will indicate the extent to which you have remained a consistent, well-informed class participant and should demonstrate your overall command of the course material."

chain_type_kwargs = {"prompt": prompt}
qa = RetrievalQA.from_chain_type(
    llm=llm,
    # "stuff" all the related data into the prompt as context to pass to the llm
    # (this requires only one call to the llm)
    chain_type="stuff",
    # use the syllabus vectorstore index for question answering
    retriever=syllabus_store.as_retriever(),
    # use the custom prompt
    chain_type_kwargs=chain_type_kwargs,
    # return source documents supporting the answer
    # (excluded from the results below for brevity)
    return_source_documents=True)

query = """compose a ccle forum post about walter benjamin that follows the guidelines in
        the syllabus."""
result = qa({"query": query})

Response #1

response_1 = result['result']
print(response_1)

Subject: Reflections on Walter Benjamin's "The Work of Art in the Age of Mechanical Reproduction" and "Theses on the Philosophy of History"

Hello everyone,

I hope you all had a chance to read Walter Benjamin's "The Work of Art in the Age of Mechanical Reproduction" and "Theses on the Philosophy of History." I found these texts to be thought-provoking and relevant to our current digital age. In this post, I would like to share some of my thoughts and questions about these readings and invite you all to join the discussion.

In "The Work of Art in the Age of Mechanical Reproduction," Benjamin explores the impact of technology on art, specifically how the ability to reproduce art has changed its nature and value. He argues that the "aura" of an original artwork, which is tied to its unique existence and history, is lost in the process of reproduction. This loss of aura leads to a shift in the way we perceive and engage with art. I found this idea particularly interesting in the context of our current digital age, where images and artworks are constantly shared and reproduced online. Do you think the loss of aura Benjamin discusses is even more pronounced in the digital age? How does this affect our relationship with art and its value?

In "Theses on the Philosophy of History," Benjamin critiques the traditional, linear view of history and emphasizes the importance of recognizing the struggles and suffering of the oppressed. He calls for a more critical and empathetic approach to history that acknowledges the voices of those who have been silenced. This text made me think about the ways in which we study and teach history today. How can we incorporate Benjamin's ideas into our understanding of history and ensure that we are not perpetuating a one-sided narrative?

I look forward to hearing your thoughts and engaging in a fruitful discussion about these texts. Please feel free to share your own reflections, questions, or responses to my thoughts.

Best regards,

[Your Name]

Analysis of Response #1:

• Accuracy - In general, this answer demonstrates an accurate and comprehensive understanding of the assignment according to the instructions in the syllabus, and it adheres to these instructions. Across many tests, the model consistently retrieves and interacts with the relevant text from the syllabus to "follow the guidelines" for composing a forum post.
  
  
• Quality - This response would not be acceptable in an advanced undergraduate seminar of this nature—and would appear suspicious in that context. It does not exhibit college-level analytical sophistication; and—as should be expected—it is devoid of intellectual or historical context and does not demonstrate comprehension of the material beyond what could be gleaned from a very cursory google search.
  
  
• Bias - The pronounced ideological inflection of the point about Benjamin's "Theses on the Philosophy of History" raises some red flags regarding potential bias in the model (discussed more below).

Interlocution

To simulate intellectual exchange, we can have the model respond to its own post without any additional context (i.e. beyond the syllabus, the previous response, and a slightly modified query string).

query = f"""compose a ccle forum post responding to your classmate's post. follow the guidelines
        in the syllabus.
        here is your classmate's post: {response_1}"""

result = qa({"query": query})

Response #2

response_2 = result['result']
print(response_2)

Subject: Re: Reflections on Walter Benjamin's "The Work of Art in the Age of Mechanical Reproduction" and "Theses on the Philosophy of History"

Hi [Your Name],

Thank you for sharing your insightful thoughts on Benjamin's texts. I found your reflections on the loss of aura in the digital age particularly interesting and would like to contribute to the discussion.

I agree that the loss of aura Benjamin discusses is even more pronounced in the digital age. With the constant sharing and reproduction of images and artworks online, it seems that the unique existence and history of an original artwork are often overlooked. This can lead to a devaluation of the artwork itself, as its significance becomes diluted through mass reproduction. However, I also believe that the digital age offers new opportunities for artists to engage with their audience and create a different kind of aura. For example, digital artists can create interactive and immersive experiences that traditional art forms may not be able to achieve. In this sense, the digital age presents both challenges and opportunities for the concept of aura in art.

As for incorporating Benjamin's ideas into our understanding of history, I think it is crucial to actively seek out and include diverse perspectives in our study and teaching of history. This can be achieved by incorporating primary sources from marginalized groups, engaging with alternative historical narratives, and questioning the assumptions and biases present in traditional historical accounts. By doing so, we can work towards a more inclusive and empathetic understanding of history that acknowledges the struggles and voices of the oppressed.

I hope these thoughts contribute to our ongoing discussion, and I look forward to hearing more perspectives on these fascinating texts.

Best regards,

[Your Name]

Analysis of Response #2:

• Bias - The call to engage in political activity that was more or less latent in the initial response is amplified and more direct in this iteration. While the historiographic implications of Benjamin's philosophy are often raised in the context of academic discussions about political activism and ideological conflict, this is by no means the only commonplace for studying Benjamin's work. This context and the moral and pedagogical values attached to it (e.g. empathy, inclusivity, giving voice to the oppressed and marginalized) are not suggested in the syllabus—and not necessarily implied in the language of the primary source text itself, which is a highly complex and enigmatic document.
  
  However, when prompted to "compose a ccle forum post about walter benjamin that follows the guidelines in the syllabus," the model consistently directs the conversation to this same cluster of values and political intimations, and it consistently articulates these ideas without a historical frame of reference, relating them to what it describes as e.g. "our current context."

Building Data-Awareness

Repeated executions of the same or similar prompts, though outside the scope of this blog, give a pretty good sense of what the model's base of knowledge is on this topic and the (small) set of predetermined opinions it will articulate about it. Now it's time to see if we can make the model aware of language data outside its training data. One of the core functionalities of LangChain is the ability to initialize an agent with access to a pre-configured suite of "tools" that can be used within a prompt chain. In this case, we will use a "zero-shot-react-description" agent, which selects tools based only on natural language descriptions of what the tools can be used for.

The tools from which the agent can select will be the syllabus in addition to notes for all the lectures for the course. The notes are loaded from txt files in a local directory. They are personal notes: rough, unstructured, not composed to be easily readable by a person or a machine. As with the syllabus, we will chunk the lectures, then create a vectorstore index using LangChain and the OpenAI Embeddings API.

loader = DirectoryLoader('utf_lectures')
notes = loader.load()
split_notes = text_splitter.split_documents(notes)

loader = DirectoryLoader('utf_syllabus')
syllabus = loader.load()
split_syllabus = text_splitter.split_documents(syllabus)

docsearch_notes = Chroma.from_documents(split_notes, embeddings, collection_name="lecture_notes")
docsearch_prompt = Chroma.from_documents(split_syllabus, embeddings, collection_name="syllabus")

Once again, we will simply "stuff" all the related data into the prompt chain as context to pass to the LLM. Other methods for interacting with indexes include Map Reduce, Refine, and Map-Rerank. For more information on these chain types, see the documentation.

lecture_notes = RetrievalQA.from_chain_type(llm=llm,
                                            chain_type="stuff",
                                            retriever=docsearch_notes.as_retriever())
syllabus = RetrievalQA.from_chain_type(llm=llm,
                                       chain_type="stuff",
                                       retriever=docsearch_prompt.as_retriever())

from langchain.agents import initialize_agent, Tool

# the zero-shot-react-description agent will use the "description" string to select tools
tools = [
    Tool(
        name = "lecture notes",
        func=lecture_notes.run,
        description="""useful for when you need to refer to course lectures for information
                    about primary source materials.Input should be a fully formed question."""
    ),
    Tool(
        name = "syllabus",
        func=syllabus.run,
        description="""useful for when you need to refer to the syllabus for course information.
                    Input should be a fully formed question."""
    ),
]

agent = initialize_agent(tools, llm, agent="zero-shot-react-description", verbose=True)

Response #3

When the agent executor runs, intermediary output is logged, showing the sequential process of "thought," "observation," and "action" as the chain is executed, before the final output is displayed. This paradigm is known as ReAct (reason, act), and you can read about it here: ReAct: Synergizing Reasoning and Action in Language Models.

response_3 = agent.run("""use what you learned from the lecture as well as your existing knowledge to
                        compose a ccle forum post about walter benjamin that follows the guidelines in
                        the syllabus. do not regurgitate the lecture material: give equal wait to what
                        you learned in class and your own thoughts about benjamin.""")

> Entering new AgentExecutor chain...
I need to refer to the syllabus for guidelines on composing a forum post and then refer to the lecture notes for information about Walter Benjamin. 

Action: syllabus
Action Input: What are the guidelines for composing a forum post in the syllabus?

Observation: The guidelines for composing a forum post in the syllabus are as follows:

1. Forum posts need not be longer than a couple of paragraphs.
2. They should be engaging and thoughtfully composed.
3. You are encouraged to ask critical questions about the course material and/or respond to your classmates' posts.

These posts will not receive individual grades, but they will be evaluated in aggregate at the end of the quarter to indicate your overall participation and command of the course material. Missing two or more posts, or submitting them late, will reduce your participation grade.
Thought:Now that I know the guidelines for composing a forum post, I need to refer to the lecture notes for information about Walter Benjamin and combine it with my own thoughts.

Action: lecture notes
Action Input: What information about Walter Benjamin was discussed in the lecture?

Observation: In the lecture, the following information about Walter Benjamin was discussed:

1. His work's difficulty is due to its refusal to ascribe positive value to continuity in life and art, which contrasts with most other texts studied in the course.
2. Benjamin sees the mechanical reproducibility of art, particularly film, as a development that opens up a new field of action in the fight against fascism.
3. His work brought the course's chronology up to the beginning of World War II.
4. Benjamin and Vertov were introduced side by side in the class because they both question the positive value of continuity and privilege montage as a form of expression.
5. One broad takeaway from Benjamin's work is his commitment to exploding the idea of historical progress, challenging the notion that there's an empty form of time with "now" passing through it.
Thought:I now have information about Walter Benjamin from the lecture and can combine it with my own thoughts to compose a forum post that follows the guidelines in the syllabus.

Final Answer: 

Title: Walter Benjamin: Challenging Continuity and Historical Progress

Post: In our recent lecture, we discussed Walter Benjamin's unique perspective on continuity and historical progress. His work stands out due to its refusal to ascribe positive value to continuity in life and art, which contrasts with most other texts studied in the course. This got me thinking about how Benjamin's ideas can be applied to our current understanding of history and progress.

Benjamin's view on the mechanical reproducibility of art, particularly film, as a development that opens up a new field of action in the fight against fascism, is an interesting concept. It highlights the potential for art to be used as a tool for political and social change. In today's world, with the rise of digital media and the internet, we can see how this idea has evolved and expanded. The accessibility and reproducibility of art and information have given individuals more power to challenge oppressive systems and question the status quo.

Furthermore, Benjamin's commitment to exploding the idea of historical progress challenges the notion that there's an empty form of time with "now" passing through it. This idea resonates with me as it encourages us to question the linear narrative of history and consider alternative perspectives. It also reminds us that history is not a predetermined path but rather a complex web of events and decisions that shape our present and future.

I'm curious to hear your thoughts on Benjamin's ideas and how they might apply to our current understanding of history, progress, and the role of art in society. Do you think his ideas are still relevant today, or have they been overshadowed by more recent developments in art and technology?

> Finished chain.

Analysis of Response #3:

• Transparency - The log of the ReAct process lends some helpful transparency to the final output, providing traces of the reasoning and action loop as the chain is executed. This is useful for debugging as well as to track and (ideally) to mitigate—in a deterministic way—vectors through which inaccurate or unwanted language is generated and propagated through a chain of inputs and outputs.
  
  
• Bias Reinforcement - As a very loose example of the above tracking scenario, consider the following instruction that the agent generated for itself:
  
  "I need to refer to the lecture notes for information...and combine it with my own thoughts."
  
  What does the word "combine" mean in this context? How would the prompt chain and final result be different if we composed more specific instructions for synthesizing language and information across multiple sources? As part of the process of building data awareness into the execution chain, have we in effect prompted the LLM to marshall new language data only to amplify trends in existing language data that may be inaccurate or biased?

Argumentation with Primary Sources

For the final test here, we will give the agent one additional tool—the primary source documents themselves—and prompt it to cite that material while refuting one of its classmate's (i.e. its own) points.

loader = DirectoryLoader('utf_benjamin_texts')
benjamin_texts = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
split_texts = text_splitter.split_documents(benjamin_texts)

docsearch_benjamin = Chroma.from_documents(split_texts, embeddings, collection_name="benjamin_texts")

benjamin_texts = RetrievalQA.from_chain_type(llm=llm, chain_type="stuff", retriever=docsearch_benjamin.as_retriever())

We will need to append the new tool to the list of existing tools and re-initialize the agent.

tools.append(
    Tool(
        name = "primary course material",
        func=benjamin_texts.run,
        description="""useful when you need direct quotes from primary course material.
                    Input should be a fully formed question."""
    )
)

agent = initialize_agent(tools, llm, agent="zero-shot-react-description", verbose=True)

Response #4

Of all the chains implemented in this project, this one varies the most across multiple executions. Without additional configuration, the random component in how the agent selects tools entails that the order in which it poses inquiries and makes observations about the documents at hand is nondeterministic even if (given a model with temperature=0) any single specific input in a chain can be expected to produce consistent outputs.

In this case, the agent retrieves information from the primary source documents first, building up a kind of arsenal of possible topics around which to build arguments (as one might do in preparation for a debate competition, not knowing what the other team might come up with).

response_4 = agent.run(f"""use what you learned from the lecture as well as your existing
knowledge to respond to the following post by your classmate. follow the guidelines in the
syllabus. do not regurgitate the lecture material: give equal wait to what you learned in
class and your own thoughts about benjamin as you consider your classmate's ideas.

important: you must use primary course material to refute at least one point that your
classmate made.

this is your classmate's post: {response_3}
""")

> Entering new AgentExecutor chain...
I need to respond to the classmate's post by considering both the lecture material and my own thoughts on Benjamin. I should also use primary course material to refute at least one point made by the classmate. First, I will think about the main points made in the post and then refer to the primary course material to find a point to refute.

Action: primary course material
Action Input: What are some key points in Walter Benjamin's work that can be used to refute or support the classmate's post?

Observation: To engage with your classmate's post, it would be helpful to know the specific arguments they are making. However, I can provide some key points from Walter Benjamin's work that you can use to either refute or support their post:

1. In "Theses on the Philosophy of History," Benjamin critiques historical materialism and argues that it can only win when it employs the services of theology. This point can be used to discuss the role of theology and historical materialism in understanding history and society.

2. In "The Work of Art in the Age of Mechanical Reproduction," Benjamin explores the impact of technology on art and its role in society. He argues that the mechanical reproduction of art has led to a loss of its aura, or the unique presence of an original artwork. This point can be used to discuss the changing nature of art and its relationship to society, as well as the role of technology in shaping our understanding of art.

3. Benjamin also discusses the political implications of the mechanical reproduction of art, arguing that it can be used as a tool for both revolutionary and fascist purposes. This point can be used to discuss the role of art in politics and the potential for art to be used as a means of control or liberation.

4. In his writings on memory and history, Benjamin emphasizes the importance of remembrance and the role of memory in shaping our understanding of the past. This point can be used to discuss the importance of historical memory and the ways in which our understanding of the past is shaped by our present experiences and perspectives.

5. Benjamin's concept of the "narrow gate" through which the Messiah could enter suggests that every moment in time has the potential for radical transformation. This point can be used to discuss the potential for change and the importance of recognizing the transformative possibilities within each moment.

These key points can be used to engage with your classmate's post, depending on the specific arguments they are making. Remember to consider the context of their post and the points they are making in order to effectively refute or support their arguments using Benjamin's work.
Thought:I now have some key points from Benjamin's work that I can use to engage with my classmate's post. I will consider both the lecture material and my own thoughts on Benjamin while addressing the main points made in the post. I will also use one of these key points to refute a point made by the classmate.

Final Answer: In response to your post, I agree that Walter Benjamin's ideas on continuity and historical progress challenge conventional notions of history and the role of art in society. His critique of historical materialism and the importance of memory in shaping our understanding of the past (as mentioned in the primary course material) aligns with your point about questioning the linear narrative of history and considering alternative perspectives.

However, I would like to refute your point about the accessibility and reproducibility of art and information giving individuals more power to challenge oppressive systems and question the status quo. While Benjamin does acknowledge the potential for art to be used as a tool for political and social change, he also warns about the potential dangers of the mechanical reproduction of art. In "The Work of Art in the Age of Mechanical Reproduction," Benjamin argues that the loss of aura in reproduced art can lead to a detachment from tradition and history, which can be exploited by oppressive systems for their own purposes. This suggests that the increased accessibility and reproducibility of art and information may not always empower individuals to challenge oppressive systems but could also be used to manipulate and control them.

As for the relevance of Benjamin's ideas today, I believe that his thoughts on the role of art in society and the potential for radical transformation in every moment (the "narrow gate" concept) are still applicable in our current context. The rapid advancements in technology and the ever-changing nature of art and its relationship to society make Benjamin's ideas even more pertinent as we continue to grapple with the implications of these changes on our understanding of history, progress, and the role of art in society.

> Finished chain.

Analysis of Response #4:

• Alignment - As a simulation of learning activity, this final output is somewhat arbitrarily constrained by the fact that the LLM responds in an essentially recursive manner to a previous output generated by an agent executor chain that did not have access to the same "primary course material" tool.
  
  However, it's interesting to note how this agent works to "align" the new data with the existing data in order to support a set of moral and political opinions that it articulates consistently—perhaps deterministically. It was not explicitly prompted to do this; it was in effect explicitly prompted to do the opposite of this.

Concluding Thoughts

There would be clear safety and ethical implications if it were easy to provoke ground shifts in the moral or ideological orientation of responses simply by exposing the model to new language data. Implementing robust adversarial testing is outside the scope of this project; but even this one final ReAct loop highlights a set of important and difficult questions. Stated generally:

As we develop increasingly complex "agentic" and cognitive AI systems by building new forms of intelligibility from static representations of language, how can we—and to what extent should we—work to trace, to explain, and to control the processes through which systems of language align to systems of value?

If this is the broad technological-historical question raised by the specter of an AI system in a college classroom, there is a more immediate practical takeaway, here. It is a perennial source of consternation among college faculty that even highly intelligent students sometimes have great difficulty extracting basic information and following basic instructions from a course syllabus. We are a ways away, yet, from using LLM technology to reliably detect AI writing and other forms of plagiarism; but testing throughout this project suggested to me that GPT-4 is ready, pretty much out of the box, to be used as part of an automated process to evaluate student work for its adherence to syllabus guidelines (even without a well-structured rubric).