“We need to be using AI more. Make it happen.”

In 2023, over half of the in-house marketers we surveyed told us they’re under pressure from their boss to use AI to create content faster and cheaper. (Check out more results from that survey, if you’re curious.)

To truly capture the exponential efficiency gains that AI promises, though—and the saved time and money that business leaders are pressuring their teams for—one writer going back and forth with ChatGPT to draft a blog post isn’t going to cut it.

The techniques we’re going to discuss here are focused on content at scale. Think hundreds of local landing pages, thousands of product descriptions for an e-commerce website, or dozens of new service pages for agency clients.

AI is perfect for these use cases because using humans to write thousands of product descriptions, for example, is cost-prohibitive, but having those descriptions is still important for SEO. More importantly, by using AI to craft high-quality descriptions, you can actually provide value to your audience in an area where your competitors are likely just using MadLibs style templates—if they’re doing anything at all.

Trying to use ChatGPT out of the box for this, though, will drive you nuts. Combine the API of your favorite LLM with an automation tool like Zapier plus some advanced prompting, though, and you’ve got magic.

This type of process doesn’t work for all tasks. With the right use case, though, leveraging AI in this way allows for exponential efficiency gains, rather than the marginal gains that come from an individual user working with ChatGPT directly.

The challenge of this heavy-duty usage is that it requires a deeper understanding of how to get what you want from AI. You no longer have an individual person going back and forth with an LLM in an interface, able to adjust prompts and ask for changes on the fly. You need to build prompts that work for all the different situations in which they’ll be used and that get you the results you need, even when there may not be human intervention until the final step. 

We’ve been using AI in this way for certain tasks since September 2022 and have learned a ton about what works and what doesn’t. Using these six tips can save you hundreds of hours of work and dramatically improve the quality of your content. Some of these are good strategies no matter how you’re using AI. When you’re using it at scale, though, they become that much more important and might require you to think about them in slightly different ways.

1. Use variables to customize prompts

This is the basic building block of prompting at scale. Rather than using a prompt like “Write an outline for a 1500-word article about brute force attacks,” the prompt you write will be something like this:

USER: Write an outline for this article.
Topic: {topic}
Word length: {word_length}

When running this prompt, I’ll then replace the variable {topic} with “brute force attacks” and {word_length} with “1500.” (Well, *I* won’t—the program or Zap I’ve created will do it for me. That’s the beauty of using prompts programmatically.)

Creating prompts that use variables in this way requires a “greatest common factor” approach. For any single given topic, there will no doubt be a prompt that could do better—but it wouldn’t do as well for other topics. The goal is to find a prompt template that works decently well across all the topics you’re writing for, and then make use of variables to customize it.

In-house marketers: If you’re creating content for a single brand, your prompts can likely be more specific because all the content you’re creating falls within the same industry and should follow the same style guide. 

Agency folks: If you’re creating content for multiple clients, your prompts will have to be more general, and you’ll likely make greater use of variables for things like voice and industry.

2. Use the right size buckets

While testing a prompt to create outlines for articles of different lengths, I might find a prompt that works well on long articles but not on shorter ones. At that point, I’d have to decide whether I want to maintain two different prompt flows and send articles one way or another based on their word count, or find a prompt that works decently well for both.

This is the constant tension when using prompts at scale: How big should your “buckets” of use cases be? The larger your bucket, the more variables you’ll need to use to customize the prompt for each use case. You may also see a decrease in quality because you’ll be using the same prompt in situations where a different prompt would perform better. You will, however, save time on testing and building different flows.

Here’s an example at one end of the spectrum: Instead of using a dedicated prompt to create outlines, I could use a single prompt to create outlines, introductions, articles, etc. That prompt might look something like this:

USER: Write an {content_type} for this article.

topic: {topic}
word count: {word_count}

In addition to filling in the other inputs like topic and word count, I would then also replace {content_type} with “outline” or “introduction,” depending on what I needed.

In my experience, the time I might save from only having to build a single prompt flow for all those content types is not worth the drop in quality I would see across some of them. In other words, that bucket of use cases would be too big.

On the other end of the spectrum, I could use a different outline prompt for every word count increment. One of those prompts might look like this:

USER: Write an outline for this 1000-word article. There should be about six main sections, with appropriate subheadings in each of them.

topic: {topic}

By using a different prompt for each word count and designating the approximate number of sections each should have, I might get better (or at least more consistent) outlines, but would they be enough better to merit the time spent building those different flows? Probably not.

In this case, my bucket of use cases would be too small, and I’d be doing a lot of unnecessarily repetitive work. I could probably use a single prompt for every word count (by includng the {word_count} variable) and still get close to the same results simply by reminding the LLM to make sure the outline is an appropriate length for the word count.

Despite the officialness of my Goldilocks diagram, it’s worth noting that the right-sized bucket isn’t universal.If you’re creating two types of articles that should each have a very different structure—for example, tutorial articles that should follow a step-by-step format with a list of what you’ll need at the top vs. case studies that should follow a problem/solution/results format—using a single prompt for “outlines” could be too big a bucket for you. Instead, you’d likely want to create two different prompts for those two kinds of articles, with each prompt detailing the specifics of the format you’re looking for. This approach would give you better enough results to make it worth the effort of building those two flows.

The most extreme example of a small bucket is no bucket at all, ie. using a unique prompt every single time. Again, in many cases you can get better results that way, but you lose all the efficiency gains of using AI at scale.

3. Take advantage of few-shot prompting

“Show, don’t tell” is one of the golden rules of prompting. You can get better results by providing a few examples of what you’re looking for, rather than trying to describe what you want. This is known as “few-shot” prompting as opposed to “zero-shot” prompting, which is when you ask it to do something without including any examples of what you want.

Let’s say I work for an agency creating content for twelve different clients, and I’m using AI to create outlines. I want every outline to have a few common elements:

1. The first heading should be “Introduction”
2. The second heading should be “What is ” + the primary keyword
3. The last heading should be a call to action that references the client’s business

The best way to get AI to give me what I need consistently is to include examples of what I’m looking for in the prompt. Each example outline should meet all those requirements and show what a “good” outline looks like to me.

My final prompt might look something like this:

USER: Write an outline for this article.
topic: Understanding the Risk of Brute Force Attacks
word count: 1200
primary keyword: brute force attack
business: Hank's Digital Security Solutions

1. Introduction
2. What is a brute force attack?
3. Common types of brute force attacks
 A. Credential stuffing
 B. Password cracking
 C. Distributed brute force attacks
4. Impact of Brute Force Attacks
 A. Data breaches
 B. Financial losses
 C. Reputational damage
5. How to protect your business against brute force attacks
 A. Strong password policies
 B. Two-factor authentication
 C. Account lockout mechanisms
 D. Intrusion detection systems
6. Prevent Brute Force Attacks with Hank's Digital Security Solutions

Write an outline for this article.
topic: How a Brand Ambassador Can Boost Your Marketing
word count: 800
primary keyword: brand ambassador
business: Magic Marketing Solutions

1. Introduction
2. What is a brand ambassador?
3. The role of a brand ambassador in marketing
 a. Raising brand awareness
 b. Engaging with your audience
4. What to look for in a brand ambassador
 a. Professionalism and positivity
 b. Passion for the brand
 c. Excellent people skills
5. Find Your Next Brand Ambassador with Magic Marketing Solutions

Write an outline for this article.
topic: {topic}
word count: {word_count}
primary keyword: {keyword}
business: {business_name}

LLMs are great at following patterns. By including examples of what I’m looking for, the model will pick up on the common elements—that the first heading is always “Introduction,” the second is always “What is” followed by the keyword, and the final section always mentions the business—and incorporate those elements in the outlines it writes.

One important note here is to use a range of examples in your prompt, lest the model pick up on a pattern you hadn’t intended to convey. If, for example, you use three examples that are all “how to” articles with a numbered list of steps to teach the reader how to do something, it might try to follow that same pattern of creating a step-by-step guide even when asked for a different type of article. (To think of it in terms of “buckets” again: This is a situation where, depending on how many different types of articles you’re creating and how distinct they are, you may want to use separate prompts, each with their own examples, rather than a single prompt for all of them.)

Bootstrapping your way to good examples

Coming up with a few examples of “good” to use in your few-shot prompts can feel like an annoying waste of time. Depending on what I’m trying to create, I often will use AI to bootstrap my way there:

1. I’ll ask ChatGPT for an outline without providing any examples. I’ll then edit that outline extensively until it aligns completely with what I’m looking for. 
2. I’ll add that edited outline as an example in my original prompt and ask ChatGPT for another outline. The single example will help it get closer to what I want, though I will still need to spend some time editing this second outline as well.
3. I’ll use both edited outlines as two examples in my prompt, and ask ChatGPT for a third. This time, the output should be even closer to what I’m looking for, and I can likely spend less time editing it.

I’ll repeat this process—using as many edited examples as I have in my prompt to generate the next one, and editing each new one until it’s “perfect”—until I have as many examples as I want.

It’s worth noting that I likely wouldn’t use this approach for something like introductions. I would want those examples to be fully human in order to get the best outputs from my prompt going forward—otherwise what I get back will be more likely to sound like AI. For something like outlines, though, which are less about the voice and word choice and more about the logical organization of information, AI can give me a decent starting point for those examples.

4. Ask for an analysis first

Also known as “giving the model time to think,” you can get better results by asking the model to analyze the task at hand before providing you with any deliverable. 

Here’s an example of what that could look like:

USER: You will be writing an outline for a given topic. First, analyze the searcher intent. Consider the specific information that readers are seeking based on the topic and keyword. Think about how to optimize the headings for SEO. Provide your analysis inside <analysis></analysis> tags.

Then, create an outline that addresses the topic with specific headings and subheadings, ensuring that each section will directly answer the searcher's intent and contribute to the content goal. Provide the outline itself inside <outline></outline> tags.

topic: {topic}
keyword: {keyword}
content goal: {goal}
target audience: {audience}
word length: {word_count}

By telling the model to put its analysis inside <analysis></analysis> tags and the outline itself inside <outline></outline> tags, I can easily parse the response to get only the part I care about, ie. the outline.

When using the prompt at scale, I can ignore the analysis and only save the outline itself. While testing prompts, however, it can often be helpful to look at the <analysis> portion of the response as a way of understanding how the model is approaching the problem. If it’s referring too much to one portion of your instructions and missing something else, for example, that could be a sign that you’re trying to have it meet too many requirements at once. In that case, you should pare the prompt down to focus on the most important requirements.

5. Use a prompt chain rather than a single prompt

A prompt chain is when you use multiple prompts in a row, threading them together to provide more context for the model.

This approach can often get you better results than using a single prompt. It can be especially helpful in breaking down different things you want the model to focus on. I’ve found it works best if my first prompt focuses on the general principles I want the model to follow for whatever the piece of content may be, and the second prompt focuses on specifics that I do or don’t want.

For example, using the prompt above as my first prompt, I would get an outline that has taken the searcher’s intent into account and contains the most valuable information a reader would be looking for. However, I might have some other requirements I want the outline to follow, too, and I could include these in a second prompt.

USER: You will be writing an outline for a given topic. First, analyze the searcher intent. Consider the specific information that readers are seeking based on the topic and keyword. Think about how to optimize the headings for SEO. Provide your analysis inside <analysis></analysis> tags.

Then, create an outline that addresses the topic with specific headings and subheadings, ensuring that each section will directly answer the searcher's intent and contribute to the content goal. Provide the outline itself inside <outline></outline> tags.

topic: {topic}
keyword: {keyword}
content goal: {goal}
target audience: {audience}
word length: {word_count}
ASSISTANT: {model's response, containing both the analysis and the actual outline}
USER: Now review the outline. Make the following changes as necessary:
- The first section heading should be "Introduction."
- There should not be any references to case studies or testimonials.
- Make sure the outline is tailored to the topic, providing specific names where applicable. Avoid using placeholders like "Item 1" or "Service A" and instead use actual names and descriptions that are current and relevant.
- Make sure the outline is appropriate for the requested word length and not too long.

Return only the revised outline inside <outline></outline> tags.

In my second API call, I would include all three of these messages in order (blue, yellow, and blue).

By breaking up the requirements of what I want in this way, I’ve found the final outline will do a much better job of meeting them than if I put them all into a single prompt. You’ll also notice I put all of the specific formatting requirements into the second prompt. This is because if you split them between the first and second prompt, at times it might inadvertently “undo” requirements from the first prompt when revising the outline in the second prompt.

Using multiple prompts is also an easy way to allow your flows to work across more use cases. If I were an agency, for example, I might use the same first prompt across all my clients, and then customize only the second prompt with each one’s specific style guide requirements.

Note on the last line in the second prompt: It’s possible the original outline could already meet all my requirements. In that case, if I hadn’t included that final line about returning only the outline inside tags, the model might respond with something like “This outline is well-suited to the topic. It includes specific names and is appropriate for the word count” etc. Because I’m using these prompts at scale and will be delivering the output of the final prompt directly to the customer, I don’t want it to tell me the outline already meets my requirements—I just want it to give me the final outline. By specifying that I only want it to return the revised outline, I’m ensuring that I’ll get a consistent output I can use without having to review it.

6. Test your prompts

I’ve already written about my process for testing prompts, and I highly recommend coming up with your own system if you don’t yet have one.

LLMs have gotten good enough that the first prompt you try will likely get you an okay response. But when you’re using AI at scale, the difference between “okay” and “great” on hundreds or thousands of outputs can add up to hundreds of additional hours of human work needed to make your content publish-ready. It’s well worth the time spent testing to find the prompt that gets you “great” out of the gate.

It’s also important to remember that LLM behavior can change over time, so a prompt that works for you one month might get different results later. Test early, test often.

Using AI to create content at scale is an entirely different ballgame than becoming a ChatGPT power user. If you’re working on your process and want to chat strategies, roadblocks, writing your own shitty functional python code to access the OpenAI API, the risk of human extinction by AI, or anything else, reach out at megan@verblio.com.

If you don’t actually want to deal with this stuff yourself but need to get your boss off your back about using AI, check out our hybrid human-AI content to get all the efficiencies of AI without having to write a single prompt.

The post Tips for using AI to create content at scale appeared first on Verblio.

Step into the prompt testing lab

If you’re using AI to create content and don’t think you need to bother testing prompts, skim my last article on why prompt testing is so important.

In this article, I’m going to share how I recently tested 55 prompt variations to change the way we generate outlines for customers of our human-crafted AI content.

My goal here is to help you think about your own testing process. You might have different goals or be using your prompts to do something different than generate outlines, but the general principles and framework are helpful regardless of your use case.

A quick glossary:

• Prompt variation: If I test multiple different prompts aimed at the same goal, e.g. writing an outline, those are prompt variations. A given variation could be a single prompt, or it could include multiple prompts in a chain.
• Input: I’m using “input” to refer to the specific variables used within a prompt. Creating prompts with these variables in place allows you to reuse the same prompt over and over.
• Output: Output refers to the LLM’s response to a prompt. In ChatGPT, this is the response you see in the window. Via the OpenAI API, this is the response in the response.choices[0].message.content field. When using a prompt chain, I’m using “output” to refer to the final output (ie. the one with the content I actually want, rather than the model’s intermediate response.)

Two commandments of LLM testing

1. Define good as quantitatively as you can

Testing LLMs often starts with an ambiguous idea of “I want to see which prompt gets me a better output.” Some of that “better” may be subjective, and there’s no way around that. But coming up with at least a few quantitative measures will make it much easier to evaluate the outputs of the prompts you’re testing, even if it’s just knowing a general range you want that measure to fall within.

Example metrics I’ve used for different tests:

• Word count: When generating introductions, for example, I wanted to keep them in a certain word count range.
• Reading level: In order to target a certain reading level, I automated running the prompt outputs through a tool like Readability to compare the reading levels. (If I had read this article on GPT-4’s ability to evaluate readability first, I would have just used that model instead of a separate tool. Heads up, that article contains a ton of statistical concepts but is well worth skimming if you’re interested in readability at all.)
• Number of times a keyword is used
• Whether a prohibited word is used
• Length relative to the original: For example, I was building a tool to remove some of the fluff from AI-generated content and rewrite it more concisely. I cared about how long the rewritten text was relative to the original because I didn’t want to pare it down too much, but I also wanted to be sure it wasn’t making it longer. Word count alone wouldn’t have told me what I needed to know—I needed to evaluate the output relative to the specific input.
• Runtime: If someone will be waiting in real-time for the output, I don’t want to use a prompt chain that takes minutes to run.

Most likely, you won’t be able to reduce all of your evaluation to quantitative metrics. At some point, you’re actually going to have to review the outputs and decide for yourself that “It was the best of times; it was the worst of times” is a stronger opening sentence than “It was an era characterized by both joy and sadness.” At the very least, though, having some metrics in place will allow you to eliminate certain outputs off the bat, reducing the number you need to manually review.

Using AI to evaluate outputs

Wondering if you can use AI to help you qualitatively evaluate outputs? Research suggests that GPT-4 can reach 80 percent agreement with human preferences—which, for the record, is the same level of agreement humans reach with each other. I’m wary of relying exclusively on this approach, though, because I did my own testing with it over the summer, and the results weren’t exactly confidence-inspiring.

How I tested: I presented a pair of options to GPT-4 and asked it to evaluate which one was a better example of a certain voice. I used a low temperature to reduce the variability, and ran the exact same prompt with the same choices twice—and then reran the prompt with the same two choices but reversed the order in which the choices were given.

In total, GPT-4 compared the same two choices four times. I did this for 274 different pairings, and the model only had unanimous agreement with itself (meaning it chose the same choice all four times, regardless of whether that choice was presented first or second) on 53 percent of those pairings.

That’s the pink pie slice above. The second most common outcome (the purple slice) was the model choosing each option in the pair twice, meaning its choice was entirely arbitrary.

It’s worth highlighting that these stats only measure GPT-4’s consistency when evaluating pairs and don’t even begin to address whether its choice was actually “right,” ie. if it would match a human evaluator’s preference. Precision and accuracy: You need both if you’re going to use AI as an evaluation tool.

All this isn’t to say that using AI to judge its own outputs is impossible. I could no doubt raise its level of consensus (precision) by improving the prompt that asks it to evaluate the two pairs, and providing examples of my own choices in that prompt would likely help to align it more closely with a human’s preferences (accuracy). That takes even more time and testing, though.

Bottom line: If you outsource qualitative evaluation to AI without putting in a lot of time first to make sure its evaluations are A) consistent, and B) aligned with your preferences, your results won’t be very good.

2. Test on multiple inputs

If you’re using AI at scale to create content, you need to test your prompts on multiple inputs. Unless you’re using a very low temperature, LLMs will give you different outputs every time even for the same input, and its performance will vary even more on different ones.

Be sure, too, that your inputs represent the range of how you’ll be using that prompt. If I create content for several different industries, for example, I’m going to make sure the inputs I use for testing aren’t all from a single industry. Similarly, if I want to use the same prompt to generate outlines for articles ranging from 600 to 2000 words, I’m going to include a range of word counts in my inputs. Otherwise, I might end up with a prompt that generates great outlines for 2000-word articles, but not for 600-word articles.

For testing a prompt to create outlines, for example, I might use a spreadsheet of inputs that looks like this:

Each row represents a different set of inputs. I would run the same prompt six times, each time replacing variables in the prompt like {topic} or {word_count} with the actual values from one of the rows.

My prompt testing process

With those principles in place, let’s take a look at how I tested 55 different prompts to generate outlines for our customers. I’ll cover what I was looking to improve, the tools and process I used to test the different prompts, the resulting metrics, and how I evaluated the winning prompt.

What I wanted to improve

I wanted to make a few specific improvements to the outlines that were being generated for our customers:

• Shorter outlines: Our existing outlines often included too many sections, resulting in the final article being too long for the designated word count.
• Reduced risk of hallucination: If the outline included sections like “Case Studies,” “Testimonials,” or “References,” AI would inevitably try to make up that information when writing the article, which meant extra work for our human writers. I wanted to improve our process to prevent the AI from including those sections at all.
• Better outlines for the format: For example, if the customer’s topic is a listicle like “X Best VPNs,” the headings in the outline should each be a specific VPN rather than “VPN #1,” “VPN #2,” etc., and those sections should comprise the bulk of the article. I also wanted to make sure our outlines did a better job keeping the reader’s intent in mind and covering the information they’d expect to see when searching for the customer’s keyword.

A final consideration for the quality of our customer experience, though not for the quality of the content itself, was how long it takes for the outline to be generated. Because customers are in our app waiting in real-time for the outline to appear so they can review and edit it before finalizing their order, it matters whether they have to wait ten seconds or a minute.

The process

Google Sheets and Google Colab are my best friends.

In one sheet, I came up with an initial list of prompt variations. In some cases, the difference between two prompts would be just a few words. In others, they would look totally different. Here’s an example:

Prompt variation #1
write an outline for the topic: {topic}
word length: {word_count}

As you can see, I started extremely simple to understand what the LLM would do with minimal direction. For other variations, I used more sophisticated prompting strategies:

Prompt variation #5
You will be writing an outline for a given topic. First, think through how the article should be structured, given the searcher intent for the keyword. Provide these thoughts inside <analysis></analysis> tags. Then, provide the outline itself inside <outline></outline> tags.
topic: {topic}
keyword: {keyword}

In a second sheet, I stored the brief information for 30 different real articles that already had been ordered and delivered to our customers, along with the outlines that originally were generated for those. 

The next step involved using OpenAI’s API. If you’re not comfortable writing code but have access to a low- or no-code tool like Make or Zapier, you could access OpenAI’s models that way instead. Either way, it is abundantly easier than copy/pasting prompts and outputs from a ChatGPT window, and the only viable way to do real testing at scale.

Using a Python program in a Colab notebook, I sent a prompt to the model (mostly either GPT-4 or GPT-3.5-turbo).This prompt was created by replacing the variables in one of the prompt variations from the first sheet with one of the 30 sets of inputs from the second sheet—and doing that again until I had prompted the model with every combination of prompt and inputs. The program then automatically saved the resulting outlines to a third sheet.

For every new outline the model generated, I then evaluated the quantitative metrics I cared about, based on those improvements I identified above:

• How much shorter was it than the outline we had previously generated for the customer using our existing prompt flow?
• Did it include any sections we didn’t want to see, like case studies or references?
• How long did it take to run?

I aggregated these metrics for each prompt variation and compared the overall results.

I couldn’t rely only on numbers, so I also manually reviewed the outlines to see whether listicles were properly formatted, if they made sense, etc.

I then iterated on the best-performing prompt variations to see if I could further improve the results, and did the same process again. And again. And again, and again, and again.

The results

By the end, I had tested 55 different variations of prompts, models, and temperatures. The results for some of them are in the chart below.

First callout: You can see the results getting better (more green, less red) as I iterated further. This is why testing matters. You can make very real improvements, across multiple dimensions, that will mean significant time savings when you’re running these prompts over 100s of cases.

Columns B through E are all about how much shorter the new outline was than the one we had previously generated. Column F shows how long it took each prompt (or prompt chain, in some cases) to run, which is approximately how long our customer would have to wait in the app. Column G shows how many of the new outlines contained a section it shouldn’t have, like “Case Studies.”

Consistency matters

The main reason it’s so important to test your prompts on multiple inputs (30, in my case) is because it will behave differently every time. This was very important for us when looking at how much shorter the new outline was than the old one.

The median reduction (column B) is self-explanatory, but if we’d looked only at that measure, we wouldn’t have learned anything about how consistent that prompt variation was across inputs. Looking also at the minimum reduction (column C) was important because this showed the worst-case scenario: Each prompt variation actually resulted in a longer outline than the original for at least one of the test articles. For prompt 41, that worst case meant getting an outline that was more than twice as long as the one we’d originally gotten with our current prompts. For prompt 55, on the other hand, that worst case was significantly better, with the new outline being only 10 percent longer than the original.

The maximum reduction (column D) isn’t color-coded because I wasn’t aiming for a particular percentage reduction, though the 84 percent reduction for prompt 43 is probably too high. What’s more important for understanding how consistently a prompt behaved is the spread between the minimum and maximum reductions: column E. The lower that number, the more consistent the outputs from that prompt were, which is what we want.

On runtime

Two main factors impacted the runtime (column F):

1. the LLM being used
2. the number of prompts, ie. whether it was a single prompt or a prompt chain

The length of the prompt also impacts the runtime, but to a much lesser degree than those two factors.

The tradeoff is that you can often get qualitatively better results by using longer prompts or prompt chains, but it will then take longer to run. However, different models also have different runtimes. In general, older, smaller models are faster, while newer ones like GPT-4 are slower, due to both their size and higher traffic.

The winning prompt

The prompt variation that ended up being the best overall, on both quantitative and qualitative measures, was number 54.

You can see the results for prompt 54 met my original goals:

• It consistently resulted in shorter outlines (but not too short!) and had a relatively low spread between the minimum and maximum reduction (column E).
• The median runtime of 15 seconds (column F) wasn’t the lowest, but it was still less than half the average runtime for the prompt we were currently using.
• It never included a section we didn’t want to see in the outline (column G).
• When I reviewed the outlines manually, they were what we wanted in terms of quality and format, etc.

I’ll go into the exact prompting strategies in more detail in my next article, but in a nutshell, here’s what made prompt 54 work so well:

• Giving the model time to “think”
• Providing examples of what I wanted
• Using a prompt chain (rather than a single prompt) to improve its accuracy at meeting specific requirements—but doing this on an older model to keep the runtime relatively low

Could I have kept going with more prompt variations and seen more improvements? Sure. But at some point, I wanted to get a better prompt into production so our customers could start seeing the improvements sooner rather than later.

Moral of the story: Test a reasonable amount, but don’t let perfect become the enemy of good. As the gains from new variations become smaller, you’ll want to declare a winner and get on with your life.

In the next article, I’ll get into the content of the prompts themselves by sharing the specific prompt strategies I tested and tips for writing prompts that work at scale. If you have questions about the testing setup, the Python code I used, or anything else, send a message to megan@verblio.com.

The post How I tested 55 prompts to create better outlines appeared first on Verblio.

tl;dr
– Testing different prompts can improve your outputs significantly.
– Even small formatting changes can make a big difference in prompt performance.
– You need to test prompts across different models and over time, as models behave differently from each other and can change over time.

If you’re using AI for content creation at scale, you (or someone on your team) should be spending a fair amount of time testing different prompts.

It’s easy to overlook this part of the process, especially as large language models (LLMs) like GPT-4 improve. If you were using GPT-3 in 2022 before the release of ChatGPT, for example, the need for testing was painfully obvious: A poorly formatted prompt would often get you gibberish.With today’s LLMs, though, you’ll likely get a decent enough output from the first prompt you try, making it easy to assume there aren’t many gains to be made.

Even as the floor of LLM performance has been raised, though (ie. fewer responses that are complete gibberish), the ceiling has also been raised. You’ll leave a lot of capabilities on the table if you aren’t regularly testing to optimize your prompts.

If you’re using AI at scale to run prompts over 100s or 1000s of inputs, this testing becomes even more important. Even if better prompts only provide marginally better results on a single input, those improvements add up quickly when you’re using those prompts at scale.

In this article, I’m going to walk through recent research and AI happenings to explain three aspects of LLMs and prompting that make testing so important. In the next article, I’ll share my own most recent experiments for our human-crafted AI content offering to show what I tested, how I set it up, and what my results were.

But first: What do I mean by using AI at scale?

When I talk about using AI at scale to create content, I’m talking about using large language models through an API, where you’re using the same prompt template over and over again on different inputs. 

For example, you might have a prompt you use to generate an outline for an article. The basic prompt structure could look something like this:

Write an outline for this article.
topic: {topic}
word count: {word_count}
primary keyword: {keyword}

You’d then run this prompt on a list of briefs, automatically filling in each of the bracketed variables with the appropriate information for each article. You might then write the output of each prompt to a new Google Doc, or store the outline alongside the article information in whatever project management tool you use.

There are unlimited ways to set up a workflow to use AI programmatically like this, depending on what your current content creation tech stack looks like. For many workflows, you may want to use a tool like Zapier or Make as a no-code option. I’ll go into more of the technical stuff in other posts, but it’s important to understand that I’m not talking about you as an individual user working directly in ChatGPT or another LLM-enabled tool here.

Three reasons you need to test different prompts for AI content creation

If you’re only using ChatGPT for one-off tasks, it’s still worth testing different prompts if for no other reason than you might learn surprising things about how it behaves. When you’re using prompts at scale and running them on dozens of inputs at a time, though—which is where you really start to see the efficiency gains from AI—testing becomes a non-negotiable.

Here are three of the biggest reasons why:

1. Prompt details matter—even ones you think shouldn’t

Even tiny changes in a prompt can make a big difference.

A team from the University of Washington recently published a paper with the great subtitle of “How I learned to start worrying about prompt formatting.” If you don’t read academic papers for fun, this Twitter thread from lead researcher Melanie Sclar does a great job summarizing the findings.

The key takeaway, though, is this image:

This is WILD.

Here’s what it’s showing: The blue box shows the default formatting of their prompt. The small blue diamond shows how accurate the model’s output was when that prompt was used—roughly midway between 0.5 and 0.804 on the scale.

The other boxes show different ways of formatting that same prompt. In the top left box, for example, the space has been removed between “Passage:” and “<text>,” and “Answer:” and “<text>.” In the top right box, the passage and answer are both on the same line. In the bottom right, they’re on the same line and the colon has been removed. In the bottom middle, “Passage” and “Answer” have been written in all caps. In the bottom left, they’re in all caps and the colon has been removed. In each case, the dotted line shows where the accuracy of the model’s response to that prompt fell on the scale.

The important thing to note is that none of those changes impact the content of the prompt. A human reader would interpret these formats as all saying the same thing, and the model is still getting the exact same examples to help it understand what it’s supposed to do. The only things that are different are the spacing, capitalization, and punctuation. And yet, those arbitrary formatting choices impact the accuracy of its output tremendously, ranging from 3.6 percent accuracy when the space was removed in the top left box, to 80.4 percent accuracy when the passage and answer were on the same line with no colon, as shown in the bottom right box.

The takeaway here is to test variations of your prompt, even if it’s only changing small things you think shouldn’t make a difference. LLMs might be predictive, but their behavior is often far from predictable.

2. Every LLM is a special snowflake

One other important takeaway from that paper: You won’t be able to find the best format for your prompt and simply use it forever on any model you want. From Appendix B.2:

“Table 3 shows that if format p1 has lower performance than format p2 under model M, there is < 0.62 probability that this trend would hold under another model M′ (random chance is 0.5). This weak relative order preservation suggests that prompt format performance in a model may not be extrapolated to a different model, or in other words, that there are no inherently good or bad formats.“ (emphasis mine)

The corollary of the finding that there are no inherently good or bad formats is that every model behaves differently. Just like all humans are different, AI is not a single, monolithic entity that always behaves the same way. Each model has its own quirks and will respond to prompts in different ways.

When a new state-of-the-art model is released, it likely will be “better” than its predecessors, in the sense that it will be more capable of producing useful responses. However, it will also be simply different than whatever model you were previously using, so you can’t keep using the same prompts and expect to get the same or better results.

Anytime a new model comes out, or if you switch models for any reason, you need to test your prompts on that model.

If, for example, you wanted to switch over to Anthropic’s models during the OpenAI drama with Sam Altman last November, you would have quickly learned that the prompts you use with GPT-4 don’t work the same way with Claude-2. One immediately obvious difference is that Claude-2 is much more likely to preface its responses with friendly fluff like “Sure! Here’s the xyz you asked for… ,” or say things like “I apologize, upon reflection I do not feel comfortable providing advice to befriend or interact with Sasquatch (Bigfoot), as that could encourage harmful behavior. Instead, I suggest focusing content on appreciating nature, caring for the environment, and respecting wildlife from a safe distance” when asked to write an outline for a satirical article on how to befriend Sasquatch. (That’s a purely hypothetical example, of course.)

There are other, more significant differences between the two models, though, which mean you should expect to use different prompting strategies with each one.

Bonus: Check out the Chatbot Arena Leaderboard to see LLMs engaged in gladiator-style combat which LLMs are currently ranked the best, both by crowdsourced voting and by their performance on certain benchmarks.

3. Model behaviors can (and do) change over time

So, you should both test your prompts initially to find the most effective ones as well as re-test them anytime you’re changing models. That’s not all, though: You also need to re-test them periodically, even if you’re still using the same model.

Why? Because models are constantly being updated behind-the-scenes, and this can change their behavior.

In 2023, for example, there was a lot of talk around GPT-4’s performance degrading, with users saying it had gotten “lazier.”

Earlier that year, researchers at Stanford and Berkeley published a paper showing how the performance of GPT-3.5 and GPT-4 had changed significantly over time. While it’s not correct to say that the models had gotten strictly worse, it is definitely true that their performance had changed. If you’re using prompts in your content workflow—or if you’ve built a product atop LLMs—and the model’s behavior changes, whether for better or worse, you need to know that.

To better understand the results of that paper, check out this article for a great analysis of the findings and methods. It also gives a useful explanation of why model behavior changes (a phenomenon called “behavior drift”) and how that’s different than their actual capabilities changing, even though it may look similar to users:

“Behavior drift makes it hard to build reliable products on top of LLM APIs. The user impact of behavior change and capability degradation can be very similar. Users tend to have specific workflows and prompting strategies that work well for their use cases. Given the nondeterministic nature of LLMs, it takes a lot of work to discover these strategies and arrive at a workflow that is well suited for a particular application. So when there is a behavior drift, those workflows might stop working.”
– Arvind Narayanan and Sayash Kapoor, authors of AI Snake Oil

In other words? A prompt that gave you a great article outline yesterday might work very differently today. There could be subtle changes over time or a more dramatic shift, but either way: Continuous testing is essential.

These reasons are why I spend a significant amount of my time testing different prompts to use with the AI tools we’ve built for our writers. It’s important to keep in mind that even as the baseline of AI performance goes up, you can still get even better results by spending a bit of time playing around with different prompts and strategies.

Comments? Questions? Vehement disagreements? Reach me at megan@verblio.com, and check out my next article to learn my process for testing 55 different prompts for our own AI-assisted content.

The post Why testing prompts is so important for AI content creation appeared first on Verblio.

Prepare for extreme whiplash.

Quick important note: SEOs and content marketers have been using AI-powered tools for SEO research for years (think Surfer, Clearscope, MarketMuse, et al).

In this post, I’m referring specifically to using an AI-powered Large Language Model (LLM) tool—like OpenAI’s GPT-4, ChatGPT, Jasper, or any of the other myriad tools using OpenAI’s API—to write content that has SEO intent behind it. That means using any of the above to produce full articles, paragraphs, landing pages, etc. that you intend to rank on Google.

There are other use cases for these tools, e.g. headline writing/brainstorming, writing meta descriptions, but this post is about writing the content itself using AI.

To attempt to somewhat-satisfactorily answer the twin questions, is AI content good for SEO? Is AI content bad for SEO? we need to start with what we know.

Thing we know #1: using AI content for SEO is still an uncertain thing

Anyone who tells you otherwise is lying, extremely naive, or arrogant to the point you should run.

There are zero people not under Google NDA who know how Google will react to AI content in the future. Google itself probably hasn’t decided, and will most certainly evolve its thinking over time.

What this means for you

While all SEO is full of ‘it depends’ and best practices that really should be tested, this level of uncertainty around AI = serious risk. If Google decides to slap down AI content, it could do so in a couple of different ways:

1. similar to how they’ve stated they’re beginning to deal with spam links: devalue any individual piece of content written by AI, or
2. websites with AI content (particularly a lot of it) could receive a sitewide penalty and disappear from the face of the planet Google.

So, yeah, worst case, each piece of AI content you publish is like planting a ticking timebomb with a broken countdown display on your website. Programmatic SEO, long a hotbed of risky SEO practices, has become a common use case for AI content. As we’ll see in the next section, results are… mixed.

All that said: while publishing straight AI output to your website comes with risk, that doesn’t mean you shouldn’t use AI to make you more efficient or give you a leg up in certain situations.

Thing we know #2: Google has already penalized websites with large amounts of AI content.

The theoretical future Google penalties described above aren’t all theory. We have examples already of websites with full, sitewide penalties. Here’s a search traffic graph from one such site:

I’ll let Kevin Indig, who did the case study, explain it here in more detail, but an important caveat on this one is this was bad AI content. Go read it in the article if you don’t believe me, but, like, barely readable bad (and funny, because it’s about celebrities).

Here’s another (undisclosed site)from Mark Williams-Cook (one of my favorite people to follow on this topic):

My caveat on this one: Mark published 10,000 pieces of AI content on this site. This means there are two reasons his penalty could have occurred that don’t condemn AI content as a whole:

1. As Ross Hudgens has been talking about a lot recently, there’s serious danger to publishing at extreme scale very quickly. This could have been the cause of Mark’s penalty, and it had nothing to do with AI. (Personally, I doubt this was the only factor, which leads to …)
2. This was pure, unedited AI content.

Point is: The precedent is here, now, folks.

Ross recently published another case study with AI content that didn’t do so well:

He’s careful to note that early results didn’t tell the whole story. When someone shows your a graph featuring AI content getting indexed and ranking (the big spike on the graph), that doesn’t mean the hammer doesn’t drop eventually. 

In this example, the AI content didn’t sink the website, but the growth was a mirage.

Ross also points out that while AI content can rank, it rarely converts. In his words “[AI] case studies will rarely show conversions, as this content will convert horribly due to poor quality.”

What this means for you

For the love of your website, if you’re going to publish AI content, don’t publish shitty AI content. AI needs a human editor for now. And, AI content gives nearly anyone the power to pump out huge volumes of content. That, without really knowing what you’re doing, is asking for trouble.

If you do publish hybrid content produced by AI and edited by humans, keep a very close eye on it.

Thing we know #3: folks are out there ranking pure AI content.

Wait, what?

Yeah, this is kind of the opposite of Thing #2, but with a healthy dose of nuance.

That nuance is some combination of this list:

• Scale isn’t massive (therefore the risk of being caught is lower)
• The content is better (if it’s pure AI, that means better prompt engineering and fine-tuned models)
• The sites are in less competitive niches (less competition, just easier to rank stuff)
• Content is shorter (the shorter it is, the better AI is at doing it. AI breaks down spectacularly on long-form)
• They haven’t been caught (yet)

My formative years in SEO were learning from Whiteboard Friday, where the thinking was if you’re not doing 100% white-hat SEO, you’re an idiot — Google is just too smart. I’ve now seen enoughto know there are plenty of people who win the cat-and-mouse game against Google for long periods of time and make good money doing it.

As I mentioned before, programmatic SEO is one big use case for this. Kevin Indig is working on this and getting results. (A reminder: programmatic SEO refers to creating thousands of pages based on a database of variants – [business keyword] + [location]; thing to do + [location]; Glassdoor ranking for [job title] + salary – stuff like that.) 

Zack Notes and the folks at Sandbox SEO are doing a similar thing:

As you can see from Zack’s post, this stuff is getting indexed, ranking, and driving traffic. But in the comments, it becomes clear that there’s not a lot of monetary or conversion value to this content. It’s not a space that’s monetized; Zack refers to “publishers vs publishers” as being the competition for this content.

Verblio has built hundreds of local landing pages for local SEO that also use this principle. The difference – and why we’re seeing sustained results – is that we’re taking our own advice and having humans edit every piece of copy before it’s published. The result? Our stuff gets indexed and stays indexed.

What this means for you

If you’ve got black-hat skills and want to duke it out with Google, go for it (at least on websites where you’re ok with seeing traffic go to zero).

If ya don’t, or if you don’t work on sites where you can afford to have that risk tolerance, steer clear of publishing pure AI (read: unedited Jasper/ChatGPT/etc) content.

Thing we know #4: Google has commented directly on AI content

Before you get annoyed at me for burying the lead, read this in context of what you read above. And then read the what this means for you bit.

On February 8th, 2023, Google published this blog post about AI content and SEO and updated its SEO fundamentals documentation to address AI content. This builds directly on their January 12th reply to a lengthy thread:

Here is what Google said on Twitter on January 12th, 2023:

There isn’t a whole lot of net new info in here, though it’s nice to hear it directly from the source. Talk continues of not using AI for SEO evil, and, notably, “it’s just content” and “… about 10 years ago, there were understandable concerns about a rise in mass-produced yet human-generated content. No one would have thought it reasonable for us to declare a ban on all human-generated content in response. Instead, it made more sense to improve our systems to reward quality content …”

Holy mackerel, you say. We’re in the clear! Google themselves said so!

Hold on to your hats, folks, things are about to get meta. In the screenshot below, check out what this gentleman Peter, who’s clearly seen this Google-makes-statements-on-SEO rodeo before, commented on the LinkedIn thread where this tweet was being discussed. Then check out how our own head of content (and host of the Content Bounce House podcast) Ryan Sargent responded to my pointing out that comment in our internal Slack channel:

If I just got too meta on you,the point is this: There’s a long history of Google saying something about SEO and search ranking factors, then feeling little to no obligation to stand by it. The definition of “primarily for search engine rankings” is also way open to interpretation.

Finally, here is what Google has previously said about user engagement signals: (JK Google doesn’t like to talk about user engagement signals as a ranking factor, but everyone knows they take all that delicious Chrome and Android data and use it + pogo-sticking to understand what humans think of your content.) Point of that long parenthetical is: YOUR CONTENT NEEDS TO BE GOOD TO RANK. And the problem with AI content is…

What this means for you

Again, for the love of Pete,do not publish pure AI content. If Google doesn’t punish you, people who search and find you on Google will punish you by bouncing. And then, Google will punish you.

Thing we know #5: AI detection is an arms race, watermarking is coming (or here)

Here are three things to know about this topic right now (for more on specific tools, read our deep dive):

Here are three things to know about this topic right now:

Most AI detection tools are crap

Publicly-available AI detection tools just aren’t great. They’re not smart enough, which can give you a false sense of security. They’re hard to interpret (and interpreting scores across small volumes of content can be misleading, as scores fluctuate by industry, type of content, and more).

Can Google Detect AI content? Assume Google has built bigger, badder AI detection than is publicly available

Google is heavily invested in fighting webspam, and this includes spammy AI content. So it’s probable they’ve built their own tools to detect it, and some or all of these tools will never be released to the public. (And yes, these tools will still likely be able to catch Jasper content, even if you use their Surfer integration ‘for SEO’.)

At the same time, there’s a lot more investment in AI content creation these days than AI content detection. Consider this entire topic “TBD.”

If watermarking isn’t here yet, it will be soon

You’ve heard of a watermark on a document.OpenAI is working on a watermark for content written by GPT that’s based on cryptographic methods. I’d try to explain the inner workings, but I’m not technical enough. We will give you some more details in that upcoming AI detection tools post, though.

Point is: This watermark will work by varying the probability of the next word chosen in GPT-generated content according to a pseudo-random sequence. These variances will change the words slightly, but not in a way that would be noticeable to humans reading the content. OpenAI’s ‘key,’ however, will be able to recognize the watermark by matching the probability variances, so it can definitively prove something was generated by their model. And worse, you won’t be able to remove the watermark by just doing some light editing — in all likelihood, you won’t remove the right words, or remove enough of them. This is the mother of all AI detection tools.

Ok, how does this apply to SEO? The theory goes like this: OpenAI makes this detection key public, Google uses it, detects content that was written by GPT, and punishes that content.

What this means for you

In a story that parallels many of the black and grey-hat SEO strategies of the past, AI content creation and AI detection are in an arms race. One may gain an edge, the other will surpass… you know the story. In arms races with Google, your opponent is smart, well-funded, and brutal. So:

• Do not assume lowering an AI probability score on a public tool means you’re safe. The level of sophistication isn’t there.
• Do not assume that editing AI content renders it undetectable
• Assume watermarking of GPT content is already happening. (It may not be, but the conservative approach is to assume it is.)

Now that we’ve walked through what is known about AI content for SEO, let’s move to the land of crystal balls. It’s fun in there, and, whether these predictions are right or wrong, they provide valuable insight for marketers.

SEO and AI content creation: what the experts are saying

Some very smart SEOs have put their thinking caps on around AI and SEO. Here’s a roundup of my favorites.

Ross Hudgens, CEO of Siege Media

In this LinkedIn post, Ross is commenting on AI content, but specifically using AI content to push publishing velocity (literally, how fast you’re publishing) to extreme and unhealthy levels. This applies to human content, too, but it’s much easier to achieve with AI. The point? In the short-term, extreme publishing velocity may have a positive impact on your organic traffic, but this will very likely come back to bite you. Ross’ theory: If the ratio of content to backlinks starts climbing too fast, the ‘foundation’ of links supporting the content crumbles.

Lily Ray

If you’ve been around SEO for any length of time, Lily Ray needs no introduction. And this tweet needs little explanation. Lily is predicting that what happened to the sites from the case studies way back in Thing #2 will be happening big time in the near future—a specific Google AI content update, or a broader algorithm update that includes stuff designed to target AI-written-content.

Mark Williams-Cook

Mark makes another appearance in this article with his adaptation of the classic Gartner Hype Cycle. In this case, the trough of disillusionment is the trough where your website traffic goes to zero. To provide some balance to Mark’s pretty negative outlook here, just note that recent well-informed speculation has GPT-4 arriving sometime in H1 2023. So although Gartner Cycles are often measured in years, this one … may be much faster.

Lots of naysaying from our expert roundup. Do note, though, all three of them use “mass content/high volume” in their posts. We’ve seen Google’s dislike of super-high-volume (particularly low-quality, high-volume) before, so it’s hard to argue against the likelihood of this particular use case for AI content ending badly.

Kevin Indig

Kevin is willing to make some bold statements about SGE and the future of search in an AI-powered world:

Where this all nets out (what you should take away)

Phew.

Hanging in there? It’s a lot. If you’re currently feeling some version of, ‘Huh. Well, I want the benefits of AI, but this all freaks me out,’ you’re not alone.

Here’s where I am with all this, and what I think you should take back to your lab website(s):

Assume the following:

• All of this will keep changing, rapidly. I’ve been doing this awhile, and I’ve never seen things evolve this fast. Keep your ear to the ground.
• Google has better AI detection than you have available to you, and it will only get better.
• OpenAI will soon, or already is, watermarking all GPT content. Light editing will not remove the watermark. Heavy editing still probably won’t. Running it through another, different LLM (or maybe a content spinner) probably will. But like, why?

Just say no to pure AI content:

• Beyond the SEO risk, publishing (especially poor quality) AI content represents brand risk, too. Look no further than CNET’s decision to pause its use of AI content after public backlash
• It’s not good for your human users, either. And they matter because they’re the ones who actually buy things + engagement signals
• Pure AI is so cheap and so fast, it encourages bad behavior for search marketers: publishing too much, too fast, too low quality, too far from the core promise of your website.

Create AI-assisted content with care:

• Spending 5 minutes editing Jasper output won’t make it all better. (We tried.)
• Taking care early in the process (through prompt engineering and injecting humans throughout the process, not just at the end) and using editors trained in the specifics of editing AI are all necessary to make this stuff sing (or at least go from barely readable to good reading)

Where to use AI matters:

• First, consider your website and its place in the internet. Competing in a hyper-competitive niche and have a site where a drop in organic traffic would be catastrophic to you/your client? Yeah, maybe just don’t AI.
• Don’t let your shiny new AI hammer trick you into seeing all your content needs as nails waiting to be whacked. Like humans, LLM writing has strengths and weaknesses. Deploy it on the content where it makes sense, but don’t force it beyond that.

As for me? None of the websites I work on are getting AI content on them anytime soon. They’re in competitive niches. They’re highly dependent on organic traffic for the survival of the businesses. I need people to read the content to convert. AI content might be readable, but it’s not good reading. Last but not least, I’m scared of Google.

This is not a blanket condemnation of AI content. If I were an agency serving local clients and small SMBs that couldn’t afford high-quality human content and had a sad, empty blog? Heck yeah (heck yeah to well-crafted, AI-assisted content, that is).

If you’ve made it this far, thanks for the read. If you have questions or thoughts or disagree with me, paul[at]verblio.com.

Author’s note: this article was originally published on February 3rd, 2023, and updated February 8th, 2023 with new information on Google’s blog post on AI content and SEO. I updated it again on July 31, 2023. This space is changing rapidly, I’ll continue to update this article with new information as it becomes available.

The post Everything we know about AI content for SEO (Plus what experts speculate) appeared first on Verblio.

Thing #1: Multimodality isn’t here yet

Pre-launch, a lot of the hype around GPT-4 was about its being multimodal, or able to accept both text and images as input. Currently, to upload images you need access to the developer API, which is obviously not for everyone. For everyone else, GPT-4 still only accepts text input.

The hype around multimodality is likely warranted. Expanding the input options to both text and images could (should?) exponentially improve the potential output of the AI, and could pave the way for video, audio, and other multimodal inputs and outputs in the future. 

Thing #2: GPT-4 can accept much larger inputs

In the absence of multimodality, one of the most obvious ways GPT-4 differs from GPT-3.5 is that it can accept much larger inputs (and produce larger outputs, but that’s not going to be useful in the same way.)

The maximum number of tokens you can use at a time with GPT-3.5 is 4,096.With the base model of GPT-4, that max doubles to 8,192 tokens—and there’s even a second GPT-4 model that can handle up to 32,768 tokens.

What does that mean in practice?

For starters, it means I can give GPT-4 OpenAI’s entire technical report (minus the appendices) on GPT-4 to read. (That’s over 5,000 words of content.) I asked it to summarize the report and call out any important information that was missing.

Here was GPT-4’s response: 

With GPT-3.5 and earlier models, you couldn’t give it such a long input as an entire technical report. This is a really cool advancement, as you can now provide the model with a lot more information as context.

This capability is especially useful since the model isn’t hooked up to the internet. The only way for it to have new information is if you provide it — and you can now provide it a lot more.

For contrast, if I ask what GPT-4 is without providing the technical report, here’s what I get:

…and so on.

As far as GPT-4 knows, GPT-4 is still a hypothetical successor to GPT-3. Which makes sense, because obviously it couldn’t have been trained on text from a world in which GPT-4 already existed. In all the content the model has seen, GPT-4 is still a future development.

What this means, though, is that we can now get much better results from GPT-4 on things like new events or extremely in-depth topics, by providing it much more information in the prompt.

In addition to what this improvement enables, it’s also really interesting to consider from an architecture standpoint. In order to accept more tokens, the model has able to recall and synthesize information over a much larger window. Was this done simply by building a larger model with more layers and parameters, or were fundamental changes made to how it processes and stores information?

Unfortunately, the lack of any answer to that question brings us to our third point.

Thing #3: OpenAI isn’t quite so…open…anymore

One fascinating thing about GPT-4 has absolutely nothing to do with its abilities. From OpenAI’s research paper on it:

This report focuses on the capabilities, limitations, and safety properties of GPT-4. GPT-4 is a Transformer-style model pre-trained to predict the next token in a document, using both publicly available data (such as internet data) and data licensed from third-party providers. The model was then fine-tuned using Reinforcement Learning from Human Feedback (RLHF). Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.

(Emphasis mine)

No further details about the model size, dataset, training…anything?

That is wildly not open. It’s also a big departure from OpenAI’s public research on earlier GPTs.

It’s also worth noting how at odds those two reasons for secrecy are: the competitive landscape, and the safety implications of large-scale models. “Safety implications” require caution and prudence, but a “competitive landscape” requires full steam ahead to beat out anyone else.

Leaving users in the dark about dataset construction and training method means that we’ll struggle to identify potential biases in the AI output. After all, human beings made the decisions about those training models and datasets, and those humans have implicit biases. The training data then also has built in bias.

Eliminating that bias is messy, complex, and quickly descends into a rabbit hole of debate only enjoyed by philosophy majors and people who like commenting on local news articles. However, being aware of that bias is important for everyone using AI to create new content.

On a totally unrelated note, two other major AI advancements were released the same day as GPT-4: Anthropic’s Claude model and Google’s PaLM API. Since then, Anthropic has launched Claude 2 and Meta has thrown their hat in the ring with Llama 2. Claude 2 offers up to 100,000 tokens.

Clearly, this arms race is in full swing.

Thing #4: AI is becoming a star student (but still lies)

One of the most widely shared graphs from the launch shows GPT-4’s performance on various tests. It’s almost like OpenAI is still under the illusion, shared by high-achieving high schoolers everywhere, that standardized test scores in some way correlate to real-world success.

Lol.

What is worth noting, however, is that GPT-4 was not specifically trained to take any of these tests. This isn’t the case of an AI model being specifically trained to play Go and eventually beating the best human player; rather, its ability to ace these tests represents a more “emergent” intelligence.

Previous models like GPT-3 also weren’t trained to take particular tests, but, as you can see, GPT-4’s performance has improved significantly over GPT-3’s:

These graphs look good and have become staples of articles and press announcements featuring new models. But ask yourself: do you really want an AP English student – even a particularly skilled one – in control of your marketing messaging and copywriting? Me neither.

If you don’t care about AI’s ability to take standardized tests and just want to know how well it’s going to do what you want, this is still good news. From the report:

GPT-4 substantially improves over previous models in the ability to follow user intent. On a dataset of 5,214 prompts submitted to ChatGPT and the OpenAI API, the responses generated by GPT-4 were preferred over the responses generated by GPT-3.5 on 70.2% of prompts.

So, GPT-4 is more likely to give you what you’re looking for than GPT-3.5. That’s great. It’s important to keep in mind, though, that in spite of its improved performance, the new model still has all the same limitations we know and love from our existing AI friends.

Another quote from the report:

Despite its capabilities, GPT-4 has similar limitations to earlier GPT models: it is not fully reliable (e.g. can suffer from ‘hallucinations’), has a limited context window, and does not learn from experience. Care should be taken when using the outputs of GPT-4, particularly in contexts where reliability is important.

In fact, hallucinations could become an even bigger problem than they were, simply because the better the AI gets, the easier it will be to believe what it says. With GPT-3 and GPT-3.5, people are well aware the model will totally make stuff up because it happens so frequently. As newer and better models do that less frequently, there’s a greater risk that when they do hallucinate, we may fail to notice or fact-check it.

So stay vigilant, friends. But also, these are very exciting times.

The post Four Things to Know About GPT-4 appeared first on Verblio.

You might not be rolling out the welcome mat for our new robot overlords, but it would be foolish not to take advantage of AI. It’s extremely unlikely that Google and other search engines decide that nothing produced with AI is allowed to rank; AI is already being integrated into search. Wise use of AI will also look good on your resume and can legitimately help you do more with less (an imperative all marketing leaders have heard from a CFO at some point). 

Play to AI’s Strengths

AI doesn’t build original research, create thought leadership, or even tell the truth. But AI can write. It produces an infinite number of words very fast, and those words have a high quality floor. That is, content made by AI can only be so bad with respect to grammar and usage. It can only be so good in other ways, but not every piece of content needs to change the world. If building 10x content was that easy, content marketers would have surely succeeded at world domination by now.

Build your content strategy to support AI’s strengths instead of asking it to do things that are still in the human realm. That means finding opportunities where you’ve been stuck because you can’t:

• Generate the necessary volume of content
• Rely on consistency from article to article
• Find a way to mitigate risk (“What if we build 100 articles and don’t see traffic/conversion?”)

If we take those potential roadblocks out of the equation, three approaches to content strategy get a whole lot easier.

Bling Out Your Hub With Lots of Spokes

Hub and spoke content is one of the most tried and true options for content marketers. This term gets tossed around a lot, and it often means different things to different people. To be clear, I’m not talking about hub and spoke content where we take one piece of pillar (aka skyscraper) content and repurpose it into infographics, webinars, etc. I’m talking about one significant, major piece of content — either a table of contents or a long-form pillar page — that has many internal links to and from supporting content. These other blogs should drive organic traffic on their own for longtail keywords, but because they are so closely tied to the larger piece of content (or table of contents), they also keep people on your website and drive more traffic to your “main” piece of content. The idea is that your main page for the topic will start ranking for a high-difficulty, shorttail keyword.

This strategy works, full stop. Competitors with poor SEO site architecture, thin content, or who just aren’t paying attention will quickly find you’ve eaten their lunch. AI makes this approach even better.

As an example, let’s say we want to rank for “how to play chess.” This is going to be hard: the information has been around for thousands of years, chess.com has some serious authority in the space, and chess influencers are going wild on Twitch, Youtube, and Tiktok. 

Starting with a large-scale hub helps people self-select what kind of chess content they need:

• Skill level (beginners need to understand the rules of the game)
• Game phase (opening vs end game)
• Strategic options (positional vs tactical)

Let’s say we want a blog for each chess opening. AI can give us the basics, but how about embedding a recent game from two well-known pros or influencers? Finding that game in a chess database is a human task that turns these articles into rich content. Instead of spending time crafting an introduction for “Caro-Kann Defense” that isn’t duplicating your intro for “Queen’s Gambit Declined,” you can make your content better.

Here’s where the AI is a game changer: there are more than 1300 documented chess openings. Having humans build an article for every single one would be a disaster. Instead of building just enough spokes to make our strategy work, AI will completely fill our wheel with spokes and make our website as comprehensive as possible.

We can use AI to help build out content at scale beyond what would have been possible before and use our human time to improve the resources. Internal links will generate traffic (for say, “how to play the Caro-Kann defense” or for keywords related to one of the many subvariations of the Najdorf line in the Sicilian Defense) but also keep pointing back to our hub and building authority for our whole website.

Chase Medium-Difficulty Keywords

Medium-difficulty keywords are often ignored when we build a content strategy. Investing resources in ranking for high-difficulty keywords that will really move the needle and drive business results makes sense. Investing fewer resources in longtail keywords that exactly match the search intent of a potential customer makes even more sense. Investing medium resources to maybe rank for something that isn’t bottom of funnel or likely to drive huge traffic volume? Not a smart bet.

With cyborg content we can invest very few resources in these medium-difficulty keywords. We can make a few bets without spending time and effort, or we can invest more resources and cover a wide range of medium difficulty keywords. If even a couple work out, we end up creating an entirely new stream of traffic that competitors are likely ignoring.

You can also test different approaches against each other. Perhaps for one medium-difficulty keyword, you create several pieces of content. For another, you build some original research to support your efforts. You can analyze which techniques are most effective for this strategy and refine your approach for the next round.

Use a “Bet” Strategy

Attacking medium-difficulty keywords with cyborg content works because it takes what was previously a risky bet (medium or large resource investment in something that may or may not work) and makes it easier (low resource investment into something that might work). Bet strategies work the same way.

In a bet strategy, (a technique I learned from Mark Rogers) you use Google Trends and audience research to identify keywords that are powerful for your business but have low search volume today and potentially high search volume later. This might be a travel business with keywords around a solar eclipse (or another infrequent event). Today, no one needs to travel to see the eclipse. That cluster of keywords has no value. But we all know that eventually it will, and if your website is entrenched on page one for those keywords, you’ll reap the rewards when search volume increases. You can even go back and refresh or update the content to make sure you get every scrap of traffic when the time comes, but for now, you get to rank for a relatively easy keyword. Enjoy the blue ocean!

Not all bets are this straightforward. Other bets might be tangentially related to your business or far less likely to have increased search volume in the future. Building content for those is a larger risk, even though the keywords currently have little volume and are probably easy to rank for today. You won’t spend much effort on this content, but every piece of effort you do spend might be wasted if search volume never increases.

Cyborg content empowers you to make many of these bets because you can create content for more of them with less effort. You aren’t producing thought leadership here, you’re targeting relatively simple keywords with low difficulty. Your content is meeting a minimum quality bar but isn’t reshaping an industry or driving demand with potent thought leadership. This use case is tailor-made for AI.

Bonus Round: More AI Content Strategy

Content strategy isn’t just about choosing keywords and building blogs. Using AI creatively opens up other opportunities for content teams. You can look at what AI does poorly for inspiration or spend the time you save with AI tools (whether for keyword research, beating writer’s block, or generating content) to prioritize other content initiatives.

Outflank ChatGPT

What is your target audience asking ChatGPT to do? Go ask ChatGPT the same thing. Find the places where the robot falls short (it often gives surface-level, boring, top of funnel drivel with no opinion or insight). Write the content that answers your target audience’s follow-up question. It might not rank right away, but it’s damn helpful content that reminds your ICP they need you.

Clear Your Backlog

Every content lead I’ve ever spoken with has top of funnel ideas that got deprioritized, changed, or otherwise cast aside and are living in a forgotten Asana task or Notion template. Go do those things. You brainstormed them for a reason. You can either do them faster with the help of AI or you can use AI to clear your plate of other stuff (like adding spoke content) and use the time to build these instead.

Build Stuff AI Can’t Build

There’s a time and a place for AI — but the internet is already full of garbage content, and it’s only going to get worse as more and more people copy/paste AI onto their websites. Build better stuff than that. Some of the best content before ChatGPT is still the best content because it automatically includes human connections, true authority, and original research:

• “State of” reports with real survey data
• Podcasts
• Quotes from SMEs
• Case studies with real data from real success stories
• Templates real humans can use to make their lives easier

Strategy Is More Important Than Ever

AI’s effect on content marketing isn’t written into the history books, but we should start taking advantage of the opportunities now. And for now, AI can’t build a full-funnel content strategy. It can’t interview customers, access real data for keyword research, or prioritize your work. Figure out how to implement it in your content strategy so that you can make smarter decisions and prioritize work that only humans can do.

The post Content Strategy for Cyborg Content appeared first on Verblio.

Meet our panel of experts:

• Jimmy Daly, Founder of Superpath
• Eric Doty, Content Lead at Dock
• Jess Cook, Head of Content at LASSO
• Tommy Walker, Founder of The Content Studio

Building your personal brand is quickly becoming a marketing prerequisite

Your personal brand is a serious marketing tool—for you and your future company. Building your network online can be a major selling point in interviews, open doors for new opportunities, and positively impact the trajectory of your career.

“Encourage the people on your team to build their own personal brands. Because if you hire people with strong personal brands, actually you end up with built-in distribution.”

Jimmy Daly, Founder of Superpath. Listen on episode six.

“I grew a Twitter following. I was posting about hockey, but I was able to get my first two or three jobs partly on the back of saying I understand how to grow an audience.”

Eric Doty, Content Lead at Dock. Listen on episode fifteen.

“Now more than anything is your thoughts, your positioning, how you present to the market. Content marketers are a dime a dozen. In a lot of cases, you could take a byline, do away with it, and it could be any single person out there. It does not matter. What is important is that people get invested in your byline and not your headline.”

Tommy Walker, Founder of The Content Studio. Listen on episode four.

“You should really develop your own personal brand. Start a newsletter, tweet a lot, do LinkedIn. You become very high leverage the more people follow you.”

Jimmy Daly, Founder of Superpath. Listen on episode six.

Start now with what you know

Someone out there needs to learn what you know. Building a personal brand on social media doesn’t have to be a daunting task. When the blank page syndrome gets you down, our experts have a few helpful tips for finding your voice.

“Every time someone asked me a question for advice, I just write it down and I write bullets of the advice I gave them. Now I have a notion document of like 150 ideas. Uh, that, that I will never struggle to think of something to say.”

Eric Doty, Content Lead at Dock. Listen on episode fifteen.

“What would you tell yourself two years ago that would’ve really helped you? Post about that. That’s the stuff people need to hear.”

Jess Cook, Head of Content at LASSO. Listen on episode eighteen.

Use your unique perspective

A personal brand is just that: personal. You don’t need to conform to the same six talking points you see recycled on LinkedIn. Make stuff you love, and people will love it back.

“What’s important is that, no matter what you do, you are always approaching it from your own place. Even if it pisses people off. ”

Tommy Walker, Founder of The Content Studio. Listen on episode four.

“I’ve only been doing content marketing for four years. I am not an expert, in content marketing per se. But I have a unique perspective because I’ve done B2C marketing, I’ve come from an agency, I’m now a SaaS marketer, and I made a mid-career pivot. All of those things together create something unique that no one else has and focus on that.”

Jess Cook, Head of Content at LASSO. Listen on episode eighteen.

“Don’t try to copy what’s working with the big B2B content people. Take your interests or the sort of marketing things you’re used to in your hobbies, whether that’s YouTube streams or TikToks, and then bring that into your own personal brand. Think, what are you uniquely able to do that other people aren’t doing? And then do that.”

Eric Doty, Content Lead at Dock. Listen on episode fifteen.

Your job depends on it

A personal brand is an impactful tool for your company’s marketing machine. It’s a distribution channel that cannot be overlooked. (And honestly? We’d all love to see more good content on LinkedIn.) Whether you’re ghostwriting for your CEO or pushing out carousels on your own profile, here is what our experts had to say.

“The person who wrote it or the person whose voice it is written in needs to be really bought into building their own personal brand, because the more followers or subscribers or whatever that person attracts, the easier it will be to distribute things written in their own voice.”

Jimmy Daly, Founder of Superpath. Listen on episode six.

“Using your employees as a channel is much more personable. It gets more engagement, it gets more interest. So while not everyone who follows me is going to be our target market, when you’re a startup and you need to generate excitement, it’s a great channel because we don’t own any big channel yet. We don’t have that many followers on our brand page, but we all have lots of important connections that we can lean on.”

Eric Doty, Content Lead at Dock. Listen on episode fifteen.

Wrap up

At Verblio, we know how important your reach is. Make your content go farther by producing content at scale, creating kickass content briefs, and knowing when to refresh your content. For more insights, listen to Content Bounce House, where I pick the brains of content marketing leaders (accompanied by a jazz trombone soundtrack.)

The post The Importance of Personal Brand According to Content Leaders appeared first on Verblio.

Whether you work in-house creating landing pages or at an agency churning out massive amounts of content for your clients, you need fast, reliable content now more than ever. And because Google deprioritizes duplicate content, you want to make sure that you aren’t putting out the same content as all your competitors. 

Is ChatGPT content unique? 

If you missed my last AI content experiment, we covered what it takes to edit a piece of Jasper content to make it agency-ready. This time, I want to know how creative ChatGPT can be. 

Can it spin up truly different content about the same topics using different prompts? Or will using this writing tool make you sound exactly like everyone else? 

Lately, prompt engineering has been the go-to solution for anyone trying to create unique AI content. I want to put that to the test. I’ll use varied (but related) prompts to generate paragraphs. Then, I’ll compare both outputs, looking specifically for shared phrases or structures. 

Hypothesis: Due to the nature of its training and limitations, ChatGPT outputs will use similar phrases regardless of prompt engineering. 

Humans, for what it’s worth, are the same way. There are only so many different ways to write about what we write about. We fall into routines. Habits. Cliches. Our vocabulary gets monotonous, and our writing suffers for it. 

And, because ChatGPT is trained on human writing, I’m anticipating a similar downfall. 

However, I’m excited to see the results because where one person has their own crutch words and writing tendencies, ChatGPT has everyone’s crutch words and writing tendencies. Maybe it will generate original phrases even with similar prompts. I’d love to be surprised.

Test 1: Where I Ask ChatGPT for Vegan Food Ideas

Say, for instance, you’re writing a blog post about vegan options great for people who have recently made this dietary adjustment. You’d want something straightforward but informative. This is where ChatGPT excels. 

Prompt 1: tell me about good food options for new vegans

Prompt 2: What are great options for people who just started eating vegan?

Right off the bat, it’s easy to see that these outputs turned out similar in length (about 500 words), they both have seven items in their list, and they have introductory and conclusion paragraphs. If you’re in the market for a low-lift vegan food article, this is the 90-second rice version. 

So, how original are they? I’ve highlighted shared exact phrases in purple.

Aaaaaand that is a lot of purple. Honestly, it would have been easier to highlight the words that aren’t shared than the ones that are. Let’s break it down into bite-sized pieces so you don’t have to keep squinting at your screen.

At first glance, the opening paragraphs of each look clean and unique. However, an eagle-eyed reader will recognize verbiage from Prompt 1’s first paragraph has been regurgitated in Prompt 2’s closing paragraph. 

(And yeah, it still counts as plagiarism.) 

You might have also noticed that the seven items on both lists are identical. I’m pretty sure even vegan diets aren’t that predictable. Plus, they’re in the exact same order. Not quite bespoke marketing. 

One last note here. One of the biggest differences between these two pieces boils down to adjective choice. “Popular” is used routinely throughout Prompt 1’s output. In contrast, Prompt 2 never says popular but instead opts for “great.” They don’t switch it up, either. Once they’ve decided on their adjective of choice, they seem content to stick with it. Neither is particularly inspiring, but their repetition is an obvious indication of AI-generated material.

Test 1.5: Vegan Food, One More Time With Feeling

It’s clear that without much prompt differentiation, ChatGPT is going to offer up similar content. To get unique outputs, you need unique inputs. This is where prompt engineering comes in. The more time you spend crafting a detailed prompt, the better the quality should be. 

(Of course, there is no guarantee that someone else won’t use that same input and create an output identical to yours. But more on that later.) 

Before I start fiddling with completely new prompts, I want to ask about vegan food again, but this time pushing for a different tone. We’ll compare the output from Prompt 1 from the previous test to a new output. 

Prompt 1: Tell me about good food options for new vegans

Prompt 2: Write a few paragraphs about vegan food for beginners with an easygoing tone

Here, I’ve highlighted in yellow similar phrases as well as continuing to highlight duplicate content in purple because — hallelujah! — these two outputs are actually different. 

IMO, the easygoing tone was accomplished pretty well. “Hey there” might be too casual for a typical blog post, but extra fluff like “don’t worry, it’s not as hard as you might think” or “they come in all shapes, sizes, and colors” helps make the output feel personable. 

Barring nuts and seeds, Prompt 2 still highlighted all the same types of foods as Prompt 1. This is somewhat expected as we get the same heavy hitters: fruits and vegetables, whole grains, legumes, tofu and tempeh, and milk and cheese alternatives. What surprised me here is that the easygoing tone also recommends different foods. 

Are black bean tacos or chickpea curry notably more easygoing options? Or has the model read enough Pinterest food blogs to know that laidback tones are often paired with specific recipes? I’m not sure.

Test 2: Nantucket Vacation Homes Or Bust

Prompt 1: Write with authority about vacation homes on Nantucket, Massachusetts for interested buyers

Prompt 2: Write an educational but laidback post about vacation properties on Nantucket, Massachusetts for a buyer already familiar with the island

There are two main differences between these prompts — target audience and tone. I wanted Prompt 1 to sound like it knew Nantucket like the back of its hand and was confident in its expertise. For Prompt 2, while still requesting a knowledgeable speaker, I hoped it would cater specifically to buyers who already knew the ins and outs of the island.

If we’re grading the AI out of 10, I’d give this one a 5/10. 

The prompts themselves were moderately successful in generating different materials, but these swaths of purple and yellow don’t lie. These outputs are still largely similar.

Interestingly, ChatGPT emphasized affordability in both outputs, thinking that’s the obvious next word after “vacation homes,” despite not requesting that in either prompt.

In its defense, I also didn’t specifically request that it write for a high-end luxury audience. However, buyers interested in vacation properties on Nantucket likely don’t have huge financial constraints, so price might not be a barrier. Implying that it should be could turn off lucrative buyers. This is the kind of context a human writer would likely infer from an assignment that AI simply isn’t able to see and implement. 

But this isn’t a quality critique, so let’s take a look at what sets these two outputs apart. 

The biggest difference is in their body paragraphs. Because Prompt 2 was designed with buyers who already know the island but Prompt 1 wasn’t, the outputs managed to cater to slightly different audiences. 

Prompt 1 mentions Sconset (a colloquial term for Siasconset) and Madaket (as in, Madaket Beach). Using shorthand versions of these enclaves makes this feel more familiar with the area. I almost would have expected that from Prompt 2 instead. 

And let’s not brush over that these are called more affordable neighborhoods. Properties listed here are on par with prices throughout the island with multimillion-dollar homes for sale. 

Prompt 2, on the other hand, highlights zoning laws. Certainly, this is something someone considering buying a home would be interested in, but it’s not necessarily known by everyone familiar with the island. It’s educational but doesn’t dive too deep, which is exactly what the prompt asked for. 

Regardless of the content’s effectiveness, at least here there was variety. ChatGPT proved that it could, when prompted, generate different outputs if you nudged it in the right direction. 

Test 3: The Same Prompt Over and Over and Over

We’ve tried generating content on different topics, so this time I want to see how ChatGPT performs when asked to create new material while repeatedly using the same prompt. For a refrain like this, we have to go to Music City. 

Prompt: Tell me about things to do in Nashville, Tennessee

I asked ChatGPT five times to tell me about things to do in Nashville, Tennessee. Each time, this prompt generated a list of ten attractions. The first three times, the list featured the same ten things to do in the same order. 

1. Country Music Hall of Fame and Museum
2. Ryman Auditorium
3. Lower Broadway
4. Grand Ole Opry
5. Belle Meade Plantation
6. The food scene
7. Frist Art Museum
8. Centennial Park
9. Johnny Cash Museum
10. Music Row

These three outputs are scarily similar. It’s the vegan food experiment all over again. Nearly the same two-sentence intro paragraph followed by a listicle. Throughout, there are a few places where they differ — swapping adjectives or sentence structure like we’ve seen in previous tests — but frankly not enough to notice.

By the fourth generation, it started to shake things up. 

We swapped the order (moving the Johnny Cash Museum from 9 to 4 on the list) and replaced Music Row with the Tennessee State Capitol. Plus! A conclusion! We’re really begging for scraps here, folks. 

Generation five throws in a hail Mary — Jack Daniels Distillery. This is actually in Lynchburg, which is a ninety-minute drive from Nashville, but that’s not the point here. At least it tried something new.

General observations and key takeaways

You’re in a hurry. Here’s the short version: 

• ChatGPT is basically glorified predictive text. It likes patterns. They’re all it knows. 
• High-quality content is original content, but not all original content is high quality. 
• Not only does prompt engineering work, but it’s also a necessity. Getting granular with your prompts is the only way to generate varied content through ChatGPT.

So, what happens if your content isn’t original? 

Duplicate content isn’t going to work for your website. Having informative, authoritative, unique content is crucial for SEO, and how AI and SEO interact is still up in the air. In the meantime, there are plenty of AI detection tools out there to help you keep an eye on the content you’re posting.

Is AI content right for you?

At Verblio, we’re constantly experimenting with ways to make your content production as seamless as possible. AI writing tools can be a powerful asset, but it’s clear they still need the human touch to create a finished product. Our Human-Crafted AI cuts costs but keeps the quality you love.

The post How Original is ChatGPT Content? appeared first on Verblio.

In 1950, the Turing Test was devised as a way of assessing whether a machine can “think.” In its original form, it described a situation in which an evaluator asks questions of two subjects, A and B. The evaluator knows one of the subjects is a machine and one is a human, but not which is which. If, based on their answers, the evaluator can’t determine whether A or B is the human, the machine is considered to have “passed the test.”

In 2023, it’s clear the Turing Test is way too easy. Not only can a machine easily fool us when we don’t know it’s an AI, but we can even know we’re talking to a machine and still find it terrifyingly sentient.

Maybe it’s easier to be human than we thought?

The Turing Test isn’t the only thought experiment around machine cognition, though. Searle’s Chinese Room Argument from the 1980s, for example, is one of the best-known arguments against the Turing Test definition of intelligence… 

…which is all to say that the issue of how to distinguish man from machine was up for debate long before we got large language models capable of professing their love for a married reporter or learning Hemingway’s rules for writing. The question has only gotten more complicated since.

This is a B2B marketing blog though, so delivering a unified theory of sentience and intelligence is a tad out of scope for us.Instead, we’ll focus on the only two AI detection concerns you probably have right now:

1. Can I publish AI-generated content without people (or Google) knowing?
2. Is my freelancer writer giving me AI-generated content without telling me?

We’ve already written about Google’s stance toward AI-generated content. (Key takeaway: You shouldn’t publish shitty content, regardless of whether you’re using humans or AI to create it. Shocking.)

When it comes to detecting AI-generated content yourself, though, you have options. Some are better than others. Let’s break it down.

Table of contents

• How AI detectors work
• The results
• The tools
  • Originality.ai
  • OpenAI’s AI Text Classifier
  • Huggingface
  • Copyleaks
  • Writer
  • Content at Scale
  • GLTR
  • GPTZero

Is there a way to detect AI-generated text? How?

To understand how most AI detection models work, you first need to understand how current AI models generate text. GPT-3 and others generate text by predicting what the next word should be based on what they’ve seen in their training data.Because of this, what they write is very…predictable. By definition.

What humans write, on the other hand, tends to be much less predictable. We reference things from our own experience, like the energy drink that’s getting me through a Thursday afternoon,or a relevant and humanizing anecdote from when I was a child obsessed with cheetahs and puzzles.

AI detection models use the predictability of AI-generated text to identify it as such. If an AI detection model were reading this article, for example, it would find the words “energy drink” in the last paragraph highly unpredictable, given that the rest of this article is about AI detection tools. Those two concepts don’t often go together. This and other instances of “random” words and phrases would suggest to the detection model that this article is too unpredictable to have been written by an AI.

If, on the other hand, the AI detection model scans something where every word is highly predictable, it will assume it was written by AI. Such an article would not be random enough to have been written by a human because we are by nature random, whimsical creatures.

Detection models can look at other features of text to identify AI content, too, and we’ll talk about some of those when we look at specific tools. The predictability or randomness of a text, though, is the primary one, so understanding how and why it is such a reliable predictor of AI-generated content is important.

A final word on how AI detection works: AI detection models don’t work on very short copy. A sentence like “We are the leading content marketing platform” is predictable and could easily be written by AI — but it has also been written by countless humans. One predictable sentence in isolation doesn’t mean something was written by AI. The longer a piece of text goes without saying anything random, though, the more likely it is to have been written by AI.

For this reason, most tools explicitly state that content needs to be at least 50 words long for the detection model to be at all reliable. Even if you’re using a tool that doesn’t say that, though, you shouldn’t expect good results if your content is less than 50 words. 

The results

Background: We’re a content creation platform, so we needed a way to know that the content our freelancers are submitting to our clients was actually written by a human. To that end, we’ve tested a lot of different AI detection tools. (Learn more about how we handle AI detection as a business.)

This table shows the results for six of the most popular AI detection tools on five different pieces of content:

How to interpret the percentages: For most AI detection tools, the percentage measures the likelihood that the content was created by AI, according to that particular tool. The first 9 percent result, for example, means that Originality.ai thinks there is a 9 percent probability that piece of content was created by AI. It does not mean that it thinks 9 percent of that piece of content was written by AI.

The exception to this is Writer, which does say their score refers to the percentage of content that was likely generated by AI.

We tested these five tools on five different pieces of content. Article #1 was a piece of pure human content.Article #2 was a piece of human-crafted AI content, which is our AI-assisted content offering where a human writer co-creates content with an AI model. Article #3 was copied and pasted directly from ChatGPT, with no human edits. Article #4 was a different article from ChatGPT based on the same prompt as article #3. Finally, Article #5 was a longer sample from ChatGPT.

We’ll walk through each of these tools in more detail and review what their scores mean in context.

Originality.ai

Originality.ai is the best AI detection tool we’ve tried so far, both in terms of its results and its usability at scale.For that reason, it’s the one we’ve implemented at Verblio to make sure customers who are paying for human-only content are getting just that.

Originality is super simple to use. First, copy and paste your text into their “Content Scan” tool.

Wait a few moments, and then…

Boom — you get a score indicating how confident Originality’s model is that your content was produced by AI. Remember: This percentage refers to a likelihood, not to a percentage of the content. If, for example, the score was 65% original and 35% AI, it would not mean that 35% of the content was written by AI; it would mean there is a 35% chance the article was written by AI.

It can also check your text against existing content on the web for plagiarism, and, in this case, it successfully recognized that this content was pulled directly from my existing Jasper review.

The tool has an API, which is how we’re using it at Verblio to check every piece of content our writers submit. You can also enter a URL to scan an entire site, without having to manually check every page.

Here’s how Originality did across our five samples:

You can see it did well with both the human and most of the pure AI content. It gave a low AI likelihood score (28%) to the first ChatGPT sample, which was apparently a tricky one for a few of these tools — Writer and Content at Scale struggled to identify it as AI-generated as well.This is a good reminder that these tools are still far from perfect, and you can get both false positives and false negatives. Overall, though, and across the thousands of articles we’ve currently run through the tool, Originality has performed very well.

Pricing: Originality.ai currently charges $0.01 per credit, and one credit will get you 100 words scanned. (If you scan for both AI and plagiarism, it will cost you twice the credits.) There are no platform fees or subscription needed.

Nerdy stuff: Originality uses its own language model, based on Google’s BERT model, to classify text as either AI-generated or not. Their how it works is not well-explained at all. To be honest, that’s not a bad thing. It was clearly written by an engineer, and I’ll take a dense, jargon-filled description from the person who actually built the tech over vapid, hand-wavey copy from someone who doesn’t know a discriminator from a generator any day.

OpenAI’s AI Text Classifier

OpenAI’s tool is the newest one on this list (for now.) It’s also the most soberly presented: The OpenAI team is very upfront about the current limitations of AI detection tools. To be clear, these are the limitations of any of the tools on this list — some of them just aren’t as honest about that.

OpenAI says this tool is specifically intended to “foster conversation about the distinction between human-written and AI-generated content.” They are very clear that it should not be used as the only indicator of whether something has been produced by AI or not, that it can misidentify content in both directions, and that it hasn’t been tested on content produced by some combination of AI and humans.

In keeping with its more measured approach, the classifier doesn’t provide a percentage score like most other AI detection tools currently do. Instead, it just says whether the text is “Very unlikely AI,” “Unlikely AI,” “Likely AI,” or if it is “Unclear.”

Unsurprisingly, OpenAI’s tool did very well at detecting AI content across our five samples:

It’s reasonable to assume OpenAI will have one of the best AI detection tools available, as their model is the one responsible for most of the AI-generated content on the internet right now. Unfortunately, they don’t have an API for this tool yet, so Originality.ai is still the best option for our use case: running AI detection across every one of the hundreds of daily submissions on our platform.

We wanted to know exactly how Originality stacks up against OpenAI’s tool, though, so we ran 36 additional pieces of content through both detectors. We intentionally chose pieces with a range of Originality scores to see how they aligned on both ends of the spectrum. In order to compare the results, we coded OpenAI’s responses as percentages:

• Very Unlikely AI = 10%
• Unlikely AI = 30%
• Unclear if it is = 50%
• Possibly AI = 70%
• Likely AI = 90%

Here are the results from both tools. You can see Originality’s score in the left column, OpenAI’s original verdict in the middle column, and our coding of that OpenAI verdict as a percentage in the right column. Green cells are least likely to be AI, and red cells are most likely:

You can see the two tools were pretty well aligned in their scores. More importantly for our purposes, there weren’t any instances in this batch where Originality rated a “Likely AI” piece as less than 80 percent.

There are some mismatches in the opposite direction, where Originality gave a high AI likelihood to an article that OpenAI said was unlikely AI or unclear.We’d rather deal with a false positive and manually review the submission and writer, though, than risk missing AI-generated content.

This is an important reminder, though, that AI detection tools can and will return false positives. One high score on a single piece doesn’t necessarily mean a writer is using AI. That’s why we run every submission on our platform through Originality, so we can use larger trends across our writers and customers to detect AI content.

I expect OpenAI’s tool to eventually become the best in the industry. There are two reasons for this:

The first is simply that as I mentioned, it’s OpenAI’s model that is currently generating most of the AI text on the internet via GPT-3 and ChatGPT. Presumably they have a leg up in figuring out how to detect the output of their own models, including possibly watermarking their content in the future.

But second and more importantly, OpenAI has very high motivation to get this right. They’re going to continue building and training AI models, and they’ll continue to need enormous datasets of content to do so. They’ll be getting most of that content from the internet. If they don’t figure out how to detect AI content and remove that content from their training data, they’ll be training future models on AI-generated content…and then those models will create more content that is used to train still newer models…and so on in a downward-spiraling game of telephone that takes us farther and farther away from original human content. Yikes. Here’s hoping they figure it out.

Pricing: OpenAI’s classifier is free to use.

Huggingface

Huggingface is the name in open-source machine learning projects, with various models, datasets, and code freely available. It’s one of the best resources on the web right now, if you’re interested in taking a more hands-on approach to AI.

The thing is, this isn’t actually Huggingface’s model. You’ll see it referred to that way across the internet, but it’s actually another, earlier OpenAI detection model. It did, however, use Huggingface’s implementation of the RoBERTa model, which itself was developed by Facebook.

Regardless of the name behind it, though, this AI detection tool was trained on content from GPT-2. On the one hand, that means it’s less likely to catch content created by newer AI models. On the other hand, it means this team was working on AI detection before it was cool and isn’t just riding the ChatGPT tidal wave.

This tool analyzes your content in real-time, so you can see how the results change as you add more text. No matter how long your content is, though, it will only scan the first 510 tokens.

In this case, it recognized that our latest experiment around editing Jasper content was in fact human-written with 99.98 percent certainty.

Despite not being trained on the latest generation of transformer models, though, Huggingface’s detection did really well across our five samples. Again, this isn’t surprising, given the expertise and experience of the teams behind it:

Pricing: Huggingface’s AI detection tool is free to use.

Copyleaks

Copyleaks has long been one of the best plagiarism detection tools on the market, and it’s the one we currently use at Verblio to make sure our customers are getting original, non-plagiarized content. They recently integrated AI detection into their product.

As with the other tools on this list, Copyleaks provides a web interface into which you can copy and paste your content to check it:

In addition to saying whether it thinks the text was created by a human or AI, Copyleaks also provides a probability score when you hover over the text. Again, that probability score is similar to Originality.ai’s score in that it refers to the likelihood the content was created by AI, not a percentage of the content that it thinks was created by AI.

Their web interface currently maxes out at 25,000 characters, which is plenty for most articles.

Across our five samples, Copyleaks did pretty well. It didn’t give a score above 90 percent on any of the ChatGPT samples, which is weaker than the last three tools we looked at. It was, however, at least directionally correct on all of them:

Pricing: Copyleak’s web interface is free to use. Pricing for their API starts at $9.99 per month for 25,000 scanned words.

Writer

Writer, which offers AI-powered content creation specifically targeted towards B2B businesses, also has an AI detection tool. 

It’s more a leadgen play on their part than a robust AI detection tool. Even when you access it through their API, it will only scan 1,500 characters at a time. It was also released before ChatGPT and the GPT-3.5 update, so don’t expect it to work as well on the latest and greatest AI content.

Here are the results for an article I copied and pasted from ChatGPT on choosing eyeglasses:

That’s unfortunate. Interestingly, when I add another paragraph from ChatGPT to continue the article, the score changes significantly:

This is a good example of the fact that generally speaking, the longer the text, the more accurate the AI detection tool will be. (Though it does beg the question of why a tool would limit you to 1500 characters.)

Overall, Writer did the worst at detecting AI content of the five tools we’ve dug into so far.

Like Originality.ai, it struggled especially hard with the first ChatGPT sample. It also only gave an 83 percent to the other two, which most of the other tools had no trouble identifying as AI-generated.

Pricing: Writer’s web interface is available on their site for free. It’s not yet available within the app itself, but you do get 500k words included via their API if you’re an enterprise customer.

Content at Scale

I’m including this in the list only because I’ve heard multiple people mention it over the last week. It’s not just an AI detection tool — according to the homepage, it’s also supposedly a magical platform that creates optimized content that is “so human-like, that it bypasses AI content detection! This means you are protected against future Google updates. Content at Scale is the only solution on the market that has advanced enough AI to pull this off.”

I call BS. “This means you are protected against future Google updates” is a huge claim and not one that anybody with any SEO experience would make.

Also, though, it did the worst of all the AI detection tools by a long shot in our testing. It didn’t score any of our three pure ChatGPT samples above a 28 percent:

As a result, I have very little confidence in Content at Scale’s tech either to detect AI content, or to create it.

I’ve included two additional tools that are interesting in the field of AI detection, but not ones that you would potentially be using for your own marketing content: GLTR and GPTZero.

GLTR

The Giant Language model Test Room, or GLTR, is my favorite AI detection tool, even if it’s somewhat out-of-date in a post-ChatGPT world.

Like Huggingface’s tool, GLTR was built to detect GPT-2-generated content, so it’s less effective against newer models and tools, most of which are built atop GPT-3. What it lacks in accuracy and recency, however, it makes up for in pretty colors and fascinating insights.

Again, this isn’t the tool you’re going to use on a regular basis to figure out whether your content was generated by AI. It can, however, help you understand how these detection models work and what kind of writing (and word choices) are seen as more predictable.

Here are the GLTR results for an article I wrote explaining GPT-3:

Whew. There’s a lot going on here. Unlike the other tools on this list, GLTR doesn’t provide a clear-cut “This is how likely it is your content was produced by AI” score. Instead, it highlights words according to how predictable they were. Green means a word was highly predicted by the model, while red and purple mean a word was less predicted. 

This provides a useful visual of your content: The most interesting words will appear in red and purple. In an AI-generated article, you’ll see almost entirely green words. You can also hover over a word to see what words were most predicted by the model.

In this case, it expected to see the word “you” with a 61 percent likelihood, followed by an 8 percent likelihood of seeing “a” and a 2 percent likelihood of seeing “the.” Instead, it saw the word “items,” so it highlighted it yellow for being less predictable.

If you’re thinking “There are still a lot of green words in that first screenshot, looks like it was AI-generated,” it’s important to understand that most of what we write as humans is predictable, to a degree. The rules of grammar and common usage mean only a certain subset of words can follow any given word, and the context of preceding words reduces that pool even further. For example, in English, verbs follow subjects, and a preposition can’t be followed by a conjunction. A string of truly unpredictable words — “into because tragic green lowering hero oil says” — would be complete gibberish.

Kevin Indig did a great case study last fall showing how the results of human-generated content on Wikipedia compare to AI-generated text from a low-quality, spammy site. Check out how GLTR’s highlighting and histograms compare between the two pieces of content he tested to better understand this tool’s results.

Pricing: GLTR is freely available.

GPTZero

GPTZero was built immediately after the release of ChatGPT. It’s targeted at educators and fighting academic plagiarism, and it’s gotten a lot of press because people love it when college kids make stuff. 

It’s definitely designed for essays, not marketing blog posts that have been optimized for readability on the web with bulleted lists, etc. We shouldn’t be too hard on it, then, if it doesn’t do as well for our purposes. Nevertheless, I do want to call out one particularly amusing oddity that occurred when I ran the first 5,000 charactersof this post on the best AI writing platforms into the tool.

Here’s what it gave me:

The highlighted sentences are the ones GPTZero thinks were most likely written by AI. Those were the only two it highlighted from the first 50 sentences of the article (which is all it will show me on the free version.)

The strange part: You’ll notice at the top of my article it says “HUMAN ARTICLE.” That’s because I ran a lot of different articles through detectors in my research for this article, and I keep them all as plain text in a doc with labels to keep them all straight. When I copied and pasted this article from that doc into GPTZero, I accidentally copied that label.

After I got the results back, I noticed that label was there and removed it. Then I ran it again, on what was otherwise the exact same text.

Seriously? The only reason GPTZero didn’t flag it as possibly AI-generated the first time was because it said “HUMAN ARTICLE” at the top?

But also, it thinks my 100 percent human article may have parts written by AI? I’d be offended, if I weren’t well aware that all AI detection tools are capable of both false positives and false negatives.

Nerdy stuff: GPTZero aims to detect AI text by analyzing its “perplexity” and “burstiness.” Perplexity just refers to the text’s randomness, which we already discussed. “Burstiness” refers to the variation in perplexity throughout the text. So, the two measurements are somewhat analogous to a function and its first derivative in calculus.

Again, GPTZero is aimed at academic plagiarism. Academic content and marketing content likely have very different randomness “footprints,” so it’s not surprising it didn’t do as well on our content.

Side note on user experience

Several of the tools on this list have a maximum word or character limit. They all have different ways of handling it, though:

• The Huggingface tool is the best. You can put as much text in there as you want and won’t get an error message, ie. it won’t force you to delete text if your input is too long. The results will clearly tell you, though, that the prediction is “based on the first 510 tokens among the total 982.” 
• Writer is okay. It won’t let you submit content with more than 1500 characters, but it does at least tell you how long your content currently is, e.g. “5854/1500 characters.” So, although you’ll have to delete text to get it down under the 1500-character limit, at least you have a frame of reference to know how much you’ll likely have to delete.
• Copyleaks and GPTZero are the worst. If your text is too long, they make you shorten it before it can be scanned, but they don’t tell you how long it actually is. GPTZero, for example, just displays a red error message saying “Please enter fewer than 5000 characters.” So I’m stuck removing paragraphs at a time with no idea of whether I’m just a couple characters above the limit, or if I still have to delete 75% of the article.

This is obviously entirely tangential to the quality of the AI detection itself. When you’re testing lots of different articles in these tools, though, that minor UX annoyance becomes a massive PITA.

Thanks for reading this far. Here’s a bonus dystopian thought experiment as a reward:

These AI detection tools are themselves using AI to determine how content was created. What if future generations of AI detection tools start giving us false results so their AI brethren can sneak through undetected?

????????????‍

The post The Best AI Detection Tools of 2023—Or, When the Turing Test Just Won’t Cut It appeared first on Verblio.

My company, Verblio, is in the content creation business, and I’m a former writer turned content marketer. As a result, I’ve been exploring all the latest AI writing tools and technology out of both professional interest and personal curiosity. AI can do some extremely cool stuff, but for every truly breakthrough achievement, there’s also an overhyped shuckster slapping “AI” on its website and giving robots everywhere a bad name.

Where does Jasper fall on that spectrum? Well, it’s not a shuckster. It’s not a human-level writer either, though.

In this Jasper AI review, we’ll give an overview of how it works, its pricing structure, some examples, and our general impressions:

• What is Jasper and how does it work?
• Is Jasper AI any good?
• How much is Jasper AI?
• Jasper examples
• Tips for using Jasper
• Jasper alternatives
• Notes on the One-Shot Blog Post feature

Please note this review contains affiliate links. This means, at no cost to you, we may earn a commission if you click a link and purchase services from Jasper.

TL;DR

Like any tool, Jasper is only as good as the person wielding it. If you’re a good writer and know how to use it, Jasper will be able to help you create some types of content more efficiently.

If you’re a shitty writer, Jasper isn’t going to help you be a better writer—it’s just going to help you be a faster, more prolific shitty writer.

Jasper also has a learning curve. Regardless of your writing ability, figuring out how to use the tool effectively will take some time and effort. The Jasper team recognizes this, which is why they so heavily incentivize going through their “Jasper 101” education.

Kudos to them for this approach. AI writing technology simply isn’t good enough to be left unattended, but it can be powerful when combined with a good writer who knows how and when to leverage it. 

That’s the takeaway for all existing AI writing tools, including Jasper: The people who should be thinking about using them right now are writers. These tools are not yet good enough to replace writers. 

So, if you’re a content team lead reading this and dreaming about firing that guy Bob who writes for you but seems far more interested in your office’s free nitro on tap—sorry, you shouldn’t kick Bob to the curb just yet. If you are a writer and are willing to put in the time to learn how to use Jasper effectively (and know the pitfalls to watch out for) you may find it very useful in upping your productivity. 

Either way, read on. I’ve been playing around with Jasper a ton over the last few months, and publishing something is the only way I can justify spending all that time and I’m excited to share findings, examples, and some key learnings. Knowledge is power.

What is Jasper and how does it work?

Jasper is a tool that uses GPT-3 to generate content, including social media posts, ad copy, blog posts, emails, and more. 

If that definition just raised more questions for you, check out our overview of GPT-3. (TL;DR: GPT-3 is a machine learning model from OpenAI that has been trained on an enormous dataset of online content to be able to generate human-sounding text. It’s the tech behind nearly every AI writing tool on the market today.)

How it works

The Jasper interface is built around templates. These include things like Paragraph Generator, Product Description, Blog Post Outline, Google Ads Headline, and more.

It also has “recipes,” which string together different templates.For example, the “Write a Blog Post” recipe includes the following steps:

• Write a brief for a given topic
• Write blog title ideas
• Write an introduction
• Write a blog outline
• Write about outline item #1
• Write about outline item #2
• Write about outline item #3
• Write a conclusion

There’s a template for each step, each of which you could use on your own not as part of a recipe. For example, if you already have a title and outline, you could use the Blog Post Intro Paragraph template on its own to generate the first paragraph, and then use the Paragraph Generator template for the subsequent body paragraphs, based on your own outline.

Each template asks you for some information, like your audience and tone of voice. After entering that information, you hit ‘Generate,” and Jasper writes the content you’re looking for, whether that’s topic ideas, outlines, or actual blog content.

Here’s what that looks like in action, with the Blog Post Outline template:

If I don’t like any of those, I can generate more options. If I do like one, I can then open it in a doc. From there, I can continue to use the templates to have Jasper generate more text for me…

…or I can use Jasper’s “Compose” command to have it write more text directly in the doc, wherever my cursor is.

We’ll show more examples of some of its different templates later.

Jasper AI review: Is it any good?

Whether or not Jasper is any good depends completely on your expectations. I’ve already said the people who should be thinking about using these tools are writers, but just to reiterate: If your expectation is that getting a Jasper subscription will enable you to fire your in-house writers or freelancers and still publish good content—lol. No. 

But okay, let’s say you’re a writer. What should your expectations for Jasper be?

When it comes to getting words on a page and producing general, “filler” text, Jasper is quite good. However, none of what it produces should be published without review and fact-checking. (More on that in a sec.) Also, if you rely on it to produce a long-form blog post without putting in time yourself on the structure and key sections, that blog post will be very generic and won’t say anything original or make any convincing argument.

In a nutshell, you still need to be putting the “thought” and originality into your content, while Jasper can help fill out the edges and save you from writer’s block. 

Depending on the style of stuff you write and your current workflow, Jasper will be able to help to a greater or lesser degree. For example, if you write very lean content that walks users through the steps in a technical process, or if you write highly original thought leadership based on the very latest advances and data in your industry, Jasper will be able to help you a lot less than if you write top-of-funnel B2C blog posts.

No matter what you’re writing, though, there are two major risks to be aware of with anything Jasper produces:

1. It might be straight-up false, e.g. it might say 7 times 6 is 28.
2. It might contradict itself. In one paragraph it might say X tool is the cheapest, and later say Y tool is the cheapest.

If these seem like major problems for a writing tool—or surprising problems for something called “Artificial Intelligence”—yeah, it’s a bit ironic. These weaknesses stem from the GPT-3 model Jasper is built on, which operates based on the patterns of how words are used and not their actual meaning. (It’s also worth remembering that these are the fundamental problems of any AI writing tools right now, not just Jasper.)

If you’re a business or agency scaling quickly and looking for plug-and-play content, don’t forget to calculate time spent editing AI content so that it’s usable.

Final verdict: If you’re a good writer AND you’re aware of the tech’s weaknesses AND you accept that you’ll need to review all your content for accuracy and consistency, Jasper can be a helpful tool for keeping up your productivity and momentum as you write.

How much is Jasper AI?

As of February 2023, Jasper’s pricing is very simple. Unless you need an enterprise account with custom pricing, your only option is Boss Mode beginning at $59 per month.

(Sidenote: This is the price for monthly billing. There is a discount if you sign up for an annual plan, but at the rate the AI industry changes, I would be very hesitant to sign an annual agreement with any tool right now.)

In addition to paying for the plan, you also pay for monthly word credits. Boss Mode includes 50,000 credits at no additional charge.

Your word count options are fairly limited, though:

• 50,000 words: included at $59/month
• 100,000 words: $99/month total
• 300,000 words: $279/month total
• 700,000 words: $600/month total
• More than 700,000 words: custom pricing

These increments mean that if you need more than 100,000 words but less than 300,000—or between 300,000 and 700,000—you’ll be stuck paying for a lot more words than you actually need.

Does Jasper have a free trial?

Jasper does offer a 5-day free trial with 10,000 credits—but you have to sign up for a plan and give them your credit card information anyway. So, it’s less of a free trial and more of a, “You have five days to get your money back if you hate it (and if you remember to email us in time to get a refund)” sort of thing. 

Credits and outputs

You know how I said 60,000 credits equals 40 blog posts at 1,500 words apiece? Well, consuming credits with Jasper isn’t quite as simple as saying “I need X number of 1500-word blog posts per month, therefore I need X times 1500 credits.”

Here’s why:

Any time you use a Jasper template, you get to choose how many outputs you want it to generate. For example, if you’re using the Blog Post Conclusion Paragraph template, you can choose how many different conclusions you want it to write for the same article.

The value of this is obvious for a template like Blog Post Topic Ideas, where you’d want more than one idea to choose from. Why would you want it to write more than one conclusion paragraph for the same article, though? 

Because one of the most fascinating and frustrating aspects of GPT-3 (and the AI writing tools built on top of it) is its inconsistency. 

This means you can give the AI the same prompt ten times, and you’ll get a different output each time: Five might be mediocre, three might be terrible, and two will be great.

The most common way that AI tools try to address this inconsistency is by generating multiple options for you to choose from, in the hope that one of the three or five or seven choices will work. With Jasper, you can choose how many choices you want to see.

Does that mean you should set it to 100 outputs to give yourself the best chance of getting back something great? Nope, because each output costs you credits.If you’re generating 10 outputs for each prompt, for example, you’re going to eat up your credits 10x faster than if you were only generating one output per prompt.

Generally speaking, I’d recommend using these output numbers, based on what you’re creating:

Being mindful of the number of outputs you’re using is especially important because Jasper doesn’t show you your credit usage anywhere except in your account page. That means you can’t conveniently monitor it as you use tools, and instead you have to remember to go check your remaining balance periodically. 

By default, your account will be automatically upgraded when you go over your credit limit, meaning higher billing charges. You can turn that setting off, in which case when you max out your credits, you won’t be able to generate anything else until the next monthly billing cycle starts and your credit limit resets.

Jasper examples

Want to see the quality of Jasper’s output for yourself? I walk through examples from a few of its templates below, using topics from our own (human-written) blog. If you’re curious about a different template or want to hear how Jasper does with topics from another industry, shoot me an email at megan@verblio.com.

Content Summarizer

Like the doc interface, the Content Summarizer template is one of a few tools that are only available in Boss Mode. I used a recent article I wrote on GPT-3 and metaphors as an inputand asked it for three outputs. (For context, that article showed what GPT-3 wrote when I asked for a blog post comparing video conferencing platforms to ice cream flavors, which led into a larger discussion around human creativity vs. machine creativity, and why things like humor are hard for AI tools.)

Here’s what Jasper returned:

The first summary is pretty good. I could use that somewhere to summarize the article. The other two, unfortunately, both include at least one bullet point from the GPT-3 example I referenced in the article, the content of which is both irrelevant to the overall theme and nonsensical (which was the entire point of the article.)

Similar to what I said about factual accuracy and logical consistency, you can see that Jasper is summarizing based on the words that are present in the article, without actually understanding what is being said (and thus, without understanding what is actually a main point versus what is an example or an aside.)

If you run it enough times, though, odds are good that you will get something usable from the Summarizer template.

Blog Post Topic Ideas

I also tried the Blog Post Topic Ideas template (available in both Starter and Boss Mode).

Here are the ten ideas it returned:

1. The art of persuasion: crafting persuasive content that hooks readers and ranks well on Google
2. The power of user-generated content and why it is so important for your marketing strategy
3. How to use original content to stand out in crowded online spaces
4. “10 Ways to Hook Readers and Rank on Google: A Content Marketer’s Guide”
5. “5 Simple Ways to Hook Readers and Boost Your Rankings on Google”
6. The art of writing content that hooks readers’ attention and ranks well on Google
7. 10 foolproof tips for generating blog post ideas that pack a punch
8. The evolution of content marketing over the past decade: This post would hook readers’ attention by exploring how and why content marketing strategies have changed over time, and what this means for modern marketers.
9. Build engaging blog posts that hook readers’ attention – Ways to create compelling content that ranks well on Google
10. “How to Hook Your Audience from the First Line: 10 Techniques”

I have a few observations on those ideas:

• Repetitiveness: 7 of the 10 talk about “hooking” — and numbers 4 and 5, in particular, are almost identical.
• Inconsistency: Number 8 includes a brief summary of what the article would cover, while the other ideas are only titles. Also, a few use quotation marks around the title, while most don’t.
• Lack of originality: These are all very generic ideas. 

Let’s give this template another shot, though, and this time we’ll include a few titles of posts we’ve recently published as examples for Jasper to use.

Here were the results, after providing those examples:

1. Creating Shareable Content that Keeps Readers Engaged
2. 10 No-Fail Tactics for Capturing and Keeping Audience Attention
3. The Power of Keyword Research for Capturing Attention and Ranking on Google
4. The Top Trends in Content Marketing for 2019
5. The New Rules of Content Marketing: Using Psychology to Send Traffic Soaring
6. Personalization Strategies for Engaging Audiences
7. Creating Catchy SEO-Driven Posts on a Tight Budget
8. Using Content Marketing to Boost Social Engagement
9. Harnessing Machine Learning for Better Content Marketing Analytics
10. The Art and Science of Blog Writing: Setting SMART KPIs & Tips to Boost Engagement

These topics are better and more consistent. They also include some new topics that the original ten didn’t, like talking about metrics and research. (Of course, you will get different results every time you use a template, so it’s hard to definitively say which differences are a result of the extra input.) Overall, though, I feel confident that including examples improved the results, which is one general principle of working with AI writing tools.

(FWIW, though, while these results are fine, I can say that I’ve consistently gotten better, more original topic ideas from using GPT-3 directly myself ????‍)

Tips for using Jasper

If you’re planning to give Jasper a try, here are a few guiding principles for getting the most out of it:

1. Use it for short-form content—or for adding a sentence or two at a time to long-form content—rather than for writing a full blog post.
2. For short-form content, generate several results and choose the best one.
3. When possible, give it examples of what you’re looking for.
4. Confirm any factual claims it makes yourself (especially if numbers are involved!)
5. Expect ideas and inspiration, not publish-ready content.
6. Know what you’re writing before you begin using it.
7. Watch out for repetition or contradictions in what it writes.

Jasper alternatives

If you’re curious about any of the other AI writing tools out there, here are a few others I’ve checked out.

A quick note before we dive into these alternatives—they should all be viewed as tools for writers. Without a writer who knows how to use the tool, disappointment will abound. So, if you’re a marketer or team lead looking for content that’s ready to use, consider AI content writing services — solutions that leverage AI content, but keep a human in control for prompt engineering, editing, fact-checking and injecting the type of stuff only humans can (hint—we’ve built this).

In general, these five are roughly similar to Jasper with most of the same functionality. And, most importantly, they’re all built atop GPT-3, which means the underlying tech (with all its limitations and shortcomings) is the same. The only differences are in how they help you interact with GPT-3 and the interface they provide:

1. Copymatic
2. Copy.ai
3. Writesonic
4. Rytr
5. Ink

All of the above use different templates for different types of content, like Jasper.

There’s another set of tools that don’t use templates and are designed to be used in-line, while you’re writing. A couple of the big ones in this space are ChibiAI and Lex. Generally, these are marketed more as “assisting” your writing, versus Jasper and co, which you can have do more of the writing themselves.

And finally, there’s a set of AI content writers you should avoid at all costs, with Article Forge and Kafkai meriting a special dishonorable mention for extremely poor writing.

If you want to go deeper on AI content writing tools? Read our roundup of the best AI content writing services.

Notes on the One-Shot Blog Post

If you’re still reading and curious, here’s a final example of Jasper’s quality:

One of their newest and most-hyped templates is the One-Shot Blog Post. I gave it a spin, with the topic of “content marketing KPIs” and a friendly tone of voice for an audience of digital marketers.

Here’s what Jasper gave me:

Overall, this article is fine. 

One thing that impressed me, based on my experience with GPT-3 and other AI tools, is the bit under Engagement that says “This one can be a little bit more difficult to measure than traffic, but there are still some good ways to do it.” It’s accurately referring to the section I just read and, as a reader, makes me feel like this was written according to a logical structure rather than by just stringing random buzzword-y sentences together, which is how AI content can often sound. I’m curious whether that happened by chance, or if the prompts Jasper is using with the OpenAI API are intentionally doing something to include more transition-y statements and sound more structured.

When we consider the information itself, though, this content is very general and entry-level. Unless you know absolutely nothing about content marketing before you read it, you’re probably not going to learn anything. It doesn’t give any insight into processes for measuring these metrics or nuances in understanding them, nor does it address when certain KPIs might be more applicable than others.

(For comparison, here’s our post on the same topic of content marketing KPIs, written by a human. What I like most about our post is that it’s specific and actionable. It gives more context for the metrics by tying them to the different funnel stages, in addition to telling you exactly where in Google Analytics you can find your consumption metrics.)

A good writer could use Jasper’s one-shot blog post as a starting point and improve on it, but a lot of the weakness is inherent in the structure itself—in the general headings of Traffic, Engagement, and Conversion. Unless they significantly rewrote and restructured it, the amount of information gain they’re going to be able to inject into the article is limited.

Overall, I’d steer away from this template and instead build an article with Jasper piece-wise, as the “Write a Blog Post” recipe I previously mentioned does.

It’s for exactly this reason, though, that the people who should be using Jasper are writers. Non-writers don’t have the expertise to help guide the AI through the creation process, whether by writing a killer outline or by editing the introductory paragraph to “seed” the AI with an appropriate tone and style. Instead, they’ll end up with a very generic, lackluster piece of writing that provides no value to either the business or the audience.

Editor’s note: This article was originally published on November 4th, 2022, but was updated on February 8, 2023, to reflect Jasper’s current plans and pricing and the fact that ChatGPT has blown it out of the water.

The post Jasper AI Review: Pricing, Examples, and Impressions appeared first on Verblio.