They say that ChatGPT-4 is 10 times more powerful than its predecessor, but does all this extra power have any bearing on its writing skills?

I was sceptical from the get-go, because when I went on the pricing page to upgrade to Plus, there was no mention of ChatGPT-4 being a better writer.

Then I remembered that this ‘large language model’ can do millions of things, so it was rather selfish of me to expect the folks at OpenAI to advertise their premium product in a way that speaks to my specific needs.

So I went ahead and upgraded my plan.

Over the course of about a month, I got ChatGPT-3.5 and ChatGPT-4 to write 150 articles each for my brand new home decor blog, and then waited a further three months to see how much traffic they generated.

This means it’s not just me judging whether GPT-4 is a better writer than the free version—we also get to see if Google prefers one over the other!

Speed

Winner: ChatGPT 3.5

You would think that the premium version would get things done much faster than the free version, wouldn’t you? ChatGPT-4 can be painfully slow at times.

When I first encountered GPT-4’s constant pausing and stuttering, I actually got a bit excited because I thought this meant it was working harder to produce an absolute masterpiece. But I was wrong.

Worse still, I was getting a lot more “There was an error generating your response” messages than what I experienced using the old version, which really slowed down my workflow.

Speed isn’t really a key measure of writing ability, but I thought it’s worth spreading the word about ChatGPT-4’s slowness in case you are someone who produces content at scale.

Word count

Winner: ChatGPT 3.5

ChatGPT-4’s inability to hit the 500-word mark was a huge source of frustration for me.

I know some SEO experts insist that Google doesn’t require at least 500 words in order to index and rank a piece of content, but this is and always has been my minimum threshold, and I wasn’t prepared to change my ways for this so-called powerful chatbot.

The 3.5 series routinely hits this threshold on the first go, whereas I often have to ask 4.0 to elaborate on certain points just to beef up the text, again leading me to wonder “why am I paying for this?”.

Originality

Winner: ChatGPT-4

GPT4’s access to real-time information was definitely advantageous for my home decor blog.

Decor trends change all the time and so it was great that GPT4 could talk about the latest trends, which in turn made the articles more fresh and original compared to what GPT3.5 was outputting.

The Plus version is also multimodal, which means I was able to feed image prompts to GPT4 and ask for suggestions on keywords and content ideas etc. It felt like I was tapping into a whole new part of GPT4’s brain when it was analyzing images of decor schemes.

Unfortunately, both versions were as bad as each other when it came to the presence of common ChatGPT phrases. I dealt with this swiftly to give my AI-generated articles the best possible chance of indexing and ranking.

Indexing

Winner: draw

With the recent Google updates wreaking havoc across the SEO world and de-indexing millions of pages, I was extra curious to know if ChatGPT’s writing could meet the new indexability standards.

I can report that neither 4.0 or 3.5 ran into any indexing issues, thanks largely to the fact I removed all the ‘classic’ ChatGPT phrases from the content. You know the ones—”In this comprehensive article, we will explore…”

If you are experiencing indexing issues, I strongly recommend checking through your content for common phrases and learn about the words to avoid. Leaving them in there is like asking Google to blacklist your website!

Ranking

Winner: ChatGPT-4

A total of 300 AI articles were given three months to rank and generate traffic on Google. The figures for the final month are as follows:

• 454 web clicks
• 142 image clicks

Bringing in nearly 600 clicks a month is a solid start for a blog this young and a niche this competitive. But what we really want to know is how the numbers split between ChatGPT-3.5 and ChatGPT-4.

Articles written by GPT4 are responsible for 61% of the web clicks and 95% of the image clicks. I should point out that the GPT4 articles featured AI images (users of Plus gain access to DALL·E) and the GPT3.5 articles featured free stock images.

I totally underestimated how well AI images perform in Google Images. More pertinently, the 4.0 series edged out 3.5 for the most web clicks, which suggests Google prefers content written by the former.

Verdict

Focusing squarely on the quality of writing, I’m not overly impressed with GPT4, especially when you consider it’s supposed to be 10 times more powerful. Interacting with it was really stressful at times, and I’m not sure the results were worth the stress.

Objectively speaking, 4.0 outperformed 3.5 in terms of traffic, but I think this is down to the additional functionalities that come with the Plus plan and not so much that it’s a better wordsmith.

For instance, the ability to ask GPT4 to analyze a photo and provide keyword opportunities leads me to believe that GPT4 is just better at playing the SEO game, which is why those articles received more clicks.

It also helped that the GPT4 articles used original AI images as opposed to the GPT3.5 articles which relied on second-hand stock images.

So, I don’t think ChatGPT-4 is ‘naturally’ a better writer. Rather, the additional tools that come with the Plus plan can make your content more SEO-friendly.