There’s a lot to be excited about in tech today. Innovation seems the weekly norm. But no tool seems to get quite as much press as OpenAI’s ChatGPT.
- It can write like Shakespeare…
- It can analyze complex data sets in a flash…
- It can write code…
- It can do math homework…
- It can translate language…
Heck, by Christmas it’ll probably be able to wrap our presents!
Of course, the AI tool has some notable drawbacks, like bias, plagiarism, and just plain making up nonsense. However, it’s still helpful for certain use cases.
We’ve covered how the tool can ramp up an already-rockstar executive assistant’s productivity. But, because our team is training extensively on this and other AI tools, we wanted to push its limits.
So, five top-caliber EAs from our team tested ChatGPT’s abilities with tasks they perform every day in these areas:
- Interpreting real-time data
- Human empathy
- Attention to detail
- Nuance and intent
- Following specific instructions
Here’s how our battle of the humans vs. the AI unfolded, along with a “✅Pass or ❌Fail” verdict for each use case.
Test #1: Can ChatGPT Interpret Real-time Data To Make Informed Decisions?
Our first test was using real-time data to make informed decisions.
Our EA reported: “In planning an itinerary, ChatGPT hit all the tourist spots. But what it doesn’t note is critical: [X location] is closed for renovations and you can’t actually tour it other than the outside!”
Oy! This blunder reveals a critical flaw: while the AI is great at generating lists with whip-crack speed, it struggles to integrate real-time, context-specific information.
This oversight might seem small, but imagine taking an Uber to a fun location after a demanding day, only to find a construction fence! That’s a lot of wasted time, disappointment, and frustration.
This is not the experience a great EA delivers. Ever.
It failed because of its inability to handle real-time data and its nuanced interpretation. ChatGPT, as efficient as it is with handling massive data, often stumbles when it comes to contextual understanding.
While generative AI is making impressive strides, it still has some distance to go in taking real-time, highly-relevant data into account when making decisions.
✅Pass or ❌Fail?
❌Fail
Test #2: Can ChatGPT Mimic Human Empathy?
Our second test was testing its ability to have empathy for, and be considerate of, everyone involved in a cross-timezone meeting.
Here’s what our EA found:
“I asked ChatGPT to find a good meeting time for a few clients; it suggested 10AM EST. But that works out to 4AM HST… I don’t think the Hawaii participants would appreciate having to get up that early for a meeting!”
This is a big one for us—as care and kindness are bedrocks of our culture!
While it accurately suggested a meeting time, it failed to consider how the time zone differences would affect actual humans. Sorry, Hawaii-based folks.
In a business context, an executive would either face disgruntled clients, no shows, or even missed opportunities. As advanced as AI might be, it’s still no match for human empathy and consideration. Failure to understand and adapt to the human element has real consequences.
This sets off a chain of future problems, too. What about other interpersonal dynamics beyond respecting personal time and preferences? This is something even the best prompt can’t reproduce to the standard premium EAs hold themselves to.
✅Pass or ❌Fail?
❌Fail
Test #3: Can ChatGPT Work With High Attention To Detail?
Our third test seemed pretty simple. Can ChatGPT handle the nitty-gritty details? It can handle thousands of cells of data in seconds—surely it could handle a few requirements, right?
Here’s what our EA reported:
“ChatGPT came up with 3 of the meeting spaces I had already looked into but some of the facts were not correct. For instance, it said one of the locations only could hold up to 14 people but the room actually holds 22 people!”
ChatGPT offered up locations already on our EA’s radar—a fair move. But, it stumbled on a critical detail: misjudging the capacity of a meeting room.
An oversight like this is more than a minor hiccup in the business realm. It’s not a huge issue for 14 participants to meet in a room that can hold 22. But what about 20 people arriving at a venue that only fits 14—with plenty having flown in for the occasion?
That’s the issue! Inattention to detail creates delays, wastes valuable time, and is a poor look on the company hosting the meeting.
✅Pass or ❌Fail?
❌Fail
Test #4: Can ChatGPT Understand Subtleties, Nuance, And Intent?
Test four dug into the AI’s ability to grasp a human’s intent behind the prompt.
Here’s what our EA shared:
“I asked for free things to do around a nearby city. Even though I tried wording it differently several times, it kept showing me the same attractions that have an entrance fee.”
What happened here? The assistant was on the lookout for budget-friendly recreational options, yet ChatGPT persistently offered paid attractions. Clearly, this misses the mark.
This scenario sheds light on a particular struggle faced by AI. It often stumbles over the crucial nuances that play a significant role in real-world contexts. Whether it’s accommodating a client’s preferences or planning a team activity within a limited budget, EAs need to nail the minutiae.
Even the most sophisticated AI can struggle to comprehend subtle differences in language or intent. In this case, ChatGPT efficiently provided a list of attractions, but overlooked the “free” part of “fun.”
This showcases where the human touch is indispensable, bringing nuance and understanding, grasping subtlety and intent, and having a deeper understanding of who the client is as a person.
✅Pass or ❌Fail?
❌Fail
Test #5: Can ChatGPT Follow Specific Instructions?
Our fifth and final test deals with the complexities of travel, itineraries, and following instructions.
Here was our EA’s summary:
“I asked ChatGPT to modify its trip suggestions by removing all museums. The revised itinerary had at least one museum on it!”
Our assistant had a straightforward request: a travel itinerary excluding museums. However, ChatGPT’s updated itinerary disregarded this specification, resulting in a museum still being part of the plan. Every time.
This highlights a significant hurdle AI might face: comprehending and acting upon specific instructions. Again, deleting a museum visit from an itinerary doesn’t seem like the end of the world. However, our tests were designed to pinpoint failures so we could extrapolate to more serious consequences.
Scale inability to follow specific instructions to operations, planning, or any other key business output, and you have much higher consequences from a seemingly innocuous error. An error, by the way, that might be difficult to track down and correct.
✅Pass or ❌Fail?
❌Fail
AI + Humans = Winning
And there we have it. A glimpse into the world where AI and human minds collide rather than collaborate. These tests from the trenches of executive assistance show us that while AI tools, like our friend ChatGPT, are genuinely impressive, they aren’t ready to fly solo just yet.
Our AI assistant can handle a heap of tasks, churning out itineraries and suggestions at lightning speed. But, it can sometimes get things wrong that have real consequences. Some are small, but others are severe.
Powerful tools like ChatGPT can boost human productivity. But, it takes a highly-trained executive assistant to get the right output and scan for inevitable errors.
Whether it’s planning a meeting that doesn’t have your Hawaii clients waking up at 4AM, or tailoring an itinerary that ditches museums, getting these details right isn’t optional. It’s required. Because it’s what we do. And that’s why we need the human touch.
So, where does that leave us? We think in a pretty exciting place.
AI and humans, working together, can achieve amazing things. AI takes the grunt work, freeing up humans to do what they do best: think, create, empathize, and grasp the nuances that make us human.
In a nutshell, AI is a powerful tool, but it’s not the be-all and end-all. The magic really happens when AI and humans work together. That’s when we’re on track to see some real game-changing stuff in the business world.
In the end, it’s not about AI vs Humans. It’s about AI + Humans.
That’s the future we’re looking forward to.
