Breaking Hashnode with Passmark

Firstly, let me just say how excited I am that Hashnode is finally hosting hackathons again. I participated in my first ever hackathon here in Hashnode. And I have to be honest, I thought they were done with hackathons since the last one before this ran in early 2024.

This was wonderful news. Here's to many more.

So... I thought what better way to celebrate their return than by making them the center of my article and therefore the core subject of my tests in this Breaking Apps Hackathon.

Passmark

Passmark is an open source library that sits on top of Playwright that you can use to test your websites and web apps end-to-end with natural language, without the complication of learning specific built-in assertion APIs (e.g. expect(page).toHaveTitle("Playwright")).

The main advantage is you can put yourself in the frame of mind of the user that WILL be using your apps. You can ask Passmark to navigate and test (or break) your website like you would tell your friend.

Example:

test("has my name", async ({ page }) => {
  test.setTimeout(60_000); // increase timeout for AI execution

  await runSteps({
    page,
    userFlow: "View portfolio page",
    steps: [
      { description: "Navigate to https://ansellmaximilian.github.io/" },
    ],
    assertions: [
        { assertion: "Ansell Maximilian is visible and in clear big text" },
        { assertion: "A summary is placed below my name and isn't overflowing over anything." }
    ],
    test,
    expect,
  });
});

Then, you'd just run the following command:

npx playwright test

If all goes well, you should see something like this:

Let's do what it suggested and run that command.

npx playwright show-report

This will open up a page with a detailed report on your test(s).

Here's mine. I only have one, so I'll click it and I'll be redirected to its detail page. Here's what it looks like:

How cool is that? It listed AI summaries explaining, in natural language, the logic behind the test passing/failing. In this case, the test succeeded and the AI reasoned in a human readable language why it passed it, just like a user would.

See that mention of accessibility snapshots? Passmark used it to determine whether or not the text I wanted it to assert was clear, just like I asked it to.

Testing Hashnode with Passmark

Okay, enough introductions let's get started on what we came here to do: breaking (or attempting to break) Hashnode. Now a little bit of a disclaimer: Hashnode is a live production app with built-in protections against bots and automation, as it should. So, in these example tests I will not be attempting to test or break features behind authentication. Typically, you'd use Passmark with your own app anyways, where you'd disable bot protections until you deploy to production.

Starting Small

Let's start with something small. I'll skip the basic stuff we just did, which is asserting the existence of text or titles, etc.

How about something that requires interaction? That will be our next logical step from just visual confirmation. Let's play with Hashnode's sidebar:

How about we test its collapse mechanism:

test("sidebar collapse mechanism", async ({ page }) => {
  test.setTimeout(60_000); // increase timeout for AI execution

  await runSteps({
    page,
    userFlow: "Collapse the sidebar",
    steps: [
      { description: "Navigate to https://hashnode.com/" },
      { description: "Collapse the sidebar" },
    ],
    assertions: [{ assertion: "Sidebar is collapsed" }],
    test,
    expect,
  });
});

Result:

You see the pattern right? Tests pass or fails and the AI clearly explains what happened and why it gave the result it gave.

More Complicated Tests

Okay, it handled simple sidebar collapse/expand states very well. Let's give it something harder to test. Something a lot more dynamic.

What if the Hashnode team wanted to make sure this Breaking Apps hackathon, or any other currently ongoing hackathon, is properly featured on the home page and that its countdown is properly showing the time remaining until deadline?

const FEATURED_HACKATHON_DETAILS = {
  title: "Breaking Apps Hackathon",
  description:
    "A 4-week open hackathon to test your apps, test the internet, and share Passmark — the open-source AI library for regression testing. $4,000+ in prizes. Free AI credits included.",
};

test("featured hackathon is visible", async ({ page }) => {
  test.setTimeout(100_000); // increase timeout for AI execution

  await runSteps({
    page,
    userFlow: "View the featured hackathon on the homepage",

    steps: [{ description: "Navigate to https://hashnode.com/" }],

    assertions: [
      { assertion: "The featured hackathon is visible on the page" },
      {
        assertion: `The featured hackathon has the correct title: ${FEATURED_HACKATHON_DETAILS.title}`,
      },
      {
        assertion: `The featured hackathon has the correct description: ${FEATURED_HACKATHON_DETAILS.description}`,
      },
    ],

    test,
    expect,
  });
});

 

First, we set up a config object for the featured hackathon we want to assert. Then in our first test case, we use that config object to dynamically assert that Hashnode's home page has the correct hackathon being featured, this time being the Breaking Apps hackathon.

But because we've set it up such that the test takes assertion values from the object we defined, if ever we will need to assert for a different hackathon in the future, all we have to do is edit the config object (without altering the actual test cases).

test("featured hackathon shows correct countdown", async ({ page }) => {
  test.setTimeout(300_000); // increase timeout for AI execution

  await runSteps({
    page,
    userFlow: "View the featured hackathon countdown",

    steps: [
      { description: "Navigate to https://hashnode.com/" },

      { description: "Click the featured hackathon" },
      {
        description:
          "Look at the submission requirement and take not of the deadline date",
      },
    ],

    assertions: [
      {
        assertion: 'A "Live now" countdown badge is visible for the hackathon',
      },
      {
        assertion:
          "Deadline date exists and is visible in the submission section",
      },
      {
        assertion: `The coundown badge shows the correct time remaining based on the current date (${Date.now()}) and the hackathon deadline within ~1 hour accuracy`,
      },
    ],

    test,
    expect,
  });
});

In the other test case, we make sure that the countdown badge shows the correct time remaining based on the deadline and the current date.

Notice how I took advantage of natural language to determine the deadline simply by asking Passmark to look at the deadline date itself instead of explicitly saying what it is.

Then without much processing or formatting I simply put Date.now(), which returns milliseconds since the Unix epoch, instead of converting to the correct timezone that the deadline uses (PT).

Here are the results of the two test cases:

Simple success case. But the human language reasoning is extra nice instead of a binary success/fail.

This one is a lot more interesting. First, it confirmed that the hackathon detail page does indeed contain the countdown badge.

Then, it also confirmed it has the deadline date, which we required for the next step.

Finally, it asserted that the countdown is indeed correct based on the current date and the deadline that I asked it to take note of. Notice how it converted the timestamp into a readable date without me asking it to. And than it compared it to the deadline and confirmed that the test passed.

You can see clearly here where AI could really elevate your testing. Instead of explicitly setting the deadline date yourself or maybe navigating the DOM to find it and then parsing it in the correct format, you can simply ask the AI to find it.

This way, you're testing for the existence of the deadline itself, the existence of the countdown, and the fact that both should be synced.

Tests with Data and Input

Let's up the ante.

test("search feature works", async ({ page }) => {
  test.setTimeout(300_000); // increase timeout for AI execution

  await runSteps({
    page,
    userFlow: "Use the search feature",

    steps: [
      { description: "Navigate to https://hashnode.com/" },

      {
        description: "Click the search menu in the sidebar",
        waitUntil: 'The "Search Hashnode..." dialog is visible',
      },
      {
        description:
          "Search for user but don't click on any of the search results",
        data: { query: "ansell max" },
        waitUntil: "Search results are visible",
      },
    ],

    assertions: [
      {
        assertion:
          "Search results are visible and display posts by Ansell Maximilian",
      },
      {
        assertion:
          "Search results are visible and display the expected user with username @ansellmax",
      },
    ],

    test,
    expect,
  });
});

Here I'm testing if the search feature works and that my account shows up. Some new things to note here is the usage of waitUntil and data.

waitUntil is simply a natural-language wait condition that has to complete before the test proceeds. Basically, a way for you to say "wait until this happens before moving on to the next steps". Again, keeping with the philosophy of testing by asking the test runner to do things like you would a real human.

In our case here, we used it to halt the steps on two occasions: one to wait until the search dialog is visible and the other to wait until the search is done loading, just to make sure the test isn't asserting the loading state.

data is like your typical payload you'd include in regular tests, but way more flexible. Instead of explicitly defining data and then calling specific properties you defined, you'd simply have to give it meaningful key-value pairs that the AI would know how to use. Like waitUntil, we can use natural language instead of programmatic values.

In our test case, we used it to tell the AI to use that data to fill in the search input. Here we put query with a string. But query could just as easily have been searchValue. As long as it makes sense in natural language within the context of the test, you can use it.

Trying to Break Hashnode

I gotta be honest: Hashnode is hard to break. I mean it is a production ready application that I'm sure has done its own extensive testing internally.

So I'm going to be...

I saw that when you go to the page for a tag, e.g. #breakingappshackathon, the list of total posts doesn't always accurately represent how many posts there actually are. So let's try to check if Passmark can detect that.

For example in #breakingappshackathon, currently as of writing this, it shows there are 17 posts, but I'm only counting 10. And when I click "Load more" no new posts show. Of course this could very well be intentional. Maybe some posts are shadowbanned. But let's see if Passmark can detect this, what I noticed manually, abd correctly assert it.

Because being able to fail test cases correctly is just as important as passing them for a testing library. And it would be nice to end things off with actually "breaking" things.

test("tag page displays correct content", async ({ page }) => {
  test.setTimeout(300_000); // increase timeout for AI execution

  await runSteps({
    page,
    userFlow: "View the tag page",

    steps: [
      { description: "Navigate to https://hashnode.com/" },

      {
        description: "Click the search menu in the sidebar",
        waitUntil: 'The "Search Hashnode..." dialog is visible',
      },
      {
        description: "Search for a tag",
        data: { query: "#breakingappshackathon" },
        waitUntil: "Search results are visible",
      },
      {
        description:
          "Click on #breakingappshackathon tag in the search results",
        waitUntil: "The tag page is visible",
      },
      {
        description:
          'On the bottom of the list, click "Load more" to load more posts',
        waitUntil: "Load more post button is gone",
      },
    ],

    assertions: [
      {
        assertion:
          "Under the #breakingappshackathon tag, there is text showing how many posts are under the tag, e.g. '18 posts'",
      },
      {
        assertion:
          "Search results are visible and display posts related to the #breakingappshackathon tag",
      },
      {
        assertion:
          "The amount of posts matches the 'n posts' number shown in the beginning of the page",
      },
    ],

    test,
    expect,
  });
});

Again, I probably don't need to explain things anymore, but you can see it very clearly how human and non-robotic my instructions are.

Here was the result:

Now not only did it correctly fail my test case, as per the parameter I've set up for it, it also provided detailed insight in its reasoning for doing so. Look at the last AI Summary. It described the failure exactly how I noticed it while manually testing it. It correctly counted 10.

Repo

If you'd like to check out the test suite I used to create the article, here's the public repo over at Github. You can clone the repo, run npm install and then run:

npx playwright test --project chromium // or omit --project entirely

Conclusion

Obviously this was mostly for fun and not meant to be a legitimate test suite for Hashnode, as I'm sure they have their own comprehensive set internally. The point of this post was to introduce you to Passmark in a fun way and to show you the potential of including AI in your end-to-end flow using Passmark.

Typically, you'd use your own application to test with Passmark; that way you'd truly be able to use Passmark end-to-end without being blocked by bot detection and other things.

But from these simple examples, I hope you can see the benefit of using natural language, instead of explicit API calls, to test your websites. By using natural language to describe steps and assertions, you put yourself in the mind of a user and you skip a lot of boring programmatic setup and documentation reading.