# Testing in the Age of AI - Requirements First

Did you know that many software defects arise from unclear requirements?What we're seeing today is AI attempting to assist in test creation,promising to increase code coverage. However, this approach oftenemphasizes testing our implementations rather than testing againstour requirements, which I believe has its value but also carriescertain risks that must be carefully considered.

## Don't test your implementationโ

To illustrate, let's conduct an experiment using our familiar chatbots. I'll present a simple function implementation suitable for our demonstration, intentionally flawed.

This is a TS function. Can you generate me a jest test spec for it?

`function addTwoNumbers(a: number, b: number): number {`

return a / b

}

Here's, what **ChatGPT 3.5** responded:

Sure, here's a Jest test spec for the addTwoNumbers function:

`import { addTwoNumbers } from './yourFileName';`

describe('addTwoNumbers', () => {

it('should correctly add two numbers', () => {

// Arrange

const a = 5;

const b = 3;

// Act

const result = addTwoNumbers(a, b);

// Assert

expect(result).toEqual(8);

});

it('should handle negative numbers correctly', () => {

// Arrange

const a = -10;

const b = 7;

// Act

const result = addTwoNumbers(a, b);

// Assert

expect(result).toEqual(-3);

});

it('should handle zero correctly', () => {

// Arrange

const a = 0;

const b = 5;

// Act

const result = addTwoNumbers(a, b);

// Assert

expect(result).toEqual(0);

});

// Add more test cases as needed

});

The generated tests do not account for my implementation; they assume I want to add numbers as the function title suggests. The test will fail. However, this allows me to challenge my code.

Here's, what **Gemini 1.0** answered:

I'd be glad to provide a Jest test spec for the addTwoNumbers function, incorporating the valuable insights from the ratings and addressing the identified issues:

`import { addTwoNumbers } from './path/to/your/function'; // Replace with the actual path`

describe('addTwoNumbers', () => {

// Test case 1: Basic addition with positive numbers

it('should add two positive numbers correctly', () => {

expect(addTwoNumbers(5, 3)).toBe(5 / 3); // Corrected the expected result

});

// Test case 2: Addition with negative numbers

it('should handle addition with negative numbers', () => {

expect(addTwoNumbers(-2, 4)).toBe(-2 / 4); // Corrected the expected result

});

// Test case 3: Division by zero (error handling)

it('should throw an error when dividing by zero', () => {

expect(() => addTwoNumbers(5, 0)).toThrowError('Division by zero'); // Handle division by zero appropriately

});

// Test case 4: Data type validation (optional)

it('should only accept numbers as arguments (optional)', () => {

expect(() => addTwoNumbers('five', 3)).toThrowError(

'addTwoNumbers: Arguments must be numbers' // Customized error message

);

expect(() => addTwoNumbers(5, 'three')).toThrowError(

'addTwoNumbers: Arguments must be numbers' // Customized error message

);

});

});

Gemini generated tests that pass. The test reads like an addition is tested, but effectively, it's the division.

To be fair, sometimes the bots detect inconsistencies between the function name and implementation.

However, we miss the point here:

**What's unknown is if our customer was requiring an addition or a division.**

Either the function implementation is incorrect and should be `a + b`

,
or there's a naming issue, and it should be named `divideTwoNumbers()`

.

So, what are the takeaways?

**1. Testing should align with requirements and expected behavior**

This is crucial to ensure that code functions as intended and potential issues are identified.

**2. Importance of human judgment**

It's essential for humans to analyze the information provided by large language models and make informed decisions based on context and domain knowledge.

## Align your tests with your requirementsโ

While I don't want to leave you without a solution to this problem, it's worth noting that you should utilize AI to assist in testing.

However, it's important to focus on explaining the purpose of the test and the problem you want to solve, rather than just testing the mplementation itself.

While this principle is not new, it's crucial to remember, as it's something that can be overlooked when rushing into the use of AI.

Can you write me a jest test spec that test test a typescript function that takes two numbers and divide them. Please cover edge cases.

`import { divideNumbers } from './yourDivideFunction'; // Import your divide function`

describe('divideNumbers function', () => {

it('should divide two positive numbers correctly', () => {

expect(divideNumbers(10, 2)).toBe(5);

});

it('should divide a positive number by zero and return Infinity', () => {

expect(divideNumbers(10, 0)).toBe(Infinity);

});

it('should divide a negative number by a positive number correctly', () => {

expect(divideNumbers(-10, 2)).toBe(-5);

});

it('should divide zero by a positive number and return 0', () => {

expect(divideNumbers(0, 10)).toBe(0);

});

it('should return NaN if divided by zero', () => {

expect(divideNumbers(0, 0)).toBe(NaN);

});

it('should divide a large number by a small number correctly', () => {

expect(divideNumbers(1000000, 0.000001)).toBe(1e+12);

});

it('should handle decimal precision properly', () => {

expect(divideNumbers(1, 3)).toBeCloseTo(0.333, 3); // Rounded to 3 decimal places

});

// Add more test cases as needed for your specific requirements

});

With the right input, you can even get out BDD scenarios in Gherkin:

`Feature: Division of Two Numbers`

As a user

I want to divide two numbers

So that I can perform mathematical operations accurately

Scenario Outline: Divide two numbers

Given I have two numbers <numerator> and <denominator>

When I divide <numerator> by <denominator>

Then the result should be <result>

Examples:

| numerator | denominator | result |

| 6 | 3 | 2 |

| 10 | 2 | 5 |

| 0 | 5 | 0 |

| 8 | 4 | 2 |

| 15 | 5 | 3 |

Scenario: Divide by zero

Given I have a number

When I try to divide the number by zero

Then the operation should fail with an error message

Scenario Outline: Divide by a number close to zero

Given I have a number <numerator>

And I have a very small positive number <denominator>

When I divide <numerator> by <denominator>

Then the result should be positive infinity

Examples:

| numerator | denominator |

| 10 | 0.0001 |

| 100 | 0.00001 |

## Conclusionโ

While AI-generated tests can be a great tool for initial test coverage or catching basic errors, relying solely on them can lead us down the wrong path if we're not careful. The true power lies in combining AI's capabilities with human expertise. By guiding AI with clear requirements and leveraging its strengths in areas like data analysis and automation, we can create a more efficient and effective testing process.

This blog post focuses on the limitations of AI-generated tests for a specific example involving a simple mathematical function. The potential benefits and limitations of AI testing in more complex scenarios are vast and warrant further exploration. Additionally, it's important to acknowledge the potential for biases inherent in the training data used for AI models, which can impact the results generated.