The emergence of Generative Artificial Intelligence (GenAI) has revolutionised the landscape of content creation and natural language processing. It is common knowledge that GenAI is capable of automating tasks that previously required exhaustive human intervention, such as, for example, the writing of highly technical texts or very specific prior knowledge. In this context, we at Quanter asked ourselves: would it be possible to use AI to generate software project estimates from requirements written in natural language?
The challenge was clear. Natural language requirements, besides not having all the necessary detail, are often written differently by different people. This makes it very difficult to convert that ambiguous input into an accurate and methodologically valid estimate. To complicate matters further, AI, by its nature, is not designed to be deterministic, i.e. it does not always generate the same output for the same input. However, estimation methodologies, such as function points, require a rigorous and repeatable approach.
Initially, we explored the use of prompt engineering as a possible solution, but quickly discovered that it did not provide the consistent results we needed. We decided to train a model through fine-tuning. We opted for OpenAI’s ChatGPT, the most advanced tool on the market in this regard, which allowed us to fine-tune the model to obtain the expected accuracy.
Today, at Quanter, we have managed to develop a tool that synthesises multiple key features: natural language processing, multilingual capability, fast estimation based on standards such as ISO/IEC, and high efficiency in converting ambiguous requirements into market value estimates.
Evolution of AI in software project estimation
The evolution of artificial intelligence in the IT sector has followed an exciting course in recent years. From the first machine learning algorithms, designed for specific tasks such as data analysis and automation of repetitive processes, to the emergence of generative AI, the technology has been gaining ground in the field of project management and, more recently, in software estimation.
In the past, software project estimation relied heavily on human expertise and traditional tools. However, with the advent of generative AI, a new range of possibilities has opened up. Today, we can automate not only operational tasks, but also complex estimates based on text requirements, saving companies time and resources.
At Quanter, we have seen how artificial intelligence can dramatically improve accuracy and speed in estimating.
Challenges in incorporating generative AI into Quanter
Incorporating generative AI into Quanter has been a challenging process that we have approached with rigour. We knew that the key to success lay in choosing the right technology and tailoring it to our specific needs. We opted for OpenAI’s LLM model, ChatGPT, as it is the most advanced on the market, with a remarkable ability to handle natural language processing and multiple language understanding.
Training dataset
The next step was to create a set of examples that reflected real-life situations in which the AI had to work. This involved designing complex cases based on ill-structured project requirements with ambiguities, which represent a common challenge in the real world. Through fine-tuning the model, we were able to get the AI to not only understand these requirements, but to generate accurate estimates aligned with function point methodologies (ISO/IEC standard and recommended by the European Commission).
Data security
One of our major focuses during development has been data security to protect our clients’ sensitive information, ensuring that data is not shared with third parties and is kept within a controlled environment. This privacy protection is crucial in a business context where information security is a key asset.
Prompt hacking and auditing
Another challenge has been dealing with prompt hacking, a technique in which users attempt to manipulate model behaviour by intentionally designing ambiguous or malicious inputs. At Quanter, we conduct constant audits and have implemented measures to mitigate these practices. We want to ensure that AI is not exploited for inconsistent or inappropriate results, but remains within the parameters set for accurate estimation.
Deterministic AI
A critical aspect is that generative AI, by its very nature, tends to be creative rather than deterministic. However, software estimation methodology demands predictable and repeatable results. At Quanter, we have developed a system to verify each answer generated by AI, ensuring that it conforms to a deterministic standard, even when the model deals with vague or ambiguous requirements.
Multilingual capabilities
One of Quanter’s most outstanding features is its ability to work in multiple languages. Our tool allows users to write requirements in their native language, be it English, Spanish or Italian, for example. We have recently conducted successful tests in Japanese, which shows the potential to expand Quanter’s functionality to more international markets. Moreover, we continue to work on integrating other languages into our platform.
Hallucinations
One of the biggest challenges with generative AI models is the phenomenon of hallucinations, where the AI generates incorrect or out-of-context responses. To avoid this, we have tuned our model to produce less than 1% of hallucinations. This achievement has been possible thanks to a thorough fine-tuning process and the implementation of quality control measures on each response.
Roadmap for improving QuanterIA
The development of QuanterIA does not stop here. We have an ambitious roadmap that aims to bring generative artificial intelligence to all levels of software estimation.
Extended methodology
In the future, we plan to incorporate new estimation methodologies that complement the use of function points. We want to include the ability to identify non-functional impacts and other key aspects of project planning, which will give our users an even more complete and robust view of the work to be done.
Intelligent Wizard
We are developing a GPT-based wizard that will allow users to query information on methodologies and standards applicable within their organisation. This conversational wizard will help resolve queries in real time and provide best practice recommendations for estimating and project management.
Improving the quality of requirements
A key part of our future focus is to improve the quality of requirements written by users. Our AI model will be able to generate a clear and complete version of the requirements, displaying quality indicators that will alert users to possible inconsistencies or ambiguities. This will not only make the estimation more accurate, but will also help teams deliver higher quality products.
Test case generation
Another line we are developing is the generation of the test cases necessary to test the functionality described in the user requirements. Thus, we would not only have an estimate of cost and effort, but also the set of tests needed to validate such functionality. This capability will allow development teams to save valuable time in the testing process, providing a more efficient flow from planning to implementation.
The future of AI estimation
Looking ahead, we believe artificial intelligence will continue to evolve to transform software project estimation. As generative AI models are refined, they will be able to process even more complex requirements, adapt to new methodologies and deliver increasingly accurate results. At Quanter, we are committed to being at the forefront of this transformation, offering our users a powerful, secure and efficient tool that optimises software project estimation, now and in the future.
