What Grade is BERT in? Seq2Seq ``Directly Tackling'' Primary School Math Word Problems

Data Processing

First, let’s observe the format of the ape210k dataset:

{
    "id": "254761",
    "segmented_text": "Xiao Wang wants to dilute 150 kg of pesticide with a 20% drug content into a pesticide with a 5% drug content. How many kilograms of water need to be added?",
    "original_text": "Xiao Wang wants to dilute 150 kg of pesticide with a 20% drug content into a pesticide with a 5% drug content. How many kilograms of water need to be added?",
    "ans": "450",
    "equation": "x=150*20%/5%-150"
}

{
    "id": "325488",
    "segmented_text": "The radius of a circular flower bed is 4 meters. Now the flower bed is to be expanded, increasing the radius by 1 meter. How many square meters has the area of the flower bed increased by?",
    "original_text": "The radius of a circular flower bed is 4 meters. Now the flower bed is to be expanded, increasing the radius by 1 meter. How many square meters has the area of the flower bed increased by?",
    "ans": "28.26",
    "equation": "x=(3.14*(4+1)**2)-(3.14*4**2)"
}

As we can see, we are primarily interested in the original_text, equation, and ans fields. The original_text is the problem description, the equation is the calculation process (usually starting with x=), and ans is the final answer. We hope to train a model that generates the equation from the original_text, which can then be evaluated using Python’s eval() function to obtain the ans.

However, we need to perform some preprocessing because the equation provided by ape210k cannot always be evaluated directly. For example, in the case above, 150*20%/5%-150 is an invalid expression for Python. The processing I performed is as follows:

1. For percentages like a%, uniformly replace them with (a/100);

2. For mixed fractions like a(b/c), uniformly replace them with (a+b/c);

3. For proper fractions like (a/b), remove the parentheses in the problem text to become a/b;

4. For the ratio colon :, uniformly replace it with /.

After this processing, most equations can be evaluated directly, and the results can be compared with ans to retain only the problems with consistent results. Additionally, there is room for improvement: the resulting expressions might contain redundant parentheses (i.e., removing them doesn’t change the value). Therefore, I added a step to remove parentheses by iterating through each pair; if removing them yields the same result, they are discarded. This results in shorter average expression lengths, and shorter sequences are generally easier to generate.

Ultimately, we obtained the following usable dataset:

The remaining items are mostly incorrect or garbled problems, which are ignored for now.

Model Introduction

The model itself is straightforward: it takes original_text as input and equation as output, using “BERT+UniLM” as the basic architecture to train a Seq2Seq model. If you have any doubts about the model, please read “From Language Models to Seq2Seq: Transformer is All About Masking”.

Project Link: http://github.com/bojone/ape210k_baseline

My training was conducted on a single 22G TITAN RTX card. The optimizer used was Adam with a learning rate of 2 \times 10^{-5}. The Base version used a batch_size of 32 and required about 25 epochs, with each epoch taking approximately 50 minutes (including evaluation time on the validation set). The Large version used a batch_size of 16 and required about 15 epochs, with each epoch taking approximately 2 hours.

Regarding the Large version, since UniLM utilizes the weights of the MLM (Masked Language Model) part, we cannot use the RoBERTa-wwm-ext-large released by HFL, as the MLM weights in that version are randomly initialized (though its Base version is normal and usable). For the Large version, I recommend using the weights released by Tencent UER. The original weights are in PyTorch format; I have converted them to TensorFlow format, which can be downloaded here (Extraction code: l0k6).

The results are shown in the table below:

The results of the Large model are significantly higher than the 70.20% reported in the ape210k paper “Ape210K: A Large-Scale and Template-Rich Dataset of Math Word Problems”, indicating that our model is a decent baseline. I suspect that using Seq2Seq techniques to mitigate the Exposure Bias problem (refer to “Analysis and Countermeasures of Exposure Bias in Seq2Seq”) could further improve the model. Additionally, introducing a copy mechanism might enhance the consistency between input and output numbers. One could also try to further shorten the sequence length (e.g., replacing the four-character 3.14 with the two-letter pi). These are left for everyone to try!

Standardized Output

From a purely modeling perspective, our task is complete: the model only needs to output the expression, and evaluation only requires checking if the eval() result matches the reference answer. However, from a practical application standpoint, we need to further standardize the output. That is, we must decide whether the output should be a decimal, integer, fraction, or percentage based on the problem. This requires us to: 1. Decide when to output which format; 2. Convert the result according to the specified format.

The first step is relatively simple. Generally, one can judge based on keywords in the problem or the equation. For example, if there are decimals in the expression, the output is usually a decimal. If the problem asks “how many cars,” “how many items,” or “how many people,” the output should be an integer. If it asks for a “fraction” or “percentage,” the output should be formatted accordingly. The more difficult part involves rounding problems, such as: “Each box of cake costs 7.90 yuan. How many boxes of cake can 50 yuan buy at most?” This requires us to floor the result of 50/7.90, but sometimes ceiling is required. Surprisingly, there are no rounding problems in ape210k, so this issue does not exist there. If encountered in other datasets where rule-based judgment is difficult, the most direct method is to include rounding symbols in the equation for the model to predict.

The second step might seem complex, especially for fractions. Average readers might not know how to keep the fractional result of an expression. If you directly run eval(’(1+2)/4’), you get 0.75 (in Python 3), but sometimes we want the fractional result 3/4. In fact, maintaining fractional calculations belongs to the domain of CAS (Computer Algebra System), which performs symbolic rather than numerical computation. Python has a tool for this called SymPy. Using SymPy, we can achieve our goal. See the following example:

from sympy import Integer
import re

r = (Integer(1) + Integer(2)) / Integer(4)
print(r)  # Output is 3/4 instead of 0.75

equation = '(1+2)/4'
print(eval(equation))  # Output is 0.75

new_equation = re.sub('(\d+)', 'Integer(\\1)', equation)
print(new_equation)  # Output is (Integer(1)+Integer(2))/Integer(4)
print(eval(new_equation))  # Output is 3/4

Summary

This article introduced a baseline for solving math word problems using a Seq2Seq model. The main idea is to directly convert problems into evaluatable expressions using “BERT+UniLM,” followed by sharing experiences on result standardization. Using the UniLM based on BERT Large, we achieved an accuracy of 75%, surpassing the results released in the original paper.

So, what grade do you think it can reach now?

Please include the original link when reposting: https://kexue.fm/archives/7809

For more details on reposting, please refer to: Scientific Space FAQ