Custom checkers

Some exercises may have multiple correct outputs, so sometimes we need to write a checker to check whether the output is correct or not.

1. Default Checker

OJ supports DMOJ's default checkers, you can see them here

2. Custom Checker (Python)

A checker written in Python needs to implement a function like this:

def check(process_output, judge_output, **kwargs):
# your code

In there:

  • process_output: output of the submission
  • judge_output: answer

**kwargs can contain the following variables:

  • submission_source: source code of the submission
  • judge_input: input of the question
  • point_value: the score of the test being graded
  • case_position: order of tests
  • batch: batch of test cases (with tests graded by subtask), in most tests we don't need to care about this variable.
  • submission_language: language of the submission
  • execution_time: execution time, in seconds

Return

The check function has 2 ways to return:

  1. Return True or False, if the answer is correct then return True, otherwise return False
  2. Return with CheckerResult, returning this way can return feedback to the submitter. Usage is as follows: CheckerResult(passed, points_awarded). In which:
  3. passed has a True/False value depending on whether the result is correct or not.
  4. points_awarded: is the number of points received for that test.
  5. feedback: information given to the submitter.

For example

Below is a sample checker for the problem POST - A plus B.

from dmoj.result import CheckerResult

def check(process_output, judge_output, judge_input, point_value, **kwargs):
# convert from bytes to text
process_output = process_output.decode("ascii")
judge_output = judge_output.decode("ascii")
judge_input = judge_input.decode("ascii")

# read data as normal
a, b = judge_input.split(' ')
output_sum = int(process_output)
if a + b != output_sum:
return CheckerResult(False, 0, f"{a} + {b} != {output_sum}")
return CheckerResult(True, point_value, "Ok answer is correct")

3. Custom Checker (C++)

To write a C++ checker, we need to write a C++ program that accepts 3 command line parameters in order input_file, output_file, answer_file, corresponding to the paths to the input, output, and answer files.

Return

The program (main function) returns the following values:

  • 0 if AC (100% of score)
  • 1 if WA (0 points)
  • 7 if a partial score is received. Then print to stderr a real number in the range [0, 1] representing the score ratio. Note that the score must be printed in the first line of stderr
  • If the program returns a different value (exit code other than 0, or run time error), the submission will receive 0 points (considered WA).

The information written to stdout will be printed to the screen for feedback, but this information can only display about 20 characters. If you want the user to see more feedback, print the feedback to stderr (note that it should be printed after the score on the first line).

For example:

The following program is used to grade the problem: Let ~n~ be a positive integer. Print out two natural numbers ~a~, ~b~ such that ~a + b = n~.

If you print out ~a + b = n~ and ~a, b \ge 0~ then you get ~100\%~ of the score, if ~a + b = n~ but one of the two numbers ~a, b~ is negative then you get ~50\%~ of the score.

#include <bits/stdc++.h>
using namespace std;

int main(int argc, char** argv) {
ifstream inp(argv[1]);
ifstream out(argv[2]);
ifstream ans(argv[3]);

int n, a, b, c, d;

    inp >> n;
    out >> a >> b;
    ans >> c >> d;

    if (a + b == c + d) {
        cout << a << " + " << b << " = " << c << " + " << d << endl;

        if (a >= 0 && b >= 0) {
            cerr << "This line will be show to the contestants as a extended feedback";
            return 0; // AC
        }
        else {
            cerr << 0.5 << '\n'; // in ra điểm ở dòng đầu tiên
            cerr << "Thís line will be show to the contestants as a extended feedback";
return 7; // PARTIAL
}
}
else {
cout << "a + b = " << a + b << " != " << n << endl;
return 1; // WA
}
}
```Currently only non-interactive papers can be graded.


OJ also supports checkers written in [testlib.h](https://github.com/MikeMirzayanov/testlib/blob/master/testlib.h), but can currently only grade non-interactive papers.

Here is a checker written in `testlib.h` for the above problem:

```c++
#include "testlib.h"

using namespace std;

int main(int argc, char* argv[]) {
    registerTestlibCmd(argc, argv);

    int n = inf.readInt();
    int a = ouf.readInt();
    int b = ouf.readInt();
    ensuref(a + b == n, "%d + %d != %d", a, b, n);
    if (a < 0) {
        quitf(_points, "0.5 \n a = %d < 0", a);
    }
    if (b < 0) {
        quitf(_points, "0.5 \n b = %d < 0", a);
    }
    quitf(_ok, "%d + %d = %d", a, b, n);
}