Results of ACE to Arnold test suite
The test machines
How to read this ?
- In the first column
- In the column "Results with ACE"
The facts!
Bonus

Results of ACE to Arnold test suite

These results are related to the great test suite created by Kevin Thacker for Arnold.

The tests are not enough to ensure a perfect emulation, but they give an idea of the overall quality of the emulation.

The good news is that ACE is performing perfectly with most of the tests, and the failing ones are generally related to secondary features never used in CPC softwares.

Anyway, all the tests should pass 100% at the end!

The test machines

Here is the list of the real hardwares used to run the tests.

Type	Description	Comments
0	Amstrad CPC6128 with CRTC HD6845SP¹⁾	French firmware (f3)
1	Amstrad CPC6128 with CRTC UM6845R²⁾ Amstrad CPC6128 with CRTC UM6845R³⁾	French firmware (f3)⁴⁾ English firmware (v3)
2	Amstrad CPC6128 with CRTC MC6845P⁵⁾ Amstrad CPC664 with CRTC MC6845P⁶⁾	English firmware (v3) English firmware (v2)
3	Amstrad 6128plus revision A Amstrad 6128plus revision D Amstrad 6128plus revision G	French firmware (f4)⁷⁾ French firmware (f4) French firmware (f4)
4	Amstrad CPC6128 version O	French firmware (f3)

How to read this ?

Each test was run on the real hardware and then on ACE for each CRTC type. Each line is reporting the results for one test, and the columns are giving information about these results.

In the first column

This is the global status for the test.

”” means that the test makes no difference between ACE and the real hardwares.
”” means that the test is reporting differences which are not critical for emulation quality. It should be fixed but it is not a priority unless a known software requires it.
”” means that the test is reporting errors that shall be fixed in ACE. Some softwares might already be impacted by this flaw.

In the column "Results with ACE"

The sub-columns are for the well-known 5 CRTC types.

For both “on screen report” and “interactive” tests:

“” means that the test is totally successful.
”” means that the test partially failed.
“” means that the test totally failed.
an empty column means that the test was not perfomed yet.

For “visual” tests:

if a grab is present, it fully matches CPC display and the test is totally succesful.
”” means that the test partially failed.
“” means that the test totally failed.
an empty columns means that the test was not perfomed yet.

Please note that the test suite is not perfect yet. Some “on screen report” test are failing on real CPC, and are then marked successful on ACE when they are failing exactly in the same manner. Also, some “visual” test are not totally stable on real CPC. Refer to the additional notes on the test to understand the details.

The facts!

Test name	Results with ACE 1.16					Kind of result	Comments
	0	1	2	3	4
8kscreen						Visual (static)
brunword	⁸⁾	⁹⁾	¹⁰⁾	¹¹⁾		On screen report	Failing in the same manner on CPCs and ACE, which is good.
clr						Visual (color is cycling)
clrbit5						Visual (color cycling at startup)
colours						Visual (static)
cpcborder						Visual (static)
cpccol						Visual (color cycling)
cpcpen						Visual (static)
cpctest	¹²⁾	¹³⁾	¹⁴⁾	¹⁵⁾		On screen report	Some tests are failing in the same manner on CPCs and ACE, which is good.
cpu	¹⁶⁾	¹⁷⁾	¹⁸⁾	¹⁹⁾		On screen report	Only one test is failing. This is a known bug on Z80 CMOS where IFF2 is copied into P/V flag before being updated when an interrupt occurs during LD A,R or LD A,I. This Z80 bug is not emulated by ACE and won't be emulated anytime soon (totally useless and over complicated to implement in a efficient way).
gaint						Visual (static)
crtc0_3_4_r8						Visual (static)
crtc1_r6						Visual (static)
crtc3_4_status	²⁰⁾	²¹⁾	²²⁾	²³⁾		On screen report	Some tests are failing in the same manner on CPCs and ACE, which is good. On CRTC 3, cursor blinking and cursor start/End are not emulated at all (then, status cannot be reported on Reg3). It might me added in the future, but since it is totally useless it is a top low priority.
crtcinp						Visual (static)
crtctest	²⁴⁾	²⁵⁾	²⁶⁾	²⁷⁾		On screen report	Some tests are failing in the same manner on CPCs and ACE. Basically, everything is working apart of some stuff related to reg8. Please note that test code needs to be fixed to work as expected (LD A,'3' should be removed at CRTC selection: poke &400d,asc(”?”)). On real CPC+ this test is triggering the memory refresh issue; some parts of the (display) memory is destroyed during tests. CRTC register 8 is not fully implemented yet in ACE, the related tests should be fixed when it is done. Also, R8 - count lines sometimes fails on real CPC with CRTC 1 too!
crtcstatus		²⁸⁾				On screen report
crtc_r1						Visual (static)
crtc_r12						Interactive	On CRTC 1 the changing area is &6E-&6A and not &6E-&69 as stated in the test. On CRTC 3 the changing area is &70-&6C and not &72-&6C as stated in the test.
crtc_r12_r5						Interactive
crtc_r12b						Interactive
crtc_r13						Interactive
crtc_r1_2						Visual (static)
crtc_r4		²⁹⁾				Interactive	On CRTC 3 the changing area is &B0-&AC and not &B1-&AA as stated in the test.
crtc_r4b						Interactive
crtc_r5						Visual (static)
crtc_r5b						Interactive	Sick mode of CRTC 1 is triggered at &5EA and not &5EC as stated in the comments of the test
crtc_r5c						Interactive
crtc_r5d	³⁰⁾					Interactive
crtc_r5e						Visual (static)
crtc_r5e2		³¹⁾				Interactive
crtc_r5ep						Visual (static)
crtc_r5f		³²⁾				Interactive	On CRTC 1 changed are triggered at &285-&281 and not &286-&281 as stated in the comments of the test
crtc_r6						Visual (static with some part of lines blinking)	This test is not stable on CRTC 1.
crtc_r6b						Interactive
crtc_r6c		³³⁾				Interactive
crtc_r6d		³⁴⁾				Interactive
crtc_r7						Interactive	This is not mentionned in the test comments but VSync alignement regarding display words is changing between ranges &2AE-&2AA and &2AB-…) on CRTC 1
crtc_r9	³⁵⁾	³⁶⁾		³⁷⁾		Interactive
crtc_r9b	³⁸⁾	³⁹⁾				Interactive	On CRTC 3 the changing area is &90-&8C and not &91-&8C as stated in the test.
dkram						On screen report	DK'Tronics bugs emulation of ACE should be activated.
flood						Visual (static)	Small issue on CRTC 2… well… not a big deal. To be fixed one day or another.
flood2						Visual (static)	Small issue on CRTC 2…
ppi						On screen report	In fact the test works with no issue in PPI mode 0 (the only one usefull on CPC) but mode 1 and 2 are not emulated by ACE. Since these modes are totally useless it is not a priority by now to add them. Also, some tests are failing in a similar way on both the real hardware and ACE. In addition, I noticed that on some CPC the VBL can be triggered from the PPI (at least with the bizarre Gate Array). I guess also there is a bug in one of the tests: the Hi-Z test on PSG register write does not do what is expected (it selects register FF for reading).
ppiaudio						Audio and visual (border color cycling)
psg						On screen report
psgexer						Audio feedback
type1ctrl						On screen report	Some tests are failed in a similar way on both the real hardware and ACE.
rtestopcodes				⁴⁰⁾		On screen report	IM 0 emulation on CPC+ is missing; but it is actually totally useless and not a priority.
inout						On screen report
modetrig						Interactive
vblank						Visual (static)
dmatest	N/A	N/A	N/A		N/A	On screen report	Two tests are failing. The only one that might have some impact is “dma int request test (dcsr bits)” which is triggering an issue when expecting bit 1 from DCSR to be reset at some point. The second failing test is “CRTC R0 length and dma” for one single value (when R0=1) which is very unlikely to happen in real life.
dmatiming	N/A	N/A	N/A		N/A	Visual (static)
asic_test	N/A	N/A	N/A		N/A	On screen report
asic_raster	N/A	N/A	N/A		N/A	Visual (static)

More tests to come…

Bonus

Here is a small AREXX script to compile, load and run a test from Arnold Test Suite into ACE.

Usage:

RunTest.rexx <testname>

Script:

/* RunTest.rexx - Compile and run Kevin Thacker Arnold Test Suite
 *
 * Syntax: RunTest.rexx <test name>
 *
 * $VER: RunTest.rexx 1.3 (28.02.18) © 2016-2018 by Philippe 'OffseT' Rimauro <offset@cpcscene.net>
 * Requires ACE 1.13 or later
 *          Pasmo Z80 assembler (must be in the path)
 */

OPTIONS RESULTS
SIGNAL ON BREAK_C

PARSE ARG test".asm"
IF test="" THEN DO
    PARSE ARG test
END

IF test="" THEN DO
    CALL FatalError('No test name specified!','My mistake!')
    EXIT
END

/* External libraries */
LoadLib("rexxsupport.library")

appname = "ACE Rexx"

/* Look for file test */
IF ~EXISTS(test".asm") THEN DO
    CALL FatalError('Test 'test' not found!','Gasp!')
    EXIT
END

/* Look for ACE running */
IF ~SHOW('P','ACE.1') THEN DO
    CALL FatalError('ACE CPC Emulator process not found!','Oops!')
    EXIT
END

/*
 * Main
 */
ADDRESS "ACE.1"

/* Display ACE version */
"VERSION"
SAY "ACE version" RESULT "found."

/* Reset ACE */
SAY "Cleaning up everything..."
"BREAKPOINTS ACTIVATE 0"
"START"
"RESET HARD"

/* Let the firmware boot */
DELAY(100)

/* Pause ACE while working */
"PAUSE"

/* Assemble code */
SAY "Assembling source code..."
ADDRESS "COMMAND" "pasmo --amsdos "test".asm T:"test".bin T:"test".symbol"

/* Check for assembling error... */
IF RC~=0 THEN DO
    "START"
    EXIT
END

/* Load code into ACE! */
SAY "Loading code into ACE..."
"LOAD FILE T:"test".bin "

/* Start ACE again */
"BREAKPOINTS ACTIVATE 1"
"START"

/* Auto-launch the assembled program */
SAY "Auto-launching the assembled program!"
"KEYBOARDSTROKE CALL 256"

EXIT

/*
 * External lib loader
 */
LoadLib:
    PARSE ARG lib

    IF SHOW("L", lib)=0 THEN
        IF ADDLIB(lib, 0, -30, 0)=0 THEN
            CALL FatalError(lib' not found!*NPlease install it.','Okay!')
    RETURN RC

/*
 * Fatal error
 */
FatalError:
    PARSE ARG message,button

    ADDRESS "COMMAND" 'RequestChoice >NIL: TITLE "'||appname||'" BODY "'||message||'" GADGETS "'||button||'"'
    EXIT

¹⁾ , ³⁾ , ⁵⁾ , ⁶⁾ Gate Array 40010

²⁾ bizarre Gate Array (40010 compatible)

⁴⁾ Hey! That's the one that printed Quasar CPC pages!

⁷⁾ Hey! That's the one on which most of Quasar CPC pages were created!

⁸⁾ , ⁹⁾ , ¹⁰⁾ , ¹¹⁾ 8-)

rom test &4000 en/dis-FAILED (got 33 expected 0D)

¹²⁾ , ¹³⁾ 8-)

CPC int (HSYNC width)-FAILED (got 00 expected 01)

¹⁴⁾

Deadlock due to reg2=0 specific case not emulated)

¹⁵⁾

CPC int (HSYNC pos)-FAILED (got 00 expected 01) / 8-)

CPC int (HSYNC width)-FAILED (got 00 expected 01)

¹⁶⁾ , ¹⁷⁾ , ¹⁸⁾ , ¹⁹⁾ 8-)

ld a,r/ld a,i p/v int test-FAILED (got 04 expected 00 - FAIL)

²⁰⁾ , ²²⁾ 8-)

RC==0 and RC==MR-FAILED (got 01 expected 00) / 8-)

VBlank-FAILED (got 40 expected 00) / 8-)

Vsync end-FAILED (got 01 expected 00) / 8-)

Cursor flash rate-FAILED (got 00 expected 10) / 8-)

Cursor active on scanline-FAILED (got 01 expected 00)

²¹⁾

RC==0 and RC==MR-FAILED (got 01 expected 00) / 8-)

VBlank-FAILED (got 00 expected 40) / 8-)

Vsync end-FAILED (got 01 expected 00) / 8-)

Cursor flash rate-FAILED (got 00 expected 10) / 8-)

Cursor active on scanline-FAILED (got 01 expected 00)

²³⁾

VBlank-FAILED (got 40 expected 00) / :-(

Vsync end-SUCCEEDED instead of FAILED (got 01 expected 00) /

Cursor flash rate-FAILED (got 00 expected 10) instead of SUCCEEDED /

Cursor active on scanline-FAILED (got 00 - OK) instead of (got 01 expected 00)

²⁴⁾

vsync r8 test (r8=1)-FAILED (got 00 expected 01) /

Deadlock: vsync r8 test (r8=3) / 8-)

vsync r8 htot/2 test-FAILED (got 01 expected FF) /

Deadlock: R8 - count lines

²⁵⁾

vsync r8 test (r8=1)-FAILED (got 00 expected 01) / 8-)

vsync r8 test (r8=3)-FAILED (got 00 expected 01) / 8-)

vsync r8 htot/2 test-FAILED (got 01 expected FF) /

R8 - count lines-FAILED (got 9B expected 99) instead SUCCEEDED

²⁶⁾

vsync r8 test (r8=1)-FAILED (got 00 expected 01) / 8-)

vsync r8 test (r8=3)-FAILED (got 00 expected 01) / 8-)

vsync r8 htot/2 test-FAILED (got 01 expected FF) /

Deadlock: R8 - count lines / 8-)

vsync length test-FAILED (got 10 expected 01) /

hsync pos test-FAILED (got 01 expected 00) instead of SUCCEEDED (only a few value are failing, probably related to wrong CRTC 2 emulation when R2=0) /

hsync length measure (type 2)-FAILED (got FF expected 2F) instead of SUCCEEDED (Probably related to wrong CRTC 2 emulation when R2=0 too)

²⁷⁾

vsync r8 test (r8=1)-FAILED (got 00 expected 01) / 8-)

vsync r8 test (r8=3)-FAILED (got 00 expected 01) / 8-)

vsync r8 htot/2 test-FAILED (got 01 expected FF) / 8-)

R8 - count lines-FAILED (got xx expected yy)

²⁸⁾ Almost ok, just a few values are shifted

²⁹⁾ Ok, except &AF where “sick mode” is not emulated

³⁰⁾ , ³¹⁾ , ³²⁾ , ³³⁾ , ³⁴⁾ , ³⁶⁾ , ³⁹⁾ Ok, except one area triggering which is shifted by 1 microsecond

³⁵⁾ , ³⁸⁾ Ok, except one area triggering which is shifted by 1 microsecond, and the flikering value which is not flickering

³⁷⁾ Ok, except the extra line of the second changing area which is not displayed properly

⁴⁰⁾ IM 0 not emulated