Hey Nu.Be, it seems we are talking apples and oranges here.
If you are using any kind of DJ software it will have an option to record your set. This has nothing to do with the kind of control gear you are using.
If you are wanting to use gear WITHOUT a computer/laptop attached, you would need some kind of recording device, like the Tascam you borrowed.
The X1800 has a record out option. It is RCA (red/white) between the RCA master out and the Jack Booth outputs. This is an analogue output and should go into the RCA (red/white) on the left-hand (input) side of the back of the Tascam. For this you would use a standard RCA/RCA audio cable of good quality. We are not talking monster cable with gold contacts, but something a bit above the standard cables that come with home hifi gear.
The record output of the mixer will give you the same signal the master output gets, with the exception of master volume control. This means that you play your mixes all night long and you can use the master volume level to make it louder or less so over the PA, without changing the actual recording level. This way, at the end of recording, it will seem like the mix was made with the same overall level, regardless of where the master level setting was at any given time.
The discussion branched out into whether or not it would be possible to use the digital out from the mixer (directly on the other - right - side of the booth outputs and also an RCA, just a single one), If it is possible, you would connect this output with a DIGITAL audio cable with ONE RCA on either side (careful, this is not half of a regular RCA audio cable, but an actual other cable. Typically they will have yellow connectors instead of red/white.) and run it into the digital in of the Tascam. The confusion has risen because maybe the digital output of the mixer is only suited for linking two X1800s and not for pre-master stereo recording purposes.
Interesting as that discussion might be though, for your current and direct need, hooking up the mixer to the Tascam using the analog RCA audio cable option is quick and simple. Play a few tunes, open the channel faders, read the Tascam manual how to set input level (0dB is always your friend) and start recording.
Done.
Clearly I am slightly prejudiced towards Denon gear, but the “come back to Pioneer, because Rekordbox offers recording options doesn’t hold true”. Again, just my personal opinion and 2 cents worth.
Hope you will try the analog option with your Tascam and let us know if you got what you set out to get.