I'm in a bit of a rush right now so I can't go into detail or double check what I'm about to say but...
When an A/D converter puts out a digital signal, it doesn't start as 44.1 kHz or 48, 96, etc. I think (I could be wrong) but it starts out as a 1 bit file with an extremely high sample rate. The waveform is represented as accurately as possible in an extremely primitive form (1 bit = just 1s and 0s, on and off).
The A/D converter isn't done yet though. It uses a built in sample rate converter (the same thing you'll find in software, like r8brain) to send the audio to the computer at the desired sample rate.
The built in sample rate converter can potentially sound terrible.
If your built-in SRC algorithm truly sounded bad, then I promise you that it would impossible to improve it by running an additional SRC algorithm on top of it.
This is assuming I accept your theory about AD conversion requiring a built-in SRC algorithm, which I do not.
I do appreciate your post, though. I just don't agree with the assumptions you made going into your test.