To begin with, it has been developed in ~1998 for Amiga, way before accuracy in emulators became important and even possible.
The main flaw of the format, though, is that it does not actually consider ZX Spectrum enviroment, i.e. there is no BASIC ROM in the emulated memory space by default, and most of beeper music players rely on having it in place. If you don't take it into account while ripping/converting music, it may easily get the drum sounds changed, or lost them althogether. Or, in worst case, all timings will be screwed up (Special FX/Fuzz Click engine has this issue), or code won't work at all (ROMBeep is an example). To make it work correctly, you need to include portions of the ROM into AY file, so you need to determine which portions actually used, which is not so easy, as to do this you need to understand how an engine works. Alternatively, to save your time, you can include whole ROM (in fact, if I remember correctly, it is not even possible, first two bytes will be missing anyway), and basically redistribute a copy of it with every AY file.
All this could easily be avoided would the format specs were explictly saying that memory space should have ROM pre-loaded (distrubuted with players, not with music).
There are other things that could be improved and cleaned up as well.