Topic: MDAL Dev Thread

I guess this isn't very interesting for most people, but since I'm currently putting a lot of time into the MDAL project, I thought  I'd post some updates about that here once in a while.


Backstory

Last year saw the specification of the MDAL standard (standards, actually) and the development of libmdal, which eventually became the back end for bintracker. While the concept itself was reasonably successful, the actual implementation - surprise surprise - turned out to be too limited to support pretty much anything other than the engines that are currently available in bintracker. In fact, the current MDAL can't even support a simple engine like Huby. So, a redesign is needed.


New Standards

In the past months, I've been drafting new standards for MDAL configurations (MDCONF) and the actual MDAL module language (MDMOD). These drafts have now progressed to a state where I can share some details. First up, here are some links:

MDMOD v2 Specification Draft
MDCONF v2 Specification Draft

Obviously nothing is set in stone yet. However, I believe that the MDMOD specification is already quite mature. On the surface it still looks pretty similar to MDMOD v0, but there are a few major breaking changes. These were necessary to facilitate some of the more complex features provided by MDCONF v2. MDMOD v2 does add some boilerplate, but it also adds some syntactic sugar. Overall, it should still be quite human readable/editable even though the focus in v2 will be on machine processing.

The most notable change is that typing and scoping are no longer implicit. In MDMOD v0 you could write something like

:pattern1
  NOTE=a4, FX=fxtable1
:fxtable1
  FXCMD=val
  ...

and MDAL would automatically infer that "pattern1" is a block of type pattern and fxtable is a block of type "fx". In MDMOD v2, you will need to write this as

:PATTERNS(1)[pattern1] {
  NOTE=a4, FX=1
}
:FX(1)[fxtable1] {
  FXCMD=val
  ...
}

The stuff in angular brackets is just a name and can be omitted. The good news is that, assuming the block type PATTERNS only provides NOTE and FX fields, you would also be able to write pattern1 like this:

:PATTERNS(1) {
  a4, 1
}

meaning the field names can be omitted. You still need to specify the field name if you only update some of the fields on the given step, though. Also, there's a new short hand for empty steps. This

  .
  .
  .
  .

becomes this:

  .4

Another big news is that sequences (now called ORDER) are now matrix sequences like for example in Famitracker. Garvalf really wanted this I think... and he was absolutely right that this is a good idea. Sequences are now entirely virtual, so in bintracker you will still be able to use the classic "glob" style (and in addition, the new matrix style or automatic sequences like in 1tracker). Also, syntax for sequences and blocks will be exactly the same in MDMOD v2. So, assuming we have a config that specifies something with 3 channels, you can write the sequence as

:ORDER {    // no need to specify the instance if there is only one
  $00, $01, $02
  $00, $01, $02
  $00, $03, $04
}

or you could write

  $00, $01, $02
  .
  CH2 = $03, CH3 = $04

MDCONF v2, on the other hand, is very much a departure from v1. There is a much higher level of abstraction, with the aim to seperate the input from the output configuration as much as possible. I'm still very much in the process of working out the details, but it's starting to take shape, at least.

One of the most exiting news is that with MDCONF v2, you can define a recursive data structure. This will make it possible to support things like SID subtunes, or the SEQUENCE-CHAIN-PATTERN approach used by LSDJ and Slocum's Music Kit for the Atari 2600. Generally, the approach for v2 is to define input and output Groups (which are an entirely abstract construct), which contain any number of Fields (still derived from Commands like in v1), Blocks (patterns, tables, etc), and, most importantly, other Groups. Anyway, don't want to go too much into detail, as this is of course all super dry and theoretical. If you are really interested, read the specifiaction draft.


Where to go from here

I'm now a couple of months into designing a new libmdal to work with the v2 standards. The new lib is meant to be a fully-fledged tracker backend, so in theory it could be used to build trackers other than bintracker (though I doubt anyone will do it). I'm afraid there's still a few more months worth of work ahead of me, so don't expect to see the results any time soon. Once it's in a working state, I'm intending to write a dedicated GUI library to work with libmdal. Together with a few salvaged bits from the current bintracker, these will eventually become the new, all-powerful bintracker. Yay!

In any case, as mentioned earlier, nothing is set in stone yet. If you have any ideas, suggestions, questions, I'm more than happy to hear them.

Re: MDAL Dev Thread

Great news! The new libmdal compiler produced it's first correct output today. Still cheating a bit by hard-coding a few parameters, but nevertheless it's a major step forward. Testing with Huby at the moment which may not sound all that exiting. However, Huby has some not-so-common quirks so it's good for testing a number of different features (fixed pattern size so MDAL source patterns must be resized, size of order list must be stored in data, which requires multiple compiler passes, and so on).

Still a lot of work to do, but for now I'm very happy smile

Re: MDAL Dev Thread

More progress. Got rid of most of the hard-coded compiler parts, so the compiler generator is getting there slowly. (Yes, you read correctly: the new libmdal generates a custom compiler for each target engine, rather than using one big standard compiler with loads of conditionals like in libmdal v1).

Also, I was able to implement a new, much more robust parser using comparse, thanks to its friendly author who gave me an in-depth tutorial last weekend.

Last but not least I did some much needed refactoring, finally breaking up that horrible 3000 lines-of-code core module into various submodules.

Unfortunately in the coming weeks I'll have to work on another, unrelated project, so development will be stalled again until at least the end of the month.

Re: MDAL Dev Thread

Unexpectedly got some more free time before that other project I mentioned. So here's a little sneak preview:

https://i.ibb.co/CPVzhg9/btng001.png

Re: MDAL Dev Thread

Another update with good news! I reimplemented the compiler generator. It's still a bit rough around the edges, but it's already much more robust than the previous version, and does not require any hard-coded work-arounds anymore. The latest incarnation of the MDCONF Standard replaces the XML-based format with an s-expression based one, which is much more compact. Needless to say it also allows me to drop all XML related dependencies (though ssax/sxml is a breeze to work with, generally). Also, MDAL now has a built-in multi-target assembler (though it only supports z80 atm), which means there's no need to provide pre-compiled binaries anymore.

Next step is to work some more on Bintracker again, so I get a better idea what kind of things MDAL needs to provide on the tracker API front.