SegaXtreme

Home	Forums	What's new	Resources

New Sega Genesis Music Format

cgfm2 - Aug 22, 2005

		cgfm2		Aug 22, 2005
			A few years ago I worked on a format for storing ripped Sega Genesis music. It worked, and I ripped around 30 games to get an idea of the different types of music drivers used. I never quite finished the project as writing the user interface always held me up. But there's definitely a need to get this completed, as GYM (being good for what it was) isn't too ideal. The main problem with GYM is that storing DAC samples eats up huge amounts of file space. By using ripped code and data you only have to store the minimum amount of data (typically a few hundered kilobytes at the most, and then you have all music and SFX in one file) So I'd like to discuss this project and get input from other people about how to continue. A large number of games work like this, with three main subroutines to control sound output: - init: 68000 copies Z80 driver to sound RAM and resets Z80 - vblank (or 'update' if you prefer): 68000 does nothing (Z80 runs off internal timers or vblank) - request: 68000 writes sound code into sound RAM which the Z80 checks eventually and plays new music or sfx To support this kind of music driver, all you have to do is locate and rip the init/request routines, and then the ROM data blocks used by the Z80 (typically for DAC samples). It is trivial to modify an emulator to report sound RAM access and bank selection to figure these parts out. The less common types are: - 68000 and Z80 work in tandem. E.g. Gauntlet has the 68000 run the PSG while the Z80 does everything else. Treasure games let the Z80 do DAC samples and the 68000 handles FM+PSG sound. - 68000 does everything, Z80 idles. For these you need to rip the 68000 music driver which is difficult but not impossible. A lot of games conveniently bundle the 68K, Z80 and music data in the same block of ROM so it's easy to rip out a chunk and have all the important bits. Some games like Thunder Force IV have a lot of interaction where the 68000 dynamically copies sections of code and RAM out of the Z80 memory space, so this required a lot of the original game logic to be analyzed and reproduced, because it was tightly integrated with the game code. I think this is about the only way such games can be handled. Now, on to playing back the ripped bits and pieces: The original player I developed fit in a few kilobytes of a Genesis ROM that had joystick control to change a sound number. It would call the 'init' subroutine, the 'update' subroutine on each VBlank if necessary, and the 'request' subroutine when a new sound number was needed. For most games I could call the original subroutines directly, or make some minor modifications. The original ROM data was loaded into ROM space at the same addresses the game used (e.g. if a bank of DAC samples were at $70000, that's where they went into the player ROM) This is because the 68000 code cannot be easily relocated (the init or request ones can, but a full 68000 driver cannot) and the Z80 code would have to be changed to specify relocated ROM bank addresses that are fed to the bank selection shift register. In a few cases my player code had to be moved around to accomodate the original game's used locations. I'll discuss a proposed file format next. Because the ripping process basically involves a lot of hex editing, 68000 coding, and sorting through emulator output logs, you end up with various odds and ends that need to be combined into a single file. I think the best way to handle this is to have the user create a plain text file divided into sections that contains this information, which a tool would compile into a binary file. Something like this: (sonic.txt) [game] rom = "/roms/sonic.bin" // where the source ROM is that the tool rips data blocks from vmode = NTSC; // tell the player what the refresh rate is [interface] // tells the player where the functions are to interact with the music driver init = 0x0abcd; // address of init subroutine request = 0x00abcd; // address of sound request subroutine update = 0x00abcd; // address of vblank update routine, not always used [data] // list of offsets and sizes of data blocks to extract 0x70000, 0x8000 // data block, 32K from $70000 0x10000, 0x10000 // another data block, 64K from $10000 [info] // textual information about the game that somebody listening to the music would want to see composer = "foobar" copyright = "1992 Sega" [tracklist] // each entry is request code, duration, loop point, name 0x8A, 2:34, 0:20, "Title Screen" 0x8B, 3:34, 2:10, "Stage 3" /* end */ This would be combined into a binary file format, using the source ROM specified. You could then do things like: - Play music in an emulated environment like a WinAMP plugin. Each 'track' would have a name, associated sound code passed to the request function, and optionally duration and loop point information. - Play music on the Genesis, using a converter tool to make a ROM out of the ripped data. Maybe some profiling to determine what portions of 68K RAM and ROM were used by the music driver would be necessary to place the player code without interfering, but it's all possible to do. Random thoughts: Maybe the tracklist should have a BGM and SFX indicator so a player can exclude SFX or place them after all the background music since most people only want to hear that. The binary file format could easily be reversed to give the textual description file (with the source ROM instead perhaps being a CRC of the original one used) and the data blocks saved individually. So a bad rip could be later worked on by someone else in the future instead of having to keep the original specification file around. Some way to specify bits and pieces of data that aren't part of the source ROM is needed, for when fragments of 68000 code or replacement data are custom made for a particular rip. So, what are people's thoughts about this? The things I've outlined have worked in the rips I've done, but to proceed with an 'official' format I want to include other features people want so we don't have a million different formats floating around. As for the actual discussion of the ripped music file's structure and header, I'll discuss that later depending on the level of interest involved. I did want to keep it in individual sections with a ID tag and version number so older players can skip over sections that they don't understand. Also any ideas about supporting 32X and/or Sega CD sound emulation (if only for Snatcher and Darxide) are welcome. Here is the original pack of rips in a Genesis ROM format that work on genecyst. Some have problems becausing the starting sound code doesn't match the original game (you'll have to cycle through a bunch of numbers first) and others have RAM sharing issues with the player so the joystick input or playback might be a bit weird. http://cgfm2.emuviews.com/segaxtreme/gsfpack.rar... (2.8MB) Have fun.

		cgfm2		Aug 22, 2005
			Nearly forgot this bit of trivia. Check out the Treasure Land rip (sounds great in Kega Fusion), it contains an early revision of the Shinobi III soundtrack that differs from the final version. I guess the Treasure people used the same sound code that had been developed by the Shinobi III staff or something along those lines. Weird, but cool.

		Borisz		Aug 22, 2005
			All I can say is to check out the VGM format at SMSpower.org. http://www.smspower.org/... And perhaps incorporate this kind of ripping into the VGM format if possible (no need for yet another, third format, VGM is already pretty nice and can handle multiple systems).

cgfm2

Aug 22, 2005

	
	
		
		
			Originally posted by Borisz@Mon, 2005-08-22 @ 05:17 PM



All I can say is to check out the VGM format at SMSpower.org.



http://www.smspower.org/...



And perhaps incorporate this kind of ripping into the VGM format if possible (no need for yet another, third format, VGM is already pretty nice and can handle multiple systems).



[post=138585]Quoted post[/post]

But VGM is still logged, so this would not solve the DAC problem at all. VGM is efficient, but you'd still have huge multi-megabyte log files per song.

Plus all the trimming/looping can be problematic, this is why there are no YM2413 VGM rips (there were some earlier ones, but they aren't offically accepted until the problem is fixed)

		Mask of Destiny		Aug 22, 2005
			At least on a few of the 68K based drivers, the Z80 is still used for playing DAC samples. Sonic 1 and Moonwalker are like this. I don't know if this is actually true, but it would seem logical that Sega provided a driver or two to developers. If this is the case it stands to reason that a large number of games would be using the same or at least similar music data formats. It would be nice to be able to rip this directly as it would be better suited to playing on the real hardware either in some kind of jukebox like ROM on a flash cart or a Sega CD based rip player. The music formats for the Sonic games have been largely figured out by the hacking community, it might not be a bad idea to see if other non-Sonic games use the same format or one derived from it.

		Kaneda		Aug 22, 2005
			yeah...a lot of games use GEMS, the sega music driver I know there is at least 3 sub versions (1.0, 1.1, 1.2) It's very basic and use mainly the 68000 but it's ok for 'normal' games...I don't think Treasure used it for example

		fonzievoltonov		Aug 22, 2005
			Assuming the 32x doc, gems (or something like that) was ported to 32x too... the 32x docs explains how the system communicates with the sound driver etc etc, you should take a look (might help?).

		LocalH		Sep 9, 2005
			I'm glad to see this is being discussed again. This is something that I've been waiting for with baited breath. I do propose a few things that I think would make the format quite versatile. For the actual final product of a rip, I propose a chunk-based format, with longword-size chunk identifiers. There should be a header chunk with all the pertinent metadata (PAL/NTSC, 68k/Z80/both, credits for the tune but not the ripper, number of tracks, number of 68K chunks, etc). You should then have one or more 68K chunks. Each chunk would begin with the identifier, directly followed by the longword starting address of the chunk, and then another longword denoting the length of the following data. You don't need separate Z80 chunks since the Z80 accesses 68k ROM through the 32K window, and RAM is injected by the 68k during initialization. This way, you could easily add your own custom glue code in it's own selfcontained chunk, and point any of the init/request/play vectors at your code if necessary. If the play routine is not needed on the 68k, then it could be handled by a simple no-op routine. As for reordering the sound IDs, I propose that this be done with custom glue code, where you essentially use a lookup table to translate the rearranged track order into the original track IDs. This is the way it is done on the other systems with ripped formats that I'm familiar with (C64, NES), in order to keep the actual player code and data as close to unmodified as possible. As for using the rips on real hardware, I propose that the glue code for this be provided as source, so that it can be assembled wherever it needs to be. With the chunk-based format I discussed, you will easily be able to parse the file and create a map of used ROM space, and thus easily find a place to put the outfit code. This would also allow easy use in a homebrew/demo project, such as I plan on doing in the near future. One thing - I would advocate that we also need some flexibility to support multispeed tunes in the future, as on the C64 - I know there aren't any right now, and I don't know if there would be that much of a benefit to using the technique, but it still might be nice to have the flexibility in case we ever see multispeed tunes. I know that the PSID format includes a bit to denote CIA or VBL timing, and I also believe that it is proper form to load the CIA timer in your 6510 glue code rather than hardcoding it into the format, but we wouldn't have to worry anything about that, we'd just need to denote how many times per frame the play routine needs to be called. As far as I know, no Genesis driver requires being called on a specific scanline, since most, if not all of them run off the Vint anyway, so you shouldn't need to do any VDP emulation, unlike the C64 where SIDPLAY2 is essentially a C64 emulator with no video display. This also wouldn't be applicable to Z80-only drivers, as they would generally just run once per frame on the Vint.

cgfm2

Sep 10, 2005

	
	
		
		
			Originally posted by LocalH@Fri, 2005-09-09 @ 03:08 PM



I'm glad to see this is being discussed again. This is something that I've been waiting for with baited breath. I do propose a few things that I think would make the format quite versatile.

I liked all of your suggestions. A lot of these I was going to implement (I didn't discuss the actual file format details as mentioned earlier, but I can elaborate on that later).

	
	
		
		
			One thing - I would advocate that we also need some flexibility to support multispeed tunes in the future, as on the C64 - I know there aren't any right now, and I don't know if there would be that much of a benefit to using the technique, but it still might be nice to have the flexibility in case we ever see multispeed tunes. I know that the PSID format includes a bit to denote CIA or VBL timing, and I also believe that it is proper form to load the CIA timer in your 6510 glue code rather than hardcoding it into the format, but we wouldn't have to worry anything about that, we'd just need to denote how many times per frame the play routine needs to be called. As far as I know, no Genesis driver requires being called on a specific scanline, since most, if not all of them run off the Vint anyway, so you shouldn't need to do any VDP emulation, unlike the C64 where SIDPLAY2 is essentially a C64 emulator with no video display. This also wouldn't be applicable to Z80-only drivers, as they would generally just run once per frame on the Vint.

I don't quite grasp the concept of multispeed tunes.

The emulated player environment would be required to emulate everything - if the Z80 driver wanted to poll the scanline counter at address $7F08 (no game would ever do this, but..) it should be supported.

So what exactly in the file format needs to be present to support multispeed tunes? I'm putting the burden of what bits & pieces of the Genesis hardware that are emulated on the player, and at the moment I think that would be enough.

The only timing data I was going to include was the display type (PAL vs NTSC) and possibly some data for the VDP registers because there are slight timing differences in the 224 vs 240 scanline modes and interlaced displays. However I'm doubting the latter because games play the same music while changing resolution all the time, and this is something outside of the sound driver and dependant on the main game.

The other thing is that for the most part SID tunes are single-song rips, so you can specify the timing method per song. But for the Genesis a single binary file would include all music and SFX, so the player environment would have to support whatever resources the music driver needs (essentially a Genesis emulator with no video, as you said)

		LocalH		Sep 10, 2005
			Multispeed tunes are basically used on the C64 to update the SID registers more than once per frame - with the heavy use of arpeggios on the SID, if you update the registers more often, then you get what I consider to be 'better' sounding arps, and usually advanced drum instruments (which also use arps) sound better as well. I don't know if there are many Genesis tunes that use arps, with the increased channel count of the FM+PSG parts, but with talk about appropriating MVS Tracker for the Genesis (I actually started thinking about this the other day, although I don't know enough about the 2612 to make it work), we might start seeing more original Genesis tunes that use such features (I know that I'd like to make some chiptunes for the Genesis, where arps are very commonplace). Well, like I said, I wouldn't be sure of the implementation details, as there are currently no multispeed tunes that exists that I know of, and in the modern PSID format, the glue code generally handles setting up the CIA timer itself, if needed - the header only denotes whether to use the CIA timer or the VIC-II interrupt facility (both of which could feasibly be used for multispeed tunes, but CIA is pretty much the defacto standard for such tunes, unless the tune specifically requires each play call to be on specific scanlines). Also, there are plenty of multi-tune SID rips that also include SFX - most of these are game tunes, but there are also some demo tunes that are multi-tune rips (although I've noticed that the trend for those has been to split them up and create a single file for each tune). Game rips definitely are multi-tune if there are multiple tunes in the game. The PSID format includes a bitfield that sets the CIA/VIC-II flag for each tune, so I'd imagine something similar could be done here, only there is no CIA or other timer source in the Genesis that I know of, so it might be a moot point.