ScoreHero
Home | Forum | Wiki
Inbox [ Login ]Inbox [ Login ]
SearchSearch MemberlistMemberlist
ProfileProfile Log inLog in
DS (OTD/OTMH/BH) Customs Pre-Guide & Discussion
Goto page 1, 2, 3, 4  Next
 
Post new topic   Reply to topic    ScoreHero Forum Index -> Software
View previous topic :: View next topic  
Author Message
codemann8  





Joined: 10 Apr 2007
Posts: 1048
Location: Oshkosh, WI

PostPosted: Fri Mar 04, 2011 10:07 pm    Post subject: DS (OTD/OTMH/BH) Customs Pre-Guide & Discussion Reply with quote

Foreward
I'm calling this a "pre"-guide as we are not to the point of completely being able to make customs, but from what I've seen and looked into, it's definitely possible. We just need more information, and more people to play around with it.

I am providing you here with what is known and what is unknown. I will show you how to customize some aspects of the game with the knowledge that we currently have. As time goes on, I will update the OP until we master the process.

Generally speaking, making customs work in games is a difficult but a possible process. Games work because of the binary data behind it that tells it what to do. Making customs is simply done by modifying that binary data to get it to do what it is you want it to do. And that's what I'm about to show you.

By the end of the day you could have this:


During this process, you will need the following things:
  • NDS Rom - You'll need to obtain a copy of an original copy of the game...the method of how you get this cannot be discussed here
  • Hex Editor - Any will do, but it's best to get one with features like:
    • Jump to a location/offset by specifying where
    • Easily see the values of each byte/word/dword in decimal format
    • Convert between hexadecimal and decimal
    • Highlight sections of bytes to keep track of different areas of importance
    • Ability to switch endianess (DS is little-endian)
    I use 010 Editor, it's not free, but it's the best I've ever seen...loaded with features to make things easier!!! There is a free 30-day trial available, but I can tell you, once you go 010, you can't go back...but if you must be a pirate about it, be my guest but I will not help you with that.
  • A DS Emulator - I highly recommend DeSmuME...it can emulate the Guitar Grip unlike most others...and it's loaded with tons of development tools (memory dumps, RAM Watches, sound viewers, etc) for research
  • Nitro Explorer 2 - This is a useful tool to let you know the offsets of each of the files contained in the NDS file as well as extracting them
  • DSDecmpGH - This is a modified version of DSDecmp that decompresses the compressed files that are extracted from NDS files that are GH.
    • I cannot publicly release the source code because it is 99% an exact copy of DSDecmp, but the original source code for that can be found at the Google Code page for DSDecmp, link above.
    • We still need help to write a recompressor program so we can make original customs
  • Audacity - This tool is useful to decode some of the audio files
History
GHOT (GH:On Tour) was the first game of the Guitar Hero series of the DS. This game consisted of a GOB file which contained the rest of the files in the game. Most people gave up on customs after realizing the difficulty of repacking the GOB into a usable NDS rom. Since that time, three new DS games have been released (OTD "Decades", OTMH "Modern Hits", & BH "Band Hero"). These games don't use GOB files to consolidate their files, instead they utilize the standard mechanism of NDS file structure. This makes it easier to get the NDS file to work the way we want it to.

I will be discussing customs in OTD, OTMH, and BH only. As I said, OT uses GOB files so that won't work.

Technical Overview
Customizing any game with new songs consists of three areas of concern:
  • Audio
  • Note Charts
  • Metadata (Text)
  • File/Data Integrity
Audio
One of the main struggles of getting customs to work is the sound. This is unfortunately the only thing we are not 100% about, but we're figuring out a lot of stuff every day. I have at times been able to hear the music that I have put it.

I've mostly dealt with replacing DirtyLittleSecret songs with other songs from the games to ensure that problems aren't with the data itself, but with the way the data is organized in the NDS file. Otherwise, if your own custom files don't work, you won't know if it is because your file is bad or the game isn't able to handle your file because of where it is or what size it is.

Each song in the game uses 2 types of files:
  • HWAS
    • This is the main music file. This is the band/auxiliary music that plays but without the instrument tracks. If you play the game and miss all the notes, you will hear only the music from the rest of the band but not your current instrument. When you start to play your notes, then you hear your instrument in addition to the rest. For each song there is only one of these file types.
  • OGG
    • This is the music track that plays your instrument sound. There is one file of this type for each instrument in every song.
HWAS
Much debate has surrounded what kind of encoding was used to originally make the file. We know that it is of a certain type of encoding called ADPCM. The flavor of that is most likely IMA, although is nearly identical to VOX. Looking at the header of the file, we have been able to determine that the header of the file is 512 bytes in length while the rest is the sound data. The header consists of information in this format:
Code:
HWAS Header (0x200 / 512 bytes)
- - - - - - - - - - - - - - - - - - - - - - -
0x0 - "sawh" - "hwas" in little endian
0x4 - Memory Allocation Length (always 32768, explanation later)
0x8 - Sample Rate (always 19996)
0xC - # of Channels (always 1, it is a mono track)
0x10 - Loop Start Sample (always 0, offset from the start of the data section)
0x14 - # of Samples (length of data portion, in bytes)
0x18 - Loop End Sample (end of music, offset from the start of the data section)
0x1C - 0s until end of header

HWAS DATA Chunks
- - - - - - - - - - - - - -
These repeat until the end of the file:
4 bytes - Unknown extra 4 bytes (Predictor/Step Index [affects the volume], least significant nibble always zero?)
32764 bytes - Raw IMA ADPCM Data
(Note: The last chunk of bytes does not have to be the full 32768 bytes)

Note: An HWAS file ends on an even 0x200 block, so dummy data (like 00's or FF's or 88's) will need to fill the rest of the space.

We need to know more about that extra 4 bytes in there, it is definitely a predictor and/or a step index, which is common among audio, but usually differently implemented in audio.

If you Import this file into Audacity as Raw Data, you can hear the audio file, but with noticeable spikes and glitches (because of those extra 4 bytes). The options you should choose are:
  • Encoding: VOX ADPCM
  • Byte order: Little-endian
  • Channels: 1
  • Start offset: 512 bytes (to ignore the header)
  • Amount to import: 100%
  • Sample rate: 19996 Hz
OGG
OGG is a pretty standard format in games. You can open this in Audacity without any difficulty. If someone finds a good page that explains the exact format of this file, it would be greatly appreciated, and I'll display it here.

Note Charts
Charts consist of 3 types of files:
  • QGM files - contains the gems (notecharts)
  • QFT file - contains the fretbars
  • QSIG file - contains the timesigs
QSIG is simple, a 2 byte file...two 0x04's...indicating a time signature, a numerator and a denominator...however, I am unsure which is which since they are all 4/4 time.

QFT is also simple, these are chunks of 4 bytes. Each chunk is a fretbar, indicating the number of milliseconds from the start of the song.

QGM is the trickier one, but still pretty trivial. For each gem (or note) in the game, there is an 8-byte chunk associated with it. The format is as follows:
Code:
QGM File Structure
- - - - - - - - - - - - -
4 bytes - Offset of gem/note (in milliseconds)
2 bytes - Length of gem/note sustain (in milliseconds)
2 bytes - Information about gem/note (color, if it is a star note, etc)
    It's been awhile since I looked into it last so this will need updating

Ever since GHOT:D, these QGM files are compressed using a LZ77-type compression. These files can be decompressed with GHDecmpGH, this is a variation of GHDecmp. There is not any recompressor available as of yet, one will need to be made first.

Metadata (Text)
This is the text and information that you visibly see in the game. This is the easiest thing to replace. I will explain the specifics later in the guide, but there are sections inside the ARM9 file that define the text. A piece of that looks like this:
Code:
Song Information
- - - - - - - - - - - - -
0x20 bytes - Song Name (as it is displayed on the song selection screen), 00's indicate unused space
0x20 bytes - Artist Name, 00's indicate unused space
0x4 bytes - Unknown, seems to be a couple of 00's and 01's
0x4 bytes - Length of song in seconds (for display purposes only)
0x4 bytes - Year of the song (again, for display)
0x4 bytes - Offset in milliseconds that the preview starts
0x4 bytes - Length in milliseconds that the preview lasts for

NDS File Structure
The format of this file is actually quite straight-forward. It consists of (not necessarily in this order):
  • Header
  • ARM9 File
  • ARM7 File
  • FNT - File Name Table
  • FAT - File Allocation Table
  • Banner File - Icon & Title
  • File Section
Header
This contains all the information necessary for the DS to start a game. The list below shows the format of the header, indicating where the rest of the important pieces are. Each game can vary in which order each of the data is organized but all is defined in the header the same way.
Code:
NDS Header Format
- - - - - - - - - - - - - -
0x00 Game title
0x0C Game code
0x10 Maker code
0x12 Unit code
0x13 Device type
0x14 Device capacity
0x16 (8 bytes blank space)
0x1E ROM version
0x1F reserved
*0x20 ARM9 ROM offset
0x24 ARM9 entry address
0x28 ARM9 RAM address
*0x2C ARM9 code size
*0x30 ARM7 ROM offset
0x34 ARM7 entry address
0x38 ARM7 RAM address
*0x3C ARM7 code size
*0x40 File name table offset
*0x44 File name table size
*0x48 FAT offset
*0x4C FAT size
0x50 ARM9 overlay offset
0x54 ARM9 overlay size
0x58 ARM7 overlay offset
0x5C ARM7 overlay size
0x60 ROM control info 1
0x64 ROM control info 2
0x68 Icon/title offset
0x6C Secure area CRC
0x6E ROM control info 3
0x70 (16 bytes blank space)
0x80 Application end offset
0x84 ROM header size
0x88 (36 bytes blank space)
0xAC PassMe autoboot detect
0xB0 (16 bytes blank space)
0xC0 Nintendo Logo (156 bytes)
0x15C Logo CRC
0x15E Header CRC
0x160 (160 bytes blank space)

*'s indicate areas of our interest

ARM9/ARM7
I don't have much information about these files, but this is where the actual "game" is. This is the code and logic that runs when you start the game. The only thing I have found of interest to any of us looking into customs in the ARM9, is the ability to modify the text that is visible in the game, such as song names, artists, year. Another piece I found is how to change the starting point and length of the song to use as a preview on the song selection screen.

FNT
This is a list of the actual filenames of each of the files in the File Section. This part is a little tricky and I haven't been able to programmatic-ally parse this section, but I do understand it. Basically it consists of two parts: The folder definition (structure) and the filenames. Each folder and file has an ID in which you can label each one and are all uniquely defined. IDs for files start at 0x0 and IDs for folders start at 0xF000.

The FNT starts with the folder structure, which is what's tricky. Each folder has an 8 byte chunk, pointing to the first file or folder in that directory. The first 8 byte chunk is different to the rest of the chunks, as it belongs to the root directory which always has an ID of 0xF000. The format is as follows:
Code:
FNT Folder Definition (at offset 0x0)
- - - - - - - - - - - - - -
The root folder: ID 0xF000
4 bytes - Offset to a list of entries (files and folders), also the length of the folder definition
2 bytes - ID of the first file found, not folder, in this directory
2 bytes - Number of non-empty folders, including the root

The remaining folders
4 bytes - Offset to a list of entries (files and folders)
2 bytes - ID of the first file found, not folder, in this directory
2 bytes - ID of the parent folder

I will explain how to use this after I explain the filename portion, which is much simpler. Basically, each of the folder, as shown above, will point to a list of entries (files and folders) starting with the first entry. The structure of that is as follows:
Code:
FNT Filename List
- - - - - - - - - - - -
1 bit - 0 if it is a file, 1 if it is a folder
7 bits - length of the file/folder name
The following bytes are the file/folder name, each character is 1 byte...if the length is 0, then there are no more files in this directory.

If it is a folder, the 2 bytes that follow is the ID of that folder.
Note: if it is a folder, you will read the next entry in the FNT Folder Definition to determine which folder is it's parent.

If it is a file, there are no additional bytes to read and you should continue on and read the next byte to determine if the next entry is a file or folder.  The ID of the file is the ID that was defined in the FNT Folder Definition entry that got you to this point, but it increments as you go through each file. ex, if a folder contains 3 files, the first file listed will get the ID that was defined in the folder definition, lets say it was 0xF, the next 2 files would be 0x10 and 0x11

Reading the FNT must be done with a depth-first approach.

FAT (File Allocation Table)
Once the names and IDs of the files are understood, the next part is to direct you to the actual data in those files. Each file has an 8-byte chunk in order of the IDs of the files listed in the FNT. The first 4 bytes is the offset to the file, the next 4 bytes is the offset to the end of the file.

Banner
This section includes the game logo and game title information. There is nothing in this section of interest to us.

File Section
This is the data portion of the NDS, all the files are throughout. There is no structure in here that is the same for every DS game. The FAT points to files this exist in this section and that's all that is important to us. No file ever spans across each 0x1000 mark, as I'm sure the DS reads in the files in 0x1000 byte chunks at a time. This will be important to know when changing file sizes.

Some files are compressed using a variation of the LZ77 algorithm, these files can be decompressed with DSDecmpGH. These compressed files are easy to spot as they always begin with the same header:
Code:
LZ77 Compressed Files
- - - - - - - - - - - - - - - -
5 bytes - Magic Number (always 4C 08 00 00 45)
3 bytes - Size of compressed file
1 byte - Type of LZ77 (always 0x10)
3 bytes - Size of original decompressed file
X bytes - The data

Other File of Interest (fsindex-us.bin)
Of the files in the NDS, there is one file that proves interesting but I do not have enough information on it. The file is fsindex-us.bin, a file kind of like the FAT but internally handled and used by the game itself. Which means if you make changes to the file sizes, you must make changes to this file as well. Unfortunately I don't know much about this file so our capabilities to make customs are hindered. I did find a couple of interesting things however:

If you are able to, open this file in a hex editor and display 12 bytes per row instead of the usual 16, you should notice the first 4-byte column increases throughout the file. I don't know what this means yet but it could relate to some sort of an offset to a file. Because of this, I believe that each 12-byte line refers to a file from the FAT.

I have also found that of each 12-byte line, bytes 0xD - 0xF indicate the amount of space that the game allocates for a file in memory...for example, if a file is compressed, when the file is decompressed, it will allocate the decompressed size, not the original size.

I have not been able to figure out how to find each file's record here, the only way I've been able to find anything is by searching for the 3-byte filesize of whatever file I am trying to change and replace each one until I find the one that works. More info on when you need to change this later.

Customization (How-To's)
Important Note: Always make backup copies of your NDS files. Also, make changes to one thing at a time, it makes it a ton easier to determine what works and what doesn't.

Modifying Metadata (Text)
This is about the easiest thing to do. All you really need to do is search the file for the name of the song you want to change. If you search for the name as you see it in the game (with spaces), you should find two instances within the ARM9 section. The first one is the information that shows just before the song starts in gameplay, the second is what shows on the song selection screen. Changing this is easy, all you need to do is replace the text with what you want it to say, what you have to be careful of is how to deal with text replacements that are different sizes.

The first part should end with 0x0A's until the next part begins. If the text you are entering is too long and starts to interfere with the next text portion, you must just delay that text, but note that each text piece starts with 2 bytes that must stay intact, I don't know what these 2 bytes mean. Just know that these text portions are separated by 0x0A's.

Modifying Charts
Based on the structure defined earlier, it is pretty trivial to make custom note charts, you will need a QSIG file, a QFT file, and several QGM files (one for each instrument/difficulty combo). However, because there is no recompressor, we cannot insert customs into the game yet (at least I think we cannot, we might be able to add it as plaintext to see if that works).

What I have tried doing, and successful in doing so, is bringing another QGM file from another song and putting it in place of another QGM file. In order to do this you need to change 3 things:
  • The QGM file itself
  • The entry in the FAT
  • The entry in the fsindex-us.bin file
I usually pick DirtyLittleSecret Easy Guitar to do my modifying. It's the first song in the game and easy charts have the smallest files.
  1. Use Nitro Explorer to open the NDS, you'll easily be able to find the specific offset and size of the files of the chart you want to look at and replace.
  2. Go to that place in the NDS file using a hex editor.
  3. Note the two sizes in the old QGM file (the compressed and decompressed sizes)
  4. Paste in another QGM file in place of this one. (Note for now that you might be affecting the next file that comes after this one if the file is longer than the original, but we shouldn't care during experimental times)
  5. Note the two sizes in the NEW QGM file (the compressed and decompressed sizes)
  6. Using the offset that Nitro Explorer reported for the old file, search the NDS file for this offset
    • Remember, search in little endian
    • ex. For offset 0x23 E7 64 F0, you should search for F0 64 E7 23.
  7. Look for the only search result within the FAT
  8. That offset shouldn't change, the following 4 bytes should change to reflect the end offset of the new file. It will be the offset you just searched for plus the compressed size of the new file.
  9. Now the game knows the correct information about the file, but we are still missing the part that tells the game how much space in memory to allocate for the chart
  10. Search the NDS file for the decompressed size of the old file in 3 bytes (yes, include the 00 for the last byte)
  11. Only look for search results that exist inside the fsindex-us.bin file, there most likely will be many, what you need to do is try the next step out for each search result.
  12. Replace these 3 bytes with the decompressed size of the new file
  13. Try playing the game until one of the changes works.
Modifying Audio
We have two routes we can go, the easy way or the better way. As we know the audio consists of 1 HWAS file and many OGG files. The HWAS as of know we know a ton about but am unable to successfully insert the audio without having distortions because we don't know what the predictor is.

So, the easy way would be just create a blank HWAS file and just put all of the music in one of the OGG files. The bad part of that, it whatever instrument you do this to, if you play the game under that instrument, once you start missing notes, the music stops playing entirely. The idea is to use an instrument you don't like or will never play.

The better way would be to look further into what that predictor is in the HWAS, that way we can play the song on all instruments without having the music mute.

HWAS The easy way:
Just replace bytes 0x5 and 0x6 in the HWAS file, which should be 00 80, with 08 00. What this does, is tells the game to only allocate 8 bytes of space in memory, versus 32768 bytes. Because it is only 8 bytes, the game can never get enough of a sample to actually play the audio. Using DeSmuME, you can verify this change by opening the Sound State window and looking at sound #08, this can even be done in the song selection screen (preview). The sound length will show double the value you edited in the HWAS file; double because it always has to have the current chunk of audio along with the next chunk ready.

HWAS The better way:
As we know, there are problems with the HWAS audio if you try decoding it as is. I did find something interesting. I found a way to make the sound clean after you extract it.
  1. First, open up any of the HWAS files in your Hex Editor. I'm using DirtyLittleSecret_song.hwas as it is the first song in the game and just easier to access from the start.
  2. Note all the information in the header as I described above.
  3. Remove the header from the file altogether (the first 512 bytes).
  4. Remove the first X bytes from the files, where X is the Loop Start Sample (it is most often 0 so you probably can ignore it) as noted in the header.
  5. Keep the first X bytes and remove the rest, where X is the Loop End Sample, as noted in the header.
    • Note: The file as it is now is the true data portion. If you open it in Audacity, with a start offset of 0, it will play the song as it would in the game. More specifically, the start and end will have no extra added onto it. But the glitches are still apparent.
  6. This next step is tricky but necessary in order to get rid of the sound spikes and glitches. For every X bytes in the file, you must remove the first 4 bytes, where X is the Memory Allocation Length. To test this easily, you could get away with just removing the first couple of them and try it out in Audacity. The file should then be free of glitches.
    • Note: I haven't thoroughly tested this step, but it seemed to work perfectly. It is unclear if those 4 bytes are used for anything or if it's just junk. Let me know if you find anything interesting about it.
To make your own custom audio, you could encode your own audio file as a Raw (headerless) VOX ADPCM and a sample rate of 19996 Hz. Then attach your own 512 byte header and supply the information needed. You would also then also need to add in an extra 4 bytes per 32764 bytes (I'm not sure what this extra 4 bytes mean, but it seems to do with predictors and step indexes...this is what needs to get figured out). This is the better way but more research is needed.

Editing the OGG:
Making your own isn't very difficult. You can just open an MP3 in Audacity and save as OGG. But you'll have to make sure that the sample rate matches the one you're replacing.

I haven't had much luck with this myself, purely because of lack of trying. But like the notecharts, you will need to edit the entry in the FAT and fsindex-us.bin.

End Notes
A lot of this is still very much a work in progress but definitely hopeful. I'm looking to have a few people play around with this to discover things that I am not able to.

Helping Out
If you are up to the task, any help in researching these files would be greatly appreciated and noted. I will include and consolidate our findings in this guide. Here's a todo list:
  • fsindex-us.bin - #1 needed thing: What maps the files in the FAT to this file? Of each 12 bytes, what do they all mean and used for?
  • HWAS 4-byte predictor - Required for creating custom HWAS files
  • OGG files - We need people to try playing around with these
  • Recompressor - Using the source code of DSDecmp, I will need help writing a recompressor, this is required for making custom notecharts
  • Much later - A packer tool that we can use to insert custom easier without manual editing...much more research is needed before this can be done.
Good Luck and Happy Customs.


Last edited by codemann8 on Sun Apr 10, 2011 11:06 pm; edited 19 times in total
Back to top
View user's profile Wiki User Page Send private message Send e-mail Visit poster's website XBL Gamertag: codemann8 PSN Name: codemann8
ComicBookGuru  





Joined: 16 Dec 2007
Posts: 1678

PostPosted: Sat Mar 05, 2011 6:17 pm    Post subject: Reply with quote

Yeah, the HWAS files were listenable in Audacity. I forget if you could make your own, though I don't imagine it would be too hard.
_________________
Yep, I learned through Rocksmith!
Back to top
View user's profile Send private message
codemann8  





Joined: 10 Apr 2007
Posts: 1048
Location: Oshkosh, WI

PostPosted: Sat Mar 05, 2011 8:52 pm    Post subject: Reply with quote

Ok, I just read more into the HWAS thing...I did DL Audacity and imported the file as Raw Data, I picked:
Encoding: VOX ADPCM
Endian: Little
Channels: 1 (Mono)
Bit Rate: 19996Hz

You can hear the song (background music w/o guitar/bass) but there are artifacts in there that shouldn't be there.

The bit rate, channels, and endianess are the things we are 100% sure of...the encoding is the only unknown, we do have to keep in mind that we don't know what kind of sound engine the DS uses, also, how good of quality are the speakers on a DS anyways, its possible it was encoded this way. But at the very least, we know it is a PCM type audio, like you said, I don't think it would be very hard to make our own, and shouldn't be hard to test by taking a different audio file and encoding it with VOX ADPCM, add a 512 byte header with the same information in a HWAS header and put it in place of an existing song and see how it plays in an emulator.
_________________
Frets and Fireworks GH:A Expert Tourney Winner
AAA-2 Season 5 Overall Points Champ & Playoff Runner-Up!!!
My FCs



Back to top
View user's profile Wiki User Page Send private message Send e-mail Visit poster's website XBL Gamertag: codemann8 PSN Name: codemann8
opiumtest5  





Joined: 03 Aug 2009
Posts: 91

PostPosted: Mon Mar 28, 2011 7:08 pm    Post subject: Reply with quote

about those hwas files... if they sound decent on the ds itself then whats the point in trying to decode them to a fromat the ds is not familiar with? and you just said that it wouldnt be hard to make our own hwas files, then if the original hwas files sound kind of laggy with the decoder you used then wouldnt it be a possibility to encode the (for example) mp3 file into an hwas file with the same decoder you used to be able to hear the song?

and another thing, those chart files may be hard to decompress but the point is how to compress a chart into the same file like the one you are trying to decompress: say youre trying to make an accustomed dimension chart and compress it to the exact same format as the dimension chart thats already in there? how is that for a try? or am i way off in the purpose of this topic?

you said those other files were easy to edit... wich means that it would somehow relate to the QGM file? because i believe the guys who made OT make theyre files all in the same way?

you wrote a tool to be able to decompress it. and youre working on a recompresser... i dont think thats nessecary because i think (and i guess most people think) that its not nessecary to be able to edit the file up. yet replace it with a newly made file with the exact same properties and size would do just the trick wouldnt it? i dont know if it works that way but its worth a try?

now for making al of those files that make the song and the chart... what if those files were made up from building parts of the charts we see on the pc/xbox/ps/wii? it would make sense to check if its the same thing.

and what if you make the file one bite bigger... would the song still work? if it does then that means the game can accept longer charts and longer songs so you can play for example exilelord's epidox on the ds (not that anyone would ever make it to the end ). but of course it seems pointless to try.

i have just brought up some theories of what i think should be at least tried 2 years ago... because it seems the previous lets say ''hackers'' didnt really make anything of it except for the awesome ability to ungob .gob files. but what am i talking about, its youre project and youre choice how to handle things

ps. i would like to be a tester! by emulator and by hardware ds user!
_________________
Im not going for high scores or 5 stars, im already satisfied when i beat the song
_________________________________________
i do not release charts that are forbidden by the scorehero mods. i only release yet to be charted songs.
Back to top
View user's profile Send private message
codemann8  





Joined: 10 Apr 2007
Posts: 1048
Location: Oshkosh, WI

PostPosted: Mon Mar 28, 2011 11:23 pm    Post subject: Reply with quote

opiumtest5 wrote:
about those hwas files... if they sound decent on the ds itself then whats the point in trying to decode them to a fromat the ds is not familiar with? and you just said that it wouldnt be hard to make our own hwas files, then if the original hwas files sound kind of laggy with the decoder you used then wouldnt it be a possibility to encode the (for example) mp3 file into an hwas file with the same decoder you used to be able to hear the song?

well, just a couple of things...

I wouldn't exactly call the sound decent, it was a little crackly just on the noticeable side, but overall the sound is distinguishable...but right, we essentially could still use an mp3 and encode it with the codec we think it is, edit the header, and rename it to HWAS and it'll be close, maybe too close to tell on a DS, thats where my thinking was.

If someone is truly concerned about sound quality, they could just make the guitar track OGG file like normal and then on the bass track OGG put both the bass track sound AND the background music in addition and then when you play guitar and miss notes, all the music will play minus the guitar like normal, but when you play bass and you miss, it will just quit playing sound altogether until you get back on track. But maybe you don't like bass, so you'll never play it. And then jam a blank HWAS file in there because the band music is already in the bass OGG file.

...really you could just create two custom songs, one called Custom Guitar and the other Custom Bass and just flip flop the roles of the OGG files. Further analysis could be done if you could have technically 2 custom songs but make each of them playable in only one of the instruments while both could point to the same Name/Artist/Year of the song, that way it would appear to be one song.
opiumtest5 wrote:
and another thing, those chart files may be hard to decompress but the point is how to compress a chart into the same file like the one you are trying to decompress: say youre trying to make an accustomed dimension chart and compress it to the exact same format as the dimension chart thats already in there? how is that for a try? or am i way off in the purpose of this topic?

yep that's pretty much it, except that it is not hard to decompress the existing content, that is not anymore...it was just an unknown type until a few weeks ago...
opiumtest5 wrote:
you said those other files were easy to edit... wich means that it would somehow relate to the QGM file? because i believe the guys who made OT make theyre files all in the same way?

My guess is that they use a specific naming so that all the game needs to know is the song filename prefix. That way it knows to find: xSongNameHerex_frets.qft and so on. But yes, after I decompressed all the files, they seem to be identical in format (other than the fact that they have additional files for drum and vocal charts...drum charts look exactly the same as the guitar and bass charts...the vocal files are definitely different and unknown...but having blank or invalid files in place for these really shouldn't cause any issues, you just wont be able to play them)
opiumtest5 wrote:
you wrote a tool to be able to decompress it. and youre working on a recompresser... i dont think thats nessecary because i think (and i guess most people think) that its not nessecary to be able to edit the file up. yet replace it with a newly made file with the exact same properties and size would do just the trick wouldnt it? i dont know if it works that way but its worth a try?

the compressed QGM files are the chart files, they definitely need to be modified if you expect to change the fret notes for a song...on GHOT these were left uncompressed by the game makers so it was easy to find and manipulate, the games after that were made with compressed QGM files...so chart editing was off the board until I found a decompressor, so it no longer isn't impossible to look at the source code of that program to figure out how to reverse the process...but I wasn't going to put any effort into it until I saw public interest, thats where that was left. but yes you are right, it would definitely be worth a shot to just add the modified files as is (that is without compression) and see if it will be smart enough to recognize that it doesn't need to decompress it. I will definitely try that out soon.
opiumtest5 wrote:
now for making al of those files that make the song and the chart... what if those files were made up from building parts of the charts we see on the pc/xbox/ps/wii? it would make sense to check if its the same thing.

Its not, unfortunately, but I'm sure that many of the people who originally looked into making customs studied this format, figured it out and made their own custom QGM files but never got to put them to use after being unsuccessful in re-gob'ing the GOB. (but like I said, this is not a problem in the other games)
opiumtest5 wrote:
and what if you make the file one bite bigger... would the song still work? if it does then that means the game can accept longer charts and longer songs so you can play for example exilelord's epidox on the ds (not that anyone would ever make it to the end ). but of course it seems pointless to try.

This is something I haven't looked into much but maybe I should. This could be a really easy thing to change, it could be extremely hard, we don't know yet...but I've been told this info is found in the fsindex-us.bin and arm9.bin files. Those files would most likely be the file/s you would need to edit to add your own new files to the file system list with the new sizes, could be larger or smaller. But I don't have enough information to make the call myself. But it is then a default action to leave the file alone and keep the filenames and song names the same, but you're right, they would have to be less-sized.
opiumtest5 wrote:
i have just brought up some theories of what i think should be at least tried 2 years ago... because it seems the previous lets say ''hackers'' didnt really make anything of it except for the awesome ability to ungob .gob files.

...but remember, the GOB files are only used on GHOT; the people who worked on customs for that couldn't find a way to re-gob after adding new files, the only way was to, like you said, replace the files with identical named customs that were less in size than the original....and the same would apply to this game, but instead of GOB files enforcing the file limitations, the BIN files do that.
opiumtest5 wrote:
ps. i would like to be a tester! by emulator and by hardware ds user!

This is slow progress if any, there hasn't been much interest in it since I brought the conversation back up after I made the breakthrough of the compression. I will try using uncompressed files before writing any kind of recompressor...a recompressor would definitely 100% make this work but it would be a waste of time if plaintext proves to be good anyways...but if it doesn't work, I can't say that I will write it if there is no interest in it...looking at the previous discussion threads about this it seemed there was PLENTY of people involved, they just must not be around anymore.

I will do my part and try some of this stuff out first as was mentioned. You could as well if you know how and interested and let me know what you find. Also anyone who is silently reading this thread and care.

TODO list: (I'll add it to the OP)
Try using uncompressed QGM files in place of compressed ones
Look into BIN files and the fs structure definition
Encode an MP3 into a HWAS file and listen to the quality

Also, I have all the uncompressed game files for all GH DS games on my FTP site under /codemann8/gamefiles/<GameName>/ds/<FilesAreHere> ... just for reference and DS custom research.

I am curious as to how you get modified ROMs back onto DS hardware...I normally am not involved with a lot of homebrew discussions.
_________________
Frets and Fireworks GH:A Expert Tourney Winner
AAA-2 Season 5 Overall Points Champ & Playoff Runner-Up!!!
My FCs



Back to top
View user's profile Wiki User Page Send private message Send e-mail Visit poster's website XBL Gamertag: codemann8 PSN Name: codemann8
opiumtest5  





Joined: 03 Aug 2009
Posts: 91

PostPosted: Tue Mar 29, 2011 6:45 am    Post subject: Reply with quote

how to get roms into hardware you say? well there is hardware that allows you to dump roms and such and i would be one of those people who ''posses'' this hardware... and on top of that i have a flashcard and a guitar-grip, drum skin, etc.

though dont get me wrong i have the original game cart for GHOT GHOTD and BH so im not entirely a pirate.

so now back to youre question... i read my micro sd card included in the purchase of my flashcard and drag 'n drop every file in there... so thats why i thought i could be a tester so i could show you if it to be work on a normal ds see.
_________________
Im not going for high scores or 5 stars, im already satisfied when i beat the song
_________________________________________
i do not release charts that are forbidden by the scorehero mods. i only release yet to be charted songs.
Back to top
View user's profile Send private message
codemann8  





Joined: 10 Apr 2007
Posts: 1048
Location: Oshkosh, WI

PostPosted: Wed Mar 30, 2011 12:24 am    Post subject: Reply with quote

opiumtest5 wrote:
how to get roms into hardware you say? well there is hardware that allows you to dump roms and such and i would be one of those people who ''posses'' this hardware... and on top of that i have a flashcard and a guitar-grip, drum skin, etc.

though dont get me wrong i have the original game cart for GHOT GHOTD and BH so im not entirely a pirate.

so now back to youre question... i read my micro sd card included in the purchase of my flashcard and drag 'n drop every file in there... so thats why i thought i could be a tester so i could show you if it to be work on a normal ds see.

ehh, yeah I got all the real stuff too but its the only way to do customs I suppose.

I have a question, how can you repack a DS game? I used DSLazy, DSBuff Final...I can't seem to get it to work. Here's what I do:

1) I have a NDS rom (orig.nds) that works perfect in any emulator, but I've become favored towards Desmume.
2) I use DSBuff to Unpack the contents to a folder
3) I literally make zero changes to the files
4) Then I Repack into a new NDS file (new.nds)
5) I load new.nds up in Desmume and I get a white screen

No$GBA - The orig.nds loads fine but theres no Guitar Grip option
-- The new.nds doesn't work, I get an error: "Undefined opcode - with no debug vector defined" and it closes

DSemu - doesn't work with either, seems to "run" but its a black screen throughout

I can try the same thing in DSLazy, but it seems like they all use the same command line tools behind the scenes anyways. But all the same outcome.

EDIT: I also tried making an exact copy of orig.nds, and I opened it up in HEX and edited it directly...changing every instance of DirtyLittleSecret to DirtyLittleSecrit...just to see if it would work...Desmume shows me the Must have Guitar Grip message...so I put it on and now it blinks black real quick and goes back to white and stays there

I then studied the NDS file structure for hours and realized how straightforward it is, and I see no reason why the outcomes are different. I know the filesizes are different because the repack process just puts everything back without the requirement that a new data section has to start on an offset of increments of 1024. The FAT is just updated appropriately. IMO, there IS a ton of space wasted in original NDS roms.

The things I don't understand is the fsindex-us.bin/arm9.bin/arm7.bin files. If I'm not mistaken, the arm9/arm7 just include actual game logic, which we're not changing nor wanting to. The fsindex-us.bin file is unknown but it is withing the scope of the Data portion, so that can't have a direct impact on the game working without the Grip and not working with it.

You have any ideas on why I can't get this to work, I'm sure you may have tried some things with the OT games.
_________________
Frets and Fireworks GH:A Expert Tourney Winner
AAA-2 Season 5 Overall Points Champ & Playoff Runner-Up!!!
My FCs



Back to top
View user's profile Wiki User Page Send private message Send e-mail Visit poster's website XBL Gamertag: codemann8 PSN Name: codemann8
opiumtest5  





Joined: 03 Aug 2009
Posts: 91

PostPosted: Wed Mar 30, 2011 6:56 am    Post subject: Reply with quote

on an emulator... nothing works on an emulator... theyre incompatible with the newly written data for the games. so it means that the emulators cant recognize the new crucial data that has been added to the games.

emulators need to be written to every game in order to run it youknow

so the only way to test new things is to run it from a ds... wich is where i come in. but the repacking problem you have? its the same with me -_-

when i repack stuff the emulator sais that the rom has crashed and the ds just gives me the white screen of death. so what does this mean? it means that when we repack our ds files the whole ds id code changes with it because i couldnt load up cheats as well... it just didnt recognize the game properly. this will lead us to the following conclusion: repacking our ds files this way is impossible... wel have to find a way to edit, dlete or add files to the nds just not by taking the nds apart...

another suggestion is to never allow your nds packer to make new nds files but try overwriting the orig.nds instead... making at least 3 backups carefully... but even that seems pointless with all these errors coming up... what we could also do is look at the codes inside the nds file and see where its broken OR we can use an existing tool to fix the header.

after this problem is solved we can finally go on and test some stuff!
_________________
Im not going for high scores or 5 stars, im already satisfied when i beat the song
_________________________________________
i do not release charts that are forbidden by the scorehero mods. i only release yet to be charted songs.
Back to top
View user's profile Send private message
codemann8  





Joined: 10 Apr 2007
Posts: 1048
Location: Oshkosh, WI

PostPosted: Wed Mar 30, 2011 8:57 am    Post subject: Reply with quote

opiumtest5 wrote:
on an emulator... nothing works on an emulator... theyre incompatible with the newly written data for the games. so it means that the emulators cant recognize the new crucial data that has been added to the games.

emulators need to be written to every game in order to run it youknow

so the only way to test new things is to run it from a ds... wich is where i come in. but the repacking problem you have? its the same with me -_-

when i repack stuff the emulator sais that the rom has crashed and the ds just gives me the white screen of death. so what does this mean? it means that when we repack our ds files the whole ds id code changes with it because i couldnt load up cheats as well... it just didnt recognize the game properly. this will lead us to the following conclusion: repacking our ds files this way is impossible... wel have to find a way to edit, dlete or add files to the nds just not by taking the nds apart...

another suggestion is to never allow your nds packer to make new nds files but try overwriting the orig.nds instead... making at least 3 backups carefully... but even that seems pointless with all these errors coming up... what we could also do is look at the codes inside the nds file and see where its broken OR we can use an existing tool to fix the header.

after this problem is solved we can finally go on and test some stuff!

im getting SOOO close, I have this so far...the audio is horribly scratchy but you can hear it:

it is definitely doable...im editting the NDS file directly, no tools...the text stuff is easy....for some reason tho, the audio is just really bad...I tried the ogg files too...the ogg file I made sounds perfect in audacity, just doesnt on the game
_________________
Frets and Fireworks GH:A Expert Tourney Winner
AAA-2 Season 5 Overall Points Champ & Playoff Runner-Up!!!
My FCs





Last edited by codemann8 on Thu Mar 31, 2011 1:15 am; edited 2 times in total
Back to top
View user's profile Wiki User Page Send private message Send e-mail Visit poster's website XBL Gamertag: codemann8 PSN Name: codemann8
codemann8  





Joined: 10 Apr 2007
Posts: 1048
Location: Oshkosh, WI

PostPosted: Wed Mar 30, 2011 9:52 am    Post subject: Reply with quote

Double Post DELETED: Site was going crazy at the time
_________________
Frets and Fireworks GH:A Expert Tourney Winner
AAA-2 Season 5 Overall Points Champ & Playoff Runner-Up!!!
My FCs





Last edited by codemann8 on Wed Mar 30, 2011 9:31 pm; edited 1 time in total
Back to top
View user's profile Wiki User Page Send private message Send e-mail Visit poster's website XBL Gamertag: codemann8 PSN Name: codemann8
opiumtest5  





Joined: 03 Aug 2009
Posts: 91

PostPosted: Wed Mar 30, 2011 2:41 pm    Post subject: Reply with quote

MY GOD! you dont even need my help anymore :O
well then its now officially proven that customs on on tour is possible... now we just need to make some more charts! and we need to find out how to get nice sounds!

codemann like always you kick a$$ im serious :P

anyway maybe you could try making a guide now because i want to try out some damn nice songs on the ds now!

i also see you followed the advice by edditing the nds directly! but you could also still try and use a tool to fix the header youknow... it works with other hacks too so it wouldnt hurt to give it a shot
_________________
Im not going for high scores or 5 stars, im already satisfied when i beat the song
_________________________________________
i do not release charts that are forbidden by the scorehero mods. i only release yet to be charted songs.
Back to top
View user's profile Send private message
ComicBookGuru  





Joined: 16 Dec 2007
Posts: 1678

PostPosted: Wed Mar 30, 2011 6:41 pm    Post subject: Reply with quote

Wow, that's awesome!
_________________
Yep, I learned through Rocksmith!
Back to top
View user's profile Send private message
codemann8  





Joined: 10 Apr 2007
Posts: 1048
Location: Oshkosh, WI

PostPosted: Wed Mar 30, 2011 9:53 pm    Post subject: Reply with quote

Either tonight or the next I can throw together a guide. I tried just about everything multimedia-wise (modifying text, modifying OGG, modifying HWAS) and can say that is possible (audio and text)

as for the charts part, I did try putting a uncompressed chart in there and updated the FAT and it failed, so that is out, we will need a compressor program. But charts would still possible.

...what I haven't tried is when your custom file (whether audio or chart) is larger than the space available...reason being, if this will be a manual process early on, we would have to scoot the following files in the NDS back x amount of bytes and modify the FAT for each file, which would be a pain...but programatically possible...its a question of what other parts of the system care about file size and name changes...

I know for a fact that if you change the filenames, the game doesn't load, it's like theres another file internally that's keeping track of the files in the data section...its most likely the fsindex-us.bin...but really, filesnames don't mean anything behind the scenes...we never see it, so that doesnt have to change.

An interesting note on fsindex-us.bin: if you have a hex editor with adjustable line feed sizes, instead of showing 16 bytes per line, show 12 instead and you'll notice one of the 4-byte columns increments throughout the file...remember, little endian...and curiously enough, after setting it to 12bytes per line, there are 3080 lines...it just so happens there are 3082 files in the FAT...theres a reason for being a couple short, fsindex itself IS one of those files so it wouldn't store that, and who knows which other files are ignored because they are loaded initially

im not sure if the header needs fixing, it works fine if I leave it alone...editing it directly works...im sure if u edited the OT GOB file directly it would probably work there too...I can see where unpacking and repacking doesnt work

EDIT: also a quick note since you mentioned chart QGM files and wanting you try out something, if u dont know already, the format is quite simple...there is no header, it is just straight data...the file is 8-byte chunks, each represent a note/chord, like so:
Code:
QGM FILE
----------
In Little Endian:
4 bytes - Start of note (in milliseconds)
2 bytes - Length of note, if its a sustain, else just 0 (in ms)
2 bytes - Info on the note (can't remember the exact details but wouldnt be hard to figure out (things like note color (in flag format) and star notes)

_________________
Frets and Fireworks GH:A Expert Tourney Winner
AAA-2 Season 5 Overall Points Champ & Playoff Runner-Up!!!
My FCs



Back to top
View user's profile Wiki User Page Send private message Send e-mail Visit poster's website XBL Gamertag: codemann8 PSN Name: codemann8
opiumtest5  





Joined: 03 Aug 2009
Posts: 91

PostPosted: Thu Mar 31, 2011 6:49 am    Post subject: Reply with quote

hey thats really helpful! now if i study this carefully... wich will take some time i will be able to create my own charts from scratch... but thats just it i think that would be hard if you dont have any audio corresponding to it...

so you got a song in there... and you can play it... you replaced it with another song meaning that thats the only way to get them in there... is there a way then to expand the size of the nds file tenfold? or should we compress everything tenfold... nah just forget it 25 songs is enough for me :P
_________________
Im not going for high scores or 5 stars, im already satisfied when i beat the song
_________________________________________
i do not release charts that are forbidden by the scorehero mods. i only release yet to be charted songs.
Back to top
View user's profile Send private message
codemann8  





Joined: 10 Apr 2007
Posts: 1048
Location: Oshkosh, WI

PostPosted: Fri Apr 01, 2011 6:26 am    Post subject: Reply with quote

I cannot say for sure whether we can change data sizes or not yet. I've made some good discoveries so far and I'd like to write up a guide so you all are able to do what I have been doing. But I'll probably get to that this weekend.

Interesting note on fsindex-us.bin:
I already mentioned that the data is in 12 byte alignments...that much I know...I still don't know much about where each 12-byte section starts and ends (because idk what the header indicates....but I will begin assuming that they align with the end of the file
I have found something very interesting...

I want you to look at 3 files, fsindex and the original compressed QGM files for TheMiddle-Rhythm-Easy and DirtyLittleSecret-Guitar-Easy...both those QGM files are the same filesize when compressed...so my idea was to copy and paste the bytes from TheMiddle in place of DirtyLittleSecret...I loaded it up and it got past the VV logo and failed before the main menu...I was scratching my head for some time and after awhile I noticed the headers on the compressed QGM files...

All compressed QGM files start with 4c 08 00 00 45 ... the next 3 bytes are the compressed file size (both files are 0x430, so 30 04 00) then there is a 0x10 (indicating the type of compression) and then 3 more bytes (indicating the file size after the files are decompressed)...

Both the files differ in decompressed size, TheMiddle is 0x710 and DirtyLittleSecret is 0x638...so what I did was searched fsindex for all instances of 38 06 00...I found 4, if you replace the one @ offset 0x5C19 and change it to 10 07 00...the game will start up fine AND you will get TheMiddle's notechart when playing DirtyLittleSecret on guitar easy!!!!!

Custom charts FTW!!!!

My guess is that at least one function of fsindex is to tell the DS how much memory to allocate for each file so then it can guarantee enough room when it gets decompressed...before it was too small so when it tried to decompress, I'm sure it collided with other data and crashed...now I also did put a number bigger than 0x710 and that did seem to work too, at least it loaded...I didn't test it all the way thru, my guess is it will cause some funky behavior at the end of the file...we dont want that

Now, I did try copying the chart from one memory location to another (like towards the end where theres a ton of wasted space...and I updated the FAT to point to the new offset and it seemed to work...but then I went back to the original location and filled it with zeros (just to be sure) and it failed right away, the logos didn't even show...this is probably something else in the fsindex thats referring it back to the original location still...more analysis needed there

Interesting note on HWAS:
It seems to be encoded IMA ADPCM but a slight variation...even if I run the original NDS, the sound likes to bug out every 10 seconds or so, thats because Desmume isn't handling the decoding 100% correctly since they are treating it like a standard IMA ADPCM file. I watched the memory while the song was playing as it looks like it allocates 0x8000 in memory to play the files, it seems to load it in 0x200 chunks, possibly the data size...I found a value in the ARM9 that controls how many bytes to read for the preview and also the duration of the preview (those can be changed) also, if you want to "blank" the HWAS out, all you need to do is modify the DWORD at 0x4 to be 08 00 00 00 ...this value is as I mentioned above, it tells the DS how much memory to allocate to play the song...its small enough that it isn't able to get a full sample to play yet not too small that when it tries to read that it won't crash...another option is to zero out the file starting at 0x200 (the header size), a better (to avoid crashing) would be to look at the value at 0x18 and only zero out that many bytes starting at 0x200 (that is the loop end sample)...fyi the loop start sample is always 0 (@ 0x10)...the value at 0x14 is the number of samples...and of course we knew the sample rate is 19996 @ 0x8 and channel count @ 0xC...I will clean up this explanation in the guide, theres a bunch to say about it...but I'm not understanding the spikes when trying to decode it...it must be using something like 4-bit/8-bit interlaced sound and converting 512 or 1024 bytes at a time or something or something, other than that it seems fine...

OGG I haven't been as successful, I can get sound to play for a short time and then it freezes...im not familiar with OGG very much but I'm sure the game is just strict as far as what kinds of metadata they allow so its a matter of figuring out what it'll take...

TEXT is the easiest, just search for what you want to replace and do it...but you have to be careful to end your words with either a 00 or 0A depending on the exact area, its real easy to see when u need it...u can change the Song Name, Artist, Year, song length (for display), preview length, preview start

Unknowns to be looked at:
fsindex - priority #1...this is most likely our way to get longer songs/charts
OGG - getting it to work, may depend on fsindex too
HWAS - not very important, probably can do everything in OGG
_________________
Frets and Fireworks GH:A Expert Tourney Winner
AAA-2 Season 5 Overall Points Champ & Playoff Runner-Up!!!
My FCs



Back to top
View user's profile Wiki User Page Send private message Send e-mail Visit poster's website XBL Gamertag: codemann8 PSN Name: codemann8
Display posts from previous:   
Post new topic   Reply to topic    ScoreHero Forum Index -> Software All times are GMT
Goto page 1, 2, 3, 4  Next
Page 1 of 4

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum




Copyright © 2006-2024 ScoreHero, LLC
Terms of Use | Privacy Policy


Powered by phpBB