RE Notes

[Necrolis]

Seeing as various things might require RE of the Diablo binaries, I decided to open this topic to track any notes relating to such (note: I'm starting from scratch as I don't have the time to go dig up long lost refs that might not be for 1.09). Currently this is just focusing on the tile renderer.

Diablo.exe
Functions

Code: Select all

401947	D1Error
401a65	D1DDrawError
401a88	D1DSoundError
401b3d	D1ErrorDialog
401fbd	D1DrawAutoMap
403a8e	D1DrawSkillButton
40435b	D1DrawHealthGlobe
404475	D1DrawManaGlobe
4052c8	D1DrawInfoText
4056d8	D1DrawCharacterSheet
406058	D1DrawText
406234	D1DrawLevelUpButton
406508	D1DrawRedMask
406667	D1DrawSkillMenu
406b08	D1DrawGoldTransaction
40a391	D1LoadLevelGFX
40a4e1	D1CreateLevel
4153a0	D1InitializeGraphics
4154b5	D1CreateDDrawBuffers
4155c2	D1CreateDDrawSurface
41561a	D1InitializeDirectDraw
415695	D1LockDrawSurface
415720	D1DrawSurfaceUnlock
416274	D1DrawImage
41685a	D1DrawCursorTown
416b19	D1DrawObjectBaseDungeon
416ba9	D1DrawArchTile
416ec0	D1DrawObjectHilight
41759b	D1Malloc
4175e8	D1Free
417618	D1BufferFile
417c99	D1DrawUnitHilight
417df8	D1DrawInfraLitObjectsDungeon
417f78	D1DrawPartialLitObjectsDungeon
4180aa	D1DrawMessageBox
419e8b	D1DrawPause
41a565	D1DrawHelp
41b8c4	D1DrawInventory
4236a6	D1DrawItemMenu
440153	D1ProcessNetworkPackets
448e33	D1CreateDDrawPalette
448eab	D1LoadGammaConfig
448f20	D1LoadSystemPalette
4525cd	D1DrawQuestLog
452f8b	D1DrawCompositeUnitDungeon
453160	D1RenderDungeon
453272	D1DrawWorldDungeonHighRes
453477	D1DrawAllFrontTiles
4538e2	D1DrawUnitsDungeon
453ed9	D1DrawUnitDungeon
453fcc	D1DrawLitObjectsDungeon
454d9d	D1DrawAllBackTiles
455bd4	D1DrawWorldDungeonLowRes
455ec7	D1DrawUIPanel
455f56	D1DrawCursor
456124	D1PresentScene
4564f9	D1RenderMain
45708b	D1InitializeSound
45e2a5	D1DrawFrontUnitsTown
45e5b0	D1DrawAllFrontTilesTown
45e939	D1DrawRoofUnitsTown
45ec49	D1DrawRoofTilesTown
45f013	D1DrawBackUnitsTown
45f323	D1DrawAllBackTilesTown
45f65d	D1DrawWorldTownHighRes
45f856	D1DrawWorldTownLowRes
45faab	D1RenderTown
462c7d	SFileClose
462c84	SFileSize
462d06	SFileOpen
462d48	SFileRead
4652c5	D1DrawTilesBack
46886b	D1DrawTilesFront
4696be	D1DrawBlankTile
469b10	sprintf
469d20	D1FillPalette
46a1e0	memcpy
46acb0	lstrcmpi

Globals

Code: Select all

4b7e48	AutoMapOpen
4b84dc	GoldTransactionOpen
4b8968	SkillMenuOpen
4b896c	CharacterSheetOpen
4b8c98	VisibleButtons
52569c	HighResMode
525740	PauseOpen
52a510	ScreenSurfaceBuffer
52a518	DirectDrawInterface
52a51c	DirectDrawPalette
52a528	DirectDrawSurfaceSwap
52a52c	DirectDrawSurface
52a548	DoubleBufferScreen
52a549	UseSoftwareRendering
52a54c	DDrawLibrary
52b9f1	MessageBoxOpen
53cd48	DrawSurface
5b70dc	CurrentTilesetTIL
5b70e0	CurrentTilesetMIN
5bb1ed	CurrentLevelType
5bdb0c	CurrentTilesetCEL
5c690c	TownSmallCEL
634498	HelpOpen
634cb8	InventoryOpen
634ec4	ItemMenuOpen
69b7b0	RedMaskOn
69bd04	QuestLogOpen

DDraw VTable Refs

Code: Select all

448e68	DDraw::CreatePalette
448e92	DirectDrawSurface::SetPalette
415600	DDraw:CreatePalette
415401	DDraw::SetCooperativeLevel
41543a	DDraw::SetDisplayMode
41545d	DDraw::SetDisplayMode
4154ca	DirectDrawSurface::GetCaps
415501	DirectDrawSurface::Lock
415511	DirectDrawSurface::Unlock
415583	DirectDrawSurface::GetPixelFormat
4155a9	DirectDraw::CreateSurface

Debugging in fullscreen apps is a bit of a pain if you don't have dual monitors, I recommend this DDraw wrapper, as its so far the only one I've found that doesn't crash or have major issues (all the Diablo specific ones crash on my system...), just make sure to run in opengl mode as GDI doesn't render correctly.

(note: if you use the labelmaster plugin for olly you can copy these directly into a txt and load them up).

[untitled]

Hi Necrolis!

Thanks for taking the initiative to share some RE notes. It seems like you have a thorough approach to reversing, as indicated by the succinct names you've managed to give to functions and global variables. It is only through deeper understanding that we may express things in brevity without loosing clarity.

I have also been struggling to debug the fullscreen game, and the solution I came across is less elegant than yours, but works. I used the Multiple Desktops application from Sysinternals to swap desktops as the game was running, while using one monitor for the game and another one for OllyDbg.

I don't know which forums you are lurking on, but I just posted a note about the debug release of Diablo 1 to http://www.lurkerlounge.com/forums/thread-16351.html. Wheybags knows about this version since before, but I'm not sure how many others have heard about it. I would be happy to try and contribute and build a shared knowledge base of anything related to the Diablo 1 game and its game engine. Jarulf's guide is of cause a wonderful reference, but there are still things to discover. For instance, how do the Dungeon Random Layout Generation algorithms really work? I know you are working on the rendering aspects of the game.

Hope we can figure these things out together :)

Cheers /u

[wheybags]

For instance, how do the Dungeon Random Layout Generation algorithms really work?

badly :3

Seriously though, should send him a pm, that should give him an email.
He probably doesn't check these forums that regularly.

[untitled]

For instance, how do the Dungeon Random Layout Generation algorithms really work?

badly :3

Seriously though, should send him a pm, that should give him an email.
He probably doesn't check these forums that regularly.

This is true. I tried to access the compose PM page, but got the following error: "We are sorry, but you are not authorised to use this feature. You may have just registered here and may need to participate more to be able to use this feature." No worries though, I found necrolis contact info on GitHub.

[Necrolis]

Yeah I don't visit the forums regularly mainly cause there isn't much going on and with work I haven't been able to do much of anything (hell, my "optimized" renderer branch is so far behind I might as well start from scratch ). I idle in the IRC channel pretty much 24/7 however (GMT+2, though I tend to start late and work late). I'm just gonna respond here rather than via email to to keep this open for others that may be interested.

Thanks for taking the initiative to share some RE notes. It seems like you have a thorough approach to reversing, as indicated by the succinct names you've managed to give to functions and global variables. It is only through deeper understanding that we may express things in brevity without loosing clarity.

I find there is an art to naming things, be it when RE'ing or writing code of my own (personal/work/research); normally I ended up renaming things multiple times till I'm happy. My RE "skill" comes from years with D2's engine, D1 is relatively simple compared to that (and D2 is simpler when compared to things like BF3 or PoE, but that is due to the leap to 3D).

I have also been struggling to debug the fullscreen game, and the solution I came across is less elegant than yours, but works. I used the Multiple Desktops application from Sysinternals to swap desktops as the game was running, while using one monitor for the game and another one for OllyDbg.

DDraw is a PITA, I was lucky that the wrapper I found worked on my system and got D1 running in windowed mode; I never thought about using virtualized desktops; however ollydbg does tend to lockup certain things (especially when kernel/syscalls are involved).

I don't know which forums you are lurking on, but I just posted a note about the debug release of Diablo 1 to http://www.lurkerlounge.com/forums/thread-16351.html. Wheybags knows about this version since before, but I'm not sure how many others have heard about it. I would be happy to try and contribute and build a shared knowledge base of anything related to the Diablo 1 game and its game engine. Jarulf's guide is of cause a wonderful reference, but there are still things to discover. For instance, how do the Dungeon Random Layout Generation algorithms really work? I know you are working on the rendering aspects of the game.

I normally lurk on the Phrozen Keep

I've actually never heard about the debug build (not that I really looked), the annotations/assertions make life slightly easier, but sometimes aren't to be trusted (eg: in D2, the messages are actually the inverse of their conditional, some are just wrong ). The other problem is that 1.09 may be vastly different from 1.0. I know the guys from TheDark mod and the makes of TheHell mod released some notes many years ago, but they seem to have been lost to net-decay... (I may have a copy somewhere; though I was more interested in getting the source of TheDarkGraphics), the source to TheHell2 was also temporarily released, but it seems sir.krist had to pull that down . There is also a (win)DDT, but I'm not sure how much of this was reversed code...

I'm pretty much able to get anything out for the code if needed, the rendering things where what I started on based on the tickets open on GH (they are also the most complex IMO, due to working with DDraw and the CEL decoder which is treated like a software renderer in the engine and has WAY to many special cases...). As for documenting it, I don't use IDA or the like, normally I use labelmaster to just export my notes, though I made a custom olly plugin to create JSON db's instead that preserved some info and was process wide (though not that many games apart from D2 utilize many custom dll's...).

I think the best is just a simple page with some form of markdown (if you want nice hilighting and type cross refs) that has a section for structs, a section for global variables and a section for functions (including the full signature rather than just the names and VA/RVA); one could possibly group by overall use (gameplay/sound/render/network). I've yet to find anything like that though, would probably just end up making it. The nice thing is Diablo 1 will never have another patch so working with 1.09(b?) is rather easy and future proof; same goes for Hellfire which might be tackled much later on.

[untitled]

I find there is an art to naming things, be it when RE'ing or writing code of my own (personal/work/research); normally I ended up renaming things multiple times till I'm happy. My RE "skill" comes from years with D2's engine, D1 is relatively simple compared to that (and D2 is simpler when compared to things like BF3 or PoE, but that is due to the leap to 3D).

I'm thrilled to hear that you feel the same way about the art of naming things. I can easily spend an equal amount of time coding, as I do thinking about precise names for functions, methods, structures and global variables. This also extends to the practice of writing succinct documentation, which aims for brevity while trying to avoid loosing the nuances. And as you already mentioned, there is a constant process of renaming things/reworking documentation until one feels happy :)

I normally lurk on the Phrozen Keep :P

Are you still in touch with Adhin? Of all the mods I've played for D2, none has even come close to the dark feeling of Blackened. My friends and I have had countless of hours of fun playing the mod, exploring different character specs, following its development, and to some extent adding our own alterations to the mod. I cannot express my gratitude enough for all your hard work!

I've actually never heard about the debug build (not that I really looked), the annotations/assertions make life slightly easier, but sometimes aren't to be trusted (eg: in D2, the messages are actually the inverse of their conditional, some are just wrong :P). The other problem is that 1.09 may be vastly different from 1.0.

This is indeed true. Getting used to the invertedness of assert strings takes time, but they do provide a tremendous context which facilitates exploration and understanding of the code. That being said, 1.00 and 1.09 are indeed different, and 1.09 contains a substantial amount of bug fixes. Focusing on analyzing 1.09 therefore makes sense, and using the debug release to facilitate this analysis may be beneficial.

I'm pretty much able to get anything out for the code if needed, the rendering things where what I started on based on the tickets open on GH (they are also the most complex IMO, due to working with DDraw and the CEL decoder which is treated like a software renderer in the engine and has WAY to many special cases...). As for documenting it, I don't use IDA or the like, normally I use labelmaster to just export my notes, though I made a custom olly plugin to create JSON db's instead that preserved some info and was process wide (though not that many games apart from D2 utilize many custom dll's...).

It is great that you are tracking down the cause of these rendering issues. I think that by now, the CEL and CL2 image formats are fully understood. To the best of my knowledge, blizzconv should be capable of decoding all CEL and CL2 images. Its code base could be used to track down some of the rendering bugs in Freeablo.

I think the best is just a simple page with some form of markdown (if you want nice hilighting and type cross refs) that has a section for structs, a section for global variables and a section for functions (including the full signature rather than just the names and VA/RVA); one could possibly group by overall use (gameplay/sound/render/network). I've yet to find anything like that though, would probably just end up making it. The nice thing is Diablo 1 will never have another patch so working with 1.09(b?) is rather easy and future proof; same goes for Hellfire which might be tackled much later on.

I agree that the best format would be some form of markdown with cross-references and syntax high-lighting. If you would be willing, I'd love to collaborate on this. I took the liberty of creating a repository on GitHub to which I've invited you. I've included some initial notes from Diablo v1.09b, mostly as an example for now of how the structure might look like. Feel free to discuss/explore other was of structuring the information.

I was happy to discovery that the file size of the diablo.exe binary of v1.09 and v1.09b is identical, and more importantly they both have the exact same addresses for every function and global variable. Therefore, notes related to any of these two versions may co-exist and thrive together.

I hope to hear back from you Lorenzo!

Cheers /u

[untitled]

I agree that the best format would be some form of markdown with cross-references and syntax high-lighting. If you would be willing, I'd love to collaborate on this. I took the liberty of creating a repository on GitHub to which I've invited you. I've included some initial notes from Diablo v1.09b, mostly as an example for now of how the structure might look like. Feel free to discuss/explore other was of structuring the information.

Hi Lorenzo!

I just thought of something which I forgot to ask you. Assuming (and of cause hoping) that we do this collaborate effort, would you feel alright with us releasing all notes into the public domain? This is how I prefer to release the source code and other content of personal projects. If you have any concerns with this, we may discuss other alternatives.

Cheers /u

[Necrolis]

I'm thrilled to hear that you feel the same way about the art of naming things. I can easily spend an equal amount of time coding, as I do thinking about precise names for functions, methods, structures and global variables. This also extends to the practice of writing succinct documentation, which aims for brevity while trying to avoid loosing the nuances. And as you already mentioned, there is a constant process of renaming things/reworking documentation until one feels happy

You should see the convoluted names for some of my work code, and there I'm using C# and the odd bit of C++ so I can at least namespace stuff

Are you still in touch with Adhin? Of all the mods I've played for D2, none has even come close to the dark feeling of Blackened. My friends and I have had countless of hours of fun playing the mod, exploring different character specs, following its development, and to some extent adding our own alterations to the mod. I cannot express my gratitude enough for all your hard work!

I loved Blackened, its how I got in touch with Adhin in the first place. I haven't spoken to him in a while, though that's cause he's off playing xbox games till I finish rewriting the graphics driver (which is like 95% of the way there, just 2 bugs and monsters palettes need to be done); we had a build of the next beta up that allowed for a lot of stuff to be scripted with Lua(JIT) but there where some other things I needed to finish first on that front (I think it was quest related).

It is great that you are tracking down the cause of these rendering issues. I think that by now, the CEL and CL2 image formats are fully understood. To the best of my knowledge, blizzconv should be capable of decoding all CEL and CL2 images. Its code base could be used to track down some of the rendering bugs in Freeablo.

I remember noting that it was missing a few cases that TDG had and also couldn't handle tile rendering (which is what my first stop was), Ulmo's TileView however has fully documented that. I honestly cannot fathom how the Blizz North guys worked with all the odd exceptions I've seen in the CEL format...

I agree that the best format would be some form of markdown with cross-references and syntax high-lighting. If you would be willing, I'd love to collaborate on this. I took the liberty of creating a repository on GitHub to which I've invited you. I've included some initial notes from Diablo v1.09b, mostly as an example for now of how the structure might look like. Feel free to discuss/explore other was of structuring the information.

I'll probably end up making a very simple SPA with a JSON file for globals/structs/enums/functions that marks everything up, adds in the cross-ref, automatic offset deduction and search etc. keeps everything easy to parse mechanically, easy to alter and easy to keep consistent (it'll probably be generic too, maybe other find it of interest). That might be something I'll tackle this weekend.

I was happy to discovery that the file size of the diablo.exe binary of v1.09 and v1.09b is identical, and more importantly they both have the exact same addresses for every function and global variable. Therefore, notes related to any of these two versions may co-exist and thrive together.

Well that certainly makes life easier.

I just thought of something which I forgot to ask you. Assuming (and of cause hoping) that we do this collaborate effort, would you feel alright with us releasing all notes into the public domain? This is how I prefer to release the source code and other content of personal projects. If you have any concerns with this, we may discuss other alternatives.

Yeah I have no problems with that, D1 isn't supported on Battle.Net so we have no issues from that side either, if any code is to be put under license I would suggest the MIT license. My only issue is that anything I contribute is gonna be C, C++ or x86 assembly, though I see you're all setup for Go

[untitled]

I loved Blackened, its how I got in touch with Adhin in the first place. I haven't spoken to him in a while, though that's cause he's off playing xbox games till I finish rewriting the graphics driver (which is like 95% of the way there, just 2 bugs and monsters palettes need to be done); we had a build of the next beta up that allowed for a lot of stuff to be scripted with Lua(JIT) but there where some other things I needed to finish first on that front (I think it was quest related).

Wow, amazing to hear that you are still working on this! As you have high ambitions for Blackened, I can understand the need for Lua scripting to facilitate the modding of custom quests and the like.

I remember noting that it was missing a few cases that TDG had and also couldn't handle tile rendering (which is what my first stop was).

It does handle tile rendering, through the min_dump and til_dump tools. It also handles dungeon and town rendering through the dun_dump tool. These are described in step 8-10 in the usage section of the readme. The only thing that is not currently implemented is the rendering of arches. We have implemented this on a local copy of the code, but it would require some love before pushing it to the repo.

Ulmo's TileView however has fully documented that.

I didn't know that Ulmo's ViewTile was open source! I will dive into the source code later and cross-reference the implementations.

I honestly cannot fathom how the Blizz North guys worked with all the odd exceptions I've seen in the CEL format...

I know, there are at least 10 or so different functions which handle the slight variations of CEL parsing. There also seems to be multiple inconsistencies in the file formats, which have been fixed by adding exceptions in the code rather than fixing the original inconsistency in the file format. It is possible that different teams were responsible for implementing the parsing of game resources and defining the file formats.

I'll probably end up making a very simple SPA with a JSON file for globals/structs/enums/functions that marks everything up, adds in the cross-ref, automatic offset deduction and search etc. keeps everything easy to parse mechanically, easy to alter and easy to keep consistent (it'll probably be generic too, maybe other find it of interest). That might be something I'll tackle this weekend.

Cool! We can experiment with a few different approaches and later discuss the benefits and drawbacks with them after we've gotten some experience.

Well that certainly makes life easier.

Indeed! I felt a rush of serenity sweep through my body as I discovered this fact.

Yeah I have no problems with that, D1 isn't supported on Battle.Net so we have no issues from that side either, if any code is to be put under license I would suggest the MIT license.

Perfect! I agree that using a MIT or BSD license is preferable, if a license is required at all that is. Personally, I'd be happy to place my source code into the public domain, which is basically a 0-clause MIT/BSD license :)

My only issue is that anything I contribute is gonna be C, C++ or x86 assembly, though I see you're all setup for Go :P

Haha :) Don't worry. C and Assmebly are still close to my heart. It should also be possible to implement different aspects of the game engine in any given language, as long as they interact using well-defined input/output (e.g. components interacting using JSON, Protocol Buffers, etc...). This has the added benefit of decoupling various components of the game engine and keeping us honest about a modular design.

[Necrolis]

It does handle tile rendering, through the min_dump and til_dump tools. It also handles dungeon and town rendering through the dun_dump tool. These are described in step 8-10 in the usage section of the readme. The only thing that is not currently implemented is the rendering of arches. We have implemented this on a local copy of the code, but it would require some love before pushing it to the repo.

I tried it when freeblo was first up on GH, so you might have fixed everything by now (also remember reading a few things in the readme that immediately struck me as incorrect..).

I didn't know that Ulmo's ViewTile was open source! I will dive into the source code later and cross-reference the implementations.

Be warned, there seem to be two versions floating around without version numbers; the latest version is the one supporting hellfire (I liked it on one og the GH tickets on tiles).

Haha Don't worry. C and Assmebly are still close to my heart. It should also be possible to implement different aspects of the game engine in any given language, as long as they interact using well-defined input/output (e.g. components interacting using JSON, Protocol Buffers, etc...). This has the added benefit of decoupling various components of the game engine and keeping us honest about a modular design.

This is on thing where you have to be careful; if you are replicating the engine, IMO it should stay the same as what it was bar possible bug fixes. Tools on the other hand can go crazy. When once there is a solid reference impl. jazzing it up is not only easier, but you can also verify it correctly.