RE Notes

[untitled]

This is on thing where you have to be careful; if you are replicating the engine, IMO it should stay the same as what it was bar possible bug fixes. Tools on the other hand can go crazy. When once there is a solid reference impl. jazzing it up is not only easier, but you can also verify it correctly.

No worries. We feel the same way, and take extreme precaution when reimplementing different aspects of the game. For the dungeon generation algorithms, we've generated test cases from the original game which cover a complete mapping of every pillar (MIN entity) and square (TIL entity) for each of the first 256 seeds. Once the reference implementation for dungeon generation has been complete it would be trivial to compare its output with the dungeon random layout generation algorithms of the original game for each possible seed, i.e. 2^32 seed values. The reason we've not created test data for all of these cases is simply because of the hard drive space it would require. The pillar test case data for one seed is roughly 50 KB and the square test case data is roughly 1.6 KB. The first 256 test cases amount to 13.3 MB. The test data for all possible seeds would require approximately 202 TB, thus making it infeasible. However, it should definitely be possible to generate these test cases on the fly and thus ensure that the reference implementation is indeed 100% accurate with regards to every possible input (seed value) and output (2d arrays of pillars and squares) combination.

We've just complete a full mapping between address ranges in the 1.09b binary and their corresponding source code file names. These mappings are made available at:
* Functions
* Global Variables (read-only)
* Global Variables (read-write)
* Global Variables (uninitialized)

As you can see, these are merely skeleton structures for now. A few function definitions and global variables have been documented so far, but mostly to provide a reference for how the structure may look like. Any suggestions, ideas or contributions are warmly appreciated! I know you mentioned the JSON mapping which could be used to produce a web page with hyperlinks, and also be used for applications such a OllyDbg. Any updates on this work?

Cheers /u

[wheybags]

Wow, that project is really cool!
I hope you do fully reverse engineer the level generation code from the original, I'd love to have an "original lvlgen" mode in freeablo. One thing is though, if you do, it'd be best if someone else wrote the implementation in freeablo, because, y'know, clean room re and all.

[Necrolis]

As you can see, these are merely skeleton structures for now. A few function definitions and global variables have been documented so far, but mostly to provide a reference for how the structure may look like. Any suggestions, ideas or contributions are warmly appreciated! I know you mentioned the JSON mapping which could be used to produce a web page with hyperlinks, and also be used for applications such a OllyDbg. Any updates on this work?

I've been following the repo when I've had time, unfortunaly health issues and some RL stuff has taken its toll so I haven't had much free time I had made a skeleton for the JSON thing (I also have a plugin I built a while back for olly 1.10 for databasing all the labels and comments for all modules in a process in JSON). Hopefully nd of this month I'll have a bit more free time to get stuck back into personal projects =)

[untitled]

I've been following the repo when I've had time, unfortunaly health issues and some RL stuff has taken its toll so I haven't had much free time :( I had made a skeleton for the JSON thing (I also have a plugin I built a while back for olly 1.10 for databasing all the labels and comments for all modules in a process in JSON). Hopefully nd of this month I'll have a bit more free time to get stuck back into personal projects =)

Hi Necrolis,

Glad to get a life sign from you :)

No worries. Of course you should focus on your health! This is always the case with hobby projects, you work on them for fun and when you have spare time.

As you may have noticed, there have been less activity in the repository during the last month or so. I've also been struggling with real life issues, which take precedence over all.

It's great that you are experimenting with a JSON version for structuring this data! I'm looking forward to evaluate the advantages and drawbacks of working with the JSON and Markdown formats. I imagine it will be easier to write tools which consume the JSON format, and these may produce Markdown pages with cross-references and such. I know that having a Markdown page which is continuously updated with recent progress is great for lifting the morale. This is especially important for large projects, as visualization of the small incremental progress may help spark enthusiasm :)

Take care.

Cheers /u

[untitled]

As you can see, these are merely skeleton structures for now. A few function definitions and global variables have been documented so far, but mostly to provide a reference for how the structure may look like. Any suggestions, ideas or contributions are warmly appreciated! I know you mentioned the JSON mapping which could be used to produce a web page with hyperlinks, and also be used for applications such a OllyDbg. Any updates on this work?

I've been following the repo when I've had time, unfortunaly health issues and some RL stuff has taken its toll so I haven't had much free time :( I had made a skeleton for the JSON thing (I also have a plugin I built a while back for olly 1.10 for databasing all the labels and comments for all modules in a process in JSON). Hopefully nd of this month I'll have a bit more free time to get stuck back into personal projects =)

Hi Necrolis!

I hope you are starting feeling better and that you may enjoy some recreational hacking again :) I tried implementing a small PoC tool which translates the Markdown notes into JSON format. The implementation is very rough, but should give us an idea of the feasibility and possibilities of such an approach, which we may compare against other approaches.

I would be happy if you could take a look and give some feedback on what specific requirements you would have from such a tool and how the JSON format you previously mentioned should be structure. Perhaps we could write tools for converting back and forth from JSON to Markdown and vice versa. This would allow us to benefit from the best of both worlds, using Markdown for presentation and JSON for parsing by other tools (which may in turn produce other formats for OllyDbg, etc).

The code is currently located at https://github.com/sanctuary/exp/tree/m ... md/md2json

As you would have guessed already, the PoC tool was implemented in Go, but as you would be using it as a stand-alone tool, I hope that feels ok for now. I'd still be more than happy to review the JSON tool you've been working on, and compare notes :)

Giving a brief example of how the tool may be used below.

Code: Select all

$ git clone https://github.com/sanctuary/notes
$ go get github.com/sanctuary/exp/cmd/md2json
$ cd notes/
$ md2json functions/towners.md

Current output for the input file functions/towners.md:

Code: Select all

{
    "0x46019B": "// towners_get_num_from_id returns the towner number (towners array index) of\n// the given towner ID.\nint __fastcall towners_get_num_from_id(towner_id towner_id);\n",
    "0x4601C1": "// towners_init_facing_cels initializes the CEL graphics for each facing\n// direction.\nvoid __fastcall towners_init_facing_cels(void *towner_cel, void **facing_cels);\n",
    "0x4601FB": "// towners_init_anim initializes the animation of the given NPC.\nvoid __fastcall towners_init_anim(int towner_num, void *active_cel, int nframes, int nticks_per_frame);\n",
    "0x46022F": "// towners_place places the given NPC on the map.\nvoid __fastcall towners_place(int towner_num, int frame_width, bool32_t targetable, towner_id towner_id, int col, int row, towner_anim_seq_id towner_anim_seq_id, int unused);\n",
    "0x4602C4": "// towners_init_gossip initializes quest gossip of the given NPC.\nvoid __fastcall towners_init_gossip(int towner_num);\n",
    "0x460311": "// towners_init_griswold initializes Griswold the Blacksmith.\nvoid towners_init_griswold();\n",
    "0x4603A0": "// towners_init_odgen initializes Ogden the Tavern owner.\nvoid towners_init_odgen();\n",
    "0x460436": "// towners_init_wounded_townsman initializes the Wounded Townsman.\nvoid towners_init_wounded_townsman();\n",
    "0x4604C6": "// towners_init_adria initializes Adria the Witch.\nvoid towners_init_adria();\n",
    "0x460555": "// towners_init_gillian initializes Gillian the Barmaid.\nvoid towners_init_gillian();\n",
    "0x4605E4": "// towners_init_wirt initializes Wirt the Peg-legged boy.\nvoid towners_init_wirt();\n",
    "0x46067A": "// towners_init_pepin initializes Pepin the Healer.\nvoid towners_init_pepin();\n",
    "0x460709": "// towners_init_cain initializes Cain the Elder.\nvoid towners_init_cain();\n",
    "0x460798": "// towners_init_farnham initializes Farnham the Drunk.\nvoid towners_init_farnham();\n",
    "0x460827": "// towners_init_cows initializes the Cow NPCs.\nvoid towners_init_cows();\n",
    "0x460976": "// towners_init initializes the NPCs of Tristram.\nvoid towners_init();\n",
    "0x4609C3": "// towners_cleanup releases resources used by NPCs.\nvoid towners_cleanup();\n",
    "0x460A05": "// towners_stop_interaction stops interaction with distant players.\nvoid __fastcall towners_stop_interaction(int towner_num);\n",
    "0x460A78": "// towners_update_logic_griswold updates the logic of Griswold each game tick;\n// stops interaction with distant players.\nvoid towners_update_logic_griswold();\n",
    "0x460A86": "// towners_update_logic_ogden updates the logic of Ogden each game tick; stops interaction with distant players.\nvoid towners_update_logic_ogden();\n",
    "0x460A95": "// towners_update_logic_wounded_townsman updates the logic of the Wounded\n// Townsman each game tick; stops interaction with distant players, and updates\n// quest events.\nvoid towners_update_logic_wounded_townsman();\n",
    "0x460B0D": "// towners_update_logic_pepin updates the logic of Pepin each game tick; stops\n// interaction with distant players.\nvoid towners_update_logic_pepin();\n",
    "0x460B1C": "// towners_update_logic_cain updates the logic of Cain each game tick; stops\n// interaction with distant players.\nvoid towners_update_logic_cain();\n",
    "0x460B2B": "// towners_update_logic_farnham updates the logic of Farnham each game tick; stops interaction with distant players.\nvoid towners_update_logic_farnham();\n",
    "0x460B3A": "// towners_update_logic_wirt updates the logic of Wirt each game tick; stops\n// interaction with distant players.\nvoid towners_update_logic_wirt();\n",
    "0x460B49": "// towners_update_logic_adria updates the logic of Adria each game tick; stops\n// interaction with distant players.\nvoid towners_update_logic_adria();\n",
    "0x460B58": "// towners_update_logic_gillian updates the logic of Gillain each game tick;\n// stops interaction with distant players.\nvoid towners_update_logic_gillian();\n",
    "0x460B67": "// towners_update_logic_cow updates the logic of the Cows each game tick; stops interaction with distant players.\nvoid towners_update_logic_cows();\n",
    "0x460B76": "// towners_update updates the logic and graphics of NPCs each game tick.\nvoid towners_update();\n"
}

I imagine we may extend these tools in the future to add cross-references, and more intricate data structures as required.

Cheers
/u

[untitled]

For those still lurking on the reverse engineering forums, a lot of progress has been made since the last update.

The notes are now accessible and searchable at http://sanctuary.github.io/notes/ per courtesy of Predelnik, and includes function signatures, global variable declarations, structure and enumerate type definitions.

Take for instance the notes for the Item struct, which now documents the struct members, and includes references to the functions and global variables that make use of the Item struct.

A brief status update as of 2018-02-14 for the notes repo:

* 35% of all functions in Diablo.exe (v1.09b) have been documented.
* 95% of all read-only global variables have been documented.
* 98% of all read-write global variables have been documented.
* 91% of all uninitialized global variables have been documented.

[qualakon]

Looks awesome untitled, looks like a lot of work.