Merge Generals and Zero Hour code into one

[xezon]

Generals and Zero Hour share the vast majority of code, but do have functional differences. It would be great if we can unite both games into one code base, while still building two distinct executables off of it with the original functionalities for the respective games.

Move Game

• GameEngine file [CORE] Unify File System code and move files to Core #1279
• GameEngine xfer [CORE] Unify and move GameEngine's Xfer files to Core #821
• GameEngine string [CORE] Move AsciiString, UnicodeString files to Core #1328
• GameEngine bezier
• GameEngine video
• GameEngine audio [GEN][ZH] Unify and move Audio files to Core #1330
• GameEngine input
• GameEngine randnum unify(common): Move RandomValue files to Core #1388
• debug [CORE] Move debug and profile libraries from Zero Hour to Core #819
• profile [CORE] Move debug and profile libraries from Zero Hour to Core #819

Move WWVegas

• WW3D2 [CORE] Move many unified WW3D2 library files to Core #814
• WWAudio [CORE] Move WWAudio library files to Core #802
• WWDebug [CMAKE][CORE] Move WWLib and WWDebug to Core #663
• WWDownload Unify code of WWDownload library and move its files to Core #785
• WWLib [CMAKE][CORE] Move WWLib and WWDebug to Core #663
• WWMath [CMAKE][CORE] Move WWSaveLoad and most of WWMath to Core #694 [CORE] Unify remaining WWMath #745
• WWSaveLoad [CMAKE][CORE] Move WWSaveLoad and most of WWMath to Core #694
• wwshade (delete) [ZH] Remove the unused WWShade library #596
• Wwutil (delete) [GEN][ZH] Remove the unused Wwutil library #786

Move Tools

• assetcull [CORE] Move assetcull, mangler, wolSetup tools to Core #721
• Autorun Unify Autorun tool and move its files to Core #739
• Babylon [CORE] Move Babylon tool to Core #720
• buildVersionUpdate [CORE] Move buildversionUpdate, textureCompress, timingTest, versionUpdate tools to Core #707
• Compress [CMAKE][CORE] Move compression library and compress tool to Core #649
• CRCDiff [CMAKE][CORE] Move CRCDiff tool to Core #684
• DebugWindow [CORE] Move DebugWindow tool to Core #748
• GUIEdit
• ImagePacker [CORE] Move ImagePacker tool files to Core #747
• Launcher + DatGen Unify Launcher, DatGen tool and move their files to Core #741
• mangler [CORE] Move assetcull, mangler, wolSetup tools to Core #721
• MapCacheBuilder [CORE] Move MapCacheBuilder tool files to Core #749
• matchbot [CORE] Move matchbot tool to Core #706
• ParticleEditor unify(particlesys): Move ParticleSys and ParticleEditor files to Core #2212
• PATCHGET Unify PATCHGET tool and move its files to Core #746
• textureCompress [CORE] Move buildversionUpdate, textureCompress, timingTest, versionUpdate tools to Core #707
• timingTest [CORE] Move buildversionUpdate, textureCompress, timingTest, versionUpdate tools to Core #707
• versionUpdate [CORE] Move buildversionUpdate, textureCompress, timingTest, versionUpdate tools to Core #707
• W3DView [CORE][GEN] Sync WWAudio with Zero Hour, move w3dview files to Core and prepare w3dview target for Generals #709
• wdump
• wolSetup [CORE] Move assetcull, mangler, wolSetup tools to Core #721
• WorldBuilder
• WW3D (max2w3d, pluglib) [CMAKE][CORE] Move max2w3d to Core #696

[xezon]

Relevant chat Notes.

feliwir:

I did a diff between both sourcetrees and the changes required to run vanilla with ZH executable are not that complicated:

• Add an option to change installation location behaviour
• Add INI fields present in Generals but not in ZH
• Look for other differences and make them configurable

But if we write everything twice now for a year and then start deleting what we've written again it's a waste of effort. The deduplication would even work with VC6 in my opinion. I think the gameplay relevant changes aren't that many. In some cases it's just refactoring which we can apply 1:1 to generals aswell

tomsons26:

First reduce the code to maintain, a lot of wwvegas goes. Then you can cleanly see what differs from what remains there.

WWVegas differs a lot: Gen uses a Ren derivative. ZH uses Ren2 or E&B derivative. HOWEVER, not that much code from it is actually used. Start with commonizing WWLib. Most of what's there is just deletable.

WPAudio can flat out be deleted. It's the audio engine used in RA2-Emperor(among a slew of other Burst/Pacific games), seems at some point it was to be in gen, headers are missing so uncompileable.

WWSaveLoad next target to remove. Generals doesn't call any of that, in this case two virtuals and inheritance is to be removed.

WWAudio. So as is it will be needed for ported over Ren W3DView. Not sure Gen handles any w3d audio chunks tho so the dialog using that would probably go away.

WWShade. Code for handling the next version of w3d file format, it was a shader based format hence the name. Westwood quietly released a few Ren2 models on ftp when Ren2 got axed, those are the only models in existence that this code handles. Max2w3d doesn't support this format so unless someone has some grand plans WWShade can go. If we had the tools and compiled game with wwshade define on then this format would be usable in-game.

Common code that is insignificantly different or exactly the same should be placed in a common place. Eliminating duplications means easier maintenance.

[tomsons26]

GUIEdit - ZH's seems like a no compromise better version so Gen version can just be scrapped
Compression - has no diffs between the two so thats movable to common
WWAudio - the same in Gen and ZH
WWDownload - string differences need to be handled, otherwise same code in both
matchbot - use ZH's
MapCacheBuilder - use ZH's
mangler - use ZH's
launcher - code is same, just strings differ
ImagePacker - use ZH's
DebugWindow - use ZH's
CRCDiff - use ZH's
buildversionupdate - use ZH's
Autorun - code about same, resolve strings, use ZH's

[andrew-2e128]

Tested against VS2022,

here is a list of files that can be removed from Code/Libraries/WWVegas/WWLib as they are not really used by the game:

CallbackHook.h
Convert.h
FONT.H
HASHLIST.H
IFF.H
LCW.H
LaunchWeb.cpp
LaunchWeb.h
MONO.H
MONODRVR.H
Notify.h
PALETTE.H
RAMFILE.H
RGB.H
RLE.H
RNDSTRAW.H
Signaler.h
WATCOM.H
_convert.cpp
_convert.h
_mono.cpp
_mono.h
_timer.cpp
_timer.h
_xmouse.h
argv.cpp
argv.h
bfiofile.h
binheap.h
blowfish.cpp
blowfish.h
blowpipe.h
blwstraw.h
bool.h
borlandc.h
convert.cpp
crcpipe.h
crcstraw.h
data.cpp
data.h
ddraw.cpp
dib.h
draw.h
dsurface.cpp
dsurface.h
fixed.h
hashtab.h
hsv.cpp
hsv.h
incdec.h
jshell.cpp
keyboard.cpp
keyboard.h
lcw.cpp
lcwpipe.h
load.cpp
misc.h
mixfile.cpp
mixfile.h
mono.cpp
noinit.h
ntree.h
obscure.h
palette.cpp
pkpipe.h
pkstraw.h
ramfile.cpp
rc4.cpp
rc4.h
rcfile.cpp
rcfile.h
ref_ptr.h
rgb.cpp
rle.cpp
rlerle.h
rndstrng.cpp
rndstrng.h
rsacrypt.h
sampler.cpp
sampler.h
search.h
sha.cpp
sha.h
shapeset.h
shapipe.h
shastraw.h
smartptr.h
srandom.cpp
srandom.h
stimer.cpp
stimer.h
stl.h
surfrect.h
timer.h
trackwin.h
trackxy.h
verchk.cpp
verchk.h
wwfont.cpp
wwfont.h
xmouse.h

if anything complains about them, remove the include, and sometimes it keeps complaining in such a case it mostly wants a windows.h include (included it via win.h)

this list applies to GeneralsMD too (except mixfile.cpp and mixfile.h)

also in ww3d2 remove the following 2 files : txt.cpp txt.h

there is actually a lot more files that can be removed if we removed referenced code that is never actually called, but this will be a lot more work (I think there are a couple of big files like that)

note, I only focused on wwlib as it is where most of unused code is, most other modules are either mostly needed or completely unused; or might make sense to keep them for now until completely replaced (like wwmath)

[xezon]

tomsons26:

At a quick glance diff

• AABTreeBuilderClass::Compute_Bounding_Box - uses different value for inits
• animobj - changes may affect animation behavior
• bitmaphandler - changes need checking if don't introduce issues
• BoxRenderObjClass::render_box uses different sorting
• composite - ZH changes need porting over i guess
• dazzle - alters a lot of code, tho may be meaningless, depends if anyone does use dazzle in mods, stock games don't
• ddsfile - ZH changes need porting over i guess
• dynamesh uses two texture arrays, dunno if w3d models from gen are compatible with this
• dx8caps dx8wrapper - are quite different
• dx8textman - needs TextureClass refactor
• hanim, hanimmgr, hcanim, hlod, hmorphanim, hrawanim, htreemgr - changes may affect animation behavior
• light - had Get_Far_Attenuation_Range(double& fStart, double& fEnd), ZH also adds Get_Far_Attenuation_Range(float & fStart, float & fEnd) const but that makes me question due to float to double promotion what is even called when
• lightenvironment - light behavior changes
• mapper - not compatible with Gen models, the mapper implementations changed in ZH
• mesh, meshdesc, meshmdl, meshmdlio - todo look into the diffs
• motchan - changes may affect animation behavior
• nullrobj - bound changes
• part_ - lots of changes dunno if compatible with a Gen W3D model
• pivot - question on what to do with LAZY_CAP_MTX_ALLOC
• scene - light behavior change
• shader - changes not compatible with Gen models
• sortingrenderer - lots of changes todo look into
• vertmaterial - mapper changes related to mapper.cpp

[tomsons26]

Things that seem to be possible to make into libs of their own

file:
	ArchiveFile
	ArchiveFileSystem
	File
	FileSystem
	LocalFile
	LocalFileSystem
	RAMFile
	StreamArchiveFile
	Win32BIGFile
	Win32BIGFileSystem
	Win32LocalFile
	Win32LocalFileSystem
	Directory?
	
xfer:
	Xfer
	XferCRC
	XferDeepCRC
	XferLoad
	XferSave
	
string:
	AsciiString
	UnicodeString

bazier:
	BezFwdIterator
	BezierSegment
	
video:
	BinkVideoPlayer
	VideoPlayer
	VideoStream
	
input:
	Win32DIKeyboard
	Win32DIMouse
	Win32Mouse

randnum:
	ClientRandomValue
	AudioRandomValue
	LogicRandomValue
	RandomValue

[xezon]

feliwir:

ZH mapper animation speed is broken. We also wanted to compare mapper versions. If the version did increment we can work around it programmatically

tomsons26:

You need to check if ZH overrides any models. Issues would only be with anything using shaders or mappers. Those would be only a handful of models, worst case 50 maybe.

xezon:

Zero Hour only overrides 95 files of Generals

w3d_generals_zerohour_differences.txt

tomsons26:

Yeah so we need to batch check for mappers and shader values that changed. If gen doesn't use any then its fine as far as retail goes, mods, well those will be broken if they use mappers or shaders.

OmniBlade:

For the mapper animation speed, what kind of ranges of values do you get for models using seconds vs milliseconds? If milliseconds is an order of magnitude larger then that could be used as a version marker unless some models go down as low as a couple of milliseconds?

tomsons26:

Mappers and shaders aren't the only diffs, as i listed them all previously. I don't know the implications of a lot of those changes. My view now is to move what is same to core, somehow magic out so cmake can find the files for Gen and ZH. Then leave what is unique in game dirs. I don't see a clean way of handling merging what is left into Generals. Not at this point anyhow. In some cases ifdef ZERO_HOUR would be a solution but there is contradicting code as well across the games.

[xezon]

If we want to commit to unifying the games, then I do think we need to merge as much as possible, even if that means changing a few things in Generals. Reason being, if the diff remains large and we touch classes and functions in Core, then the divergences of both Generals and Zero Hour would need to be accommodated for those Core changes, which could be a substantial amount of work forever going forward. Ideally only game specific components, like the Spectre Gunship, would be unique on one side.

But there is another problem. If data requirements for Zero Hour eventually diverge from Generals, then everything needs to be built to always keep supporting the legacy Generals data, for all eternity. But perhaps that needs to be done anyway, because of Community Mods. Aka we would not want to break some formats because it would break Mod compatibility. Otherwise, we would also need to maintain Generals data to migrate it to new code requirements and all customer Mods would need to do the same. That to say, maintaining compatibility to more than just pure Zero Hour will take way more effort now and in the long run. Are we ok with this?

[xezon]

tomsons26:

pluglib, max2w3d, take zh's
afaik the plugin only uses pluglib
wolsetup - can also be scrapped entirely since wol isn't used in gen
timingtest
texturecompress
patchget - can be scrapped, not sure it has a use today?
matchbot
mangler
crcdiff
launcher, autorun has string differences, with that resolved it can be moved
in wwvegas:
wwdownload has string differences, with that resolved it can be moved
wwaudio move pr already exists

[xezon]

tomsons26

Reviewed Babylon, move ZH's
All the diffs are are renaming Noxstring and Nox to Babylon, no actual code behavior changes at all